From patchwork Wed Aug 25 02:21:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D06AC4320A for ; Wed, 25 Aug 2021 02:22:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4C14C6127B for ; Wed, 25 Aug 2021 02:22:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237088AbhHYCWz (ORCPT ); Tue, 24 Aug 2021 22:22:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236512AbhHYCWy (ORCPT ); Tue, 24 Aug 2021 22:22:54 -0400 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEDF4C061757 for ; Tue, 24 Aug 2021 19:22:09 -0700 (PDT) Received: by mail-pg1-x531.google.com with SMTP id y23so21611937pgi.7 for ; Tue, 24 Aug 2021 19:22:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=osyt+1YtfCPmJML6Js9+aETSUzA98PVwdKrLEsDt8+A=; b=mCEM/Oc5AYlhIqsLuEyZGYQujUiuSLb9O7is9exzmwFzAM/jICrXnuZky73AWdS7Bl KvvN/yo7hACSHeoPSwqMOy4MfzUjzdnZF1TGsSPuOwFyY2VDc0QaYoCifGJ7U8Mxb9nA hZUrYN2MTpFpbcT8OUqcf1cGInDTlTZisAig+BKFyJVYzuWVeNGrswwMyYMnAdy2r/C2 V0sC/OlXPFNZeVGzQJK83qn5h2nOaDwcCKG2P289uIJsQyaJXzvV/4D5+YP6SxQhwvBr EeTg5aVnZvgfSyU90X/a4y/aktHvojW8j8rMR90k9Na+YEE0AWtMsARC5WcI6wDRWKIs udCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=osyt+1YtfCPmJML6Js9+aETSUzA98PVwdKrLEsDt8+A=; b=FJ6xyiXVlI+2iUk1ehLndFwaoHma8njx6w9e1pCWThSaH1G/0Iup9J10zpvYNLTu1x vFhot5jEXD+lRSTm+jeE11WCf25Ce+43rFe97r7DUTwHc0YXci6gh6btcnaVMtb4M70w aUOgqCy/On8/0g1E8iDb15+Vaa2iNzC+QKtsADp4sFIqtJ4HQezXjFkPwsbbuSFAARi+ dUc6dD6qa2d9rP2ADCJqjzw1Kkgb4FsAlKg/hrjxPDQyORMOuxn3EJy32MigG+uUqqwD 8dldHARckuCK3eZs+0kmKbpNY6kHAmDbZJSIAiL/yFp053PtSnMBWkyOGQLE9YL3KX+I ee4A== X-Gm-Message-State: AOAM533aRzCwB7Au+g//zMHemxMhWZMiiQIKiiU5HjT+OZWUTwTkbaIS sYwCV+GkeeD52N3GGs/BDko= X-Google-Smtp-Source: ABdhPJwIRSQqV+AMsXLqWoWFHA7jAMgWEKQ8WpJGUqIOYPwgnj+6WY+trW3hYX3egxjOWSgxHx7i3g== X-Received: by 2002:a05:6a00:91:b0:3ef:90cd:162d with SMTP id c17-20020a056a00009100b003ef90cd162dmr828438pfj.8.1629858129358; Tue, 24 Aug 2021 19:22:09 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:09 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 01/14] pack-objects.c: introduce new method `match_packfile_uri_exclusions` Date: Wed, 25 Aug 2021 10:21:45 +0800 Message-Id: <73a5b4ccc11b8b0604dc3414129069ab41be45d0.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- builtin/pack-objects.c | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 6d13cd3e1a..31556e7396 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1188,6 +1188,24 @@ static int have_duplicate_entry(const struct object_id *oid, return 1; } +static int match_packfile_uri_exclusions(struct configured_exclusion *ex) +{ + int i; + const char *p; + + if (ex) { + for (i = 0; i < uri_protocols.nr; i++) { + if (skip_prefix(ex->uri, + uri_protocols.items[i].string, + &p) && + *p == ':') + return 1; + + } + } + return 0; +} + static int want_found_object(const struct object_id *oid, int exclude, struct packed_git *p) { @@ -1335,19 +1353,10 @@ static int want_object_in_pack(const struct object_id *oid, if (uri_protocols.nr) { struct configured_exclusion *ex = oidmap_get(&configured_exclusions, oid); - int i; - const char *p; - if (ex) { - for (i = 0; i < uri_protocols.nr; i++) { - if (skip_prefix(ex->uri, - uri_protocols.items[i].string, - &p) && - *p == ':') { - oidset_insert(&excluded_by_config, oid); - return 0; - } - } + if (ex && match_packfile_uri_exclusions(ex)) { + oidset_insert(&excluded_by_config, oid); + return 0; } } From patchwork Wed Aug 25 02:21:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F521C4338F for ; Wed, 25 Aug 2021 02:22:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 02FE06127B for ; Wed, 25 Aug 2021 02:22:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237205AbhHYCW6 (ORCPT ); Tue, 24 Aug 2021 22:22:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237110AbhHYCW5 (ORCPT ); Tue, 24 Aug 2021 22:22:57 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A100C061764 for ; Tue, 24 Aug 2021 19:22:12 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id g14so19997741pfm.1 for ; Tue, 24 Aug 2021 19:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E2fQgFrL8MGUlHMeZZiudDcR4oGhamfYtVSLvxVVWcU=; b=LTB/1/5+s5Tv2cENEjXf6iwtRSDbJCKMd6R9mUxenhwR0YF5tQWDcry0+i938B6mFl eBCFpSi1llElaiZZdFrj7M0QjX7HrNDbElxF7HC+XHaSPQe7iQM0f6lHPwS227GuGV4C 5exkiMaooZQfszql0bGq9VBafXG/87gGbx4ybujxn1vEoG0GB/R5yZWdsELVEiCo/nYP UbloOifH3zb89OR1NNn65GL2F6IMoAl8ud41JyEs474YbjYibh6haJsybpIYUucOqxoC Yozo2fDaRJWRsaCGAQbk3ankzjIqEekAb4SDBBkxcuJOS9ijAfyMPgZFCW1vpmcoFxzU wVIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E2fQgFrL8MGUlHMeZZiudDcR4oGhamfYtVSLvxVVWcU=; b=kOfJt9T02cVismN/FCGqW9wZtvKQ32CAeh0R9089zDvzMRlKgb+mmiVXSW+wofrgek 2wHadKMUBOu9nkOW1+2gxFfMLdZBnpfTc0rqm62RCH3N5/l5ktfaeovxn9gk07yH7Vgu s7dftZZm6NvauU7XhX8EiTJ3gn037oD/XmZdHyUTtkbzolmq+x3bSjPsBEtwdLdn2tPl uF04Qf3XTk2Mj1QjD7IDyh8ku1kmWDIqTa2qna8RxHQysnpUB3w36K8p5ry2uVfLSlii DldNFkUUNnKTuE73tK0tSZlHLje200eseQcZlhF34ZGMlq58t7kSr9iKOeysYWcwE7uV mXGA== X-Gm-Message-State: AOAM530c0wLIvUu2GClZYGRckHk4ek+HsUVXBXp7sE0Be9i+pcNKl4mZ Ljm7Ui4AjeALaPqQjtGU29w= X-Google-Smtp-Source: ABdhPJzE6Eu/fXxbjupMg6Ib1A76jMw2coRd4MChtNHW6kZqYZmNzPXmDbang+TBTrYaM+pJmWQuxQ== X-Received: by 2002:a63:1b45:: with SMTP id b5mr40036602pgm.302.1629858132047; Tue, 24 Aug 2021 19:22:12 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:11 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 02/14] Add new parameter "carry_data" for "show_object" function Date: Wed, 25 Aug 2021 10:21:46 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the pack-objects process, "show_object" function will be called to find the object and show the process("show_object_fn" in "list-object.h"), the function definition contains three parameters: 1. struct object *obj(contains object type, flags, and oid). 2. const char *name(the object name). 3. void *show_data(function to show progress info). This commit adds a new parameter: "void *carry_data", the reason is mainly based on scalability and performance considerations when showing an object, space for time, avoid costly temporary calculations in the "show" phase. For example, carry the ownership relationship between blob or tree object and the referred commit to avoid redundant and expensive calculations. Signed-off-by: Teng Long --- builtin/describe.c | 4 ++-- builtin/pack-objects.c | 15 ++++++++------- builtin/rev-list.c | 2 +- list-objects.c | 8 ++++---- list-objects.h | 2 +- pack-bitmap.c | 8 ++++---- reachable.c | 8 ++++---- 7 files changed, 24 insertions(+), 23 deletions(-) diff --git a/builtin/describe.c b/builtin/describe.c index 40482d8e9f..045da79b5c 100644 --- a/builtin/describe.c +++ b/builtin/describe.c @@ -485,9 +485,9 @@ static void process_commit(struct commit *commit, void *data) pcd->current_commit = commit->object.oid; } -static void process_object(struct object *obj, const char *path, void *data) +static void process_object(struct object *obj, const char *path, void *show_data, void *carry_data) { - struct process_commit_data *pcd = data; + struct process_commit_data *pcd = show_data; if (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) { reset_revision_walk(); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 31556e7396..5f9ec3566f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3068,7 +3068,7 @@ static void show_commit_pack_hint(struct commit *commit, void *_data) } static void show_object_pack_hint(struct object *object, const char *name, - void *_data) + void *show_data, void *carry_dataa) { struct object_entry *oe = packlist_find(&to_pack, &object->oid); if (!oe) @@ -3252,7 +3252,7 @@ static void show_commit(struct commit *commit, void *data) propagate_island_marks(commit); } -static void show_object(struct object *obj, const char *name, void *data) +static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data) { add_preferred_base_object(name); add_object_entry(&obj->oid, obj->type, name, 0); @@ -3274,7 +3274,7 @@ static void show_object(struct object *obj, const char *name, void *data) } } -static void show_object__ma_allow_any(struct object *obj, const char *name, void *data) +static void show_object__ma_allow_any(struct object *obj, const char *name, void *show_data, void *carry_data) { assert(arg_missing_action == MA_ALLOW_ANY); @@ -3285,10 +3285,10 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void if (!has_object(the_repository, &obj->oid, 0)) return; - show_object(obj, name, data); + show_object(obj, name, show_data, carry_data); } -static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data) +static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *show_data, void *carry_data) { assert(arg_missing_action == MA_ALLOW_PROMISOR); @@ -3299,7 +3299,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name, if (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid)) return; - show_object(obj, name, data); + show_object(obj, name, show_data, carry_data); } static int option_parse_missing_action(const struct option *opt, @@ -3547,7 +3547,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs) static void record_recent_object(struct object *obj, const char *name, - void *data) + void *show_data, + void *carry_data) { oid_array_append(&recent_objects, &obj->oid); } diff --git a/builtin/rev-list.c b/builtin/rev-list.c index b4d8ea0a35..1cad33d9e8 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -266,7 +266,7 @@ static int finish_object(struct object *obj, const char *name, void *cb_data) return 0; } -static void show_object(struct object *obj, const char *name, void *cb_data) +static void show_object(struct object *obj, const char *name, void *cb_data, void *carry_data) { struct rev_list_info *info = cb_data; struct rev_info *revs = info->revs; diff --git a/list-objects.c b/list-objects.c index e19589baa0..427228a3ba 100644 --- a/list-objects.c +++ b/list-objects.c @@ -60,7 +60,7 @@ static void process_blob(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, path->buf, ctx->show_data); + ctx->show_object(obj, path->buf, ctx->show_data, NULL); strbuf_setlen(path, pathlen); } @@ -191,7 +191,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); + ctx->show_object(obj, base->buf, ctx->show_data, NULL); if (base->len) strbuf_addch(base, '/'); @@ -207,7 +207,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); + ctx->show_object(obj, base->buf, ctx->show_data, NULL); strbuf_setlen(base, baselen); free_tree_buffer(tree); @@ -335,7 +335,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - ctx->show_object(obj, name, ctx->show_data); + ctx->show_object(obj, name, ctx->show_data, NULL); continue; } if (!path) diff --git a/list-objects.h b/list-objects.h index a952680e46..ab946d34db 100644 --- a/list-objects.h +++ b/list-objects.h @@ -6,7 +6,7 @@ struct object; struct rev_info; typedef void (*show_commit_fn)(struct commit *, void *); -typedef void (*show_object_fn)(struct object *, const char *, void *); +typedef void (*show_object_fn)(struct object *, const char *, void *, void *); void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *); typedef void (*show_edge_fn)(struct commit *); diff --git a/pack-bitmap.c b/pack-bitmap.c index 3ed15431cd..516eb235da 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -459,9 +459,9 @@ struct bitmap_show_data { struct bitmap *base; }; -static void show_object(struct object *object, const char *name, void *data_) +static void show_object(struct object *object, const char *name, void *show_data, void *carry_data) { - struct bitmap_show_data *data = data_; + struct bitmap_show_data *data = show_data; int bitmap_pos; bitmap_pos = bitmap_position(data->bitmap_git, &object->oid); @@ -1268,9 +1268,9 @@ struct bitmap_test_data { }; static void test_show_object(struct object *object, const char *name, - void *data) + void *show_data, void *carry_data) { - struct bitmap_test_data *tdata = data; + struct bitmap_test_data *tdata = show_data; int bitmap_pos; bitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid); diff --git a/reachable.c b/reachable.c index 77a60c70a5..521b39edef 100644 --- a/reachable.c +++ b/reachable.c @@ -47,14 +47,14 @@ static int add_one_ref(const char *path, const struct object_id *oid, * The traversal will have already marked us as SEEN, so we * only need to handle any progress reporting here. */ -static void mark_object(struct object *obj, const char *name, void *data) +static void mark_object(struct object *obj, const char *name, void *show_data, void *carry_data) { - update_progress(data); + update_progress(show_data); } -static void mark_commit(struct commit *c, void *data) +static void mark_commit(struct commit *c, void *show_data) { - mark_object(&c->object, NULL, data); + mark_object(&c->object, NULL, show_data, NULL); } struct recent_data { From patchwork Wed Aug 25 02:21:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF6C2C432BE for ; Wed, 25 Aug 2021 02:22:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 932F561184 for ; Wed, 25 Aug 2021 02:22:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237165AbhHYCXA (ORCPT ); Tue, 24 Aug 2021 22:23:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237293AbhHYCXA (ORCPT ); Tue, 24 Aug 2021 22:23:00 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 387FBC061764 for ; Tue, 24 Aug 2021 19:22:15 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id y11so19998367pfl.13 for ; Tue, 24 Aug 2021 19:22:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8uHVEWNAxsgm1qYxleTs0BTAlHJZcZoB0UcOymXT2Gk=; b=fnFpCyzteFTJidZo9qlq6iesEsKBKesBZwqIzHxHlmlAdC0aQeEnqQ0Zp/CGVEDfd3 1nXRXeLFlOZxad/NerGAEFLeYBpDPW4NAkPBc4y22CwiXuFvuvs+TJT5PB+vlu2OpvmW cgYXIg/h6fFfbveuLsb/f6akAn6/ikq1XjvIjgQRCB0285qmtsUmtPl0pH3NBv3MFy9D 8TMMNpx4+3OtF8v7j+LuOq2Wq4R+NP50mYqsNCEcENieASjlOiNHtNjfIGku5765MDET 9y1CSm0uc3ETL9OKs3X03h0R/wnhQGGe3PbrfjmcPYOWmXgN0oRZwLf6NwVIhtqXUz7Z y46g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8uHVEWNAxsgm1qYxleTs0BTAlHJZcZoB0UcOymXT2Gk=; b=NKkPed7aBc8em4vKlnW1RDq9LFAkcQXWfxrpMlE9ybwWBQ9k7AsAoK6d6MjmPYJMM8 aT91erLehI1QYh8Lj8alPD51Vh3KF0lYq4AFTBf6d85ts8ijeJI0/CWJ1qZIHdw6FrG/ o6KOFxO/VNkQ7TQEFBFwAeCJDi/dx/OLfLemLlfRatRPpq7TFvN9tsYr+/hvaGFXt4qt vAlwONeEx9DPNDA+f76QA6Xl+JtG9jNpt8obgaoSFXGEFjjtPMuOeadtYouRiJVlSomh +51FcjnMpfZ+2fyVY0cShzAhLami31V+leppwm/s3dbm0eJOabwQD/oLnFQvYqeJgc+I +27A== X-Gm-Message-State: AOAM533394UsZ5ydiiLYPlqKNOk00sgLtuH67IQeHlSg3iDXRh4HJ4Wy CRFaogyHOzIbDhwfya3SCcQ= X-Google-Smtp-Source: ABdhPJyk+DP8q2rk9vIPI4v2dzg9e1BfUSo3s00cZtKpTlzX+poNrffBNLKpUzmsU5moVcsaVjjTIw== X-Received: by 2002:a63:ec45:: with SMTP id r5mr39965010pgj.440.1629858134617; Tue, 24 Aug 2021 19:22:14 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:14 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 03/14] packfile-uri: support for excluding commit objects Date: Wed, 25 Aug 2021 10:21:47 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Currently packfile-uri supports the exclusion of blob objects, but in some scenarios, users may wish to exclude more types of objects, such as commit and tree objects, not only because packfile itself supports storing these object types, but also on the other hand, to make configuration items maintainable and simpler. This commit is used to support the recursive exclusion of a commit object, which means that if the exclusion of a commit is configured as packfile-uri, the commit itself and all the objects it contains will also be recursively excluded. In addition, to support this feature, a new configuration `uploadpack.excludeobject` is introduced. The reason for bringing a new configuration is for two considerations. First, the old configuration supports a single object type (blob), which limits the use of this feature. Secondly, the name of the old configuration is not abstract enough, this make extension difficult. If different object types use different configuration names, the configuration items will be bloated and difficult to maintain, so the new configuration is more abstract in name and easy to extend. Although a new configuration has been introduced, the old one is still available and compatible with the new configuration. The old configuration `uploadpack.blobpackfileuri` only supports excluding blobs. The new configuration `uploadpack.excludeobject` not only supports excluding blob objects, but also supports excluding commit objects, as well as recursively excluding tree objects and blob objects they contain. Signed-off-by: Teng Long --- builtin/pack-objects.c | 50 +++++++++++++++++++++++++----------------- list-objects.c | 37 +++++++++++++++++-------------- object.c | 15 ++++++++++--- object.h | 4 +++- revision.c | 34 ++++++++++++++++++++-------- revision.h | 3 +++ upload-pack.c | 7 ++++++ 7 files changed, 101 insertions(+), 49 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 5f9ec3566f..63f3aed70a 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1311,11 +1311,14 @@ static int want_object_in_pack_one(struct packed_git *p, static int want_object_in_pack(const struct object_id *oid, int exclude, struct packed_git **found_pack, - off_t *found_offset) + off_t *found_offset, + struct object *referred_commit) { int want; struct list_head *pos; struct multi_pack_index *m; + struct configured_exclusion *ex; + struct configured_exclusion *referred_ex; if (!exclude && local && has_loose_object_nonlocal(oid)) return 0; @@ -1351,9 +1354,13 @@ static int want_object_in_pack(const struct object_id *oid, } if (uri_protocols.nr) { - struct configured_exclusion *ex = - oidmap_get(&configured_exclusions, oid); + if (referred_commit) { + referred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid); + if (referred_ex && match_packfile_uri_exclusions(referred_ex)) + return 0; + } + ex = oidmap_get(&configured_exclusions, oid); if (ex && match_packfile_uri_exclusions(ex)) { oidset_insert(&excluded_by_config, oid); return 0; @@ -1393,7 +1400,8 @@ static const char no_closure_warning[] = N_( ); static int add_object_entry(const struct object_id *oid, enum object_type type, - const char *name, int exclude) + const char *name, int exclude, + struct object *referred_commit) { struct packed_git *found_pack = NULL; off_t found_offset = 0; @@ -1403,7 +1411,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type, if (have_duplicate_entry(oid, exclude)) return 0; - if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) { + if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) { /* The pack is missing an object, so it will not have closure */ if (write_bitmap_index) { if (write_bitmap_index != WRITE_BITMAP_QUIET) @@ -1429,7 +1437,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid, if (have_duplicate_entry(oid, 0)) return 0; - if (!want_object_in_pack(oid, 0, &pack, &offset)) + if (!want_object_in_pack(oid, 0, &pack, &offset, NULL)) return 0; create_object_entry(oid, type, name_hash, 0, 0, pack, offset); @@ -1569,7 +1577,7 @@ static void add_pbase_object(struct tree_desc *tree, if (name[cmplen] != '/') { add_object_entry(&entry.oid, object_type(entry.mode), - fullname, 1); + fullname, 1, NULL); return; } if (S_ISDIR(entry.mode)) { @@ -1637,7 +1645,7 @@ static void add_preferred_base_object(const char *name) cmplen = name_cmp_len(name); for (it = pbase_tree; it; it = it->next) { if (cmplen == 0) { - add_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1); + add_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL); } else { struct tree_desc tree; @@ -2839,7 +2847,7 @@ static void add_tag_chain(const struct object_id *oid) die(_("unable to pack objects reachable from tag %s"), oid_to_hex(oid)); - add_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0); + add_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL); if (tag->tagged->type != OBJ_TAG) return; @@ -2994,7 +3002,7 @@ static int git_pack_config(const char *k, const char *v, void *cb) pack_idx_opts.flags &= ~WRITE_REV; return 0; } - if (!strcmp(k, "uploadpack.blobpackfileuri")) { + if (!strcmp(k, "uploadpack.excludeobject") || !strcmp(k, "uploadpack.blobpackfileuri")) { struct configured_exclusion *ex = xmalloc(sizeof(*ex)); const char *oid_end, *pack_end; /* @@ -3007,11 +3015,11 @@ static int git_pack_config(const char *k, const char *v, void *cb) *oid_end != ' ' || parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) || *pack_end != ' ') - die(_("value of uploadpack.blobpackfileuri must be " + die(_("value of uploadpack.excludeobject or uploadpack.blobpackfileuri must be " "of the form ' ' (got '%s')"), v); if (oidmap_get(&configured_exclusions, &ex->e.oid)) - die(_("object already configured in another " - "uploadpack.blobpackfileuri (got '%s')"), v); + die(_("object already configured by an earlier " + "uploadpack.excludeobject or uploadpack.blobpackfileuri (got '%s')"), v); ex->pack_hash_hex = xcalloc(1, pack_end - oid_end); memcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1); ex->uri = xstrdup(pack_end + 1); @@ -3040,7 +3048,7 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; ofs = nth_packed_object_offset(p, pos); - if (!want_object_in_pack(oid, 0, &p, &ofs)) + if (!want_object_in_pack(oid, 0, &p, &ofs, NULL)) return 0; oi.typep = &type; @@ -3233,7 +3241,7 @@ static void read_object_list_from_stdin(void) die(_("expected object ID, got garbage:\n %s"), line); add_preferred_base_object(p + 1); - add_object_entry(&oid, OBJ_NONE, p + 1, 0); + add_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL); } } @@ -3242,7 +3250,7 @@ static void read_object_list_from_stdin(void) static void show_commit(struct commit *commit, void *data) { - add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0); + add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL); commit->object.flags |= OBJECT_ADDED; if (write_bitmap_index) @@ -3254,8 +3262,9 @@ static void show_commit(struct commit *commit, void *data) static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data) { + struct object *referred_commit = carry_data; add_preferred_base_object(name); - add_object_entry(&obj->oid, obj->type, name, 0); + add_object_entry(&obj->oid, obj->type, name, 0, referred_commit); obj->flags |= OBJECT_ADDED; if (use_delta_islands) { @@ -3406,7 +3415,7 @@ static void add_objects_in_unpacked_packs(void) QSORT(in_pack.array, in_pack.nr, ofscmp); for (i = 0; i < in_pack.nr; i++) { struct object *o = in_pack.array[i].object; - add_object_entry(&o->oid, o->type, "", 0); + add_object_entry(&o->oid, o->type, "", 0, NULL); } } free(in_pack.array); @@ -3422,7 +3431,7 @@ static int add_loose_object(const struct object_id *oid, const char *path, return 0; } - add_object_entry(oid, type, "", 0); + add_object_entry(oid, type, "", 0, NULL); return 0; } @@ -3841,7 +3850,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) N_("respect islands during delta compression")), OPT_STRING_LIST(0, "uri-protocol", &uri_protocols, N_("protocol"), - N_("exclude any configured uploadpack.blobpackfileuri with this protocol")), + N_("exclude any configured uploadpack.excludeobject or " + "uploadpack.blobpackfileuri with this protocol")), OPT_END(), }; diff --git a/list-objects.c b/list-objects.c index 427228a3ba..968d842ceb 100644 --- a/list-objects.c +++ b/list-objects.c @@ -24,7 +24,8 @@ struct traversal_context { static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, - const char *name) + const char *name, + struct object *referred_commit) { struct object *obj = &blob->object; size_t pathlen; @@ -60,7 +61,7 @@ static void process_blob(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, path->buf, ctx->show_data, NULL); + ctx->show_object(obj, path->buf, ctx->show_data, referred_commit); strbuf_setlen(path, pathlen); } @@ -97,11 +98,13 @@ static void process_gitlink(struct traversal_context *ctx, static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, - const char *name); + const char *name, + struct object *referred_commit); static void process_tree_contents(struct traversal_context *ctx, struct tree *tree, - struct strbuf *base) + struct strbuf *base, + struct object *referred_commit) { struct tree_desc desc; struct name_entry entry; @@ -129,7 +132,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } t->object.flags |= NOT_USER_GIVEN; - process_tree(ctx, t, base, entry.path); + process_tree(ctx, t, base, entry.path, referred_commit); } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid.hash, @@ -142,7 +145,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } b->object.flags |= NOT_USER_GIVEN; - process_blob(ctx, b, base, entry.path); + process_blob(ctx, b, base, entry.path, referred_commit); } } } @@ -150,7 +153,8 @@ static void process_tree_contents(struct traversal_context *ctx, static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, - const char *name) + const char *name, + struct object *referred_commit) { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; @@ -191,14 +195,14 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data, NULL); + ctx->show_object(obj, base->buf, ctx->show_data, referred_commit); if (base->len) strbuf_addch(base, '/'); if (r & LOFR_SKIP_TREE) trace_printf("Skipping contents of tree %s...\n", base->buf); else if (!failed_parse) - process_tree_contents(ctx, tree, base); + process_tree_contents(ctx, tree, base, referred_commit); r = list_objects_filter__filter_object(ctx->revs->repo, LOFS_END_TREE, obj, @@ -207,7 +211,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data, NULL); + ctx->show_object(obj, base->buf, ctx->show_data, referred_commit); strbuf_setlen(base, baselen); free_tree_buffer(tree); @@ -314,9 +318,9 @@ void mark_edges_uninteresting(struct rev_info *revs, } } -static void add_pending_tree(struct rev_info *revs, struct tree *tree) +static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct object *referred_commit) { - add_pending_object(revs, &tree->object, ""); + add_pending_object_with_referred_commit(revs, &tree->object, "", referred_commit); } static void traverse_trees_and_blobs(struct traversal_context *ctx, @@ -329,23 +333,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, for (i = 0; i < ctx->revs->pending.nr; i++) { struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; + struct object *referred_commit = pending->referred_commit; const char *name = pending->name; const char *path = pending->path; if (obj->flags & (UNINTERESTING | SEEN)) continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - ctx->show_object(obj, name, ctx->show_data, NULL); + ctx->show_object(obj, name, ctx->show_data, referred_commit); continue; } if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(ctx, (struct tree *)obj, base, path); + process_tree(ctx, (struct tree *)obj, base, path, referred_commit); continue; } if (obj->type == OBJ_BLOB) { - process_blob(ctx, (struct blob *)obj, base, path); + process_blob(ctx, (struct blob *)obj, base, path, referred_commit); continue; } die("unknown pending object %s (%s)", @@ -370,7 +375,7 @@ static void do_traverse(struct traversal_context *ctx) else if (get_commit_tree(commit)) { struct tree *tree = get_commit_tree(commit); tree->object.flags |= NOT_USER_GIVEN; - add_pending_tree(ctx->revs, tree); + add_pending_tree(ctx->revs, tree, &commit->object); } else if (commit->object.parsed) { die(_("unable to load root tree for commit %s"), oid_to_hex(&commit->object.oid)); diff --git a/object.c b/object.c index 14188453c5..6b1ce2fcde 100644 --- a/object.c +++ b/object.c @@ -322,9 +322,10 @@ void object_list_free(struct object_list **list) */ static char object_array_slopbuf[1]; -void add_object_array_with_path(struct object *obj, const char *name, - struct object_array *array, - unsigned mode, const char *path) +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, + struct object_array *array, + unsigned mode, const char *path, + struct object *referred_commit) { unsigned nr = array->nr; unsigned alloc = array->alloc; @@ -339,6 +340,7 @@ void add_object_array_with_path(struct object *obj, const char *name, } entry = &objects[nr]; entry->item = obj; + entry->referred_commit = referred_commit; if (!name) entry->name = NULL; else if (!*name) @@ -354,6 +356,13 @@ void add_object_array_with_path(struct object *obj, const char *name, array->nr = ++nr; } +void add_object_array_with_path(struct object *obj, const char *name, + struct object_array *array, + unsigned mode, const char *path) +{ + add_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL); +} + void add_object_array(struct object *obj, const char *name, struct object_array *array) { add_object_array_with_path(obj, name, array, S_IFINVALID, NULL); diff --git a/object.h b/object.h index 87a6da47c8..d63819ab91 100644 --- a/object.h +++ b/object.h @@ -52,6 +52,7 @@ struct object_array { char *name; char *path; unsigned mode; + struct object *referred_commit; } *objects; }; @@ -157,7 +158,8 @@ void object_list_free(struct object_list **list); /* Object array handling .. */ void add_object_array(struct object *obj, const char *name, struct object_array *array); void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path); - +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array, + unsigned mode, const char *path, struct object *referred_commit); /* * Returns NULL if the array is empty. Otherwise, returns the last object * after removing its entry from the array. Other resources associated diff --git a/revision.c b/revision.c index 4853c85d0b..65e0926d25 100644 --- a/revision.c +++ b/revision.c @@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit) commit_stack_clear(&pending); } -static void add_pending_object_with_path(struct rev_info *revs, - struct object *obj, - const char *name, unsigned mode, - const char *path) +static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs, + struct object *obj, + const char *name, unsigned mode, + const char *path, + struct object *referred_commit) { struct interpret_branch_name_options options = { 0 }; if (!obj) @@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs, strbuf_release(&buf); return; /* do not add the commit itself */ } - add_object_array_with_path(obj, name, &revs->pending, mode, path); + add_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit); } +static void add_pending_object_with_path(struct rev_info *revs, + struct object *obj, + const char *name, unsigned mode, + const char *path) +{ + add_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL); +} static void add_pending_object_with_mode(struct rev_info *revs, struct object *obj, - const char *name, unsigned mode) + const char *name, unsigned mode, + struct object *referred_commit) +{ + add_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit); +} + +void add_pending_object_with_referred_commit(struct rev_info *revs, + struct object *obj, const char *name, + struct object *referred_commit) { - add_pending_object_with_path(revs, obj, name, mode, NULL); + add_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit); } void add_pending_object(struct rev_info *revs, struct object *obj, const char *name) { - add_pending_object_with_mode(revs, obj, name, S_IFINVALID); + add_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL); } void add_head_to_pending(struct rev_info *revs) @@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s if (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc)) diagnose_missing_default(revs->def); object = get_reference(revs, revs->def, &oid, 0); - add_pending_object_with_mode(revs, object, revs->def, oc.mode); + add_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL); } /* Did the user ask for any diff output? Run the diff! */ diff --git a/revision.h b/revision.h index a24f72dcd1..f9c9628ed8 100644 --- a/revision.h +++ b/revision.h @@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *); */ void add_pending_object(struct rev_info *revs, struct object *obj, const char *name); +void add_pending_object_with_referred_commit(struct rev_info *revs, + struct object *obj, const char *name, + struct object *referred_commit); void add_pending_oid(struct rev_info *revs, const char *name, const struct object_id *oid, diff --git a/upload-pack.c b/upload-pack.c index 5c1cd19612..d26fb351a3 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r, strbuf_addstr(value, " packfile-uris"); free(str); } + + if (!repo_config_get_string(the_repository, + "uploadpack.excludeobject", + &str) && str) { + strbuf_addstr(value, " packfile-uris"); + free(str); + } } return 1; From patchwork Wed Aug 25 02:21:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E07D7C4338F for ; Wed, 25 Aug 2021 02:22:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B7DB461176 for ; Wed, 25 Aug 2021 02:22:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237293AbhHYCXD (ORCPT ); Tue, 24 Aug 2021 22:23:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237168AbhHYCXC (ORCPT ); Tue, 24 Aug 2021 22:23:02 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B43F3C061757 for ; Tue, 24 Aug 2021 19:22:17 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id oc2-20020a17090b1c0200b00179e56772d6so3104769pjb.4 for ; Tue, 24 Aug 2021 19:22:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9yjpYoFGBvV2gdMmkmCOsuPvp0NJ6hOG4dvt5OsmB+A=; b=mYvt2GI+k8GEIn4oWcGdQYJq/op06sRFouAVSXDt3p14x8v2jGF6Elz03Ps74iP9YZ raZC+2FlP0JT1QBb4HBwYZ32MRZ0h6IWGxvtrWWDmHJzt5QHWzL7XdBtj7vdR+QpM2CG jRKCsFPk33sFBpitpmmx8j5ZL7rrlbcVRt+cr9LAY1Tf+zCk5nX3zdgC2L1DVNnu6woQ 1zytKX5JaJKH82S1tQ6bmBJdU6I4b7XNwgmBaSEbhU756HDqAeUYTJ5DsRfXaLnjCleA vWwEEFekrYREwrJboxz5qhpN/nnWAcp24JBTZEn7UxZwWQ2VdtKMQG8n9ac9efoOy3s1 jaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9yjpYoFGBvV2gdMmkmCOsuPvp0NJ6hOG4dvt5OsmB+A=; b=JCMsNAO73Tq63ZtvKKMfjpX/UceO4vvNoSFagyTEmtE6tjn+SbMUDJqO796gmSR/2+ jZg58jc2ZT/urUM0sAnuFObvwqbjQoedGiJ0ElrxhGVJZf8JW18ZxfVJkHbIvic38GzB s4m6JZBc66o6LRANuQmtgHdf/SDFjbEMkRaBumKgqqm441MSZidJfUgp4jHj7/AqzRww 8Ak7zzurCSaS5H1e2j90mih9+Ewk8IlsMOeKIuoLX+TXoPrRzHmCUd8lwCT1rQ4x/y1d HcCBNsWybPe2Pt8PAZ4y+zisedldCSBieYyQVJAA2JYVT1cAk/pKfZMnLqRtlE9Su3S9 oYQw== X-Gm-Message-State: AOAM531xAHmD1t6dlIW9Mz7QhCKzPr+drfgN7SmzwzuqwkCZ+cp5swUZ M9xcy1lclYEYTMr1XcTjEC0mILZD93ePj4Uh X-Google-Smtp-Source: ABdhPJy5ORVamiSuIOsgLARRa/1YUuZF3BAauNhhLvwDhChJ9pxx5lkjM7IlVA3Rn8Gaq18y9WoZkw== X-Received: by 2002:a17:902:74c3:b0:132:45eb:6e9c with SMTP id f3-20020a17090274c300b0013245eb6e9cmr18281971plt.72.1629858137251; Tue, 24 Aug 2021 19:22:17 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:16 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 04/14] packfile-uri: support for excluding tree objects Date: Wed, 25 Aug 2021 10:21:48 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This commit supports the use of `uploadpack.excludeobject` to exclude tree objects, which means that when a type object is configured as packfile-uri, the tree object itself and all objects contains will be recursively excluded. Signed-off-by: Teng Long --- builtin/pack-objects.c | 30 ++++++++++++++++++++---------- list-objects.c | 32 ++++++++++++++++++-------------- object.c | 6 +++++- object.h | 13 ++++++++++++- 4 files changed, 55 insertions(+), 26 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 63f3aed70a..4ff12ec525 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1312,13 +1312,14 @@ static int want_object_in_pack(const struct object_id *oid, int exclude, struct packed_git **found_pack, off_t *found_offset, - struct object *referred_commit) + struct referred_objects *referred_objs) { int want; struct list_head *pos; struct multi_pack_index *m; + struct configured_exclusion *commit_ex; + struct configured_exclusion *tree_ex; struct configured_exclusion *ex; - struct configured_exclusion *referred_ex; if (!exclude && local && has_loose_object_nonlocal(oid)) return 0; @@ -1354,14 +1355,23 @@ static int want_object_in_pack(const struct object_id *oid, } if (uri_protocols.nr) { - if (referred_commit) { - referred_ex = oidmap_get(&configured_exclusions, &referred_commit->oid); - if (referred_ex && match_packfile_uri_exclusions(referred_ex)) + if (referred_objs && referred_objs->commit) { + commit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid); + if (match_packfile_uri_exclusions(commit_ex)) return 0; } + if (referred_objs && referred_objs->trees) { + struct object_list *p; + for (p = referred_objs->trees; p; p = p->next) { + tree_ex = oidmap_get(&configured_exclusions, &p->item->oid); + if (match_packfile_uri_exclusions(tree_ex)) + return 0; + } + } + ex = oidmap_get(&configured_exclusions, oid); - if (ex && match_packfile_uri_exclusions(ex)) { + if (match_packfile_uri_exclusions(ex)) { oidset_insert(&excluded_by_config, oid); return 0; } @@ -1401,7 +1411,7 @@ static const char no_closure_warning[] = N_( static int add_object_entry(const struct object_id *oid, enum object_type type, const char *name, int exclude, - struct object *referred_commit) + struct referred_objects *referred_objs) { struct packed_git *found_pack = NULL; off_t found_offset = 0; @@ -1411,7 +1421,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type, if (have_duplicate_entry(oid, exclude)) return 0; - if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_commit)) { + if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_objs)) { /* The pack is missing an object, so it will not have closure */ if (write_bitmap_index) { if (write_bitmap_index != WRITE_BITMAP_QUIET) @@ -3262,9 +3272,9 @@ static void show_commit(struct commit *commit, void *data) static void show_object(struct object *obj, const char *name, void *show_data, void *carry_data) { - struct object *referred_commit = carry_data; + struct referred_objects *referred_objs = carry_data; add_preferred_base_object(name); - add_object_entry(&obj->oid, obj->type, name, 0, referred_commit); + add_object_entry(&obj->oid, obj->type, name, 0, referred_objs); obj->flags |= OBJECT_ADDED; if (use_delta_islands) { diff --git a/list-objects.c b/list-objects.c index 968d842ceb..49f177cb56 100644 --- a/list-objects.c +++ b/list-objects.c @@ -25,7 +25,7 @@ static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, const char *name, - struct object *referred_commit) + struct referred_objects *referred_objs) { struct object *obj = &blob->object; size_t pathlen; @@ -61,7 +61,7 @@ static void process_blob(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, path->buf, ctx->show_data, referred_commit); + ctx->show_object(obj, path->buf, ctx->show_data, referred_objs); strbuf_setlen(path, pathlen); } @@ -99,19 +99,22 @@ static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, const char *name, - struct object *referred_commit); + struct referred_objects *referred_objs); static void process_tree_contents(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, - struct object *referred_commit) + struct referred_objects *referred_objs) { struct tree_desc desc; struct name_entry entry; enum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ? all_entries_interesting : entry_not_interesting; + struct referred_objects *referred_buf; init_tree_desc(&desc, tree->buffer, tree->size); + referred_buf = xmemdupz(referred_objs, sizeof(struct referred_objects)); + object_list_insert(&tree->object, &referred_buf->trees); while (tree_entry(&desc, &entry)) { if (match != all_entries_interesting) { @@ -132,7 +135,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } t->object.flags |= NOT_USER_GIVEN; - process_tree(ctx, t, base, entry.path, referred_commit); + process_tree(ctx, t, base, entry.path, referred_buf); } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid.hash, @@ -145,16 +148,17 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } b->object.flags |= NOT_USER_GIVEN; - process_blob(ctx, b, base, entry.path, referred_commit); + process_blob(ctx, b, base, entry.path, referred_buf); } } + free(referred_buf); } static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, const char *name, - struct object *referred_commit) + struct referred_objects *referred_objs) { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; @@ -195,14 +199,14 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data, referred_commit); + ctx->show_object(obj, base->buf, ctx->show_data, referred_objs); if (base->len) strbuf_addch(base, '/'); if (r & LOFR_SKIP_TREE) trace_printf("Skipping contents of tree %s...\n", base->buf); else if (!failed_parse) - process_tree_contents(ctx, tree, base, referred_commit); + process_tree_contents(ctx, tree, base, referred_objs); r = list_objects_filter__filter_object(ctx->revs->repo, LOFS_END_TREE, obj, @@ -211,7 +215,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data, referred_commit); + ctx->show_object(obj, base->buf, ctx->show_data, referred_objs); strbuf_setlen(base, baselen); free_tree_buffer(tree); @@ -333,24 +337,24 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, for (i = 0; i < ctx->revs->pending.nr; i++) { struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; - struct object *referred_commit = pending->referred_commit; + struct referred_objects *referred_objs = pending->referred_objects; const char *name = pending->name; const char *path = pending->path; if (obj->flags & (UNINTERESTING | SEEN)) continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - ctx->show_object(obj, name, ctx->show_data, referred_commit); + ctx->show_object(obj, name, ctx->show_data, referred_objs); continue; } if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(ctx, (struct tree *)obj, base, path, referred_commit); + process_tree(ctx, (struct tree *)obj, base, path, referred_objs); continue; } if (obj->type == OBJ_BLOB) { - process_blob(ctx, (struct blob *)obj, base, path, referred_commit); + process_blob(ctx, (struct blob *)obj, base, path, referred_objs); continue; } die("unknown pending object %s (%s)", diff --git a/object.c b/object.c index 6b1ce2fcde..69ba0baf95 100644 --- a/object.c +++ b/object.c @@ -331,6 +331,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch unsigned alloc = array->alloc; struct object_array_entry *objects = array->objects; struct object_array_entry *entry; + struct referred_objects *referred_objs = xmalloc(sizeof(struct referred_objects)); if (nr >= alloc) { alloc = (alloc + 32) * 2; @@ -338,9 +339,11 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch array->alloc = alloc; array->objects = objects; } + referred_objs->commit = referred_commit; + referred_objs->trees = NULL; entry = &objects[nr]; entry->item = obj; - entry->referred_commit = referred_commit; + entry->referred_objects = referred_objs; if (!name) entry->name = NULL; else if (!*name) @@ -377,6 +380,7 @@ static void object_array_release_entry(struct object_array_entry *ent) if (ent->name != object_array_slopbuf) free(ent->name); free(ent->path); + free(ent->referred_objects); } struct object *object_array_pop(struct object_array *array) diff --git a/object.h b/object.h index d63819ab91..3785546adf 100644 --- a/object.h +++ b/object.h @@ -52,12 +52,23 @@ struct object_array { char *name; char *path; unsigned mode; - struct object *referred_commit; + /* + * referred_objects or NULL. If non-NULL, it will + * temporary storage the referred commit and trees when + * traversing the specified object. Space for time, + * reduce related computing costs (such as packfile-uri + * exclusion), clean up when the traversal is over. + */ + struct referred_objects *referred_objects; } *objects; }; #define OBJECT_ARRAY_INIT { 0, 0, NULL } +struct referred_objects{ + struct object *commit; + struct object_list *trees; +}; /* * object flag allocation: * revision.h: 0---------10 15 23------26 From patchwork Wed Aug 25 02:21:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C211BC4338F for ; Wed, 25 Aug 2021 02:22:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A5D1D61184 for ; Wed, 25 Aug 2021 02:22:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237335AbhHYCXG (ORCPT ); Tue, 24 Aug 2021 22:23:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237380AbhHYCXF (ORCPT ); Tue, 24 Aug 2021 22:23:05 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B64AC061796 for ; Tue, 24 Aug 2021 19:22:20 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id j10-20020a17090a94ca00b00181f17b7ef7so3122614pjw.2 for ; Tue, 24 Aug 2021 19:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q2AAyi3xdKM2quMoj/wE8DktKqVER/T+ZtH6I+HstiU=; b=lxarbMWWF2kIwrMh1LtY4n3NRoUFDpuTZS8k8TnElDgeAOjJr3j+pFv9U3uFWUyZbJ MYiNy6gzqbNPGe0iWWGqf23OAmPHGI5qHTtPQHa4Nr9g1AB+J5wnoH9yC3xTe9Lq3mGY 9hzTn9MzVheD5a9oWHjp5zW+O408SbbpIPUsI3kMmncL5fmgth81O/rOSM/lfnGCJrBD 7LipEqZYAkxXNbEjOljVtoIJqpTCMecx5HyfulUDhJ6nCBzQWv+BEErfOXC8KB5UL7aZ AKYYK+7f9UaESTMZupcIhh9EWKiZKm/7fcV/6/vB6D5U4ide5J3zLKtBVUmI+KXVb6jp 6zVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q2AAyi3xdKM2quMoj/wE8DktKqVER/T+ZtH6I+HstiU=; b=BuO3E9fmjTQ9pEnXRZmqwAlDSvMivRhO+Iq7ZFOG9GLdNKx4ljWpw+ugdRYvTGk73d fBQAmBJ8QahPpNtgTpLOkU+lS/mg4TyZ2GkQ8nKxFxG+suFzyQllo/XQlSN4Bw0EWEnY FhxiuN9pMI/uFLvG2I6ArFIlxzvfH1Sx+EfPNVSWdNQQWzKbRKRu/u1IQ9eWWByhdtEl wFwzHZkP/4piHy9LEgP6wk18Bs++Ss4yqwBZtHOWwXY1aeCzeAnJ580eMO0AX66e+42e ArLsmJaj1fWZSc+G/QD9vAMkrgQneVi4yjUap7/h7sly+zS7nuN2QuJ56razOWZtJSal lbgg== X-Gm-Message-State: AOAM531eAvxJlNUspSHn6245JdaNW6mltYGT0PjysxKJRG2OFL+IFKg/ TmY9FyQGBqWLMG1LXkBBYmI= X-Google-Smtp-Source: ABdhPJy5BuUVmLBg/5puTgTNAz+CF8zDP5JNsTz2YWqMXMz/Lg7N+K0+MspKjAmi1HfeqEsjBH0RIQ== X-Received: by 2002:a17:902:dcca:b0:12d:7f02:f6a2 with SMTP id t10-20020a170902dcca00b0012d7f02f6a2mr35982039pll.7.1629858139844; Tue, 24 Aug 2021 19:22:19 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:19 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 05/14] packfile-uri.txt: support for excluding commits and trees Date: Wed, 25 Aug 2021 10:21:49 +0800 Message-Id: <8e5bf4010cb0d7e8b4f6c7285fe1f5365af955e8.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- Documentation/technical/packfile-uri.txt | 32 ++++++++++++++++-------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt index f7eabc6c76..c3e4873956 100644 --- a/Documentation/technical/packfile-uri.txt +++ b/Documentation/technical/packfile-uri.txt @@ -35,13 +35,26 @@ include some sort of non-trivial implementation in the Minimum Viable Product, at least so that we can test the client. This is the implementation: a feature, marked experimental, that allows the -server to be configured by one or more `uploadpack.blobPackfileUri= -` entries. Whenever the list of objects to be sent is assembled, all such -blobs are excluded, replaced with URIs. As noted in "Future work" below, the -server can evolve in the future to support excluding other objects (or other -implementations of servers could be made that support excluding other objects) -without needing a protocol change, so clients should not expect that packfiles -downloaded in this way only contain single blobs. +server to be configured by one or more entries with the format: + + uploadpack.excludeobject= + +Value is the key of entry, and the object type can be a blob, +tree, or commit. The exclusion of tree and commit is recursive by default, +which means that when a tree or commit object is excluded, the object itself +and all reachable objects of the object will be excluded recursively. Whenever +the list of objects to be sent is assembled, all such objects are excluded, +replaced with URIs. + +Configuration compatibility +------------- + +The old configuration of packfile-uri: + + `uploadpack.blobPackfileUri= ` + +For the old configuration is compatible with the new one, but it only +supports the exclusion of blob objects. Client design ------------- @@ -65,9 +78,6 @@ The protocol design allows some evolution of the server and client without any need for protocol changes, so only a small-scoped design is included here to form the MVP. For example, the following can be done: - * On the server, more sophisticated means of excluding objects (e.g. by - specifying a commit to represent that commit and all objects that it - references). * On the client, resumption of clone. If a clone is interrupted, information could be recorded in the repository's config and a "clone-resume" command can resume the clone in progress. (Resumption of subsequent fetches is more @@ -78,4 +88,4 @@ There are some possible features that will require a change in protocol: * Additional HTTP headers (e.g. authentication) * Byte range support - * Different file formats referenced by URIs (e.g. raw object) + * Different file formats referenced by URIs (e.g. raw object) \ No newline at end of file From patchwork Wed Aug 25 02:21:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96E2CC432BE for ; Wed, 25 Aug 2021 02:22:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C7F561165 for ; Wed, 25 Aug 2021 02:22:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237439AbhHYCXI (ORCPT ); Tue, 24 Aug 2021 22:23:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237313AbhHYCXH (ORCPT ); Tue, 24 Aug 2021 22:23:07 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ED1DC061757 for ; Tue, 24 Aug 2021 19:22:22 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id h1so9704389pjs.2 for ; Tue, 24 Aug 2021 19:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r0d4YQsFc37dzg9Th5FxMivQJQ7VL1aQx42dqW780Iw=; b=uvJDnR0ybNQqJQZbiZv3C5Wo6L9R4Zr5l0IXPBCO0MkLAGyE7RvyLGyrRVvhSXEugD AbaLMDfdVSj6I60MT/AffpI1hDyUzPbcwgDIOPz7VESM8WICY/roDYZRgDqFwU0wgKb4 40RzK9yzKHHmxsVy3b/qo/rWP1FHFpD/zsG8cJ/yZ8cJO/APixL87p9N1LwS+GIEA7gU YS11CSUH6SIyjnfHMXc/DGSskecLrc+XSYF+/tz+4jGZJdVi4zARrfcgQf0SeiabN1mC +cGmo0qhfndTNHw6ynXsa7QsJHfn0df/9FEs6AdzTM5qFtWNPuZATUzN/mlgy42aOd7N SKHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r0d4YQsFc37dzg9Th5FxMivQJQ7VL1aQx42dqW780Iw=; b=tdvZyyHYLw5R35KXw1H3OjzgWcHa2ri+G6noJwRpk89mFQcQghxOX3ZnHAmahhOPgi A/pX09M0VH/1Rs7//RRORGs2iqhl8m8/FJxYMoMBqV7O8GXbdo1UhCrsS/uobHEckM3A 8bw7JRux0cPr9zhiu/hrnazkksH6XKXrOZE9+e08KVfSsZTWgTyllo/xhzG3K8xIbhFI U8hVZYSKMcd/LPgXprHN4ijJxubb+q161ldMX/SMx4JdpiMrEf6cwQL4WLq0umP9cEG3 sTk2e1NZNyI6Ea9e+tP/AVToWKXQOcYKFFAF7/dlhGOZ8ec5vlAwxJjlglgW6WnKvddI QTnQ== X-Gm-Message-State: AOAM5309QUmxRc+F9jCGYSwi/gbkkroM9iXtpxNLfTIZNxBwzZwyCNtN 3AtExuHPcb9XUOSPH8hPGB0= X-Google-Smtp-Source: ABdhPJwxKNA5ICFwBFSnxes1H58wgoj/SUhrR2DlksKjta7WinFBPLD6WCRMzW2u0P88BYEFgvXuvg== X-Received: by 2002:a17:902:e8ce:b0:132:b140:9540 with SMTP id v14-20020a170902e8ce00b00132b1409540mr16735161plg.28.1629858142170; Tue, 24 Aug 2021 19:22:22 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:21 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 06/14] t5702: replace with "test_when_finished" for cleanup Date: Wed, 25 Aug 2021 10:21:50 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 2e1243ca40..e6314b53b0 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -753,7 +753,7 @@ test_expect_success 'ls-remote with v2 http sends only one POST' ' ' test_expect_success 'push with http:// and a config of v2 does not request v2' ' - test_when_finished "rm -f log" && + test_when_finished "rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log" && # Till v2 for push is designed, make sure that if a client has # protocol.version configured to use v2, that the client instead falls # back and uses v0. @@ -776,7 +776,7 @@ test_expect_success 'push with http:// and a config of v2 does not request v2' ' ' test_expect_success 'when server sends "ready", expect DELIM' ' - rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child && + test_when_finished "rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child" && git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one && @@ -796,7 +796,7 @@ test_expect_success 'when server sends "ready", expect DELIM' ' ' test_expect_success 'when server does not send "ready", expect FLUSH' ' - rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child log && + test_when_finished "rm -rf \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" http_child log" && git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one && @@ -834,7 +834,7 @@ configure_exclusion () { test_expect_success 'part of packfile response provided as URI' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -883,7 +883,7 @@ test_expect_success 'part of packfile response provided as URI' ' test_expect_success 'packfile URIs with fetch instead of clone' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -904,7 +904,7 @@ test_expect_success 'packfile URIs with fetch instead of clone' ' test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -935,7 +935,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' test_expect_success 'packfile-uri with transfer.fsckobjects' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -959,7 +959,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' ' test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -989,7 +989,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child && + test_when_finished "rm -rf \"$P\" http_child" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -1015,7 +1015,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child err && + test_when_finished "rm -rf \"$P\" http_child err" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -1038,4 +1038,4 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled. -test_done +test_done \ No newline at end of file From patchwork Wed Aug 25 02:21:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2773C4338F for ; Wed, 25 Aug 2021 02:22:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AAA9861165 for ; Wed, 25 Aug 2021 02:22:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237380AbhHYCXM (ORCPT ); Tue, 24 Aug 2021 22:23:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237049AbhHYCXL (ORCPT ); Tue, 24 Aug 2021 22:23:11 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18842C061757 for ; Tue, 24 Aug 2021 19:22:26 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id mq3so15520776pjb.5 for ; Tue, 24 Aug 2021 19:22:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=majvrSXHl7BgoXNmFyWu//0uxk8/uZXDOVmVHxMZ18k=; b=tm/C2BDySHkBpqPdOe2vHFNS2EBUKz1rodzeEtnPO/7CrlBCHGAq3tz+HLrQOB4S5Y f2GzHX5OgXLbwkoQ0kYULiQ/he/fIlSod0/9cSfZD17kdp+Xqmn6jv11B/KRxuSHHApH sx7o6ftw+IVZqJRmZkFWrbSjfsRMlUoJgV8xGbC3doLFYSjiIE/cz7GOkLQJ+Warc1YF hCnT/Edh3ph0ZmXlph8BGPSAPsFpZoT7GnYdRHXx/BpoL2wqxIVZTy1zH22dFYx+Bpez UHCnFCw5aSM0sKFCc7mmAhmdjRw6urpZO2U0hOEnL3bxbrCYatuMqjSccpJRZ5iw9WPx D8sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=majvrSXHl7BgoXNmFyWu//0uxk8/uZXDOVmVHxMZ18k=; b=YcjbpSrjUJEBJzUqmG2t4ciXa4M1RLdC03h7Vsg8uyRKzSt9RHBevYNvi5MPsVKWl7 3KkgnEqVa4nCaWHKh+CD7mfKDDMXddHtF7AS/4jiEb/JfFpdTp6U5L0azk35n2qm5zCg zKXEwCwJklHnybQ8E50G5PrIieWgLwgadnotw0kwLyX+o+r1RPvpw4I7Tp3Nv2gZWYFP 8TmeP3K3v2lgGspOYCk3yvRjum+s9pXAsXFXrqG/qYcAYC1i2gbBMELzCgusR0/MvYqY dQnXIVxifVgb1jwbS4HhJfOmzKsxHR0rFSu1MTIeUNaUg4VBW2xIQlGCZru21LNud/CQ k/fg== X-Gm-Message-State: AOAM531JMCTFy4t0eao2/YKKjKt3C8ZX/6oefEYxGYzKrMBkxiDfYne1 SbIy0e7FstYLAns2tdHa83M= X-Google-Smtp-Source: ABdhPJy1Tc7uX9imVR3EQipMfGvk4aI7mQpjBwkXtsMbKhKyKRMwKR4vM7aqjmDd1We42oq86mOtyA== X-Received: by 2002:a17:90a:d204:: with SMTP id o4mr8009289pju.15.1629858145632; Tue, 24 Aug 2021 19:22:25 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:25 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 07/14] t5702: support for excluding commit objects Date: Wed, 25 Aug 2021 10:21:51 +0800 Message-Id: <3b5f9732b814a712094091027c8229988e2ba791.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 292 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 266 insertions(+), 26 deletions(-) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index e6314b53b0..5ad52e0cee 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -824,17 +824,47 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' ' configure_exclusion () { - git -C "$1" hash-object "$2" >objh && - git -C "$1" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && - git -C "$1" config --add \ - "uploadpack.blobpackfileuri" \ - "$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && - cat objh + objt="$1" + P="$2" + oid="$3" + version="$4" + + oldc="uploadpack.blobpackfileuri" + newc="uploadpack.excludeobject" + configkey="" + + if test "$version" = "old" + then + configkey="$oldc" + else + configkey="$newc" + fi + + if test "$objt" = "blob" + then + git -C "$P" hash-object "$oid" >objh && + git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && + git -C "$P" config --add \ + "$configkey" \ + "$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && + cat objh + elif test "$objt" = "commit" || test "$objt" = "tree" || test "$objt" = "tag" + then + echo "$oid" >objh + git -C "$P" pack-objects --revs "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh + git -C "$P" config --add \ + "$configkey" \ + "$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && + cat objh + else + echo "unsupported object type in configure_exclusion (got $objt)" + fi } -test_expect_success 'part of packfile response provided as URI' ' +part_of_packfile_response_verify() { + config="$1" P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - test_when_finished "rm -rf \"$P\" http_child log" && + test_when_finished "rm -rf \"$P\" http_child log *found" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -843,10 +873,10 @@ test_expect_success 'part of packfile response provided as URI' ' git -C "$P" add my-blob && echo other-blob >"$P/other-blob" && git -C "$P" add other-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && - configure_exclusion "$P" other-blob >h2 && + configure_exclusion blob "$P" my-blob config >h && + configure_exclusion blob "$P" other-blob config >h2 && GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ git -c protocol.version=2 \ @@ -879,9 +909,11 @@ test_expect_success 'part of packfile response provided as URI' ' ls http_child/.git/objects/pack/*.pack \ http_child/.git/objects/pack/*.idx >filelist && test_line_count = 6 filelist -' +} + +blobpackfileuri_fetch () { + config="$1" -test_expect_success 'packfile URIs with fetch instead of clone' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_when_finished "rm -rf \"$P\" http_child log" && @@ -890,9 +922,9 @@ test_expect_success 'packfile URIs with fetch instead of clone' ' echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && + configure_exclusion blob "$P" my-blob $config >h && git init http_child && @@ -900,6 +932,215 @@ test_expect_success 'packfile URIs with fetch instead of clone' ' git -C http_child -c protocol.version=2 \ -c fetch.uriprotocols=http,https \ fetch "$HTTPD_URL/smart/http_parent" +} + +test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): part of packfile response provided as URI' ' + part_of_packfile_response_verify old +' + +test_expect_success 'blob-exclusion (using uploadpack.excludeobject): part of packfile response provided as URI' ' + part_of_packfile_response_verify new +' + +test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): packfile URIs with fetch instead of clone' ' + blobpackfileuri_fetch old +' + +test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile URIs with fetch instead of clone' ' + blobpackfileuri_fetch new +' + +test_expect_success 'tree-exclusion: part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + # Dir struct + # . + # |-- A.t + # |-- my-tree + # | `-- my-blob + # `-- other-tree + # |-- other-blob + # `-- sub-tree + # `-- sub-blob + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/other-tree && + echo other-blob >"$P"/other-tree/other-blob && + mkdir "$P"/other-tree/sub-tree && + echo sub-blob >"$P"/other-tree/sub-tree/sub-blob && + git -C "$P" add other-tree && + test_commit -C "$P" A && + + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + ah=$(git -C "$P" hash-object A.t) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + othertreeh=$(git -C "$P" ls-tree HEAD other-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD other-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + otherblobh=$(git -C "$P" hash-object other-tree/other-blob) && + subblobh=$(git -C "$P" hash-object other-tree/sub-tree/sub-blob) && + + configure_exclusion tree "$P" "$mytreeh" config >h && + configure_exclusion tree "$P" "$othertreeh" config >h2 && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + # Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 3 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi + elif test_line_count = 2 out.objectlist + then + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $myblobh out + then + >myblobhfound + fi + elif test_line_count = 4 out.objectlist + then + if grep $othertreeh out + then + >othertreehfound + fi && + if grep $otherblobh out + then + >otherblobhfound + fi + if grep $subtreeh out + then + >subtreehfound + fi && + if grep $subblobh out + then + >subblobhfound + fi + fi + done && + test -f mytreehfound && + test -f myblobhfound && + test -f othertreehfound && + test -f otherblobhfound && + test -f subtreehfound && + test -f subblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 6 filelist +' + +test_expect_success 'commit-exclusion: part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/my-tree/sub-tree && + echo sub-blob >"$P"/my-tree/sub-tree/sub-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD my-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + subblobh=$(git -C "$P" hash-object my-tree/sub-tree/sub-blob) && + + configure_exclusion commit "$P" "$commith" >h && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 7 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $myblobh out + then + >myblobhfound + fi && + if grep $subtreeh out + then + >subtreehfound + fi && + if grep $subblobh out + then + >subblobhfound + fi + fi + done && + test -f mytreehfound && + test -f myblobhfound && + test -f subtreehfound && + test -f subblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist ' test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' @@ -913,9 +1154,9 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' git -C "$P" add my-blob && echo other-blob >"$P/other-blob" && git -C "$P" add other-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && + configure_exclusion blob "$P" my-blob >h && # Configure a URL for other-blob. Just reuse the hash of the object as # the hash of the packfile, since the hash does not matter for this # test as long as it is not the hash of the pack, and it is of the @@ -923,7 +1164,7 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' git -C "$P" hash-object other-blob >objh && git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && git -C "$P" config --add \ - "uploadpack.blobpackfileuri" \ + "uploadpack.excludeobject" \ "$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && test_must_fail env GIT_TEST_SIDEBAND_ALL=1 \ @@ -942,9 +1183,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' ' echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && - - configure_exclusion "$P" my-blob >h && + test_commit -C "$P" A && + configure_exclusion blob "$P" my-blob >h && sane_unset GIT_TEST_SIDEBAND_ALL && git -c protocol.version=2 -c transfer.fsckobjects=1 \ @@ -976,9 +1216,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && + configure_exclusion blob "$P" my-blob >h && sane_unset GIT_TEST_SIDEBAND_ALL && test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ @@ -1000,7 +1240,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo git -C "$P" add .gitmodules && git -C "$P" commit -m x && - configure_exclusion "$P" .gitmodules >h && + configure_exclusion blob "$P" .gitmodules >h && sane_unset GIT_TEST_SIDEBAND_ALL && git -c protocol.version=2 -c transfer.fsckobjects=1 \ @@ -1024,9 +1264,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul echo "path = include/foo" >>"$P/.gitmodules" && echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && git -C "$P" add .gitmodules && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" .gitmodules >h && + configure_exclusion blob "$P" .gitmodules >h && sane_unset GIT_TEST_SIDEBAND_ALL && test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ From patchwork Wed Aug 25 02:21:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3098C4338F for ; Wed, 25 Aug 2021 02:22:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9CE7961165 for ; Wed, 25 Aug 2021 02:22:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236920AbhHYCXV (ORCPT ); Tue, 24 Aug 2021 22:23:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237447AbhHYCXP (ORCPT ); Tue, 24 Aug 2021 22:23:15 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DA17C061757 for ; Tue, 24 Aug 2021 19:22:29 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id j10-20020a17090a94ca00b00181f17b7ef7so3122840pjw.2 for ; Tue, 24 Aug 2021 19:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qK0K62voAt5415f3QcetEjG6uyZTTo2UZTa/IL+Gmnc=; b=a0/p1jdee7S4LkDMAp0AZOHqxRYhjEs6oeZzYFktCj3IfbyFtjBS3kTrGRknUsXqcw 9B9h0w8DuDlDK5UakRZZ5xSN6ZBWDhpGhj6RaBwLYkmrt4FXLF7idZasBKjDxPJzq3EP uk3kvIhYnbZB4BuCzYsIH1RVln3rL9EnYJNdhxI51fCbmJQyHPwmA2x33+46RO17TdW7 34+2NMU+DsVjPW8nmmKGiwP1WE74fy9YqWW6Y7CLmUR1IeKJV2HtN6K9g9VFnGfV3zhx cG6BFk4TqpVApo16Eyjir3XFkiZ4B8UqopI3zmd4lkh+MujjXhgedGoXx0pI0ZGQlUsC X69A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qK0K62voAt5415f3QcetEjG6uyZTTo2UZTa/IL+Gmnc=; b=WXjjGldmJ4NT5qoHFNIuQzd13lZgs/5ujzhGDCWQ5BlyU6F+MweYRh9z2v1BqFCkgu wMdrMhnt1d0TLZ1F51bmCdZ+X1IogYxQOVsIeTgv/9N4fxcWQkOtg8ltdsGI0oZucyss 0EBMUtYVA0DZUzd2ipuSLS6JUp6h4LbnltOJcG6Gd/oGkuPkqggoKFKwWE5f6TmUvtPM qvvWuL0+d2tFShxoc5fMiFTz32YJw9o1gpFG3AlzailVFdONgnhtTU6qGcCOe0u0o8vL cbKSpJ2z/SM0zrvkmYSPikA1080RMCGGCURZ8FPDsIa9Fq93khIdCzxPpDKL+3RpUDd0 GwPA== X-Gm-Message-State: AOAM532rMu2lteyPsXgQCn6D7FdAn1LFclvBkQTx6IR32w7RSV8vXEMf i8cacg6zIPPvW9kQdzd1gkJWd6VczYHfk+Wx X-Google-Smtp-Source: ABdhPJyeZro+ucbzJZ5D0tTBq790cZJb0yPohodCs9SQMBPSux3ShZ+DKV9mXn3AphDQdNgJlMIeHg== X-Received: by 2002:a17:90a:c89:: with SMTP id v9mr7986150pja.175.1629858148589; Tue, 24 Aug 2021 19:22:28 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:28 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 08/14] Add new parameter "carry_data" for "show_commit function Date: Wed, 25 Aug 2021 10:21:52 +0800 Message-Id: <19f767038486e53a3d12e38aea3bba51e6541e7d.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- builtin/describe.c | 4 ++-- builtin/pack-objects.c | 6 +++--- builtin/rev-list.c | 4 ++-- bundle.c | 4 ++-- list-objects.c | 2 +- list-objects.h | 2 +- pack-bitmap.c | 6 +++--- reachable.c | 2 +- shallow.c | 4 ++-- 9 files changed, 17 insertions(+), 17 deletions(-) diff --git a/builtin/describe.c b/builtin/describe.c index 045da79b5c..8fb99bbda5 100644 --- a/builtin/describe.c +++ b/builtin/describe.c @@ -479,9 +479,9 @@ struct process_commit_data { struct rev_info *revs; }; -static void process_commit(struct commit *commit, void *data) +static void process_commit(struct commit *commit, void *show_data, void *carry_data) { - struct process_commit_data *pcd = data; + struct process_commit_data *pcd = show_data; pcd->current_commit = commit->object.oid; } diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 4ff12ec525..d38b24e375 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3080,7 +3080,7 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; } -static void show_commit_pack_hint(struct commit *commit, void *_data) +static void show_commit_pack_hint(struct commit *commit, void *show_data, void *carry_data) { /* nothing to do; commits don't have a namehash */ } @@ -3258,7 +3258,7 @@ static void read_object_list_from_stdin(void) /* Remember to update object flag allocation in object.h */ #define OBJECT_ADDED (1u<<20) -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, void *show_data, void *carry_data) { add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL); commit->object.flags |= OBJECT_ADDED; @@ -3572,7 +3572,7 @@ static void record_recent_object(struct object *obj, oid_array_append(&recent_objects, &obj->oid); } -static void record_recent_commit(struct commit *commit, void *data) +static void record_recent_commit(struct commit *commit, void *show_data, void *carry_data) { oid_array_append(&recent_objects, &commit->object.oid); } diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 1cad33d9e8..b5e7ba6e83 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -94,9 +94,9 @@ static off_t get_object_disk_usage(struct object *obj) } static void finish_commit(struct commit *commit); -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, void *show_data, void *carry_data) { - struct rev_list_info *info = data; + struct rev_list_info *info = show_data; struct rev_info *revs = info->revs; display_progress(progress, ++progress_counter); diff --git a/bundle.c b/bundle.c index 693d619551..143e45ce0c 100644 --- a/bundle.c +++ b/bundle.c @@ -437,9 +437,9 @@ struct bundle_prerequisites_info { int fd; }; -static void write_bundle_prerequisites(struct commit *commit, void *data) +static void write_bundle_prerequisites(struct commit *commit, void *show_data, void *carry_data) { - struct bundle_prerequisites_info *bpi = data; + struct bundle_prerequisites_info *bpi = show_data; struct object *object; struct pretty_print_context ctx = { 0 }; struct strbuf buf = STRBUF_INIT; diff --git a/list-objects.c b/list-objects.c index 49f177cb56..2e53a01458 100644 --- a/list-objects.c +++ b/list-objects.c @@ -384,7 +384,7 @@ static void do_traverse(struct traversal_context *ctx) die(_("unable to load root tree for commit %s"), oid_to_hex(&commit->object.oid)); } - ctx->show_commit(commit, ctx->show_data); + ctx->show_commit(commit, ctx->show_data, NULL); if (ctx->revs->tree_blobs_in_commit_order) /* diff --git a/list-objects.h b/list-objects.h index ab946d34db..838b8c78c9 100644 --- a/list-objects.h +++ b/list-objects.h @@ -5,7 +5,7 @@ struct commit; struct object; struct rev_info; -typedef void (*show_commit_fn)(struct commit *, void *); +typedef void (*show_commit_fn)(struct commit *, void *, void *); typedef void (*show_object_fn)(struct object *, const char *, void *, void *); void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *); diff --git a/pack-bitmap.c b/pack-bitmap.c index 516eb235da..81cf14ef8e 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -473,7 +473,7 @@ static void show_object(struct object *object, const char *name, void *show_data bitmap_set(data->base, bitmap_pos); } -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, void *show_data, void *carry_data) { } @@ -1281,9 +1281,9 @@ static void test_show_object(struct object *object, const char *name, display_progress(tdata->prg, ++tdata->seen); } -static void test_show_commit(struct commit *commit, void *data) +static void test_show_commit(struct commit *commit, void *show_data, void *carry_data) { - struct bitmap_test_data *tdata = data; + struct bitmap_test_data *tdata = show_data; int bitmap_pos; bitmap_pos = bitmap_position(tdata->bitmap_git, diff --git a/reachable.c b/reachable.c index 521b39edef..d38c31ae45 100644 --- a/reachable.c +++ b/reachable.c @@ -52,7 +52,7 @@ static void mark_object(struct object *obj, const char *name, void *show_data, v update_progress(show_data); } -static void mark_commit(struct commit *c, void *show_data) +static void mark_commit(struct commit *c, void *show_data, void *carry_data) { mark_object(&c->object, NULL, show_data, NULL); } diff --git a/shallow.c b/shallow.c index 9ed18eb884..33f878565c 100644 --- a/shallow.c +++ b/shallow.c @@ -185,9 +185,9 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth, return result; } -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, void *show_data, void *carry_data) { - commit_list_insert(commit, data); + commit_list_insert(commit, show_data); } /* From patchwork Wed Aug 25 02:21:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6D75C432BE for ; Wed, 25 Aug 2021 02:22:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9ECDD61184 for ; Wed, 25 Aug 2021 02:22:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237494AbhHYCXW (ORCPT ); Tue, 24 Aug 2021 22:23:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237542AbhHYCXU (ORCPT ); Tue, 24 Aug 2021 22:23:20 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B0A9C0617A8 for ; Tue, 24 Aug 2021 19:22:33 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id qe12-20020a17090b4f8c00b00179321cbae7so3698774pjb.2 for ; Tue, 24 Aug 2021 19:22:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IdJsBBXPrKP6PQNZtIraFQqIN5D48duU1jn8S4lBxJk=; b=pEhhM75i+OfOG2Z1zIi6HWgVCu+sNdbUlQi+Bs5bBNV3lHmR7nTiwa5rT5SEHY0TgX siCOz8NRaJZLlN0j0fL0AeyragmfRHWzQKk4yZktWS8ci8YyZ4RXASpMcOUyWcX0i0J3 wIBYkBBODZ1uo61oPBe1gzhPgKyeiWe7YQSjabK/1LGC5uB9UKBa4xMkIZs0na7wxvvR esBwJ3rpGFW4BU4nq7TYODX5+ytMVccFfL38JZxM2IDxAf9qA1KCwHOHsj9B5XrDWfIU 9GMbYhvo19+Y7AhdG1Yky4yA9wA0MOcKSF6HmNbfmOoMnfvV5b6TSrnSJBG7KJP1rY86 BLDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IdJsBBXPrKP6PQNZtIraFQqIN5D48duU1jn8S4lBxJk=; b=QzKKw5F8gLhHao9gDSkGEorlS13/9V9lNq4/IK6swtPSz1WW6cY1frLwmvQlfAEg3h bqJYSin3eEKosOckYsqVCnzri155cmZv2RFu7sMrdQMUXpAK12qoXqbRzyivuBamOBKi BEboKaMV91Zm4FYMiFrEcpKN1S2Pcgjhnc8iAKtuO9l9w5PQVu5M+ewSBrlnd++4fSmP yGVUhb2SowQ3pdmEP/R87BGPJUDrCYrmPsFRrBPk/yVEG77j9YuyaCxbDatg5LjfiLZv h1Z00dlmEiAlce7z6fnO8GAgZy61+8ZwPWS11xxcjpIa+7zaml8N+oKrJHGH5gpdvBPw EGJw== X-Gm-Message-State: AOAM532BQdxRKl1q0r/szgD9mLmAlxee0tvMUopyJ+IeKbrqFaoxxf4D XBb0POoEt7nrfnalTELJ2Kg= X-Google-Smtp-Source: ABdhPJwmk4YIorLS6RZhQOuu2QYjmGuForKf7+vaWbIWxmKhBruuUNXu1OLSrj0fzWZWozGKV3M9iw== X-Received: by 2002:a17:902:d2c3:b0:136:3916:c936 with SMTP id n3-20020a170902d2c300b001363916c936mr5889418plc.85.1629858153224; Tue, 24 Aug 2021 19:22:33 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:32 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 09/14] commit.h: add wrapped tags in commit struct Date: Wed, 25 Aug 2021 10:21:53 +0800 Message-Id: <011e5eaea3f48eeacac1614f769e8fd809e0c093.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- commit.h | 5 +++++ revision.c | 8 ++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/commit.h b/commit.h index df42eb434f..1e99e9ae8a 100644 --- a/commit.h +++ b/commit.h @@ -38,6 +38,11 @@ struct commit { */ struct tree *maybe_tree; unsigned int index; + /* + * wrapped tags or NULL. If the commit is peeled from tag(s), + * then save the wraps, otherwise will be NULL. + */ + struct object_list *wraps; }; extern int save_commit_buffer; diff --git a/revision.c b/revision.c index 65e0926d25..aecf493f46 100644 --- a/revision.c +++ b/revision.c @@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs, const char *path = entry->path; unsigned int mode = entry->mode; unsigned long flags = object->flags; - + struct object_list *wraps = NULL; /* * Tag object? Look what it points to.. */ while (object->type == OBJ_TAG) { struct tag *tag = (struct tag *) object; - if (revs->tag_objects && !(flags & UNINTERESTING)) + if (revs->tag_objects && !(flags & UNINTERESTING)) { + object_list_insert(object, &wraps); add_pending_object(revs, object, tag->tag); + } + object = parse_object(revs->repo, get_tagged_oid(tag)); if (!object) { if (revs->ignore_missing_links || (flags & UNINTERESTING)) @@ -449,6 +452,7 @@ static struct commit *handle_commit(struct rev_info *revs, */ if (object->type == OBJ_COMMIT) { struct commit *commit = (struct commit *)object; + commit->wraps = wraps; if (repo_parse_commit(revs->repo, commit) < 0) die("unable to parse commit %s", name); From patchwork Wed Aug 25 02:21:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92372C4320A for ; Wed, 25 Aug 2021 02:22:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7CC2D61184 for ; Wed, 25 Aug 2021 02:22:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237542AbhHYCXY (ORCPT ); Tue, 24 Aug 2021 22:23:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237594AbhHYCXU (ORCPT ); Tue, 24 Aug 2021 22:23:20 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9795AC0617AE for ; Tue, 24 Aug 2021 19:22:35 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id mw10-20020a17090b4d0a00b0017b59213831so3145621pjb.0 for ; Tue, 24 Aug 2021 19:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0qQjliPr0/s7baFkmaMyhWgWFFttSbuWmEbFgxprA7E=; b=LtjozlB4Fl71ep91cPQctdebOIU0mEIaUClMMUVUP5pyk5jjSV+v5t39ZSZVmErzjK gu4dF5UwgNoSWi6G2qNI2dZmSZZsqQ1NbkPexhmzxYPhK4kW0jY8GO8k8qn2z+c/5Rgn 5No7StXgmKT+V95h+iJzftXBF3FZG0ec++Huxbv8/TZAfsNWEmRh0NXJ4NYVbFFVHRcj jHy/rn+0QSn5KgEX4GV+Noin7wLgbLpnqdQ9Q5wBsIxlF8SDHmmqsM8lgct41osor7o/ oX4PaEttPagoRGz7wmmu15D8xWK0uYlTn37vh4Rx0Hyq7AfzqBiui78/+49tKaBV7Tzw xrrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0qQjliPr0/s7baFkmaMyhWgWFFttSbuWmEbFgxprA7E=; b=uJGO8RIXlwa0xOqHbeWv75Zo10EkmFtAyhciGgnPD7EwUGfLXNvikj/PPCDdJYONLI Wmuhv3qZoQ+P6A0gMMPjfbipyPUIJq2nNKos3vlpgX91jerEo6mNdCDOH+kOoKqUxWaR 4TCh5c58cGeu86RzbQIj0ToqmwiYLJ8lDiTSDTKdpPJ8iUahte1Q1qfZ9W/7y3IEZo8z fFwJotHM2iO+9CtOPIwYuA6ebyU8TRiC87ZYG3D9CKXlXYfOj9gzRWJlptj4CYJUjxqi DFpUwNHpFbODc50d3qUT7sd4sU4y5HyA1wIr4EN4EvbVGFR1r2JcSPOBeVLS7kzDKCBO u55w== X-Gm-Message-State: AOAM530SQq9upEg8j4Stmk5wzHQpXZ1KcxeuZEGGWRHhwFeWFoExApyJ BDe4T2WFYrIT1WKOWQ3nvQg= X-Google-Smtp-Source: ABdhPJxX0IEu8JqQVV6ouwcMXu5T/SiZcCV3GjJpDRkxecJhDWTIGmZTmwN5wqjerZ/S6hEp1Ojf6Q== X-Received: by 2002:a17:90a:db96:: with SMTP id h22mr7957354pjv.212.1629858155174; Tue, 24 Aug 2021 19:22:35 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:34 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 10/14] object.h: add referred tags in `referred_objects` struct Date: Wed, 25 Aug 2021 10:21:54 +0800 Message-Id: <824844499f3761b9bdc7952a5ace3721f3c92432.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- object.c | 1 + object.h | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/object.c b/object.c index 69ba0baf95..b25b2e2919 100644 --- a/object.c +++ b/object.c @@ -341,6 +341,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch } referred_objs->commit = referred_commit; referred_objs->trees = NULL; + referred_objs->tags = NULL; entry = &objects[nr]; entry->item = obj; entry->referred_objects = referred_objs; diff --git a/object.h b/object.h index 3785546adf..bd59eaa6d9 100644 --- a/object.h +++ b/object.h @@ -54,8 +54,8 @@ struct object_array { unsigned mode; /* * referred_objects or NULL. If non-NULL, it will - * temporary storage the referred commit and trees when - * traversing the specified object. Space for time, + * temporary storage the referred commit, trees and tags + * when traversing the specified object. Space for time, * reduce related computing costs (such as packfile-uri * exclusion), clean up when the traversal is over. */ @@ -68,6 +68,7 @@ struct object_array { struct referred_objects{ struct object *commit; struct object_list *trees; + struct object_list *tags; }; /* * object flag allocation: From patchwork Wed Aug 25 02:21:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41280C4320E for ; Wed, 25 Aug 2021 02:22:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 29F3A61176 for ; Wed, 25 Aug 2021 02:22:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238063AbhHYCX0 (ORCPT ); Tue, 24 Aug 2021 22:23:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237447AbhHYCXW (ORCPT ); Tue, 24 Aug 2021 22:23:22 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CCE5C061757 for ; Tue, 24 Aug 2021 19:22:37 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id qe12-20020a17090b4f8c00b00179321cbae7so3698873pjb.2 for ; Tue, 24 Aug 2021 19:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nwtCwIG/1ACR64LjTJvIyq9ilPKyGe7sZY3fnNRh6sM=; b=uQxsL1cSFUkQ4V2k6h7u6T4EjSvmhH0PlMTT/ZsyP2dFgXMgHfZo8IgGICsCwusPmU rXJcB8QDFd0zwZJpKnzNn72qQCxRBGvkKkfJT9wjqma7DqaCZ/o0BP24xpHJ5lAEItrD FLTND1L8KU53uXSkPjllFRCUa+iyqlyNONPpYxHTkCxDg+EnBXrUQbUnYQTBTijB57NX MpAtl66m6lXbb1Sumw/KoO46QxNWRK191Fvno9674WKi5csrrFsAlY8nukojepY+IJ34 kcL/VmWXIPIP9ytaCDtGM2dfDgSBC9Le53SqFJ2mR9qjkRTb5wH+QbHTeQndIZKLFn49 Y6wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nwtCwIG/1ACR64LjTJvIyq9ilPKyGe7sZY3fnNRh6sM=; b=qMsNUuJEOhuwQlU0x7QP4YqsGUwzEg5ng8847VMBd8QYjWbVpmWpeSKil0nBE+P6xV Qb4twCDelKHbliq9duggsv5eZafMEA06u0HNfemmt5tY2H8o3zNDqxQ+JI9h9ZnPZE7I v2L8pAHgCSXlCEFIGCISSmqr/QGhHSXGpEkyZ05BiN2uMgjWYHL64ZC2dhIDftHaLeKN QO+FzmzMosK14dM8NqTCjGMOffONj+rr5B29bOXu6aHp2QlKbRtXmuqiR2szcZVT0A4c 5UNBbDmSEEnmlJ6BKp0reI/ZjNhkCLYstWQDIX6lcxC9ompHd7ooKF/waN10W4yLHKC0 kPxw== X-Gm-Message-State: AOAM532QUAFI2QM3fjVLTilh5P2M/oAZNxEnYNLJr2Kk7qimK3OzRc45 wlkXacGkLCC2PbVaQ2WPjAE= X-Google-Smtp-Source: ABdhPJwtSZqwFl2nmU8wu4juCqW1p84Da/ajvrXqOBZTeV6Iyc/oxy7yXYF1RlbDRcArjjyoveldVg== X-Received: by 2002:a17:90a:191a:: with SMTP id 26mr7677612pjg.79.1629858157168; Tue, 24 Aug 2021 19:22:37 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:36 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 11/14] packfile-uri: support for excluding tag objects Date: Wed, 25 Aug 2021 10:21:55 +0800 Message-Id: <43aa811b659651611c041db8e9d6c63a9aedfce3.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This commit supports the use of `uploadpack.excludeobject` to exclude tag objects, both lightweight tag and annotated tag are supported: -If a lightweight tag (such as a commit object) have been configured, the mechanism of exclusion is the same as the commit object (the commit object and all objects it contains will be recusively excluded). -If an annotated tag (created with -a, -s, or -u) have been configured, the annotated tag and all the objects that it contains will be excluded, For an example of the annotated tag: Create an annotated tag from HEAD: git tag -a A -m "tag A description" Output the SHA () of tag "A" : git rev-parse A^{object} Dereference , output the SHA of commit: git rev-parse A^{} In the above case, when the tag object () is configured with `uploadpack.excludeobject` which means , the dereference commit object (), and all the objects that recursively contains (trees, blobs), will be excluded (using a packfile URI instead). Signed-off-by: Teng Long --- builtin/pack-objects.c | 20 ++++++++++++++++++-- list-objects.c | 9 ++++++++- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index d38b24e375..d5e3f2c229 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1317,9 +1317,11 @@ static int want_object_in_pack(const struct object_id *oid, int want; struct list_head *pos; struct multi_pack_index *m; + struct configured_exclusion *tag_ex; struct configured_exclusion *commit_ex; struct configured_exclusion *tree_ex; struct configured_exclusion *ex; + struct object_list *p; if (!exclude && local && has_loose_object_nonlocal(oid)) return 0; @@ -1355,14 +1357,27 @@ static int want_object_in_pack(const struct object_id *oid, } if (uri_protocols.nr) { + if (referred_objs && referred_objs->tags) { + for (p = referred_objs->tags; p; p = p->next) { + tag_ex = oidmap_get(&configured_exclusions, &p->item->oid); + if (match_packfile_uri_exclusions(tag_ex)) + return 0; + } + } + if (referred_objs && referred_objs->commit) { commit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid); if (match_packfile_uri_exclusions(commit_ex)) return 0; + struct commit *commit = (struct commit*) referred_objs->commit; + for (p = commit->wraps; p; p = p->next) { + tag_ex = oidmap_get(&configured_exclusions, &p->item->oid); + if (match_packfile_uri_exclusions(tag_ex)) + return 0; + } } if (referred_objs && referred_objs->trees) { - struct object_list *p; for (p = referred_objs->trees; p; p = p->next) { tree_ex = oidmap_get(&configured_exclusions, &p->item->oid); if (match_packfile_uri_exclusions(tree_ex)) @@ -3260,7 +3275,8 @@ static void read_object_list_from_stdin(void) static void show_commit(struct commit *commit, void *show_data, void *carry_data) { - add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL); + struct referred_objects *referred_objs = carry_data; + add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, referred_objs); commit->object.flags |= OBJECT_ADDED; if (write_bitmap_index) diff --git a/list-objects.c b/list-objects.c index 2e53a01458..52f38c9151 100644 --- a/list-objects.c +++ b/list-objects.c @@ -367,6 +367,7 @@ static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ + struct referred_objects *referred_objs; strbuf_init(&csp, PATH_MAX); while ((commit = get_revision(ctx->revs)) != NULL) { @@ -384,7 +385,13 @@ static void do_traverse(struct traversal_context *ctx) die(_("unable to load root tree for commit %s"), oid_to_hex(&commit->object.oid)); } - ctx->show_commit(commit, ctx->show_data, NULL); + referred_objs = xmalloc(sizeof(struct referred_objects)); + referred_objs->commit = NULL; + referred_objs->trees = NULL; + referred_objs->tags = commit->wraps; + + ctx->show_commit(commit, ctx->show_data, referred_objs); + free(referred_objs); if (ctx->revs->tree_blobs_in_commit_order) /* From patchwork Wed Aug 25 02:21:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AFB1C4338F for ; Wed, 25 Aug 2021 02:22:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4317761165 for ; Wed, 25 Aug 2021 02:22:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238442AbhHYCXa (ORCPT ); Tue, 24 Aug 2021 22:23:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237508AbhHYCXY (ORCPT ); Tue, 24 Aug 2021 22:23:24 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80FEDC061757 for ; Tue, 24 Aug 2021 19:22:39 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id c4so13410084plh.7 for ; Tue, 24 Aug 2021 19:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PSdJD/zcXLyFdHkY9uKDoUWKgqf53ytifKQnKDMMuXg=; b=jZoEWInTXOf5Oo5fOwlqQx4xp3isbtTLoWAXK56d2T9h44dt61ikeuKuaytQl/aZjm W6U3b3TZCMGxNf5kYAiBgddKDnVXFPBq3M9lXU4hcSAqXJAFyrsZSIllNyitaStaNTcY Sy1AJ4+4JU6V4aczZITzQaTyiw4apGGFtYSvR5rkyUtj+Gwi9zNXChP2sQ/jFDUKhOT8 dLuuBCw6/dssEyEr5ldia7Vk663XA+cTvabdBrx8QWn0966aeG9hUGqCDwpNULwOIpx0 8qb46CgJ1zhIpuUFNphASVTKaGCta+V78aU9SalBluyPFi0aZpldZTFIQJFNUONlW/47 73Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PSdJD/zcXLyFdHkY9uKDoUWKgqf53ytifKQnKDMMuXg=; b=CWq4yxjc2UflItvfP/orFLbrB19oIUt4B9tuDtVkDAPNrivw/zDkbNpY4TA29Va31o b20kyP/09O5ZckkQ1r2xL/nkRmFBvhOu6RAk3U32cu9OkT2qFQQbyeYOd/+KKG5O3D// f0IXXqClgDLjEdxEURq2YYXe68Vf0Tl1oCFg59IGv/eMqrQRyD3kQCajL9cBI5dZaip6 NDDlc+jL9OwFfp1f47ZCWFSdlxPo0SpEzPBo/cd3NElXJCCUZPhXYqLgawn6sHe88NzO GTIG8W/1duIelPDl+xNqMIaRMYzfAPGk/N20UU0ZSnf74QtklUYsuw2rcJR5/iUyf/mn uGqQ== X-Gm-Message-State: AOAM532WalAOznISF+frwh1lobpR/+l56116gpCVrHLnndrjoY8NwaCr gDC0Ys0HWymck5q2Yc2e1/A= X-Google-Smtp-Source: ABdhPJz1+hBD7+oEEYV4OM01f7c5EkrlvRD941ePavjizxw/f2hlr/N9+o+AEn8Y4lNyMuabTA4l4g== X-Received: by 2002:a17:90a:5583:: with SMTP id c3mr7594784pji.133.1629858159126; Tue, 24 Aug 2021 19:22:39 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:38 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 12/14] packfile-uri.txt: support for excluding tag objects Date: Wed, 25 Aug 2021 10:21:56 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- Documentation/technical/packfile-uri.txt | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt index c3e4873956..bf5939cb8d 100644 --- a/Documentation/technical/packfile-uri.txt +++ b/Documentation/technical/packfile-uri.txt @@ -39,12 +39,18 @@ server to be configured by one or more entries with the format: uploadpack.excludeobject= -Value is the key of entry, and the object type can be a blob, -tree, or commit. The exclusion of tree and commit is recursive by default, -which means that when a tree or commit object is excluded, the object itself -and all reachable objects of the object will be excluded recursively. Whenever -the list of objects to be sent is assembled, all such objects are excluded, -replaced with URIs. +Value is the key of entry, and the object type can be blob, tree, +commit, or tag. When an object is configured with `uploadpack.excludeobject` which +means that whenever the list of objects to be sent is assembled, the object (also +include the related objects in some cases, the following will introduce) will be +excluded, replaced with URIS. The mechanism for exclusion is as follows: + + * blob: exclude blob object. + * tree: exclude tree object, blobs that the tree list, and recursive into sub-trees. + * commit: exclude commit object, and recursively exclude all the reachable trees + (ditto tree exclusion) and blobs it contains. + * tag: exclude tag object itself, and the dereference commit (ditto commit exclusion) + if the tag is annotated. Configuration compatibility ------------- From patchwork Wed Aug 25 02:21:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCB0AC432BE for ; Wed, 25 Aug 2021 02:22:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C01CD61176 for ; Wed, 25 Aug 2021 02:22:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237652AbhHYCXa (ORCPT ); Tue, 24 Aug 2021 22:23:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237913AbhHYCX0 (ORCPT ); Tue, 24 Aug 2021 22:23:26 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69CA0C0613D9 for ; Tue, 24 Aug 2021 19:22:41 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id b9so9046306plx.2 for ; Tue, 24 Aug 2021 19:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6LGz2nbArDUtEP+kmAKLGRouLOLkJB3+JfDDtW4q7Sk=; b=NklSq3rmHvbRKRKDYP1rnvoVsfA+c10nwmX99detj8qKE/nv/5h0wG1pMMZw88Oe3u tSmdChMigvL25jCCvfJ1abXhhxMnC5czUTm1oB1/G5gRQAJ53im1Li9HGb06hSIiSMRw pyPnrpS82rfV1t02ggD8URhKBL7p0c5lyUb0IHeP+it9Z3+qfel2EAia4xPSosLGauBb BRBToSJM1HAXf29WJjSJxUDkuZ+pGXYY2P4B5VvSoK7MAs6fE9RtpG5ejjfe2/VN9F1d n/M5NTTnSSKjFzin/84vWFLDUtMUSoJp13LPBU7EZdzstPlpsadE9wBQGyzHc+lmc2RY Tvdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6LGz2nbArDUtEP+kmAKLGRouLOLkJB3+JfDDtW4q7Sk=; b=KzSuGOb+GtJcBkMuEJegNrNMD7wLwCO47jato3JUe7IYIcLNoR4NpravVXwX/UJWG/ PR0B6NRJ6N7zHp886hVQRJjFuhm0aF2LF1KZxgBo0VDscU6JZ2Hsur3bD8fXbTWGzhIz Bo+rd0ox3obJEUBnfH29qyThY4mAoQLn4ERI53l6z3PZryQv92NYP/9kzrMu747+bx3s WKF4KtyRkv/hPghxMXw7oN99PfzWTvpg0CACZlFm9EYQMoEJwxGtCQMNh+UYVTD0LTjD WhZO+NyZmjevMXwfGR9o+uHTzcrFriRNx72I+A8Fxev3TBTyaP4/hVKHsR8FzLQK+fnt dpUw== X-Gm-Message-State: AOAM530z0nw/WuhbBWOU2+YHBWvGLOIlFXpD3xF187slL0tpSpPqG0AB nF+RB632U1g/mpHfp0ItNwJsh9fyuTl9mcfI X-Google-Smtp-Source: ABdhPJyNQ/nmIH7rjg9TErKRT79nWRdEmnpBWac37Ffzr0muJRyaum5RNYJSr2AwKnvfas3QX+xEUQ== X-Received: by 2002:a17:90a:4a95:: with SMTP id f21mr8001271pjh.122.1629858161063; Tue, 24 Aug 2021 19:22:41 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:40 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 13/14] t5702: add tag exclusion test case Date: Wed, 25 Aug 2021 10:21:57 +0800 Message-Id: <29a52b7a0d8a2169d19aabed3a48e59dd9ae3d34.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 5ad52e0cee..09683ebfdc 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -1142,6 +1142,24 @@ test_expect_success 'commit-exclusion: part of packfile response provided as URI http_child/.git/objects/pack/*.idx >filelist && test_line_count = 4 filelist ' +test_expect_success 'tag-exclusion: part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + echo my-blob >"$P/my-blob" && + git -C "$P" add my-blob && + test_commit -C "$P" A && + git -C "$P" tag -a -m "annotated_tag" tagA && + tagObj=$(git -C "$P" rev-parse tagA) && + configure_exclusion tag "$P" "$tagObj" >h2 && + git init http_child && + GIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \ + git -C http_child \ + -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + fetch --tags "$HTTPD_URL/smart/http_parent" +' test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && From patchwork Wed Aug 25 02:21:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12456243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 041D8C4338F for ; Wed, 25 Aug 2021 02:22:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD56561165 for ; Wed, 25 Aug 2021 02:22:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238295AbhHYCXf (ORCPT ); Tue, 24 Aug 2021 22:23:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238290AbhHYCX3 (ORCPT ); Tue, 24 Aug 2021 22:23:29 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EE90C061757 for ; Tue, 24 Aug 2021 19:22:43 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id oc2-20020a17090b1c0200b00179e56772d6so3105311pjb.4 for ; Tue, 24 Aug 2021 19:22:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FZB5IVxJr55R2SVGW0MxVUnPARFLsT2THBJI8tenPhM=; b=jKu7j9jTzyvCvTJ30Nsrz4fhchwOw2FxlBsbIJjQq2nfo1CL0ZApey5iu4BgzzDdlq Ox2m4Vhypr/gXeZLeV/kbI8/mSpHTWPV+UE2xZhWJ32CYv2uOaX7lXImMpiyVNfTAlmS Gm485WFY4wREUydqULJPRnWRybdfQGc2UnJN45BfGrfLU9C3DM30x2QN+a08C+XSricR oWneSROnuFDSpGAMjFY5NvVsl3i09VYF0/mTeqEITwoGcGY99gkE6XGXDMmrSf6aWClD Q1w6om4IyJfvRJWIHuhvl3zYCUNRJd3jCvLnSSpSBdQ2BH9eG+Kp4STUmFJpocy5Jqhb HWtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FZB5IVxJr55R2SVGW0MxVUnPARFLsT2THBJI8tenPhM=; b=TmlX+Z924ZscrHsYUQKOTVJWKIc+MAO8xkgbgYECnUSmvTebG0o+vPw6VJdwFhdiO3 MiPCu8BKLj0xYv1tRTEqAdOvaHMAZlHAwtg1XtR1aqJ+7LIKAoGbNVF4uC1bacAXOlSw ooIKo9KC5dRK7r/dXH9LKfrbBSmK1iA8ri77AGApyNqjOaCzmsxrBTiDuIUV1iY8ha7q FYMZVnlpyE1LoaCqZx5SQEHJcLAHLEp6iUz18H/WfZnWtPFryj2p/tWrrfRffXdT8Cpv pppo5ovh1hYJxVEoleLhhTlA/hixwOg7eJ3fNHEp6OyynubU02LltaxLbGbbPpY9lz9X PJgg== X-Gm-Message-State: AOAM5329T/HCe8FTmQdCkIqHwTNL4h9qk6bLJg0G4E2MB3Rg1xSRDnei ZB4bQTUSChisiIJxIqhL2yI= X-Google-Smtp-Source: ABdhPJx41FK0ZGUNFq/Kqx9ICCjwvZKd5nICfM8qqPefCMBNFd045ERD+orndpdRs94XOCy+vk770A== X-Received: by 2002:a17:90a:14e4:: with SMTP id k91mr7786680pja.3.1629858163028; Tue, 24 Aug 2021 19:22:43 -0700 (PDT) Received: from localhost.localdomain ([47.246.98.155]) by smtp.gmail.com with ESMTPSA id e14sm3628683pjg.40.2021.08.24.19.22.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Aug 2021 19:22:42 -0700 (PDT) From: Teng Long To: gitster@pobox.com, dyroneteng@gmail.com Cc: avarab@gmail.com, git@vger.kernel.org, jonathantanmy@google.com Subject: [PATCH v5 14/14] pack-objects.c: introduce `want_exclude_object` function Date: Wed, 25 Aug 2021 10:21:58 +0800 Message-Id: <1a7c4c58946cc6aefe07021f366d95f174c7da21.1629805396.git.dyroneteng@gmail.com> X-Mailer: git-send-email 2.31.1.456.gec51e24953 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- builtin/pack-objects.c | 52 ++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 27 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index d5e3f2c229..503f8a5746 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1206,6 +1206,21 @@ static int match_packfile_uri_exclusions(struct configured_exclusion *ex) return 0; } +static int want_exclude_object(struct object_list *objects) +{ + struct object_list *p; + struct configured_exclusion *ex; + + if (!objects) + return 0; + for (p = objects; p; p = p->next) { + ex = oidmap_get(&configured_exclusions, &p->item->oid); + if (match_packfile_uri_exclusions(ex)) + return 1; + } + return 0; +} + static int want_found_object(const struct object_id *oid, int exclude, struct packed_git *p) { @@ -1317,11 +1332,8 @@ static int want_object_in_pack(const struct object_id *oid, int want; struct list_head *pos; struct multi_pack_index *m; - struct configured_exclusion *tag_ex; struct configured_exclusion *commit_ex; - struct configured_exclusion *tree_ex; struct configured_exclusion *ex; - struct object_list *p; if (!exclude && local && has_loose_object_nonlocal(oid)) return 0; @@ -1357,41 +1369,27 @@ static int want_object_in_pack(const struct object_id *oid, } if (uri_protocols.nr) { - if (referred_objs && referred_objs->tags) { - for (p = referred_objs->tags; p; p = p->next) { - tag_ex = oidmap_get(&configured_exclusions, &p->item->oid); - if (match_packfile_uri_exclusions(tag_ex)) + if (referred_objs) { + if (referred_objs->commit) { + struct commit *commit = (struct commit *) referred_objs->commit; + commit_ex = oidmap_get(&configured_exclusions, &commit->object.oid); + if (match_packfile_uri_exclusions(commit_ex)) return 0; - } - } - - if (referred_objs && referred_objs->commit) { - commit_ex = oidmap_get(&configured_exclusions, &referred_objs->commit->oid); - if (match_packfile_uri_exclusions(commit_ex)) - return 0; - struct commit *commit = (struct commit*) referred_objs->commit; - for (p = commit->wraps; p; p = p->next) { - tag_ex = oidmap_get(&configured_exclusions, &p->item->oid); - if (match_packfile_uri_exclusions(tag_ex)) + if (want_exclude_object(commit->wraps)) return 0; } - } + if (referred_objs->tags && want_exclude_object(referred_objs->tags)) + return 0; - if (referred_objs && referred_objs->trees) { - for (p = referred_objs->trees; p; p = p->next) { - tree_ex = oidmap_get(&configured_exclusions, &p->item->oid); - if (match_packfile_uri_exclusions(tree_ex)) - return 0; - } + if (referred_objs->trees && want_exclude_object(referred_objs->trees)) + return 0; } - ex = oidmap_get(&configured_exclusions, oid); if (match_packfile_uri_exclusions(ex)) { oidset_insert(&excluded_by_config, oid); return 0; } } - return 1; }