From patchwork Thu Sep 1 09:41:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ZheNing Hu X-Patchwork-Id: 12962183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88AD2ECAAD5 for ; Thu, 1 Sep 2022 09:41:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233832AbiIAJlf (ORCPT ); Thu, 1 Sep 2022 05:41:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233422AbiIAJlS (ORCPT ); Thu, 1 Sep 2022 05:41:18 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D714E126495 for ; Thu, 1 Sep 2022 02:41:16 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id n17so21552648wrm.4 for ; Thu, 01 Sep 2022 02:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=3fNp0xWy4ypRVVc6MHsw8lJAmPQZQlIUMjSn2xeBII8=; b=RnzkbtYG8ZWcevWVS9T6XQsUFb2Dfu4+Z31GECjQd5Zn4kw7xFM7alWdzFpO2R7XOB DSfhJ+PwmZ6FVPsqepzg2/DcQURrOz6nlUfoucU1vzAMWvcvcTmHpf3QZ2Pp3qTbUY/G 1UratLLn+DtT4JTrKEf45tNVyMjRCgW7XGWN7qCm/A9D3OdB4B8wSfagXiSz7DYeJKlM 713u7iCzbWZ27BQOFlLALZhcN/xqtwTqIsjCWoCUaM7huJWRXgybSVL80rd4GE+4NxG7 3EtC79PrvqVEjIZSYk6WodQ8ljFdRXtQyU+HXSjgpWB/v0maPVx/SxbI4SI06j+oHv6u EFZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=3fNp0xWy4ypRVVc6MHsw8lJAmPQZQlIUMjSn2xeBII8=; b=THHpo4Hn1CSzkFdRmIqVFf7fN+waGrpTFsN8IPmfjQ2nS1YZQjpsC+xe60Jw7Df+F9 PthRpF2sap+/jdWwFUD6EcLJrKAgPs8GzwjbZkFq1Ys4xzFbbARjbDMUIHRSBYR51u+y 5/8P3ODf+UmFw9OynZvyBlSR0q8K9DrAVRFeujgek3tQQEdh5h+Swnq63b9CuXTKzIm1 8StDWRYRTrga9VHuR61zJJd5cDHllMkBtHiWHZCIN6tghrRAUUjSkkpZU4VX63Ww8Bz3 pvu8Krb7EXXBDaM4nkJkbjMvp6kG92mNiQAMI5ESebmuuuzO8EsfEh51ytE1WX2wkYP5 AuNw== X-Gm-Message-State: ACgBeo3NxMWSo/yI+8AUSDAtqBRvTCLR6CdNbrIzha9t1stFdr4uKgNr VlYhJkY5hZNxVJJkbgSurXIzemyrdng= X-Google-Smtp-Source: AA6agR5kFsKkHCquhi6QL9LFdcyW1F1IkT/xfe7m1pToIx2jq6W03a5GWeOGsAmYRpZ7IQlJRqa7Og== X-Received: by 2002:a5d:5847:0:b0:221:748d:243c with SMTP id i7-20020a5d5847000000b00221748d243cmr13548110wrf.222.1662025275106; Thu, 01 Sep 2022 02:41:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 12-20020a05600c028c00b003a317ee3036sm4750590wmk.2.2022.09.01.02.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 02:41:14 -0700 (PDT) Message-Id: <19fd72c34dcd1332df638d76b0b028e9d9da3d41.1662025272.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Sep 2022 09:41:10 +0000 Subject: [PATCH 1/3] commit-graph: let commit graph respect commit graft Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Christian Couder , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Jeff King , Jeff Hostetler , Junio C Hamano , Derrick Stolee , Johannes Schindelin , ZheNing Hu , ZheNing Hu Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: ZheNing Hu From: ZheNing Hu In repo_parse_commit_internal(), if we want to use commit graph, it will call parse_commit_in_graph() to parse commit's content from commit graph, otherwise call repo_read_object_file() to parse commit's content from commit object. repo_read_object_file() will respect commit graft, which can correctly amend commit's parents. But parse_commit_in_graph() not. Inconsistencies here may result in incorrect processing of shallow clone. So let parse_commit_in_graph() respect commit graft as repo_read_object_file() does, which can solve this problem. Signed-off-by: ZheNing Hu --- commit-graph.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index f2a36032f84..89bb6f87079 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -820,6 +820,7 @@ static int fill_commit_in_graph(struct repository *r, struct commit_list **pptr; const unsigned char *commit_data; uint32_t lex_index; + struct commit_graft *graft; while (pos < g->num_commits_in_base) g = g->base_graph; @@ -833,31 +834,54 @@ static int fill_commit_in_graph(struct repository *r, set_commit_tree(item, NULL); + graft = lookup_commit_graft(r, &item->object.oid); + if (graft) + r->parsed_objects->substituted_parent = 1; + pptr = &item->parents; edge_value = get_be32(commit_data + g->hash_len); if (edge_value == GRAPH_PARENT_NONE) return 1; - pptr = insert_parent_or_die(r, g, edge_value, pptr); + if (!(graft && (graft->nr_parent < 0 || grafts_replace_parents))) + pptr = insert_parent_or_die(r, g, edge_value, pptr); edge_value = get_be32(commit_data + g->hash_len + 4); if (edge_value == GRAPH_PARENT_NONE) return 1; if (!(edge_value & GRAPH_EXTRA_EDGES_NEEDED)) { - pptr = insert_parent_or_die(r, g, edge_value, pptr); - return 1; + if (!(graft && (graft->nr_parent < 0 || grafts_replace_parents))) { + pptr = insert_parent_or_die(r, g, edge_value, pptr); + return 1; + } } parent_data_ptr = (uint32_t*)(g->chunk_extra_edges + 4 * (uint64_t)(edge_value & GRAPH_EDGE_LAST_MASK)); do { edge_value = get_be32(parent_data_ptr); - pptr = insert_parent_or_die(r, g, - edge_value & GRAPH_EDGE_LAST_MASK, - pptr); + if (!(graft && (graft->nr_parent < 0 || grafts_replace_parents))) { + pptr = insert_parent_or_die(r, g, + edge_value & GRAPH_EDGE_LAST_MASK, + pptr); + } parent_data_ptr++; } while (!(edge_value & GRAPH_LAST_EDGE)); + if (graft) { + int i; + struct commit *new_parent; + for (i = 0; i < graft->nr_parent; i++) { + new_parent = lookup_commit(r, + &graft->parent[i]); + if (!new_parent) + die(_("bad graft parent %s in commit %s"), + oid_to_hex(&graft->parent[i]), + oid_to_hex(&item->object.oid)); + pptr = &commit_list_insert(new_parent, pptr)->next; + } + } + return 1; } From patchwork Thu Sep 1 09:41:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ZheNing Hu X-Patchwork-Id: 12962184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C16EECAAD1 for ; Thu, 1 Sep 2022 09:41:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229534AbiIAJlg (ORCPT ); Thu, 1 Sep 2022 05:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233645AbiIAJlT (ORCPT ); Thu, 1 Sep 2022 05:41:19 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 042D7130627 for ; Thu, 1 Sep 2022 02:41:18 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id e20so21519180wri.13 for ; Thu, 01 Sep 2022 02:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=gU1Fn+L+0RDtC1OZCd5rrkOyjVxmCDcYGKf/sTgNgIE=; b=MWj2ZOt1fpIyHUZTRmm+hO/6CkW/IDFNeb1g13fxY6/xpdkfSB4W4zkUfqeg/H7Hf1 K7CHw0m4zKKc6lr588H7PqjUTGEheCdEwy8+pCHGqM16ZchC3ax62Ozxx8EKqCUw15Ny DbW0bih/lXfZ5T+Kv1xKJeGErI24d8brAmIbAYsCxr6l+aymWByYM/Lh1xASkP0cTDo3 JDfmVq2hWEB3xlyTQ3pNGICyJAc1j6LvHLhTz3AvwyXKjH7BzKhICnj+1OAO7erDA2kj ikwWWLYNwk9hGiTaxAL/mI1tClF8XRzOQi0EFVy7O+UX71s5OQWNxYok6cca50N/mW2u XQhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=gU1Fn+L+0RDtC1OZCd5rrkOyjVxmCDcYGKf/sTgNgIE=; b=dOydkolB8FCRMMiZ4VGCTXhoyQ9KhnnlBN45k/nmqg30vNP2CoyCt6+vlAGzECH0mq WvdrbPXkDvpF8EQ1e182w2l2qjVN/zuDotrnJMK8lrfecQrFV46wjIvXOAjyjBvV4rNm vpqM6b5TDRWkOyB8ZFZ0Z1ZHuNKJuQ+OLK6WFe2M8zvs/P1/HcV0Jc7/bDd4/ytZJmsq q+ltAaqzQ4croCYwLzDBRMRaA0lJ2HYmreuUov1cYdl8Si3OW9XcBISZ3fXWTnnPA+CJ kD+/9wG77KPeMXtSm7Paf6k0YRXkrm1JEj334LJ0ThKJQM35P80Yuu6MJ/OzFQn1asYa eImw== X-Gm-Message-State: ACgBeo1ouxbxRh09QuwClmKjtXxqixf3vHk9YknCuEJpi5A/lijhxEed +tq845v269fnsgGH4C9HsazLVG/meuc= X-Google-Smtp-Source: AA6agR7mOYTVZVGJREwNso4kGgJWfMU9x/ln3uZNkIc90ScPYUdwQSqw+ndRxrOY1LDgPmCYJfTs1w== X-Received: by 2002:a5d:40d1:0:b0:226:d793:ccac with SMTP id b17-20020a5d40d1000000b00226d793ccacmr11539445wrq.238.1662025276324; Thu, 01 Sep 2022 02:41:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l6-20020adff486000000b0021e13efa17esm14523729wro.70.2022.09.01.02.41.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 02:41:15 -0700 (PDT) Message-Id: <0df61091c194ed46fda4d70272fcbcfdbedc8770.1662025272.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Sep 2022 09:41:11 +0000 Subject: [PATCH 2/3] list-object-filter: pass traversal_context in filter_init_fn Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Christian Couder , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Jeff King , Jeff Hostetler , Junio C Hamano , Derrick Stolee , Johannes Schindelin , ZheNing Hu , ZheNing Hu Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: ZheNing Hu From: ZheNing Hu Pass traversal_context to all filter init functions, so that we can read or modify the rev_info of traversal_context in the filter initialization function. Signed-off-by: ZheNing Hu --- list-objects-filter.c | 12 ++++++++++-- list-objects-filter.h | 2 ++ list-objects.c | 10 +--------- list-objects.h | 8 ++++++++ 4 files changed, 21 insertions(+), 11 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index 1c1ee3d1bb1..76e8659ea73 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -112,6 +112,7 @@ static enum list_objects_filter_result filter_blobs_none( } static void filter_blobs_none__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter) { @@ -249,6 +250,7 @@ static void filter_trees_free(void *filter_data) { } static void filter_trees_depth__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter) { @@ -336,6 +338,7 @@ include_it: } static void filter_blobs_limit__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter) { @@ -519,6 +522,7 @@ static void filter_sparse_free(void *filter_data) } static void filter_sparse_oid__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter) { @@ -609,6 +613,7 @@ static enum list_objects_filter_result filter_object_type( } static void filter_object_type__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter) { @@ -734,6 +739,7 @@ static void filter_combine__finalize_omits( } static void filter_combine__init( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter* filter) { @@ -744,7 +750,7 @@ static void filter_combine__init( CALLOC_ARRAY(d->sub, d->nr); for (sub = 0; sub < d->nr; sub++) d->sub[sub].filter = list_objects_filter__init( - filter->omits ? &d->sub[sub].omits : NULL, + ctx, filter->omits ? &d->sub[sub].omits : NULL, &filter_options->sub[sub]); filter->filter_data = d; @@ -754,6 +760,7 @@ static void filter_combine__init( } typedef void (*filter_init_fn)( + struct traversal_context *ctx, struct list_objects_filter_options *filter_options, struct filter *filter); @@ -771,6 +778,7 @@ static filter_init_fn s_filters[] = { }; struct filter *list_objects_filter__init( + struct traversal_context *ctx, struct oidset *omitted, struct list_objects_filter_options *filter_options) { @@ -792,7 +800,7 @@ struct filter *list_objects_filter__init( CALLOC_ARRAY(filter, 1); filter->omits = omitted; - init_fn(filter_options, filter); + init_fn(ctx, filter_options, filter); return filter; } diff --git a/list-objects-filter.h b/list-objects-filter.h index 9e98814111c..0a3cb500976 100644 --- a/list-objects-filter.h +++ b/list-objects-filter.h @@ -5,6 +5,7 @@ struct list_objects_filter_options; struct object; struct oidset; struct repository; +struct traversal_context; /* * During list-object traversal we allow certain objects to be @@ -72,6 +73,7 @@ struct filter; * filter *`. */ struct filter *list_objects_filter__init( + struct traversal_context *ctx, struct oidset *omitted, struct list_objects_filter_options *filter_options); diff --git a/list-objects.c b/list-objects.c index 250d9de41cb..698e4dbe8ff 100644 --- a/list-objects.c +++ b/list-objects.c @@ -13,14 +13,6 @@ #include "object-store.h" #include "trace.h" -struct traversal_context { - struct rev_info *revs; - show_object_fn show_object; - show_commit_fn show_commit; - void *show_data; - struct filter *filter; -}; - static void show_commit(struct traversal_context *ctx, struct commit *commit) { @@ -448,7 +440,7 @@ void traverse_commit_list_filtered( }; if (revs->filter.choice) - ctx.filter = list_objects_filter__init(omitted, &revs->filter); + ctx.filter = list_objects_filter__init(&ctx, omitted, &revs->filter); do_traverse(&ctx); diff --git a/list-objects.h b/list-objects.h index 9eaf4de8449..44c598e9ce8 100644 --- a/list-objects.h +++ b/list-objects.h @@ -16,6 +16,14 @@ void mark_edges_uninteresting(struct rev_info *revs, struct oidset; struct list_objects_filter_options; +struct traversal_context { + struct rev_info *revs; + show_object_fn show_object; + show_commit_fn show_commit; + void *show_data; + struct filter *filter; +}; + void traverse_commit_list_filtered( struct rev_info *revs, show_commit_fn show_commit, From patchwork Thu Sep 1 09:41:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ZheNing Hu X-Patchwork-Id: 12962185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B26CC54EE9 for ; Thu, 1 Sep 2022 09:41:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233925AbiIAJlk (ORCPT ); Thu, 1 Sep 2022 05:41:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233699AbiIAJlV (ORCPT ); Thu, 1 Sep 2022 05:41:21 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15B6313251A for ; Thu, 1 Sep 2022 02:41:19 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id c7so14896271wrp.11 for ; Thu, 01 Sep 2022 02:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date; bh=0ZdoFthGz8MCKDxh11/mf2ZhondxWc0EelMNBObOzuc=; b=nKsqiYGGZTeuBKpV3n4x4fzEnG7p7KOEjg0W/BqJdwNZZYSjmScUTycEDxVD/6+gZ5 bJIhK2kmJalpGtEORP0cybqD8lLDGNSfM81vRW2cZoWaw758p4owPlBbjtfw/wAW3Twk t9gGB/E1iQS4/EczXI9SG76JOs6EzXPoUx28nWuKIUmEF0mYpjRYbNT2RV0CDRLMl8IC hUazIGDcJES5SYPQOMLdVH5VMd5eogHTJJ+ysaTL+BWjcfFO5kgNVwI5lk/mxLyre4x2 smdD6ie0VLGH3uuP9Do0UdXjK7w4axuscjAxxu9FZxoI9n7jsmvoT682EDXRvqndG31C GEPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date; bh=0ZdoFthGz8MCKDxh11/mf2ZhondxWc0EelMNBObOzuc=; b=jfVRFzR3Z5ovYVoNWOgBsxG4UOsTSb5NgV0pqSHbZOmD1YvZtl4Ki2GUah+aQuf7s+ JvLa1nBmh0fpzi9WpLVVAbCpbO+rwr1xhMUbu8rzlpI8k2MyHVd5WWB7KKFn1RmkUhol hmq0y4FmXAyHkx5Zf5uF5oVRoY92ifxPxWqmpG7npU+0bKqvUDaiYncuTz+F/0tz+WVu JJGUSXCu27tD6tmkg85DUdCzqC02gjzXiGkSITNndrBgFzl5jNphhwG2R3Req2AHQCtF +LoTDbxQ0LFYQ3rWaXIN/8lY4Q3kXxHAgSuCZxjgDr2Ka/qBmfSvRyaV3Eb1bxKMLDrp WXAA== X-Gm-Message-State: ACgBeo2v5nSkfAOBKojmsoQ3T0gGQTx71pt3MnlpIfCnh2a54qz7zKhK d1Tw0Ulxl2a8sDo1Hd+OABNF74ehx0Q= X-Google-Smtp-Source: AA6agR6ZfzVUB/dHTVev3n3IDx40GLFlbMaQME2l8UqxYVjvr9TKOYMBgjmdOFi0/PT4zLrUmlBJtg== X-Received: by 2002:a5d:69c9:0:b0:226:d258:2aec with SMTP id s9-20020a5d69c9000000b00226d2582aecmr13199625wrw.45.1662025277271; Thu, 01 Sep 2022 02:41:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x4-20020a5d4904000000b0021e30e9e44asm13976352wrq.53.2022.09.01.02.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 02:41:16 -0700 (PDT) Message-Id: <2f7d1490c43be2a1f363484f1c7031db56b6b673.1662025272.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Sep 2022 09:41:12 +0000 Subject: [PATCH 3/3] list-object-filter: introduce depth filter Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Christian Couder , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Jeff King , Jeff Hostetler , Junio C Hamano , Derrick Stolee , Johannes Schindelin , ZheNing Hu , ZheNing Hu Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: ZheNing Hu From: ZheNing Hu 'git clone --depth=' have a obvious disadvantages: We can't do some git commands that require looking deeper commits. We might use 'git fetch -unshallow' to lift this restriction, but that's not a good idea either: it downloads too much objects. Rethink this question: why not integrate the functionality of shallow clone into parital clone? Partial clone has a very clear advantage: it downloads objects only if the user needs them. Therefore, add a filter 'depth=' which can omits all commits whose depth is >= ( > 0), it just look like '--depth=' in git clone, but the git client doesn't treat it as a shallow clone. '--filter=depth:' cannot be used with '--depth', '--shallow-since', '--shallow-exclude', '--shallow-submodules'. Signed-off-by: ZheNing Hu --- Documentation/rev-list-options.txt | 6 ++ builtin/clone.c | 10 ++- list-objects-filter-options.c | 30 +++++++ list-objects-filter-options.h | 6 ++ list-objects-filter.c | 66 ++++++++++++++++ shallow.c | 16 ++++ shallow.h | 2 + t/t5616-partial-clone.sh | 116 ++++++++++++++++++++++++++++ t/t6112-rev-list-filters-objects.sh | 14 ++++ upload-pack.c | 14 ---- upload-pack.h | 14 ++++ 11 files changed, 279 insertions(+), 15 deletions(-) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 1837509566a..4e2905b9e1e 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -954,6 +954,12 @@ Note that the form '--filter=sparse:path=' that wants to read from an arbitrary path on the filesystem has been dropped for security reasons. + +The form '--filter=depth:' omits all commits whose depth is +>= , it just look like '--depth=' in git clone, but it +will not be treated as shallow-clone, so if you want to see some deeper +commits, you can freely do some git commands e.g. git diff to refetch +missing git objects without 'git fetch --unshallow'. ++ Multiple '--filter=' flags can be specified to combine filters. Only objects which are accepted by every filter are included. + diff --git a/builtin/clone.c b/builtin/clone.c index c4ff4643ecd..0b168ec18ef 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -916,7 +916,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix) usage_msg_opt(_("You must specify a repository to clone."), builtin_clone_usage, builtin_clone_options); - if (option_depth || option_since || option_not.nr) + if (option_depth || option_since || option_not.nr || + list_objects_filter_choice_exists(&filter_options, LOFC_DEPTH)) deepen = 1; if (option_single_branch == -1) option_single_branch = deepen ? 1 : 0; @@ -1113,6 +1114,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix) die(_("the option '%s' requires '%s'"), "--also-filter-submodules", "--recurse-submodules"); + if ((option_depth || option_since || option_not.nr || + option_shallow_submodules) && + list_objects_filter_choice_exists(&filter_options, LOFC_DEPTH)) + die(_("--filter='depth:' cannot be used with " + "--depth, --shallow-since, --shallow-exclude, " + "--shallow-submodules")); + /* * apply the remote name provided by --origin only after this second * call to git_config, to ensure it overrides all config-based values. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 4b25287886d..687010b84d4 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -31,6 +31,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c) return "sparse:oid"; case LOFC_OBJECT_TYPE: return "object:type"; + case LOFC_DEPTH: + return "depth"; case LOFC_COMBINE: return "combine"; case LOFC__COUNT: @@ -40,6 +42,23 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c) BUG("list_object_filter_config_name: invalid argument '%d'", c); } +int list_objects_filter_choice_exists( + struct list_objects_filter_options *filter_options, + enum list_objects_filter_choice choice) { + int i; + + if (!filter_options) + return 0; + + if (filter_options->choice == choice) + return 1; + if (filter_options->sub_nr) + for (i = 0; i < filter_options->sub_nr; i++) + if (filter_options->sub[i].choice == choice) + return 1; + return 0; +} + int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, @@ -97,6 +116,17 @@ int gently_parse_list_objects_filter( return 0; + } else if (skip_prefix(arg, "depth:", &v0)) { + if (!git_parse_ulong(v0, &filter_options->depth)) { + strbuf_addstr(errbuf, _("expected 'depth:'")); + return 1; + } else if (atoi(v0) <= 0) { + strbuf_addf(errbuf, _("depth %s is not a positive number"), v0); + return 1; + } + filter_options->choice = LOFC_DEPTH; + return 0; + } else if (skip_prefix(arg, "combine:", &v0)) { return parse_combine_filter(filter_options, v0, errbuf); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index ffc02d77e76..a4fd40567d2 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -15,6 +15,7 @@ enum list_objects_filter_choice { LOFC_TREE_DEPTH, LOFC_SPARSE_OID, LOFC_OBJECT_TYPE, + LOFC_DEPTH, LOFC_COMBINE, LOFC__COUNT /* must be last */ }; @@ -54,6 +55,7 @@ struct list_objects_filter_options { */ char *sparse_oid_name; + unsigned long depth; unsigned long blob_limit_value; unsigned long tree_exclude_depth; enum object_type object_type; @@ -69,6 +71,10 @@ struct list_objects_filter_options { */ }; +int list_objects_filter_choice_exists( + struct list_objects_filter_options *filter_options, + enum list_objects_filter_choice choice); + /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: diff --git a/list-objects-filter.c b/list-objects-filter.c index 76e8659ea73..5b4d8348b54 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -13,6 +13,8 @@ #include "oidmap.h" #include "oidset.h" #include "object-store.h" +#include "shallow.h" +#include "upload-pack.h" /* Remember to update object flag allocation in object.h */ /* @@ -69,6 +71,69 @@ struct filter { struct oidset *omits; }; +static enum list_objects_filter_result filter_noop( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct oidset *omits, + void *filter_data_) +{ + switch (filter_situation) { + default: + BUG("unknown filter_situation: %d", filter_situation); + + case LOFS_TAG: + assert(obj->type == OBJ_TAG); + /* always include all tag objects */ + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + + case LOFS_COMMIT: + assert(obj->type == OBJ_COMMIT); + /* always include all commit objects */ + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + + case LOFS_BEGIN_TREE: + assert(obj->type == OBJ_TREE); + /* always include all tree objects */ + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + + case LOFS_END_TREE: + assert(obj->type == OBJ_TREE); + return LOFR_ZERO; + + case LOFS_BLOB: + assert(obj->type == OBJ_BLOB); + /* always include all blob objects */ + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + } +} + +static void noop_free(void *filter_data) { + /* noop */ +} + +static void filter_depth__init( + struct traversal_context *ctx, + struct list_objects_filter_options *filter_options, + struct filter *filter) +{ + struct commit_list *result = get_shallow_commits_by_commits(ctx->revs->commits, + filter_options->depth, + SHALLOW, NOT_SHALLOW); + + while (result) { + register_shallow(the_repository, &result->item->object.oid); + result = result->next; + } + free_commit_list(result); + + filter->filter_object_fn = filter_noop; + filter->free_fn = noop_free; +} + + static enum list_objects_filter_result filter_blobs_none( struct repository *r, enum list_objects_filter_situation filter_situation, @@ -774,6 +839,7 @@ static filter_init_fn s_filters[] = { filter_trees_depth__init, filter_sparse_oid__init, filter_object_type__init, + filter_depth__init, filter_combine__init, }; diff --git a/shallow.c b/shallow.c index 8cb768ee5f8..9d1d7668ad8 100644 --- a/shallow.c +++ b/shallow.c @@ -122,6 +122,22 @@ static void free_depth_in_slab(int **ptr) { FREE_AND_NULL(*ptr); } + +struct commit_list *get_shallow_commits_by_commits(struct commit_list *commits, int depth, + int shallow_flag, int not_shallow_flag) { + struct object_array array = OBJECT_ARRAY_INIT; + struct commit_list *result = NULL; + struct commit_list *commit = commits; + + while (commit) { + add_object_array(&commit->item->object, NULL, &array); + commit = commit->next; + } + result = get_shallow_commits(&array, depth, shallow_flag, not_shallow_flag); + object_array_clear(&array); + return result; +} + struct commit_list *get_shallow_commits(struct object_array *heads, int depth, int shallow_flag, int not_shallow_flag) { diff --git a/shallow.h b/shallow.h index aba6ff58294..ed425b72796 100644 --- a/shallow.h +++ b/shallow.h @@ -34,6 +34,8 @@ void rollback_shallow_file(struct repository *r, struct shallow_lock *lk); struct commit_list *get_shallow_commits(struct object_array *heads, int depth, int shallow_flag, int not_shallow_flag); +struct commit_list *get_shallow_commits_by_commits( + struct commit_list *commits, int depth, int shallow_flag, int not_shallow_flag); struct commit_list *get_shallow_commits_by_rev_list( int ac, const char **av, int shallow_flag, int not_shallow_flag); int write_shallow_commits(struct strbuf *out, int use_pack_protocol, diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9aeacc2f6a5..c328f5d76bc 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -458,6 +458,122 @@ test_expect_success 'partial clone with unresolvable sparse filter fails cleanly test_i18ngrep "unable to parse sparse filter data in" err ' +test_expect_success 'setup src repo for depth filter' ' + git init depth-src && + git -C depth-src config --local uploadpack.allowfilter 1 && + git -C depth-src config --local uploadpack.allowanysha1inwant 1 && + test_commit -C depth-src one && + test_commit -C depth-src two && + test_commit -C depth-src three && + git -C depth-src rm -rf two.t && + git -C depth-src commit -m four +' + +test_expect_success 'partial clone with depth=1 filter succeeds' ' + rm -rf dst.git && + git clone --no-local --bare \ + --filter=depth:1 \ + depth-src dst.git && + ( + cd dst.git && + git cat-file --batch-check --batch-all-objects >object && + grep blob object >blob_count && + test_line_count = 2 blob_count && + grep tree object >tree_count && + test_line_count = 1 tree_count && + grep commit object >commit_count && + test_line_count = 1 commit_count + ) +' + +test_expect_success 'partial clone with depth=2 filter succeeds' ' + rm -rf dst.git && + git clone --no-local --bare \ + --filter=depth:2 \ + depth-src dst.git && + ( + cd dst.git && + git cat-file --batch-check --batch-all-objects >object && + grep blob object >blob_count && + test_line_count = 3 blob_count && + grep tree object >tree_count && + test_line_count = 2 tree_count && + grep commit object >commit_count && + test_line_count = 2 commit_count + ) +' + +test_expect_success 'partial clone depth filter combine with blob:none filter succeeds' ' + rm -rf dst.git && + git clone --no-local --bare \ + --filter="combine:depth:1+blob:none" \ + depth-src dst.git && + ( + cd dst.git && + git cat-file --batch-check --batch-all-objects >object && + ! grep blob object && + grep tree object >tree_count && + test_line_count = 1 tree_count && + grep commit object >commit_count && + test_line_count = 1 commit_count + ) +' + +test_expect_success 'refetch other commits after partial clone with depth filter' ' + rm -rf dst.git && + git clone --no-local --bare \ + --filter=depth:1 \ + depth-src dst.git && + ( + cd dst.git && + git cat-file --batch-check --batch-all-objects >object && + grep blob object >blob_count && + test_line_count = 2 blob_count && + grep tree object >tree_count && + test_line_count = 1 tree_count && + grep commit object >commit_count && + test_line_count = 1 commit_count && + # git log will trigger refetch commits + git log && + git cat-file --batch-check --batch-all-objects >object && + grep blob object >blob_count && + test_line_count = 2 blob_count && + grep tree object >tree_count && + test_line_count = 4 tree_count && + grep commit object >commit_count && + test_line_count = 4 commit_count && + # git diff will trigger refetch blobs + git diff HEAD^ HEAD && + git cat-file --batch-check --batch-all-objects >object && + grep blob object >blob_count && + test_line_count = 3 blob_count && + grep tree object >tree_count && + test_line_count = 4 tree_count && + grep commit object >commit_count && + test_line_count = 4 commit_count + ) +' + +test_expect_success 'partial clone with depth filter with shallow clone failed' " + rm -rf dst.git && + test_must_fail git clone --no-local --bare \ + --filter=depth:1 --depth=1 \ + depth-src dst.git 2>err && + test_i18ngrep \"fatal: --filter='depth:' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\" err && + test_must_fail git clone --no-local --bare \ + --filter=depth:1 --shallow-since '300000000 +0700' \ + depth-src dst.git 2>err && + test_i18ngrep \"fatal: --filter='depth:' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\" err && + test_must_fail git clone --no-local --bare \ + --filter=depth:1 --shallow-exclude one.t \ + depth-src dst.git 2>err && + test_i18ngrep \"fatal: --filter='depth:' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\" err && + test_must_fail git clone --no-local --bare \ + --filter=depth:1 --shallow-submodules \ + depth-src dst.git 2>err && + test_i18ngrep \"fatal: --filter='depth:' cannot be used with --depth, --shallow-since, --shallow-exclude, --shallow-submodules\" err +" + setup_triangle () { rm -rf big-blob.txt server client promisor-remote && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 8d9d6604f05..a562bdffc80 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -417,6 +417,20 @@ test_expect_success 'verify tree:3 includes everything expected' ' test_line_count = 10 actual ' +test_expect_success 'verify depth:1 includes tip commit expected' ' + git -C r3 rev-list --objects --filter=depth:1 --no-object-names HEAD >objects && + cat objects | git -C r3 cat-file --batch-check="%(objecttype)" >types && + grep commit types >actual && + test_line_count = 1 actual +' + +test_expect_success 'verify depth:2 includes two commits expected' ' + git -C r3 rev-list --objects --filter=depth:2 --no-object-names HEAD >objects && + cat objects | git -C r3 cat-file --batch-check="%(objecttype)" >types && + grep commit types >actual && + test_line_count = 2 actual +' + test_expect_success 'combine:... for a simple combination' ' git -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \ >actual && diff --git a/upload-pack.c b/upload-pack.c index b217a1f469e..d4ffebfa6ab 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -28,20 +28,6 @@ #include "commit-reach.h" #include "shallow.h" -/* Remember to update object flag allocation in object.h */ -#define THEY_HAVE (1u << 11) -#define OUR_REF (1u << 12) -#define WANTED (1u << 13) -#define COMMON_KNOWN (1u << 14) - -#define SHALLOW (1u << 16) -#define NOT_SHALLOW (1u << 17) -#define CLIENT_SHALLOW (1u << 18) -#define HIDDEN_REF (1u << 19) - -#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \ - NOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF) - /* Enum for allowed unadvertised object request (UOR) */ enum allow_uor { /* Allow specifying sha1 if it is a ref tip. */ diff --git a/upload-pack.h b/upload-pack.h index d6ee25ea98e..36add62f6bc 100644 --- a/upload-pack.h +++ b/upload-pack.h @@ -1,6 +1,20 @@ #ifndef UPLOAD_PACK_H #define UPLOAD_PACK_H +/* Remember to update object flag allocation in object.h */ +#define THEY_HAVE (1u << 11) +#define OUR_REF (1u << 12) +#define WANTED (1u << 13) +#define COMMON_KNOWN (1u << 14) + +#define SHALLOW (1u << 16) +#define NOT_SHALLOW (1u << 17) +#define CLIENT_SHALLOW (1u << 18) +#define HIDDEN_REF (1u << 19) + +#define ALL_FLAGS (THEY_HAVE | OUR_REF | WANTED | COMMON_KNOWN | SHALLOW | \ + NOT_SHALLOW | CLIENT_SHALLOW | HIDDEN_REF) + void upload_pack(const int advertise_refs, const int stateless_rpc, const int timeout);