From patchwork Tue Mar 8 14:39:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 411BEC433EF for ; Tue, 8 Mar 2022 14:39:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347447AbiCHOkY (ORCPT ); Tue, 8 Mar 2022 09:40:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347549AbiCHOkU (ORCPT ); Tue, 8 Mar 2022 09:40:20 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A87938D9D for ; Tue, 8 Mar 2022 06:39:23 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id h15so4535276wrc.6 for ; Tue, 08 Mar 2022 06:39:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7/Vet349YwWXOD88+eWNxKjf9Jhn0cuBQkzK8diKGoU=; b=lCuzi6NanSuerHJToIHJNaZFXjRLC244M6rwqF3bt3kAOSkU11lwicq7rPbLDtEFUK IcPG/z2TohZoQcNyovbTQh1wGyuHxcvU87haUxU1NU8ltLv+fUikGhrwIXk0EnUDG0Fx VrwPIIkyMh7viczhf2WFIXKUmp3XGqlutNF6zC4hdylfrEbhUF7Km1UNOGKZLMLO0nFV +XgSgkFl34g6wFydWxj2Gp5YlFwYw/WAYusd054pRoTDQMEjl9amkj5vY7Gm41OURE1C GgMO9E0FFr8dNyPsBdlzaOxMU9XdT3o+GoLK7N6Ri3lnlghrc9tGgq9ukLfyD09Dy7yX a+ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7/Vet349YwWXOD88+eWNxKjf9Jhn0cuBQkzK8diKGoU=; b=C1H/PVHdsq8dy+OGgMFQEv53SPlRGzZvKZNCyy3c1CEJ/ufj4HUGDv963qk4EJiwkt EwBQy1P2tgDJz4DGXYZ7H7Gz3C1Jh7S70fQVISvJF/WaaXJvTtdxX4WxfMHoowfor+BM ms5hd+88riMR2THguA8kHukCyaLEn3H0mFIeGNhZKUFVBgV2dZFQBiKV7BdneF3PaXCm +Nr9SHScny3o4G5WZhKAImSU7antevjRRfOXuPeo1CNaGqWNuOpwhmEBagWsZIrkS06P WqSclYng4VhAv6kiW6PVLAg9AUwqYLIqjCysB5Ap168Ckj4iKoeYIFB+imW+QJ4qZIAx 6FhA== X-Gm-Message-State: AOAM5313QkqTQJeqgghBf3zlRyjPSnQJ63ef6swLibvaSrrLxwhrRKqC gaFS5aAvGdLu45gDEhAi5xgfxuU6yeQ= X-Google-Smtp-Source: ABdhPJzxuXrLFbCvtddggPTk0A7ny5augS6Ef9x7IfTlJFuer3MhmEyOC/IvtT3yC2+dlm2TZdmwLA== X-Received: by 2002:a5d:49cb:0:b0:1f1:d7e0:3f66 with SMTP id t11-20020a5d49cb000000b001f1d7e03f66mr12887425wrs.218.1646750361785; Tue, 08 Mar 2022 06:39:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p3-20020a05600c358300b003897e440047sm2552272wmq.28.2022.03.08.06.39.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:21 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:08 +0000 Subject: [PATCH v3 01/12] index-pack: document and test the --promisor option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The --promisor option of 'git index-pack' was created in 88e2f9e (introduce fetch-object: fetch one promisor object, 2017-12-05) but was untested. It is currently unused within the Git codebase, but that will change in an upcoming change to 'git bundle unbundle' when there is a filter capability. For now, add documentation about the option and add a test to ensure it is working as expected. Signed-off-by: Derrick Stolee --- Documentation/git-index-pack.txt | 8 ++++++++ t/t5300-pack-object.sh | 4 +++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index 1f1e3592251..4e71c256ecb 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -122,6 +122,14 @@ This option cannot be used with --stdin. + include::object-format-disclaimer.txt[] +--promisor[=]:: + Before committing the pack-index, create a .promisor file for this + pack. Particularly helpful when writing a promisor pack with --fix-thin + since the name of the pack is not final until the pack has been fully + written. If a `` is provided, then that content will be + written to the .promisor file for future reference. See + link:technical/partial-clone.html[partial clone] for more information. + NOTES ----- diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index 2fd845187e7..a11d61206ad 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -315,8 +315,10 @@ test_expect_success \ git index-pack -o tmp.idx test-3.pack && cmp tmp.idx test-1-${packname_1}.idx && - git index-pack test-3.pack && + git index-pack --promisor=message test-3.pack && cmp test-3.idx test-1-${packname_1}.idx && + echo message >expect && + test_cmp expect test-3.promisor && cat test-2-${packname_2}.pack >test-3.pack && git index-pack -o tmp.idx test-2-${packname_2}.pack && From patchwork Tue Mar 8 14:39:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A318AC433EF for ; Tue, 8 Mar 2022 14:39:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243308AbiCHOk1 (ORCPT ); Tue, 8 Mar 2022 09:40:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245743AbiCHOkV (ORCPT ); Tue, 8 Mar 2022 09:40:21 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D5A838DA4 for ; Tue, 8 Mar 2022 06:39:24 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id t11so28899222wrm.5 for ; Tue, 08 Mar 2022 06:39:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sclOhVe4EV0U75UTToLOlsrVExWmGOlFRPY6iEbDWEo=; b=e1i4Pdhnfmlwh8dx+0HqO1sg5MXKYFiW3hsXxuIE958CxFYWHiDlM8LM+03BUHbt8G d0b+7m7z4FNURI+W9pqEl/6AZpPKHC3iyoasatgGhPm3Jbnqb40n43N8jAohC462uM2c 6v7KyQiDHJy21cgqYjyQOR93hcW8hxYEyaG3vnCGlEgUUYVVFQGkOaytpnIEfsu17z+u dSvPzT9yKFgFYE0FN3mq77rwyDFIwU3lN7ho3OzWM1G3v99rCV5q2+zET20a+iJaet/V vRYK16MRUDy9sKm6MWSxW5gQ+wIQPZmzgldSrkEfh+pQTmmhwOfmTwhhq6rHh/RScwXS Omyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sclOhVe4EV0U75UTToLOlsrVExWmGOlFRPY6iEbDWEo=; b=P7JJWYsY3QDdFFzNNTEWHyLLfGNZY+DymmZOcUsm77GGHVcIFlIHDy+sFDiuEM+wDE V6VxR9fyD6bswlK/agNMKbWPAyQ8tLoTpA+ERV56ame5Ah+eqwgn5k9H3INhnWxiyyUj Hxu6a01CgxkiiWCMjub3sQXjqI8rIaY8prWaPxR4ki6mGwaVguWZt3fwJrIb2F6V7wPT 1ktfPcnbZl3uuGFIhL2ughYYwt09VToUXnSJkABFhQwOwFkPyUCznv+OiC5MSLNfBfNR U6hhCAZ/9dd8i1IYorhA7hClyrx+rj7amkQPqLNMSUUpdc1vjY2lOOjRRfLddk4PT+nr XttQ== X-Gm-Message-State: AOAM5328u4zu22dIqUVYVqJ4HhgewRE5HnGia5MW8Vjmb0mlY4zAmotO 3SJzk8xtH7Nku678C0inP+or564XunM= X-Google-Smtp-Source: ABdhPJxqLKePr6yYcOcjj+yur0JopqtHUd3jukk8i2ve/DdSFkcOKY6r+MXnF7ZdMrOn/4j4ioVRJQ== X-Received: by 2002:adf:816e:0:b0:1e4:ad2b:cb24 with SMTP id 101-20020adf816e000000b001e4ad2bcb24mr12908949wrm.521.1646750362692; Tue, 08 Mar 2022 06:39:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l7-20020adfe9c7000000b001f06f8ec92dsm11545825wrn.30.2022.03.08.06.39.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:22 -0800 (PST) Message-Id: <3a88c99d9bc765bf4728fe0f0df1eed86adace0e.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:09 +0000 Subject: [PATCH v3 02/12] revision: put object filter into struct rev_info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Placing a 'struct list_objects_filter_options' pointer within 'struct rev_info' will assist making some bookkeeping around object filters in the future. For now, let's use this new member to remove a static global instance of the struct from builtin/rev-list.c. Signed-off-by: Derrick Stolee --- builtin/rev-list.c | 30 ++++++++++++++++-------------- revision.h | 4 ++++ 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 777558e9b06..6f2b91d304e 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -62,7 +62,6 @@ static const char rev_list_usage[] = static struct progress *progress; static unsigned progress_counter; -static struct list_objects_filter_options filter_options; static struct oidset omitted_objects; static int arg_print_omitted; /* print objects omitted by filter */ @@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value) } static int try_bitmap_count(struct rev_info *revs, - struct list_objects_filter_options *filter, int filter_provided_objects) { uint32_t commit_count = 0, @@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs, */ max_count = revs->max_count; - bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, revs->filter, + filter_provided_objects); if (!bitmap_git) return -1; @@ -453,7 +452,6 @@ static int try_bitmap_count(struct rev_info *revs, } static int try_bitmap_traversal(struct rev_info *revs, - struct list_objects_filter_options *filter, int filter_provided_objects) { struct bitmap_index *bitmap_git; @@ -465,7 +463,8 @@ static int try_bitmap_traversal(struct rev_info *revs, if (revs->max_count >= 0) return -1; - bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, revs->filter, + filter_provided_objects); if (!bitmap_git) return -1; @@ -475,7 +474,6 @@ static int try_bitmap_traversal(struct rev_info *revs, } static int try_bitmap_disk_usage(struct rev_info *revs, - struct list_objects_filter_options *filter, int filter_provided_objects) { struct bitmap_index *bitmap_git; @@ -483,7 +481,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs, if (!show_disk_usage) return -1; - bitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects); if (!bitmap_git) return -1; @@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) } if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { - parse_list_objects_filter(&filter_options, arg); - if (filter_options.choice && !revs.blob_objects) + if (!revs.filter) + CALLOC_ARRAY(revs.filter, 1); + parse_list_objects_filter(revs.filter, arg); + if (revs.filter->choice && !revs.blob_objects) die(_("object filtering requires --objects")); continue; } if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) { - list_objects_filter_set_no_filter(&filter_options); + if (!revs.filter) + CALLOC_ARRAY(revs.filter, 1); + list_objects_filter_set_no_filter(revs.filter); continue; } if (!strcmp(arg, "--filter-provided-objects")) { @@ -688,11 +690,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) progress = start_delayed_progress(show_progress, 0); if (use_bitmap_index) { - if (!try_bitmap_count(&revs, &filter_options, filter_provided_objects)) + if (!try_bitmap_count(&revs, filter_provided_objects)) return 0; - if (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects)) + if (!try_bitmap_disk_usage(&revs, filter_provided_objects)) return 0; - if (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects)) + if (!try_bitmap_traversal(&revs, filter_provided_objects)) return 0; } @@ -733,7 +735,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) oidset_init(&missing_objects, DEFAULT_OIDSET_SIZE); traverse_commit_list_filtered( - &filter_options, &revs, show_commit, show_object, &info, + revs.filter, &revs, show_commit, show_object, &info, (arg_print_omitted ? &omitted_objects : NULL)); if (arg_print_omitted) { diff --git a/revision.h b/revision.h index 3c58c18c63a..1ddb73ab82e 100644 --- a/revision.h +++ b/revision.h @@ -81,6 +81,7 @@ struct rev_cmdline_info { struct oidset; struct topo_walk_info; +struct list_objects_filter_options; struct rev_info { /* Starting list */ @@ -94,6 +95,9 @@ struct rev_info { /* The end-points specified by the end user */ struct rev_cmdline_info cmdline; + /* Object filter options. NULL for no filtering. */ + struct list_objects_filter_options *filter; + /* excluding from --branches, --refs, etc. expansion */ struct string_list *ref_excludes; From patchwork Tue Mar 8 14:39:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDB23C433F5 for ; Tue, 8 Mar 2022 14:39:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347549AbiCHOk2 (ORCPT ); Tue, 8 Mar 2022 09:40:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347550AbiCHOkV (ORCPT ); Tue, 8 Mar 2022 09:40:21 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 140CF38DA2 for ; Tue, 8 Mar 2022 06:39:25 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id r187-20020a1c2bc4000000b003810e6b192aso1649376wmr.1 for ; Tue, 08 Mar 2022 06:39:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=h62JVzb/uF3I+8SLeb9g0QKm9zTbW6d6LFRo3OfnaqQ=; b=NcMP5kSSNdcO6eL8kr+JDNGW8GEmFj168vKIQBL7O1LuAXrYgDTKQRAlwr78mwTnVs 8JE7XB+4YhntcPpTzifu7vl0g1nQybfjWWLknowsj2jqWoOgegfebKG969dNXveifn0Y +T+vRW55JGcZ/WAJwuMEDPQbh6e6aDVM5clnw1pOpxaokbP0AgH/GkcpH0fRyGZ/zIB1 QGkXKsAq3/TtvYO5XnfghU3N1w+qK8iTM+rHpUrAtq0u87lxhC7Nlufixkrrptb0NSGS m29OlF2EVd6BF70lPKPa9FWFUsMpt9o92SNQeO8jskkTUn+wLeHqWV4dv5mHkjhndV4G hbMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=h62JVzb/uF3I+8SLeb9g0QKm9zTbW6d6LFRo3OfnaqQ=; b=7CwtkuTLtHm6m08G3kbRESRY5+cBWJnBLZuPN/eJPBnNLlSJlX0HJkVj4cCk6CADau lDakrKEVCXCHDiSpjzR5fobm9ZokniPGY2k4393pxwRvmV7SSbjjzyjHkXHNg1RozZBe 24/gJ47fIdrN831FKYtVPAAxwCNrqxQPt9QEm7XmjbCMu8GylzcVcEVmtxOU0ivsvVKD oWK1Dn1Q2a1Oloj6GjG9f/RES/pcYar1qA/Xe74Owvhfc95eMQ4qSm4bGOoajikjVwyX hFElcIX4GnMZHx/dZ9+ylNPQFhl2NoVTEHMb9vepoomJErhHy+xjEIK8NI14MoQyWEJe Bmpg== X-Gm-Message-State: AOAM533uAHRT+0mImMBN8kd3su4oW41+EnESrJJPDpORCD0zwD05rewE rqDCbOEi/9e8bNrDjrqZNGDPu4GG5Tw= X-Google-Smtp-Source: ABdhPJzLrVZIrZgK0mMiQIgLokmZ3Yikx6ZFsIZ+PTbHgFOjS8Se2ADq5vd6Oh7BM3HQKtgO0RrPzg== X-Received: by 2002:a05:600c:240b:b0:389:b41c:57bc with SMTP id 11-20020a05600c240b00b00389b41c57bcmr3895900wmp.131.1646750363494; Tue, 08 Mar 2022 06:39:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m18-20020a05600c3b1200b003899d242c3asm2527625wms.44.2022.03.08.06.39.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:23 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:10 +0000 Subject: [PATCH v3 03/12] pack-objects: use rev.filter when possible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee In builtin/pack-objects.c, we use a 'filter_options' global to populate the --filter= argument. The previous change created a pointer to a filter option in 'struct rev_info', so we can use that pointer here as a start to simplifying some usage of object filters. Signed-off-by: Derrick Stolee --- builtin/pack-objects.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index ba2006f2212..256d9b1798f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void) static int get_object_list_from_bitmap(struct rev_info *revs) { - if (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0))) + if (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0))) return -1; if (pack_options_allow_reuse() && @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av) repo_init_revisions(the_repository, &revs, NULL); save_commit_buffer = 0; setup_revisions(ac, av, &revs, &s_r_opt); + revs.filter = &filter_options; /* make sure shallows are read */ is_repository_shallow(the_repository); @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av) if (!fn_show_object) fn_show_object = show_object; - traverse_commit_list_filtered(&filter_options, &revs, + traverse_commit_list_filtered(revs.filter, &revs, show_commit, fn_show_object, NULL, NULL); From patchwork Tue Mar 8 14:39:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 870CCC433EF for ; Tue, 8 Mar 2022 14:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347563AbiCHOk3 (ORCPT ); Tue, 8 Mar 2022 09:40:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347556AbiCHOkX (ORCPT ); Tue, 8 Mar 2022 09:40:23 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8D1338D89 for ; Tue, 8 Mar 2022 06:39:26 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id x15so28862750wru.13 for ; Tue, 08 Mar 2022 06:39:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=q8+BEG3NwnoERyq+lPMXDfKtOelfHmetpKek56q/MYw=; b=a1ci3vq0vpooecJ2NPweWk3cdVUExJpaQF5/42WKZWxMFAp75IT5Dzlp4yAYjPTFKv ylnrcq/aiH20RrYjG/u6mt2GSSB8MkD2QUqDubq6p+ynAFuiIdjUE1rMuADNEirfNLn/ epZKt0ko4kQUm1ZtZtstoEcjawRYijhOAiVyuw+dpt7Vc3iDYZe5BkbwaF/3AukNuv7t dX34zmPxCK19bnjS4Bkc+8c9hQCWqGquEXnQ/HQj/9mNi3I5wgqWQvoz+dGF93HjLsmx iaJla6x3fFvPz1szA9yYARFNmPLjcwXqTeVq0YN1MKIPuuBEVMYBvkK6l8M0jlo4inp0 uBbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=q8+BEG3NwnoERyq+lPMXDfKtOelfHmetpKek56q/MYw=; b=s+HhexJpvj2ZIOB4VZ58PzXPf8FLbzefaBqHE2q0Ea1Uk5NVB8vDqisognBZqJBp+I o9PjUuDB1WVOC+XTNr83jq/NPruskjYxNARmfaGDoQ+WW7EQSdkm1sQzX0CUFzgQ2P3k +IK9rvJJJfwitjyykVX2OyEXtpb/EHvpg4/UdpGv6LPDoZwgsPpZdMMynZGsQu3KUHBN FzVUCM2/0RBZeFytJr/WMmog7pmj+YcKuudNjVbTx4Q+KWzYc8vHrkQG2axrihdETVMZ KtEDK6v9Tw66NkDm+iNjBbg3IeUjDSKXy9dhORpTwVGilGVE4JINvMczRQ5xqYbpZmtj vLJw== X-Gm-Message-State: AOAM532bLXktkDalTIPF68nzySUnB52P+8uOkuLgEclhqLpVpvfKF/ad 07Suxq8mhXgSRcR1SzWvvNSvI7lrNGU= X-Google-Smtp-Source: ABdhPJxEdUJ8mr3GOkYgkYpZ2NxgrjuMR+kMOP0n/JxgfeGdlvcRVJ1PWkiT2TmJfhiBL24BPg0kzQ== X-Received: by 2002:a5d:56cd:0:b0:1ee:3248:525b with SMTP id m13-20020a5d56cd000000b001ee3248525bmr12023573wrw.124.1646750364545; Tue, 08 Mar 2022 06:39:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v4-20020a5d6784000000b001f1f7b814d7sm5974998wru.69.2022.03.08.06.39.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:23 -0800 (PST) Message-Id: <888774f6f28b291c928041a32c3df360cee13d10.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:11 +0000 Subject: [PATCH v3 04/12] pack-bitmap: drop filter in prepare_bitmap_walk() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Now that all consumers of prepare_bitmap_walk() have populated the 'filter' member of 'struct rev_info', we can drop that extra parameter from the method and access it directly from the 'struct rev_info'. Signed-off-by: Derrick Stolee --- builtin/pack-objects.c | 2 +- builtin/rev-list.c | 8 +++----- pack-bitmap.c | 20 +++++++++----------- pack-bitmap.h | 2 -- reachable.c | 2 +- 5 files changed, 14 insertions(+), 20 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 256d9b1798f..57f2cf49696 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void) static int get_object_list_from_bitmap(struct rev_info *revs) { - if (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0))) + if (!(bitmap_git = prepare_bitmap_walk(revs, 0))) return -1; if (pack_options_allow_reuse() && diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 6f2b91d304e..556e78aebb9 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -434,8 +434,7 @@ static int try_bitmap_count(struct rev_info *revs, */ max_count = revs->max_count; - bitmap_git = prepare_bitmap_walk(revs, revs->filter, - filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects); if (!bitmap_git) return -1; @@ -463,8 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs, if (revs->max_count >= 0) return -1; - bitmap_git = prepare_bitmap_walk(revs, revs->filter, - filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects); if (!bitmap_git) return -1; @@ -481,7 +479,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs, if (!show_disk_usage) return -1; - bitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects); + bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects); if (!bitmap_git) return -1; diff --git a/pack-bitmap.c b/pack-bitmap.c index 9c666cdb8bd..613f2797cdf 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -739,8 +739,7 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git, static struct bitmap *find_objects(struct bitmap_index *bitmap_git, struct rev_info *revs, struct object_list *roots, - struct bitmap *seen, - struct list_objects_filter_options *filter) + struct bitmap *seen) { struct bitmap *base = NULL; int needs_walk = 0; @@ -823,7 +822,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, show_data.bitmap_git = bitmap_git; show_data.base = base; - traverse_commit_list_filtered(filter, revs, + traverse_commit_list_filtered(revs->filter, revs, show_commit, show_object, &show_data, NULL); @@ -1219,7 +1218,6 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter) } struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, - struct list_objects_filter_options *filter, int filter_provided_objects) { unsigned int i; @@ -1240,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, if (revs->prune) return NULL; - if (!can_filter_bitmap(filter)) + if (!can_filter_bitmap(revs->filter)) return NULL; /* try to open a bitmapped pack, but don't parse it yet @@ -1297,8 +1295,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, if (haves) { revs->ignore_missing_links = 1; - haves_bitmap = find_objects(bitmap_git, revs, haves, NULL, - filter); + haves_bitmap = find_objects(bitmap_git, revs, haves, NULL); reset_revision_walk(); revs->ignore_missing_links = 0; @@ -1306,8 +1303,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, BUG("failed to perform bitmap walk"); } - wants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap, - filter); + wants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap); if (!wants_bitmap) BUG("failed to perform bitmap walk"); @@ -1315,8 +1311,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, if (haves_bitmap) bitmap_and_not(wants_bitmap, haves_bitmap); - filter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants, - wants_bitmap, filter); + filter_bitmap(bitmap_git, + (revs->filter && filter_provided_objects) ? NULL : wants, + wants_bitmap, + revs->filter); bitmap_git->result = wants_bitmap; bitmap_git->haves = haves_bitmap; diff --git a/pack-bitmap.h b/pack-bitmap.h index 19a63fa1abc..3d3ddd77345 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -10,7 +10,6 @@ struct commit; struct repository; struct rev_info; -struct list_objects_filter_options; static const char BITMAP_IDX_SIGNATURE[] = {'B', 'I', 'T', 'M'}; @@ -54,7 +53,6 @@ void test_bitmap_walk(struct rev_info *revs); int test_bitmap_commits(struct repository *r); int test_bitmap_hashes(struct repository *r); struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, - struct list_objects_filter_options *filter, int filter_provided_objects); uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git); int reuse_partial_packfile_from_bitmap(struct bitmap_index *, diff --git a/reachable.c b/reachable.c index 84e3d0d75ed..b9f4ad886ef 100644 --- a/reachable.c +++ b/reachable.c @@ -205,7 +205,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog, cp.progress = progress; cp.count = 0; - bitmap_git = prepare_bitmap_walk(revs, NULL, 0); + bitmap_git = prepare_bitmap_walk(revs, 0); if (bitmap_git) { traverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen); free_bitmap_index(bitmap_git); From patchwork Tue Mar 8 14:39:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52B98C433FE for ; Tue, 8 Mar 2022 14:39:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344146AbiCHOka (ORCPT ); Tue, 8 Mar 2022 09:40:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347540AbiCHOkY (ORCPT ); Tue, 8 Mar 2022 09:40:24 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1058B38D8A for ; Tue, 8 Mar 2022 06:39:27 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id u1so28854104wrg.11 for ; Tue, 08 Mar 2022 06:39:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QMqisOwNxGJZM+rj/78LU6orxR1EXwydOnTcmjLNxVs=; b=OnChC/RMfqrJ5BhRVTD4jcfvsKZ3kk/HYk9qC44bcR20sppo1E6q5Nc+HEqNyRH3xr AcFz2ixn1xqHJ4Zxk9DU9/PP08brvBe6ucVoUW+qg6DLKOtohp11FO0IRzaF7ro256ZH uko9Rfn6kKGUhsQ8e11+FQGoaOoarcEusi0YU2yB11XrCfkSZw9b+mJES5bLWIk4k9AO oic2g3XYHUT8HdCBMZAI3DYCEFBekG63TyRO3U7Lfi6f/tXRgreS7hp+grWvBfwHWYi+ bGQUktOI0ie54Pu6mq/mrQn5N9Ri+/SX+NOGeA9jhGmfnEsli/A5q3VGYwdcFrpDBUnz 1Nsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QMqisOwNxGJZM+rj/78LU6orxR1EXwydOnTcmjLNxVs=; b=0y1x0S3gi1eUqE1/q1aAgjY/c+6Nq2anBtzsGkdayizJsTSGKxBh2Asvk5GKa0zR18 PoduzRnL6uPxy1r5ghUcqp9Y+/oG4RIsGN1Q1F8DU+oJ+BkJSpyPKDT9POoXfoF+FV3B Uu6nX+sgObQ06zKTeudpcfmd8OHOBDrqvpMHASIR4uHPuiDFyjOi7ntdBPpuG1Khnbyo MQsbPKdESIP248abNMMqjgRgfebfWkoIkexvzAwPZgEKHt6KnLNTB5ZAMK5YV1ts3uC1 v4kZSrj6o/rsJindPgzbGuCOvVAPVLfCEOPN3hKxVNDXy+6ktRlIttF8h228Ej5k9/Wo uGsA== X-Gm-Message-State: AOAM533Up6TsXjwH6r0Lmx2nl0REm8JW4q1eyJF4Bkc3tJRhsray9sN4 UO4QJwzl4B/Zip8q58aviNDrch28dbE= X-Google-Smtp-Source: ABdhPJyPIqduyQfiLMacFA6edcuDnZoEuv9GHNhQkH0eD91vVMCmF3ZCASuTCOz++G488O/R8HCoEA== X-Received: by 2002:adf:ea0f:0:b0:1f0:61f6:c3df with SMTP id q15-20020adfea0f000000b001f061f6c3dfmr12504042wrm.543.1646750365449; Tue, 08 Mar 2022 06:39:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p3-20020a05600c358300b003897e440047sm2552400wmq.28.2022.03.08.06.39.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:25 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:12 +0000 Subject: [PATCH v3 05/12] list-objects: consolidate traverse_commit_list[_filtered] Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Now that all consumers of traverse_commit_list_filtered() populate the 'filter' member of 'struct rev_info', we can drop that parameter from the method prototype to simplify things. In addition, the only thing different now between traverse_commit_list_filtered() and traverse_commit_list() is the presence of the 'omitted' parameter, which is only non-NULL for one caller. We can consolidate these two methods by having one call the other and use the simpler form everywhere the 'omitted' parameter would be NULL. Signed-off-by: Derrick Stolee --- builtin/pack-objects.c | 6 +++--- builtin/rev-list.c | 2 +- list-objects.c | 25 ++++++++----------------- list-objects.h | 12 ++++++++++-- pack-bitmap.c | 6 +++--- 5 files changed, 25 insertions(+), 26 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 57f2cf49696..0432ae1e499 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av) if (!fn_show_object) fn_show_object = show_object; - traverse_commit_list_filtered(revs.filter, &revs, - show_commit, fn_show_object, NULL, - NULL); + traverse_commit_list(&revs, + show_commit, fn_show_object, + NULL); if (unpack_unreachable_expiration) { revs.ignore_missing_links = 1; diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 556e78aebb9..3ab727817fd 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -733,7 +733,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) oidset_init(&missing_objects, DEFAULT_OIDSET_SIZE); traverse_commit_list_filtered( - revs.filter, &revs, show_commit, show_object, &info, + &revs, show_commit, show_object, &info, (arg_print_omitted ? &omitted_objects : NULL)); if (arg_print_omitted) { diff --git a/list-objects.c b/list-objects.c index 2f623f82115..9422625b39e 100644 --- a/list-objects.c +++ b/list-objects.c @@ -416,22 +416,7 @@ static void do_traverse(struct traversal_context *ctx) strbuf_release(&csp); } -void traverse_commit_list(struct rev_info *revs, - show_commit_fn show_commit, - show_object_fn show_object, - void *show_data) -{ - struct traversal_context ctx; - ctx.revs = revs; - ctx.show_commit = show_commit; - ctx.show_object = show_object; - ctx.show_data = show_data; - ctx.filter = NULL; - do_traverse(&ctx); -} - void traverse_commit_list_filtered( - struct list_objects_filter_options *filter_options, struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, @@ -444,7 +429,13 @@ void traverse_commit_list_filtered( ctx.show_object = show_object; ctx.show_commit = show_commit; ctx.show_data = show_data; - ctx.filter = list_objects_filter__init(omitted, filter_options); + if (revs->filter) + ctx.filter = list_objects_filter__init(omitted, revs->filter); + else + ctx.filter = NULL; + do_traverse(&ctx); - list_objects_filter__free(ctx.filter); + + if (ctx.filter) + list_objects_filter__free(ctx.filter); } diff --git a/list-objects.h b/list-objects.h index a952680e466..9eaf4de8449 100644 --- a/list-objects.h +++ b/list-objects.h @@ -7,7 +7,6 @@ struct rev_info; typedef void (*show_commit_fn)(struct commit *, void *); typedef void (*show_object_fn)(struct object *, const char *, void *); -void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *); typedef void (*show_edge_fn)(struct commit *); void mark_edges_uninteresting(struct rev_info *revs, @@ -18,11 +17,20 @@ struct oidset; struct list_objects_filter_options; void traverse_commit_list_filtered( - struct list_objects_filter_options *filter_options, struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, void *show_data, struct oidset *omitted); +static inline void traverse_commit_list( + struct rev_info *revs, + show_commit_fn show_commit, + show_object_fn show_object, + void *show_data) +{ + traverse_commit_list_filtered(revs, show_commit, + show_object, show_data, NULL); +} + #endif /* LIST_OBJECTS_H */ diff --git a/pack-bitmap.c b/pack-bitmap.c index 613f2797cdf..cbefaedbf43 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, show_data.bitmap_git = bitmap_git; show_data.base = base; - traverse_commit_list_filtered(revs->filter, revs, - show_commit, show_object, - &show_data, NULL); + traverse_commit_list(revs, + show_commit, show_object, + &show_data); revs->include_check = NULL; revs->include_check_obj = NULL; From patchwork Tue Mar 8 14:39:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F618C433EF for ; Tue, 8 Mar 2022 14:39:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347556AbiCHOkc (ORCPT ); Tue, 8 Mar 2022 09:40:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347555AbiCHOkZ (ORCPT ); Tue, 8 Mar 2022 09:40:25 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BF8238D9D for ; Tue, 8 Mar 2022 06:39:28 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id t11so28899549wrm.5 for ; Tue, 08 Mar 2022 06:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RxrhDEZAKd1TxOQXR/q4fcpzONcBLxkiP9hpg49quRw=; b=O2fjw40CHBQE/K8AR/0TJxndePsrHxJrVyCOHyoAdgYngviJ/9CRM7pQzYmJEGBAPC 8S4zGHs1yunoTZiGT+Kgz6JLJm4LC5CQTiLw4ky0EJGB56HT3aiQkbup30d23i4veVg5 0x/O/sc768sU9ElC3tnt4GxOBDe7R86mXUUs8V69ZukeScN72ZXmSpMX9Buhkz+nx9bZ yQMH9dJftee4vZ6aF8s6aTL1e6I9BwCZObGrO8eZoZZPqZAQov8oeQQDrEXRb06VSLpE BEqNX4zx2abDjrcFboDDQOq5LQxT+rVsoTEJiDCxNkZ/9o3+MOvXC02/HzBcZxhrwUl7 U+Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RxrhDEZAKd1TxOQXR/q4fcpzONcBLxkiP9hpg49quRw=; b=Z18KxtD2aeeFkyQ84B/xxcHxOgX8S4a8xpiqZviBE6KhirOhJC9NQEb6M+zJ2o+7YE 1qaCH+7P/64vOaUIat5sR/EP8Ej8GT1SlU4bbHX82YTziuzi2x+94udFy8Ol87WazViW X9F0A5GMdTxvhaexMtIMy8ad9Xq/vlN9cW21c3S0XkSFspYoebDPBt78AO7Xts4RsS3j ULfcoEGV0a0uaCEnGlxvG0CzHZG2EXZ3xXeWYk/mMzaWckuZwWSsTYh+M6XGdQWed+gt N8n98Gs06dXAjDkYDyY3zkyMthQjO5CHKwIZy2q2hiPYvUl2qLmTqrR9Svm1JnHqbOCk AhXg== X-Gm-Message-State: AOAM533dnYEcKzYTH56AWr9zQ61CIBGXcAe43rtAS4rEdNBLuFLzJ0P3 pUTOcLyRpWovBeh2dsZUnMsbTBmo+Uw= X-Google-Smtp-Source: ABdhPJy0wIC19CMgn2eT5fs5cDm/gEBSd5jDobd6msc3u8ewD4w8hsMuQ6nryPolwpCHTuFpkOfb+g== X-Received: by 2002:a5d:5850:0:b0:1f0:2d5b:dc35 with SMTP id i16-20020a5d5850000000b001f02d5bdc35mr12385786wrf.344.1646750366418; Tue, 08 Mar 2022 06:39:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s17-20020adfbc11000000b001f023d197b8sm14004629wrg.68.2022.03.08.06.39.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:25 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:13 +0000 Subject: [PATCH v3 06/12] MyFirstObjectWalk: update recommended usage Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The previous change consolidated traverse_commit_list() and traverse_commit_list_filtered(). This allows us to simplify the recommended usage in MyFirstObjectWalk.txt to use this new set of values. While here, add some clarification on the difference between the two methods. Signed-off-by: Derrick Stolee --- Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------ 1 file changed, 16 insertions(+), 28 deletions(-) diff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt index ca267941f3e..8d9e85566e6 100644 --- a/Documentation/MyFirstObjectWalk.txt +++ b/Documentation/MyFirstObjectWalk.txt @@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two functions reside in `list-objects.c`; examining the source shows that, despite the name, these functions traverse all kinds of objects. Let's have a look at -the arguments to `traverse_commit_list_filtered()`, which are a superset of the -arguments to the unfiltered version. +the arguments to `traverse_commit_list()`. -- `struct list_objects_filter_options *filter_options`: This is a struct which - stores a filter-spec as outlined in `Documentation/rev-list-options.txt`. -- `struct rev_info *revs`: This is the `rev_info` used for the walk. +- `struct rev_info *revs`: This is the `rev_info` used for the walk. If + its `filter` member is not `NULL`, then `filter` contains information for + how to filter the object list. - `show_commit_fn show_commit`: A callback which will be used to handle each individual commit object. - `show_object_fn show_object`: A callback which will be used to handle each non-commit object (so each blob, tree, or tag). - `void *show_data`: A context buffer which is passed in turn to `show_commit` and `show_object`. + +In addition, `traverse_commit_list_filtered()` has an additional paramter: + - `struct oidset *omitted`: A linked-list of object IDs which the provided filter caused to be omitted. -It looks like this `traverse_commit_list_filtered()` uses callbacks we provide -instead of needing us to call it repeatedly ourselves. Cool! Let's add the -callbacks first. +It looks like these methods use callbacks we provide instead of needing us +to call it repeatedly ourselves. Cool! Let's add the callbacks first. For the sake of this tutorial, we'll simply keep track of how many of each kind of object we find. At file scope in `builtin/walken.c` add the following @@ -712,20 +713,9 @@ help understand. In our case, that means we omit trees and blobs not directly referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only `HEAD` in the `pending` list.) -First, we'll need to `#include "list-objects-filter-options.h"` and set up the -`struct list_objects_filter_options` at the top of the function. - ----- -static void walken_object_walk(struct rev_info *rev) -{ - struct list_objects_filter_options filter_options = { 0 }; - - ... ----- - For now, we are not going to track the omitted objects, so we'll replace those parameters with `NULL`. For the sake of simplicity, we'll add a simple -build-time branch to use our filter or not. Replace the line calling +build-time branch to use our filter or not. Preface the line calling `traverse_commit_list()` with the following, which will remind us which kind of walk we've just performed: @@ -733,19 +723,17 @@ walk we've just performed: if (0) { /* Unfiltered: */ trace_printf(_("Unfiltered object walk.\n")); - traverse_commit_list(rev, walken_show_commit, - walken_show_object, NULL); } else { trace_printf( _("Filtered object walk with filterspec 'tree:1'.\n")); - parse_list_objects_filter(&filter_options, "tree:1"); - - traverse_commit_list_filtered(&filter_options, rev, - walken_show_commit, walken_show_object, NULL, NULL); + CALLOC_ARRAY(rev->filter, 1); + parse_list_objects_filter(rev->filter, "tree:1"); } + traverse_commit_list(rev, walken_show_commit, + walken_show_object, NULL); ---- -`struct list_objects_filter_options` is usually built directly from a command +The `rev->filter` member is usually built directly from a command line argument, so the module provides an easy way to build one from a string. Even though we aren't taking user input right now, we can still build one with a hardcoded string using `parse_list_objects_filter()`. @@ -784,7 +772,7 @@ object: ---- ... - traverse_commit_list_filtered(&filter_options, rev, + traverse_commit_list_filtered(rev, walken_show_commit, walken_show_object, NULL, &omitted); ... From patchwork Tue Mar 8 14:39:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12773919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86299C433F5 for ; Tue, 8 Mar 2022 14:39:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347584AbiCHOkl (ORCPT ); Tue, 8 Mar 2022 09:40:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347565AbiCHOk3 (ORCPT ); Tue, 8 Mar 2022 09:40:29 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D54FA38D8A for ; Tue, 8 Mar 2022 06:39:28 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id u10so27262695wra.9 for ; Tue, 08 Mar 2022 06:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=kk1MRlblAYjK5sXEUGfrwQYKU7ShZoPcHeq9ONQsuPc=; b=AN5GCAGO/1HeLtc8QBEwOK0Xq9EM9a4oriratGCQzZFLt3rJNor0qiQ365wquBBNYD XP8DtgtlP25e7WgWPBhRUy6xOi5paL0Zf20UDvigwSbMwXGN/b5pvcTf4d6UloAP8a1g jg+OgMRYdSuJK65WnJXD7qORfE2AMSED1+zgFdjGgV0cLuklzZ4sQmGht/wZLPZS0FSy 3ZYwIyp1OSSGfrVFYlpyjmgBij5/Q9tMOlv+44dmsmcAM8NxJepVbcMKijgiqtwlOO96 5qx6dghNejidvvmEP/VEZAPrJiyzNWffAk+HZ54On31qV844P6qRyE1KRnapxqJenQJJ pdxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=kk1MRlblAYjK5sXEUGfrwQYKU7ShZoPcHeq9ONQsuPc=; b=cNX2pe+lsmbGRsDLE7/sCrmAhnfUPML1m3kV2OI+7TP7OFD8EB0cegkauYVcWiEYWm medGpV59NeIlyaQHct23w4wgF/9imosh8VWKFPnl96mlGALgW1PhsmgRFtk2niqMv3EE w2DpyfprXrF2+mKBAHrP/o8hA9DjbkVXnT69Lhh1pNmBggObcqL8fGBhNEsJt4XWqvxO wssx/M0y9RVYjxya6FR1d8Rg6no0YR+GrCZOr+xOhObospHkqKxn0/eaBfmF06Uhabx2 MHs9fhpimzORoOyKzJeqWCeXE5irUCRF9NBZ6XtHzvj1hiSge9YVvxQ7hkkUpGo4fvdU prrQ== X-Gm-Message-State: AOAM531TzE4W28y5ZGv6WzTDKtlZQa1WInuEzykDZpZ1G0q6p/eqpUXi ApTmhPGfh4uZFj7ocyzt556I17FIv6o= X-Google-Smtp-Source: ABdhPJyZ/db1PC1J/ppNDbV/iReOhIz0ufxZ6C17mNJfbsbNYR2kLaWVvxrDly30ShuLYmAQHfA49Q== X-Received: by 2002:adf:90e2:0:b0:1e3:f5a:553c with SMTP id i89-20020adf90e2000000b001e30f5a553cmr12185852wri.476.1646750367299; Tue, 08 Mar 2022 06:39:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n65-20020a1c2744000000b003862bfb509bsm2275307wmn.46.2022.03.08.06.39.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:26 -0800 (PST) Message-Id: <782182a26e37eb8e84aef7d8cc67cf276b2abb54.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:14 +0000 Subject: [PATCH v3 07/12] list-objects: handle NULL function pointers MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= If a caller to traverse_commit_list() specifies the options for the --objects flag but does not specify a show_object function pointer, the result is a segfault. This is currently visible by running 'git bundle create --objects HEAD'. We could fix this problem by supplying a no-op callback in builtin/bundle.c, but that only solves the problem for one builtin, leaving this segfault open for other callers. Replace all callers of the show_commit and show_object function pointers in list-objects.c to be local methods show_commit() and show_object() which check that the given contex has non-NULL functions before passing the necessary data. One extra benefit is that it reduces duplication due to passing ctx->show_data to every caller. Test that this segfault no longer occurs for 'git bundle'. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- bundle.c | 2 ++ list-objects.c | 27 ++++++++++++++++++++++----- t/t6020-bundle-misc.sh | 12 ++++++++++++ 3 files changed, 36 insertions(+), 5 deletions(-) diff --git a/bundle.c b/bundle.c index a0bb687b0f4..7ba60a573d7 100644 --- a/bundle.c +++ b/bundle.c @@ -544,6 +544,8 @@ int create_bundle(struct repository *r, const char *path, die("revision walk setup failed"); bpi.fd = bundle_fd; bpi.pending = &revs_copy.pending; + + revs.blob_objects = revs.tree_objects = 0; traverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi); object_array_remove_duplicates(&revs_copy.pending); diff --git a/list-objects.c b/list-objects.c index 9422625b39e..0af0bef1dbc 100644 --- a/list-objects.c +++ b/list-objects.c @@ -21,6 +21,23 @@ struct traversal_context { struct filter *filter; }; +static void show_commit(struct traversal_context *ctx, + struct commit *commit) +{ + if (!ctx->show_commit) + return; + ctx->show_commit(commit, ctx->show_data); +} + +static void show_object(struct traversal_context *ctx, + struct object *object, + const char *name) +{ + if (!ctx->show_object) + return; + ctx->show_object(object, name, ctx->show_data); +} + static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, @@ -60,7 +77,7 @@ static void process_blob(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, path->buf, ctx->show_data); + show_object(ctx, obj, path->buf); strbuf_setlen(path, pathlen); } @@ -194,7 +211,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); + show_object(ctx, obj, base->buf); if (base->len) strbuf_addch(base, '/'); @@ -210,7 +227,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); + show_object(ctx, obj, base->buf); strbuf_setlen(base, baselen); free_tree_buffer(tree); @@ -228,7 +245,7 @@ static void process_tag(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) tag->object.flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(&tag->object, name, ctx->show_data); + show_object(ctx, &tag->object, name); } static void mark_edge_parents_uninteresting(struct commit *commit, @@ -402,7 +419,7 @@ static void do_traverse(struct traversal_context *ctx) if (r & LOFR_MARK_SEEN) commit->object.flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_commit(commit, ctx->show_data); + show_commit(ctx, commit); if (ctx->revs->tree_blobs_in_commit_order) /* diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh index b13e8a52a93..6522401617d 100755 --- a/t/t6020-bundle-misc.sh +++ b/t/t6020-bundle-misc.sh @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' ' test_cmp expect actual ' +test_expect_success 'unfiltered bundle with --objects' ' + git bundle create all-objects.bdl \ + --all --objects && + git bundle create all.bdl \ + --all && + + # Compare the headers of these files. + head -11 all.bdl >expect && + head -11 all-objects.bdl >actual && + test_cmp expect actual +' + test_done From patchwork Tue Mar 8 14:39:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B4A8C433F5 for ; Tue, 8 Mar 2022 14:39:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347577AbiCHOki (ORCPT ); Tue, 8 Mar 2022 09:40:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347564AbiCHOk3 (ORCPT ); Tue, 8 Mar 2022 09:40:29 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFB8838DA5 for ; Tue, 8 Mar 2022 06:39:29 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id u10so27262770wra.9 for ; Tue, 08 Mar 2022 06:39:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cODEiV4ork0dtdnF/SVv9/aBWdDEJq0vadf34r48t0g=; b=H2hXAzZ8I9Chgf0RM6rJQ2BVbAMzrPc7+Cwdbr2JrrEUlbmMoyD7gu8jCbo6pFW9BR 8R6rOj2cetNjH/QG9pyGXD3n+mv08sxV2xnZn2muwYNXqxd7+ygs9ZGHZTwSpIP8qAo3 Bu9Yq0fjtRIwhvPG6iCoMq4w434zU/8Ycv3GSSjCRv5+s3M/7nnQRaWEDsLH6TFYQODd N5Obp4hJFsjDR0jWVl3MZvokgQhcH5wQj+pz/Bkz9z7evs9TJdvJtP2FHa1wDUmmubd2 fCwQPS9rm7aSOOHQweoEvucrY+ZH3i+qWUBVrrAoiORN2So/DIaQxGB26l5qvGDO+5M/ YVFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cODEiV4ork0dtdnF/SVv9/aBWdDEJq0vadf34r48t0g=; b=BJekvfDqlKhIolu2KxK/t5ckYDa40wzkj9HJzPfB++/kCSkRJVvc1Tp1kEPTy1lXoh s2Po97I4qnoog1UlNc2IfjqrLw8GyBhATpeYV91VAUpdqSwpEwoRvLhbZ3Txqkyl6m1E VlcE+FBpIpo+9BGi2KGgpjVrnCpbFNTYdZpOgC0P2xh2wJ1t0VlQkSAcTr8dVCamwWHT y6VOXJVgU73anBYNPuZLD2o+ZUY+LRK8B2U5bJq9nJ34NA2DzR2i5zKWZMgRyJe34Swy LhOy/a9lKPG6NPXZkrva7E3nO8/Bja4ScuHa56jmOserwrBOMka4eCjDGtIwEzc3kz+g BOow== X-Gm-Message-State: AOAM530Nf32422vE1d/BwD1Jcq96eBq2cUbm6nZqKITSeIAvtw87prf5 DxCTu8Vk8DsheN6UWBuvsdGCRuXNUt8= X-Google-Smtp-Source: ABdhPJytXwUlGU4keWkhq5SelTiH5pKXDs1UaaFIgQntobn+LWjU7/vGANmRLnDIRrvJlWm40QbN5A== X-Received: by 2002:adf:d210:0:b0:1f1:f999:b538 with SMTP id j16-20020adfd210000000b001f1f999b538mr6834682wrh.473.1646750368126; Tue, 08 Mar 2022 06:39:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n188-20020a1ca4c5000000b00389c692a3b5sm1592743wme.42.2022.03.08.06.39.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:27 -0800 (PST) Message-Id: <025f38290f5a705c80854a42e1abcaba9a9f336d.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:15 +0000 Subject: [PATCH v3 08/12] bundle: parse filter capability Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The v3 bundle format has capabilities, allowing newer versions of Git to create bundles with newer features. Older versions that do not understand these new capabilities will fail with a helpful warning. Create a new capability allowing Git to understand that the contained pack-file is filtered according to some object filter. Typically, this filter will be "blob:none" for a blobless partial clone. This change teaches Git to parse this capability, place its value in the bundle header, and demonstrate this understanding by adding a message to 'git bundle verify'. Since we will use gently_parse_list_objects_filter() outside of list-objects-filter-options.c, make it an external method and move its API documentation to before its declaration. Signed-off-by: Derrick Stolee --- Documentation/technical/bundle-format.txt | 11 ++++++++--- bundle.c | 15 ++++++++++++++- bundle.h | 2 ++ list-objects-filter-options.c | 17 +---------------- list-objects-filter-options.h | 20 ++++++++++++++++++++ 5 files changed, 45 insertions(+), 20 deletions(-) diff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt index bac558d049a..b9be8644cf5 100644 --- a/Documentation/technical/bundle-format.txt +++ b/Documentation/technical/bundle-format.txt @@ -71,6 +71,11 @@ and the Git bundle v2 format cannot represent a shallow clone repository. == Capabilities Because there is no opportunity for negotiation, unknown capabilities cause 'git -bundle' to abort. The only known capability is `object-format`, which specifies -the hash algorithm in use, and can take the same values as the -`extensions.objectFormat` configuration value. +bundle' to abort. + +* `object-format` specifies the hash algorithm in use, and can take the same + values as the `extensions.objectFormat` configuration value. + +* `filter` specifies an object filter as in the `--filter` option in + linkgit:git-rev-list[1]. The resulting pack-file must be marked as a + `.promisor` pack-file after it is unbundled. diff --git a/bundle.c b/bundle.c index 7ba60a573d7..41922565627 100644 --- a/bundle.c +++ b/bundle.c @@ -11,7 +11,7 @@ #include "run-command.h" #include "refs.h" #include "strvec.h" - +#include "list-objects-filter-options.h" static const char v2_bundle_signature[] = "# v2 git bundle\n"; static const char v3_bundle_signature[] = "# v3 git bundle\n"; @@ -33,6 +33,7 @@ void bundle_header_release(struct bundle_header *header) { string_list_clear(&header->prerequisites, 1); string_list_clear(&header->references, 1); + list_objects_filter_release(&header->filter); } static int parse_capability(struct bundle_header *header, const char *capability) @@ -45,6 +46,10 @@ static int parse_capability(struct bundle_header *header, const char *capability header->hash_algo = &hash_algos[algo]; return 0; } + if (skip_prefix(capability, "filter=", &arg)) { + parse_list_objects_filter(&header->filter, arg); + return 0; + } return error(_("unknown capability '%s'"), capability); } @@ -220,6 +225,8 @@ int verify_bundle(struct repository *r, req_nr = revs.pending.nr; setup_revisions(2, argv, &revs, NULL); + revs.filter = &header->filter; + if (prepare_revision_walk(&revs)) die(_("revision walk setup failed")); @@ -259,6 +266,12 @@ int verify_bundle(struct repository *r, r->nr), r->nr); list_refs(r, 0, NULL); + + if (header->filter.choice != LOFC_DISABLED) { + printf_ln("The bundle uses this filter: %s", + list_objects_filter_spec(&header->filter)); + } + r = &header->prerequisites; if (!r->nr) { printf_ln(_("The bundle records a complete history.")); diff --git a/bundle.h b/bundle.h index 06009fe6b1f..7fef2108f43 100644 --- a/bundle.h +++ b/bundle.h @@ -4,12 +4,14 @@ #include "strvec.h" #include "cache.h" #include "string-list.h" +#include "list-objects-filter-options.h" struct bundle_header { unsigned version; struct string_list prerequisites; struct string_list references; const struct git_hash_algo *hash_algo; + struct list_objects_filter_options filter; }; #define BUNDLE_HEADER_INIT \ diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index fd8d59f653a..d8597cdee36 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -40,22 +40,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c) BUG("list_object_filter_config_name: invalid argument '%d'", c); } -/* - * Parse value of the argument to the "filter" keyword. - * On the command line this looks like: - * --filter= - * and in the pack protocol as: - * "filter" SP - * - * The filter keyword will be used by many commands. - * See Documentation/rev-list-options.txt for allowed values for . - * - * Capture the given arg as the "filter_spec". This can be forwarded to - * subordinate commands when necessary (although it's better to pass it through - * expand_list_objects_filter_spec() first). We also "intern" the arg for the - * convenience of the current command. - */ -static int gently_parse_list_objects_filter( +int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf) diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index da5b6737e27..f6fe6a3d2ca 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -72,6 +72,26 @@ struct list_objects_filter_options { /* Normalized command line arguments */ #define CL_ARG__FILTER "filter" +/* + * Parse value of the argument to the "filter" keyword. + * On the command line this looks like: + * --filter= + * and in the pack protocol as: + * "filter" SP + * + * The filter keyword will be used by many commands. + * See Documentation/rev-list-options.txt for allowed values for . + * + * Capture the given arg as the "filter_spec". This can be forwarded to + * subordinate commands when necessary (although it's better to pass it through + * expand_list_objects_filter_spec() first). We also "intern" the arg for the + * convenience of the current command. + */ +int gently_parse_list_objects_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf); + void list_objects_filter_die_if_populated( struct list_objects_filter_options *filter_options); From patchwork Tue Mar 8 14:39:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7F92C433F5 for ; Tue, 8 Mar 2022 14:39:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347567AbiCHOkn (ORCPT ); Tue, 8 Mar 2022 09:40:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347568AbiCHOka (ORCPT ); Tue, 8 Mar 2022 09:40:30 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A758039143 for ; Tue, 8 Mar 2022 06:39:30 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id j26so18525928wrb.1 for ; Tue, 08 Mar 2022 06:39:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=banR2/YPRedUlMEFqceTdv5AzXLOH8Pv1nIbKygHUss=; b=GRYPGzVVqZQOa0gisGG5gxH7kONDMMiRE5GBhxQDzqnbNovysKuSOH3KNk6DXDjm/V YIMFBmjVefPFIelPYeIO5CWp/ZYBvrbxJCS7ZlxFJ4DsK4SmmDuuh2K2+V1YBMzdLTSe D9tXU2HQgNp9qT6qsDhmd1Zyb5j/oDSrXLsdJZl6Tn/7ZJDPZYXNJRwNUgIqzAKxFxQk kmzB5CVYjOlEPHpkpxzhtipx7jTAJNExArSgMh4nm+kPzJ39Ur1XR/nkSlXRh9apvMei Cx/8h/jHD78LtoSpU/3m6RIk8Nf2EpPKezKhkLS+prTkKf0n4XGW86YL4VnJag8XOVeh rtpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=banR2/YPRedUlMEFqceTdv5AzXLOH8Pv1nIbKygHUss=; b=fLgZI+IpkoFk3jEeQiwcrQ3DEtSuXMlBtmw1dNStdGeZTE7um8Sl2/sgKJYFkwWFCe zInP6K2A5IeaJSAA9SL0wtSux6xt+SBjVPzF1g3qd4RoaOeLevMcrYlHRC2rmRlNPvI/ PLgggo0nPvQ5dYl/4j/Y+mDmvxYkjToEfsaQNyIEZ+tRDCqKZ0T7p1hSwEtBeFFOzhcZ NZ7hUEi6z99xtuppwnHLBr5wKgDfCsn1sgg1C0kstBSjHDglT0uga5MVM8qxzlHcLytH gMg/4ijIkvNDNlOjfFM5bbvOsJjwYUkHJbDnyGYKv7B31eCrUsaPg9veMmNDBc49thD+ nBAg== X-Gm-Message-State: AOAM533TAvw5BVouKQjt/Uvhza5/ph5oqyEOBetn3Co6e/gPm2x2xd0W JSKnVo59JHYgOpM0BcXefURLWrjeIu4= X-Google-Smtp-Source: ABdhPJyOf9SZycgKh/IjZ+1agV2Pupp8mhSMaw9EDLwDtSJt5HEWSD7KPwrHdF9256pBlNgwQflHRw== X-Received: by 2002:adf:fbc4:0:b0:1e7:2060:d65 with SMTP id d4-20020adffbc4000000b001e720600d65mr12331133wrs.583.1646750368870; Tue, 08 Mar 2022 06:39:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 186-20020a1c19c3000000b0038990c0925fsm2551440wmz.14.2022.03.08.06.39.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:28 -0800 (PST) Message-Id: <2c8e8a6c2a50aced72d70fbc610f5b4273b00947.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:16 +0000 Subject: [PATCH v3 09/12] rev-list: move --filter parsing into revision.c Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Now that 'struct rev_info' has a 'filter' member and most consumers of object filtering are using that member instead of an external struct, move the parsing of the '--filter' option out of builtin/rev-list.c and into revision.c. This use within handle_revision_pseudo_opt() allows us to find the option within setup_revisions() if the arguments are passed directly. In the case of a command such as 'git blame', the arguments are first scanned and checked with parse_revision_opt(), which complains about the option, so 'git blame --filter=blob:none ' does not become valid with this change. Some commands, such as 'git diff' gain this option without having it make an effect. And 'git diff --objects' was already possible, but does not actually make sense in that builtin. The key addition that is coming is 'git bundle create --filter=' so we can create bundles containing promisor packs. More work is required to make them fully functional, but that will follow. Signed-off-by: Derrick Stolee --- builtin/rev-list.c | 15 --------------- revision.c | 11 +++++++++++ 2 files changed, 11 insertions(+), 15 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 3ab727817fd..640828149c5 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -591,21 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) show_progress = arg; continue; } - - if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { - if (!revs.filter) - CALLOC_ARRAY(revs.filter, 1); - parse_list_objects_filter(revs.filter, arg); - if (revs.filter->choice && !revs.blob_objects) - die(_("object filtering requires --objects")); - continue; - } - if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) { - if (!revs.filter) - CALLOC_ARRAY(revs.filter, 1); - list_objects_filter_set_no_filter(revs.filter); - continue; - } if (!strcmp(arg, "--filter-provided-objects")) { filter_provided_objects = 1; continue; diff --git a/revision.c b/revision.c index ad4286fbdde..1d612c1c102 100644 --- a/revision.c +++ b/revision.c @@ -32,6 +32,7 @@ #include "utf8.h" #include "bloom.h" #include "json-writer.h" +#include "list-objects-filter-options.h" volatile show_early_output_fn_t show_early_output; @@ -2669,6 +2670,14 @@ static int handle_revision_pseudo_opt(struct rev_info *revs, revs->no_walk = 0; } else if (!strcmp(arg, "--single-worktree")) { revs->single_worktree = 1; + } else if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { + if (!revs->filter) + CALLOC_ARRAY(revs->filter, 1); + parse_list_objects_filter(revs->filter, arg); + } else if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) { + if (!revs->filter) + CALLOC_ARRAY(revs->filter, 1); + list_objects_filter_set_no_filter(revs->filter); } else { return 0; } @@ -2872,6 +2881,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s die("cannot combine --walk-reflogs with history-limiting options"); if (revs->rewrite_parents && revs->children.name) die(_("options '%s' and '%s' cannot be used together"), "--parents", "--children"); + if (revs->filter && revs->filter->choice && !revs->blob_objects) + die(_("object filtering requires --objects")); /* * Limitations on the graph functionality From patchwork Tue Mar 8 14:39:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CEFDC433FE for ; Tue, 8 Mar 2022 14:39:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347583AbiCHOkq (ORCPT ); Tue, 8 Mar 2022 09:40:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347571AbiCHOka (ORCPT ); Tue, 8 Mar 2022 09:40:30 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96FEF39147 for ; Tue, 8 Mar 2022 06:39:31 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id q14so13326987wrc.4 for ; Tue, 08 Mar 2022 06:39:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=DQYyduXCH8AzCMoe7NeiL6VRz0Am9b0h6i4kCTzpbs0=; b=kscVmkQfG2to4aLLMkV5ryh34ifG2IC6MBXI8XY5RLCt9K92vyTBQtqmVstSZUsLCp XlTE7WGjZTpxEXwaQPOzYy0YsS/4sKHtxZaI/NkL9nUcsCvq5yi22YAcr5wCrT6jJIV9 lUG98dgJXA7TOMKJHpPCI6n473xBdGRiz3OfvaCJdn0nbQbo83gP3U3zvSeR4vlyxoE/ CTv5AROWyIMVTcxZ7BfAQCQPge2bPACzv4qDbwqTEMTaT1rbk28jsf4hWzOzbGJPFQnA 98/8JCCxAweev3QrkXQ8c5R2YiDnJO9WBmmYckeVXJNqzfLtPD9S6rgKa/YSm3gARdAr lYAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=DQYyduXCH8AzCMoe7NeiL6VRz0Am9b0h6i4kCTzpbs0=; b=ULIhBaAL0D4745QFAl/fDgAWNzy5q0xZ/RyQPvtreRgTaEoU0SqFIC6E7PbyJQXxDx bxjNmtGvhxTEi6+Sjqsk8HU0uojum/oso7NDo/osO1ViHLTyRBq0dqGHO02oaw4MOilW SiBadlesyfcul1AicErmllcUu6Gx76p3upNe1MdCnO6P8CT24do3KGSnwbBvSyHKwzTP 8+P76TvBmxKtkXl3g/xX7C59ORhL9WOfmzDpRx+HIxFt1NwW5cAuzNAhEoEziylZvqVy D2OgZJ9ByLkK+iK3h07j1sZ9plQQJniZdoQAIca0esFF+QpCy8OQ36qEqbq2zrdoDEbo 4o4Q== X-Gm-Message-State: AOAM533QYBBE//aNUCuYNjeDUDoGt3fC9+Vo4i++2HYirf4Cjjcjo9zI d2gJ3Ybch+uEuqP8ZGNfdKwpFlapywE= X-Google-Smtp-Source: ABdhPJxfrVoK+AmpfgXLcjidTDmDsAyzRmQGHPbo4DbOLraHg54HyHKMSe9ePZ1NEd+AHrcxMPhGFA== X-Received: by 2002:a5d:64c5:0:b0:1f1:e6b8:bd3c with SMTP id f5-20020a5d64c5000000b001f1e6b8bd3cmr9783403wri.686.1646750369938; Tue, 08 Mar 2022 06:39:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o3-20020a1c7503000000b0038100e2a1adsm2331754wmc.47.2022.03.08.06.39.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:29 -0800 (PST) Message-Id: <470b6f73e2805b0ed4ec83bc8eb8be45f16fbb5a.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:17 +0000 Subject: [PATCH v3 10/12] bundle: create filtered bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A previous change allowed Git to parse bundles with the 'filter' capability. Now, teach Git to create bundles with this option. Some rearranging of code is required to get the option parsing in the correct spot. There are now two reasons why we might need capabilities (a new hash algorithm or an object filter) so that is pulled out into a place where we can check both at the same time. The --filter option is parsed as part of setup_revisions(), but it expected the --objects flag, too. That flag is somewhat implied by 'git bundle' because it creates a pack-file walking objects, but there is also a walk that walks the revision range expecting only commits. Make this parsing work by setting 'revs.tree_objects' and 'revs.blob_objects' before the call to setup_revisions(). Signed-off-by: Derrick Stolee --- Documentation/git-bundle.txt | 4 +-- bundle.c | 55 ++++++++++++++++++++++++++++-------- t/t6020-bundle-misc.sh | 48 +++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+), 13 deletions(-) diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt index 72ab8139052..831c4788a94 100644 --- a/Documentation/git-bundle.txt +++ b/Documentation/git-bundle.txt @@ -75,8 +75,8 @@ verify :: cleanly to the current repository. This includes checks on the bundle format itself as well as checking that the prerequisite commits exist and are fully linked in the current repository. - 'git bundle' prints a list of missing commits, if any, and exits - with a non-zero status. + 'git bundle' prints the bundle's object filter and its list of + missing commits, if any, and exits with a non-zero status. list-heads :: Lists the references defined in the bundle. If followed by a diff --git a/bundle.c b/bundle.c index 41922565627..393216c6246 100644 --- a/bundle.c +++ b/bundle.c @@ -332,6 +332,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec * "--stdout", "--thin", "--delta-base-offset", NULL); strvec_pushv(&pack_objects.args, pack_options->v); + if (revs->filter) + strvec_pushf(&pack_objects.args, "--filter=%s", + list_objects_filter_spec(revs->filter)); pack_objects.in = -1; pack_objects.out = bundle_fd; pack_objects.git_cmd = 1; @@ -499,10 +502,37 @@ int create_bundle(struct repository *r, const char *path, int bundle_to_stdout; int ref_count = 0; struct rev_info revs, revs_copy; - int min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3; + int min_version = 2; struct bundle_prerequisites_info bpi; int i; + /* init revs to list objects for pack-objects later */ + save_commit_buffer = 0; + repo_init_revisions(r, &revs, NULL); + + /* + * Pre-initialize the '--objects' flag so we can parse a + * --filter option successfully. + */ + revs.tree_objects = revs.blob_objects = 1; + + argc = setup_revisions(argc, argv, &revs, NULL); + + /* + * Reasons to require version 3: + * + * 1. @object-format is required because our hash algorithm is not + * SHA1. + * 2. @filter is required because we parsed an object filter. + */ + if (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter) + min_version = 3; + + if (argc > 1) { + error(_("unrecognized argument: %s"), argv[1]); + goto err; + } + bundle_to_stdout = !strcmp(path, "-"); if (bundle_to_stdout) bundle_fd = 1; @@ -525,17 +555,14 @@ int create_bundle(struct repository *r, const char *path, write_or_die(bundle_fd, capability, strlen(capability)); write_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name)); write_or_die(bundle_fd, "\n", 1); - } - - /* init revs to list objects for pack-objects later */ - save_commit_buffer = 0; - repo_init_revisions(r, &revs, NULL); - argc = setup_revisions(argc, argv, &revs, NULL); - - if (argc > 1) { - error(_("unrecognized argument: %s"), argv[1]); - goto err; + if (revs.filter) { + const char *value = expand_list_objects_filter_spec(revs.filter); + capability = "@filter="; + write_or_die(bundle_fd, capability, strlen(capability)); + write_or_die(bundle_fd, value, strlen(value)); + write_or_die(bundle_fd, "\n", 1); + } } /* save revs.pending in revs_copy for later use */ @@ -558,6 +585,12 @@ int create_bundle(struct repository *r, const char *path, bpi.fd = bundle_fd; bpi.pending = &revs_copy.pending; + /* + * Nullify the filter here, and any object walking. We only care + * about commits and tags here. The revs_copy has the right + * instances of these values. + */ + revs.filter = NULL; revs.blob_objects = revs.tree_objects = 0; traverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi); object_array_remove_duplicates(&revs_copy.pending); diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh index 6522401617d..f10cf011519 100755 --- a/t/t6020-bundle-misc.sh +++ b/t/t6020-bundle-misc.sh @@ -487,4 +487,52 @@ test_expect_success 'unfiltered bundle with --objects' ' test_cmp expect actual ' +for filter in "blob:none" "tree:0" "tree:1" "blob:limit=100" +do + test_expect_success "filtered bundle: $filter" ' + test_when_finished rm -rf .git/objects/pack cloned unbundled && + git bundle create partial.bdl \ + --all \ + --filter=$filter && + + git bundle verify partial.bdl >unfiltered && + make_user_friendly_and_stable_output actual && + + cat >expect <<-EOF && + The bundle contains these 10 refs: + refs/heads/main + refs/heads/release + refs/heads/topic/1 + refs/heads/topic/2 + refs/pull/1/head + refs/pull/2/head + refs/tags/v1 + refs/tags/v2 + refs/tags/v3 + HEAD + The bundle uses this filter: $filter + The bundle records a complete history. + EOF + test_cmp expect actual && + + test_config uploadpack.allowfilter 1 && + test_config uploadpack.allowanysha1inwant 1 && + git clone --no-local --filter=$filter --bare "file://$(pwd)" cloned && + + git init unbundled && + git -C unbundled bundle unbundle ../partial.bdl >ref-list.txt && + + # Count the same number of reachable objects. + reflist=$(git for-each-ref --format="%(objectname)") && + git rev-list --objects --filter=$filter --missing=allow-any \ + $reflist >expect && + for repo in cloned unbundled + do + git -C $repo rev-list --objects --missing=allow-any \ + $reflist >actual && + test_cmp expect actual || return 1 + done + ' +done + test_done From patchwork Tue Mar 8 14:39:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D7A6C433EF for ; Tue, 8 Mar 2022 14:39:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346961AbiCHOks (ORCPT ); Tue, 8 Mar 2022 09:40:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347558AbiCHOkg (ORCPT ); Tue, 8 Mar 2022 09:40:36 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6823F39159 for ; Tue, 8 Mar 2022 06:39:32 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id h15so4536067wrc.6 for ; Tue, 08 Mar 2022 06:39:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XwVYkuxuOeTBby06SZbqGTmC3SIt7fObUONs40r0wX4=; b=ecIazw4psoxDINr/G1GXjTxEusKFTdkqX333Gc6Kh01ULHqjF5w1boCU2l+Ev4SHGL oAuCY9daLorB+czbbySJVlkBjCMCmNKlI+6+XDcsIzKdDZ76ntcXIBeweW7KeAjt/hp6 ambotf6OtgRdMFhiC56Nzl493kPipBnZ7gZLRLoOWYy0rDOaDvHGMHBwnKzpSYLsvrr9 imsemdeMU9NGi4plc2Mh+4GXjVXFJsmfrv0aOfX4YxmtyP7TFUvbD1kTA02aqI0QyioQ U0mg0z6iqpb+Bgq0i+Asw5+UtC2YMpnoSqyblA9lKjOyo3VDb93q4AUeeBiE2NdZQueT bTog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XwVYkuxuOeTBby06SZbqGTmC3SIt7fObUONs40r0wX4=; b=xw1iqq4yWYqgzjE0X2LCSdwLjHZZsaVCRm8s7cxFGxFpyXI4FsPIhaPA6MApb4aJ16 pSYj/YDpT3/27hUYlQmbk2NVVXVIXgyrCD633AcCuIlHTRT75OfuZ1Ci8eiVSwJURF5N aLsrJrlGPzdIJ0BaM+SQSqHVvJQ26lbZpkPlb76esF4fizH+O7yyCKzTOVcC0zgxmCV7 HzIDM/RmCt4f3Jlo/zgdKFpa5884t7tKHRqq7pnWDLFU9bVH+DEO6o6aR7YTGh563BtA A9ujbGZYERhrviTLyUBsOk3NgF6YBkOlvX5k5ze8Fl7G6+JLltz44wxSOYGF3Qvc2tpc nYSg== X-Gm-Message-State: AOAM531pfczHfYazp+gbnOZUNpKdFYv9vHwOHtpTr5S/aZaxfSgb/ohX Pp+lJDXciwaVci7bDOIpH8KVEny9oLI= X-Google-Smtp-Source: ABdhPJyQlJdzxWhA+47GJVboJHWykcC5M0tELgwfcMbphqz1ryYfSYbHeNSRr3lVRjcnlQL6Psm+Cw== X-Received: by 2002:a5d:4aca:0:b0:1f0:7a8c:3505 with SMTP id y10-20020a5d4aca000000b001f07a8c3505mr12706195wrs.337.1646750370820; Tue, 08 Mar 2022 06:39:30 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s8-20020adfc548000000b001f07639a704sm11326971wrf.7.2022.03.08.06.39.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:30 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:18 +0000 Subject: [PATCH v3 11/12] bundle: unbundle promisor packs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee In order to have a valid pack-file after unbundling a bundle that has the 'filter' capability, we need to generate a .promisor file. The bundle does not promise _where_ the objects can be found, but we can expect that these bundles will be unbundled in repositories with appropriate promisor remotes that can find those missing objects. Use the 'git index-pack --promisor=' option to create this .promisor file. Add "from-bundle" as the message to help anyone diagnose issues with these promisor packs. Signed-off-by: Derrick Stolee --- bundle.c | 4 ++++ t/t6020-bundle-misc.sh | 2 ++ 2 files changed, 6 insertions(+) diff --git a/bundle.c b/bundle.c index 393216c6246..322810dc1d6 100644 --- a/bundle.c +++ b/bundle.c @@ -622,6 +622,10 @@ int unbundle(struct repository *r, struct bundle_header *header, struct child_process ip = CHILD_PROCESS_INIT; strvec_pushl(&ip.args, "index-pack", "--fix-thin", "--stdin", NULL); + /* If there is a filter, then we need to create the promisor pack. */ + if (header->filter.choice != LOFC_DISABLED) + strvec_push(&ip.args, "--promisor=from-bundle"); + if (extra_index_pack_args) { strvec_pushv(&ip.args, extra_index_pack_args->v); strvec_clear(extra_index_pack_args); diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh index f10cf011519..42e8cf2eb29 100755 --- a/t/t6020-bundle-misc.sh +++ b/t/t6020-bundle-misc.sh @@ -521,6 +521,8 @@ do git init unbundled && git -C unbundled bundle unbundle ../partial.bdl >ref-list.txt && + ls unbundled/.git/objects/pack/pack-*.promisor >promisor && + test_line_count = 1 promisor && # Count the same number of reachable objects. reflist=$(git for-each-ref --format="%(objectname)") && From patchwork Tue Mar 8 14:39:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12773923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8DFAC433F5 for ; Tue, 8 Mar 2022 14:39:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347590AbiCHOkx (ORCPT ); Tue, 8 Mar 2022 09:40:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347573AbiCHOkg (ORCPT ); Tue, 8 Mar 2022 09:40:36 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8320439169 for ; Tue, 8 Mar 2022 06:39:33 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id x15so28863315wru.13 for ; Tue, 08 Mar 2022 06:39:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=kfEW0dk/GJsWNlzL6rW1CD5ocGSfZpxQ5GeCl6aCtPw=; b=ctXRrc1trZnbASvvbunink1LUkAA7FV9V/R652DbdfqrOwBZXLlTLT6pNHvjeTJ6Xy QAIz1FdQkvF8uBa60k4DGrNbzZx5xHXJuWd48adrZf0j2PU7W5UbPWX8f9szHqLWklMX YGeCxpO+x74qmph1w5TFx1hZvLBaK6KwmCT/gcy1etkAF2OxHk26s9iMLh1NHY1lwbaS 1gxuuB/7G8ZG1U+/2dKJvZYdTZDWzG0vhxzkL1azIcd2TZ9Jg8IABXysNE6c+GoGnAJ9 m+68/5Vp3x+XA4a2euNL0lKwywHiJTT/9NvZeTsvz2hfR9jOh0+48F8CzF9cYCT/7ljj TglA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=kfEW0dk/GJsWNlzL6rW1CD5ocGSfZpxQ5GeCl6aCtPw=; b=mURqmewzYOH0jBAOkWoBcTRkVuOqLryRisSjH7QTjcTt0zqqKvU04k11qHVbdMfTdV x5BePp53Fq3+/c3PRFK6y2fBwi/wSRKFrPNJdlLHp3HWONZIqYyWbbMBEWv/AZHD3nDR 4/qPBjQJ+OZTIDapHHH/2WOHfj2kXaE2Zo0qAT4E9Ce/CJqTOvExBIglVejX5abxvdgK EcO9q8Cgahd6yYS0yNsChaZuvq4Yk7hql3W1MhWYl5GXFgAZLRmMzC+ihKPvOqugDLR4 Qaats0kswKZuRyP635DK7mnzXUO56NN2myR3fVYyK0u9KESBditpDKsTWQZLy0s6+TeV yV+A== X-Gm-Message-State: AOAM530mR50UfufZUEHwbgqEgYyTBVbOqwwDgw79EcS5Ikq3Zzv1pr43 S9+9AGOjzNuzKeBWHQGl29RjTtDVDj0= X-Google-Smtp-Source: ABdhPJzMY6eP1um0RbTudrnozcrIXx2+A+2q7/JHq5gSJSrWG6vzKYY+W0ziyLlQBCUFF2yvhtTwuA== X-Received: by 2002:a05:6000:10c2:b0:1f1:e43d:c979 with SMTP id b2-20020a05600010c200b001f1e43dc979mr12049353wrx.671.1646750371710; Tue, 08 Mar 2022 06:39:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k18-20020adfe8d2000000b0020294da2b42sm3689688wrn.117.2022.03.08.06.39.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 06:39:31 -0800 (PST) Message-Id: <805e1d1172210c6a39b33edcb2cd6d21b754f821.1646750359.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 14:39:19 +0000 Subject: [PATCH v3 12/12] clone: fail gracefully when cloning filtered bundle Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, avarab@gmail.com, gitster@pobox.com, zhiyou.jx@alibaba-inc.com, jonathantanmy@google.com, Jeff Hostetler , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Users can create a new repository using 'git clone '. The new "@filter" capability for bundles means that we can generate a bundle that does not contain all reachable objects, even if the header has no negative commit OIDs. It is feasible to think that we could make a filtered bundle work with the command git clone --filter=$filter --bare or possibly replacing --bare with --no-checkout. However, this requires having some repository-global config that specifies the specified object filter and notifies Git about the existence of promisor pack-files. Without a remote, that is currently impossible. As a stop-gap, parse the bundle header during 'git clone' and die() with a helpful error message instead of the current behavior of failing due to "missing objects". Most of the existing logic for handling bundle clones actually happens in fetch-pack.c, but that logic is the same as if the user specified 'git fetch ', so we want to avoid failing to fetch a filtered bundle when in an existing repository that has the proper config set up for at least one remote. Carefully comment around the test that this is not the desired long-term behavior of 'git clone' in this case, but instead that we need to do more work before that is possible. Signed-off-by: Derrick Stolee --- builtin/clone.c | 13 +++++++++++++ t/t6020-bundle-misc.sh | 12 ++++++++++++ 2 files changed, 25 insertions(+) diff --git a/builtin/clone.c b/builtin/clone.c index 9c29093b352..623a5040b1c 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -33,6 +33,7 @@ #include "packfile.h" #include "list-objects-filter-options.h" #include "hook.h" +#include "bundle.h" /* * Overall FIXMEs: @@ -1138,6 +1139,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix) warning(_("--local is ignored")); transport->cloning = 1; + if (is_bundle) { + struct bundle_header header = { 0 }; + int fd = read_bundle_header(path, &header); + int has_filter = header.filter.choice != LOFC_DISABLED; + + if (fd > 0) + close(fd); + bundle_header_release(&header); + if (has_filter) + die(_("cannot clone from filtered bundle")); + } + transport_set_option(transport, TRANS_OPT_KEEP, "yes"); if (reject_shallow) diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh index 42e8cf2eb29..5160cb0a75c 100755 --- a/t/t6020-bundle-misc.sh +++ b/t/t6020-bundle-misc.sh @@ -537,4 +537,16 @@ do ' done +# NEEDSWORK: 'git clone --bare' should be able to clone from a filtered +# bundle, but that requires a change to promisor/filter config options. +# For now, we fail gracefully with a helpful error. This behavior can be +# changed in the future to succeed as much as possible. +test_expect_success 'cloning from filtered bundle has useful error' ' + git bundle create partial.bdl \ + --all \ + --filter=blob:none && + test_must_fail git clone --bare partial.bdl partial 2>err && + grep "cannot clone from filtered bundle" err +' + test_done