From patchwork Wed Jul 5 06:08:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8E2DEB64DD for ; Wed, 5 Jul 2023 06:08:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231346AbjGEGIl (ORCPT ); Wed, 5 Jul 2023 02:08:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbjGEGIh (ORCPT ); Wed, 5 Jul 2023 02:08:37 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2F6D10C3 for ; Tue, 4 Jul 2023 23:08:35 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-3fbca8935bfso50124345e9.3 for ; Tue, 04 Jul 2023 23:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537313; x=1691129313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i1pPBDeU9PFJ+cpOuqfIS73mPIlEz/tCTAD1Q12GMpw=; b=GCjOBCrNRPy6lTOIOXHvuU4oymV2T3GWFLqIjHrrWWpiv4HBMaGOXLXLh8pf/Gtk/C nmivvi9IDAtiaYhRGcxpy5+uYIcshQJd5cdOJ4rl8SaP7CDhX1Mu5gzyWqZYMT6J87sc gdxir64/CV8vHziUzf+rborLrh+9mYPR3KIuFEHt7AoG6jerJ+tLiSEYIUkFkSfVFYhL tLj3+sV76nIAvSelruiNzjVzn+xLCSW8quyAFsYVoxTa+hNz/B7GDfSIBwskw+waj60L HG5nOwe+jTdOz5MmDrZR+vZEK/Fx3JNcjeNkDDppmimD73KX5uvQFWO1d1MYMgM5Vsg/ Zlpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537313; x=1691129313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i1pPBDeU9PFJ+cpOuqfIS73mPIlEz/tCTAD1Q12GMpw=; b=U91L1npzHIuULiZ2PQoDOXQUR7fX2pQFSEINva/BKpZ3JgbdpvDonpw4+x0qeaR7Sm jbOBRsInOmQ0kSQM1WNioPX6lFjgV6GPq0+yC5Z55Ix+MeNF5XIhnSWvKjIvdoH8Iv37 y2khmaiKPOT66Q/HCfb7PG56/M3KA3OdPRUlmCWpXpAnMmqIpvQfhwlxMzbWD46ChY0u yrfrHnQ82adQ2MCRzIy0XWdEZOGZ+qoTNlAtPZfN3jzlwQkyRXOyF+LGx0ukpDSDBOpj JIsB41Upt9Exvtjk0pknteok5wOvu/7JSuGP/FrCYTW1ktUxMvo/dpgWtfvrK+wX68xe EIng== X-Gm-Message-State: AC+VfDw7wgNOrxxXm49+waPAH3BlPLX4NqbJfx2MPMBnVFUqwweCPZt5 m/fQpn79uHU0RJWc9o4UiZn/jP+OWg8= X-Google-Smtp-Source: ACHHUZ4mGm21H4tyuhsczuLB9pDMogiPIWvicKv2ygkPXfZOTbTLv1B25T/YLsT4URK+GtcO9tFxmg== X-Received: by 2002:a05:600c:2253:b0:3fb:a5b3:4f02 with SMTP id a19-20020a05600c225300b003fba5b34f02mr12104909wmm.36.1688537313055; Tue, 04 Jul 2023 23:08:33 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:32 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 1/8] pack-objects: allow `--filter` without `--stdout` Date: Wed, 5 Jul 2023 08:08:05 +0200 Message-ID: <20230705060812.2865188-2-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org 9535ce7337 (pack-objects: add list-objects filtering, 2017-11-21) taught `git pack-objects` to use `--filter`, but required the use of `--stdout` since a partial clone mechanism was not yet in place to handle missing objects. Since then, changes like 9e27beaa23 (promisor-remote: implement promisor_remote_get_direct(), 2019-06-25) and others added support to dynamically fetch objects that were missing. Even without a promisor remote, filtering out objects can also be useful if we can put the filtered out objects in a separate pack, and in this case it also makes sense for pack-objects to write the packfile directly to an actual file rather than on stdout. Remove the `--stdout` requirement when using `--filter`, so that in a follow-up commit, repack can pass `--filter` to pack-objects to omit certain objects from the resulting packfile. Signed-off-by: John Cai Signed-off-by: Christian Couder --- Documentation/git-pack-objects.txt | 4 ++-- builtin/pack-objects.c | 8 ++------ t/t5317-pack-objects-filter-objects.sh | 8 ++++++++ 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt index a9995a932c..583270a85f 100644 --- a/Documentation/git-pack-objects.txt +++ b/Documentation/git-pack-objects.txt @@ -298,8 +298,8 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle. nevertheless. --filter=:: - Requires `--stdout`. Omits certain objects (usually blobs) from - the resulting packfile. See linkgit:git-rev-list[1] for valid + Omits certain objects (usually blobs) from the resulting + packfile. See linkgit:git-rev-list[1] for valid `` forms. --no-filter:: diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 3c4db66478..614721684a 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -4388,12 +4388,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) if (!rev_list_all || !rev_list_reflog || !rev_list_index) unpack_unreachable_expiration = 0; - if (filter_options.choice) { - if (!pack_to_stdout) - die(_("cannot use --filter without --stdout")); - if (stdin_packs) - die(_("cannot use --filter with --stdin-packs")); - } + if (stdin_packs && filter_options.choice) + die(_("cannot use --filter with --stdin-packs")); if (stdin_packs && use_internal_rev_list) die(_("cannot use internal rev list with --stdin-packs")); diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index b26d476c64..2ff3eef9a3 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -53,6 +53,14 @@ test_expect_success 'verify blob:none packfile has no blobs' ' ! grep blob verify_result ' +test_expect_success 'verify blob:none packfile without --stdout' ' + git -C r1 pack-objects --revs --filter=blob:none mypackname >packhash <<-EOF && + HEAD + EOF + git -C r1 verify-pack -v "mypackname-$(cat packhash).pack" >verify_result && + ! grep blob verify_result +' + test_expect_success 'verify normal and blob:none packfiles have same commits/trees' ' git -C r1 verify-pack -v ../all.pack >verify_result && grep -E "commit|tree" verify_result | From patchwork Wed Jul 5 06:08:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94A73C001B3 for ; Wed, 5 Jul 2023 06:08:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231463AbjGEGIs (ORCPT ); Wed, 5 Jul 2023 02:08:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229892AbjGEGIh (ORCPT ); Wed, 5 Jul 2023 02:08:37 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D30510C8 for ; Tue, 4 Jul 2023 23:08:36 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-3fbd33a573cso36259065e9.2 for ; Tue, 04 Jul 2023 23:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537314; x=1691129314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OvPhXB4LCLkFU3+zBc7i3nD4HUnMOaMi20t4DB/wEsU=; b=mRdVwaV3foaDvANhczbRNARFbHI1/XGMgzq5daOgYsp3p3tc7jWB0Yu3hwzFNlIVPz PLnoYq7g3psY1crfMJGR3FliC0SjiYY0fV1WNBsvM66+Tj4r44tbwea8LsiIGkdBkX5a PDFMQn9SyZt7twsdQ0gopHa19Ofg1Q2BqLoxG3hkHTCgWD623v+CKaWTdn3hOogR02MV phdHNRk01S+hZ8idPxuArEzlA5Lxkd8dz0moIGkZJOMz3+1LAsXFQHTRWtGHqF/tn+gS zCgMOsy6OdPNGY3STyUAVFK11FOub+8qSZrfv6Z8j9d83Ye6huILq0Tsp/hdSTxk3+bn Ercg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537314; x=1691129314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OvPhXB4LCLkFU3+zBc7i3nD4HUnMOaMi20t4DB/wEsU=; b=K5bKiO1r8yfI/bgZ1xxj+0MlFrsuYZyGy6ZroqzX6cqgmvw30GrdmESmo12MiLC41V lAKfE5WCOvdU/Ira8QO3Dhd4/bDke5m/AGbpWT/AruHAWyntK9uRauRUUJQF2NrpsJXF 4nIxOSGG+bzvmimkXmIfXeaeEEljeNsxC8jVuk3ckrvP2/JlYyCrIMDRma9bCSYgdulE oM/McwBVAGBZrLhKu9OeElhLRwN6LMkzzIX34zDjxfzqWbl5EV1YuflKO9Omod06BpC5 w6BXyOYMcgnCYR2oqBStKjirTB9Yq94GnVRHBharlKK/JQgr08/+ErWF1HyMbpjpoa0v eQOw== X-Gm-Message-State: AC+VfDxgRlf3R6h0wnGNiHvwQNqn4kiuY3ThKC8e7lag0gZGylDD6tZU pv6qb5cuvNxmCjlTVzTumJe695ES92k= X-Google-Smtp-Source: ACHHUZ6oVzz6I/0h7/ne1uzAniqWPJbNhpRFRTAsR95wWMF2Ny9xOspSrZA3cC+CwWxH1SPfJvCBBQ== X-Received: by 2002:a7b:cd97:0:b0:3fb:ac9c:e6f with SMTP id y23-20020a7bcd97000000b003fbac9c0e6fmr12261014wmj.38.1688537313884; Tue, 04 Jul 2023 23:08:33 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:33 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 2/8] t/helper: add 'find-pack' test-tool Date: Wed, 5 Jul 2023 08:08:06 +0200 Message-ID: <20230705060812.2865188-3-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a following commit, we will make it possible to separate objects in different packfiles depending on a filter. To make sure that the right objects are in the right packs, let's add a new test-tool that can display which packfile(s) a given object is in. Signed-off-by: Christian Couder --- Makefile | 1 + t/helper/test-find-pack.c | 35 +++++++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 38 insertions(+) create mode 100644 t/helper/test-find-pack.c diff --git a/Makefile b/Makefile index fb541dedc9..14ee0c45d4 100644 --- a/Makefile +++ b/Makefile @@ -800,6 +800,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-env-helper.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-find-pack.o TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o diff --git a/t/helper/test-find-pack.c b/t/helper/test-find-pack.c new file mode 100644 index 0000000000..1928fe7329 --- /dev/null +++ b/t/helper/test-find-pack.c @@ -0,0 +1,35 @@ +#include "test-tool.h" +#include "object-name.h" +#include "object-store.h" +#include "packfile.h" +#include "setup.h" + +/* + * Display the path(s), one per line, of the packfile(s) containing + * the given object. + */ + +static const char *find_pack_usage = "\n" +" test-tool find-pack "; + + +int cmd__find_pack(int argc, const char **argv) +{ + struct object_id oid; + struct packed_git *p; + + setup_git_directory(); + + if (argc != 2) + usage(find_pack_usage); + + if (repo_get_oid(the_repository, argv[1], &oid)) + die("cannot parse %s as an object name", argv[1]); + + for (p = get_all_packs(the_repository); p; p = p->next) { + if (find_pack_entry_one(oid.hash, p)) + printf("%s\n", p->pack_name); + } + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index abe8a785eb..41da40c296 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -31,6 +31,7 @@ static struct test_cmd cmds[] = { { "env-helper", cmd__env_helper }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "find-pack", cmd__find_pack }, { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index ea2672436c..411dbf2db4 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -25,6 +25,7 @@ int cmd__dump_reftable(int argc, const char **argv); int cmd__env_helper(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); int cmd__fast_rebase(int argc, const char **argv); +int cmd__find_pack(int argc, const char **argv); int cmd__fsmonitor_client(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); int cmd__genzeros(int argc, const char **argv); From patchwork Wed Jul 5 06:08:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33AE1EB64DD for ; Wed, 5 Jul 2023 06:08:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230434AbjGEGIu (ORCPT ); Wed, 5 Jul 2023 02:08:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230017AbjGEGIj (ORCPT ); Wed, 5 Jul 2023 02:08:39 -0400 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D7CA10E3 for ; Tue, 4 Jul 2023 23:08:38 -0700 (PDT) Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-4fb96e2b573so10213667e87.3 for ; Tue, 04 Jul 2023 23:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537316; x=1691129316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=icwadf3OpgZ5ELW9RPnIlITNXr/NDVJfKlSvypx5lV4=; b=Ee0QJ4NhnRiCmevJM49plP1yXAXRjr5LIV2Y4NjX0ew54F9osHaLM3GGunQoVlPByB KFON1f/uxeeTX0VcELm0VH6fGasaBI2Aj64WTYNziPQjX2SRtSvE2X09Azvvmus8O0G2 xhLqLM3pJtMR3vFbycS4MRwzTtItjcqzRkszKQp+d7xq411WwoL2lb0oWmvQ3DmlD+r5 iMHXHgH3gISPZtERnUAYa8WYp0RpvT8yErTdTTLT97cCVZIBFBNP4dl7aOpZBydDGix3 e1iYjjrSEK2aPi2xC9BFMFH/pMbiFhyUmOM0yP8Ft/mXofUE82mpI+sOf9Kl61onf1tU CI1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537316; x=1691129316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=icwadf3OpgZ5ELW9RPnIlITNXr/NDVJfKlSvypx5lV4=; b=gxnBfxrEgpyYNL0/J7xI1v1U8cs3NwBGrLF46YjZdO4Xo+RMFH107CYkuQDvNfcvHK vRwbAWrxoACbw1nfkddBehso2psviqQRDkCn0VVk16fnxbWBENHG1Otw+48ej1WSMsFv Niiz5MiDWZB5/wWvd22lEWmpqnz7nBOiZv2EUVZRNzgX48mqiHI6+KdERS+74R85Iq7Q B9GhCtZzym+aZuKFFeaJ40BrWq/E0GDMk+zEYRo2KIEj8KEnyYcZQ6KvXVg/5zBGqaEP Eh+5kwyJRkthd+abughcpxI3+WzxQrLRcXIx5qE5acjuSqwZU+ax2UW6K1DNSC9o1UZL +vQg== X-Gm-Message-State: ABy/qLZAP4SagWDz7ZbXFx8ElR8lO6kgQ9I0Kx91LjMCQnewoSq3lXwe iM/kYIQ3EWRPGGwckSlu/u2yYvcTWC0= X-Google-Smtp-Source: APBJJlEqDcs6CPR0whdt+t+xhNwMlM6KDD+MDZP5yYDU3GD0sUIekeJ0nhguw3WiKZ2486im3kSuUg== X-Received: by 2002:a05:6512:2510:b0:4f9:a542:91c with SMTP id be16-20020a056512251000b004f9a542091cmr12851154lfb.3.1688537315498; Tue, 04 Jul 2023 23:08:35 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:34 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder Subject: [PATCH v2 3/8] repack: refactor finishing pack-objects command Date: Wed, 5 Jul 2023 08:08:07 +0200 Message-ID: <20230705060812.2865188-4-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Create a new finish_pack_objects_cmd() to refactor duplicated code that handles reading the packfile names from the output of a `git pack-objects` command and putting it into a string_list, as well as calling finish_command(). While at it, beautify a code comment a bit in the new function. Signed-off-by: Christian Couder out, "r"); + while (strbuf_getline_lf(&line, out) != EOF) { + struct string_list_item *item; + + if (line.len != the_hash_algo->hexsz) + die(_("repack: Expecting full hex object ID lines only " + "from pack-objects.")); + /* + * Avoid putting packs written outside of the repository in the + * list of names. + */ + if (local) { + item = string_list_append(names, line.buf); + item->util = populate_pack_exts(line.buf); + } + } + fclose(out); + + strbuf_release(&line); + + return finish_command(cmd); +} + static int write_cruft_pack(const struct pack_objects_args *args, const char *destination, const char *pack_prefix, @@ -713,9 +743,8 @@ static int write_cruft_pack(const struct pack_objects_args *args, struct string_list *existing_kept_packs) { struct child_process cmd = CHILD_PROCESS_INIT; - struct strbuf line = STRBUF_INIT; struct string_list_item *item; - FILE *in, *out; + FILE *in; int ret; const char *scratch; int local = skip_prefix(destination, packdir, &scratch); @@ -759,27 +788,7 @@ static int write_cruft_pack(const struct pack_objects_args *args, fprintf(in, "%s.pack\n", item->string); fclose(in); - out = xfdopen(cmd.out, "r"); - while (strbuf_getline_lf(&line, out) != EOF) { - struct string_list_item *item; - - if (line.len != the_hash_algo->hexsz) - die(_("repack: Expecting full hex object ID lines only " - "from pack-objects.")); - /* - * avoid putting packs written outside of the repository in the - * list of names - */ - if (local) { - item = string_list_append(names, line.buf); - item->util = populate_pack_exts(line.buf); - } - } - fclose(out); - - strbuf_release(&line); - - return finish_command(&cmd); + return finish_pack_objects_cmd(&cmd, names, local); } int cmd_repack(int argc, const char **argv, const char *prefix) @@ -790,10 +799,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) struct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP; struct string_list existing_kept_packs = STRING_LIST_INIT_DUP; struct pack_geometry *geometry = NULL; - struct strbuf line = STRBUF_INIT; struct tempfile *refs_snapshot = NULL; int i, ext, ret; - FILE *out; int show_progress; /* variables to be filled by option parsing */ @@ -1024,18 +1031,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) fclose(in); } - out = xfdopen(cmd.out, "r"); - while (strbuf_getline_lf(&line, out) != EOF) { - struct string_list_item *item; - - if (line.len != the_hash_algo->hexsz) - die(_("repack: Expecting full hex object ID lines only from pack-objects.")); - item = string_list_append(&names, line.buf); - item->util = populate_pack_exts(item->string); - } - strbuf_release(&line); - fclose(out); - ret = finish_command(&cmd); + ret = finish_pack_objects_cmd(&cmd, &names, 1); if (ret) goto cleanup; From patchwork Wed Jul 5 06:08:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52E97C001B3 for ; Wed, 5 Jul 2023 06:08:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231466AbjGEGIw (ORCPT ); Wed, 5 Jul 2023 02:08:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231154AbjGEGIj (ORCPT ); Wed, 5 Jul 2023 02:08:39 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4589910F2 for ; Tue, 4 Jul 2023 23:08:38 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-3fbc244d39dso84946935e9.3 for ; Tue, 04 Jul 2023 23:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537316; x=1691129316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lAi+xvrjzE7keEMAZobkXBGgLdIFZ0gxTr8MhGGeh+Y=; b=QEpWTApvs0OCr5SILydGcOANBRzPNrGYIBkjzznBhrf0xU9qocOX/QxSvNaOxSa0hr hC45ut4FDUN4gu1XvndvegrMDTtvvK1RziVsByRc20dL27zU4K8A3jDcdrYXd6ipYQUF FbiyJ7y0mFfMg+g5C8zwnfVp5If88MxVcW/di+vvtzhjo2TXs5oRDkc0MWQG62Urle/W IPCOoLvXRWi973uo/Q8BQeK7xLgnpukB+QYfsGRihtZFsmQm/+ChVAd7328Dw5xRmWmD es4DfmTa8pKOgZZnGNV70zKOFczCEmKSeIm0iCvH2TI1O8yfq74vWNBUHl0asDvcGTa2 seDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537316; x=1691129316; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lAi+xvrjzE7keEMAZobkXBGgLdIFZ0gxTr8MhGGeh+Y=; b=XB4n3qN9cgCFq/1K9+QP2GPAZet4VjsJ1/vFHVDkymU3pL4vMTp9uIV8EISDL7OS0b SEL4YZkVYU5VlCsh6mrLMTIS8V76n8xy+ODsLICUofXqQFCZUi+zMCYq0ycW4cbttFSK aAh2VlYTigQ4rqc9zpJbjAc+VciVci/NKS2LykSPV33TETnvA78h7LFgT/CyPGRNOzjw Q6LvqFrjdD85XZVFy5n7f5KRcHIPDbEe74ZRtmzPP0gbmfRrZ3h/RvHZdRkWgxDALMQN p8vI+mwQLvCjiOL5NymN9+XK8LxWV1a2vtrcgkUCSxfNB4Rs0HMnlHX2oabS7Ewc/aB7 SLzQ== X-Gm-Message-State: AC+VfDwm+iJms4pwqyIk2Oxt4V6qLZ+42DqkfoJGlkB9xY5xNseLelIN 8JQuhL3SYjZ2SP5uaMHwiAsl7wfjFUo= X-Google-Smtp-Source: ACHHUZ5iaFsSRAJk3rRUcivmr3CRxwAp+lfje99ZsGHDfuXkMqecdX1i8/0sLG3hNE+xTlipz47W8Q== X-Received: by 2002:a05:600c:215:b0:3fb:b9b7:fb0c with SMTP id 21-20020a05600c021500b003fbb9b7fb0cmr19856272wmi.31.1688537316231; Tue, 04 Jul 2023 23:08:36 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:35 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder Subject: [PATCH v2 4/8] repack: refactor finding pack prefix Date: Wed, 5 Jul 2023 08:08:08 +0200 Message-ID: <20230705060812.2865188-5-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Create a new find_pack_prefix() to refactor code that handles finding the pack prefix from the packtmp and packdir global variables, as we are going to need this feature again in following commit. Signed-off-by: Christian Couder X-Patchwork-Id: 13301686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E44FEB64DD for ; Wed, 5 Jul 2023 06:08:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229610AbjGEGI4 (ORCPT ); Wed, 5 Jul 2023 02:08:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbjGEGIl (ORCPT ); Wed, 5 Jul 2023 02:08:41 -0400 Received: from mail-lf1-x135.google.com (mail-lf1-x135.google.com [IPv6:2a00:1450:4864:20::135]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A8BD10C3 for ; Tue, 4 Jul 2023 23:08:40 -0700 (PDT) Received: by mail-lf1-x135.google.com with SMTP id 2adb3069b0e04-4fbaef9871cso5154590e87.0 for ; Tue, 04 Jul 2023 23:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537318; x=1691129318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rxke6fCOdCy2XjLu7dBwvEluN1rgbayU02IXErc9ffM=; b=r0edIGKb7SeeRAyVKQdYFd4D4axUNigoPtLCVXKV6g+kEQHxFryehpbNYexVb5pn3O Qwc8YIYoUR1lEso5Bn2Z6LLa/5BLUBrH5ER4aZkp4zk9x/9IjRAlGItUXDk2WsV9mAhU LM3i+KpKsARPQBoZKBUkE2dUJ+Sp3MRfM/e9Z9ojEESNHqSXuw1SpFZStFPDvcRv3DPN 0K8zneHa7vna8OpW9uXg9ER3umdnidmAVAUz4CF0VZXFsF2EEzl6PxynZmeEm9WNByQV qYbwznKVkA4jU83CihEzmBYG6ZIra7OzNkw/vz2Anl10DtAeQPqDVC8neX5Jdyl4Znpl VjKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537318; x=1691129318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rxke6fCOdCy2XjLu7dBwvEluN1rgbayU02IXErc9ffM=; b=SyhnuycwfuTmLpi3t3mvw5SWCgLmDTZxngUZhupHtDFZhseRMSXDw/eSeJ48w3bD35 1GtGsrZq2vIsgbxsv1n+ix9sRnVQcEWs/8sukmU2wThJqDKd0u2RfrTqLfZgxBhGCukj Z4Z/XYOBmHH94b48eowkDiKFTSsYRYEXU05KaOw3nt6hoGDCcrgiQgGJ6rSWl7uFHfQk //R/aF2h1p9ADH+CRA15we38bLIx9lOMhrhKb/Wg4W5HVeIdGaXxssQ+WYaQU61+Pe3y hzafqAsc44vx/5Fvduv06Ckwzmgem8EvJN050zUg+DJReHvFlh3kegxBo4JMWik6VJ/w BdMw== X-Gm-Message-State: ABy/qLYkrrQgLz0KWT+I80Rhod/jyP7CaMhtjg+S+jZLfEOtDU6J6ZJ1 l9EMtysKdBQaE6L4B0XePiqyMaxMZ6s= X-Google-Smtp-Source: APBJJlErVFAgfa63Y4QwI4gCEux0NBjpV9tFomfAqyrvttdcmkLp2nfCDpI1bSGjYOmaBURzri749A== X-Received: by 2002:a05:6512:3490:b0:4fb:9da2:6cec with SMTP id v16-20020a056512349000b004fb9da26cecmr9872734lfr.7.1688537317467; Tue, 04 Jul 2023 23:08:37 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:36 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 5/8] repack: add `--filter=` option Date: Wed, 5 Jul 2023 08:08:09 +0200 Message-ID: <20230705060812.2865188-6-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This new option puts the objects specified by `` into a separate packfile. This could be useful if, for example, some large blobs take a lot of precious space on fast storage while they are rarely accessed. It could make sense to move them into a separate cheaper, though slower, storage. In other use cases it might make sense to put all the blobs into separate storage. This is done by running two `git pack-objects` commands. The first one is run with `--filter=`, using the specified filter. It packs objects while omitting the objects specified by the filter. Then another `git pack-objects` command is launched using `--stdin-packs`. We pass it all the previously existing packs into its stdin, so that it will pack all the objects in the previously existing packs. But we also pass into its stdin, the pack created by the previous `git pack-objects --filter=` command as well as the kept packs, all prefixed with '^', so that the objects in these packs will be omitted from the resulting pack. The result is that only the objects filtered out by the first `git pack-objects` command are in the pack resulting from the second `git pack-objects` command. Signed-off-by: John Cai Signed-off-by: Christian Couder --- Documentation/git-repack.txt | 9 +++++ builtin/repack.c | 67 ++++++++++++++++++++++++++++++++++++ t/t7700-repack.sh | 16 +++++++++ 3 files changed, 92 insertions(+) diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt index 4017157949..d702553033 100644 --- a/Documentation/git-repack.txt +++ b/Documentation/git-repack.txt @@ -143,6 +143,15 @@ depth is 4095. a larger and slower repository; see the discussion in `pack.packSizeLimit`. +--filter=:: + Remove objects matching the filter specification from the + resulting packfile and put them into a separate packfile. Note + that objects used in the working directory are not filtered + out. So for the split to fully work, it's best to perform it + in a bare repo and to use the `-a` and `-d` options along with + this option. See linkgit:git-rev-list[1] for valid + `` forms. + -b:: --write-bitmap-index:: Write a reachability bitmap index as part of the repack. This diff --git a/builtin/repack.c b/builtin/repack.c index 4e5afee8d8..e2661b956c 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -54,6 +54,7 @@ struct pack_objects_args { const char *depth; const char *threads; const char *max_pack_size; + const char *filter; int no_reuse_delta; int no_reuse_object; int quiet; @@ -174,6 +175,8 @@ static void prepare_pack_objects(struct child_process *cmd, strvec_pushf(&cmd->args, "--threads=%s", args->threads); if (args->max_pack_size) strvec_pushf(&cmd->args, "--max-pack-size=%s", args->max_pack_size); + if (args->filter) + strvec_pushf(&cmd->args, "--filter=%s", args->filter); if (args->no_reuse_delta) strvec_pushf(&cmd->args, "--no-reuse-delta"); if (args->no_reuse_object) @@ -734,6 +737,57 @@ static int finish_pack_objects_cmd(struct child_process *cmd, return finish_command(cmd); } +static int write_filtered_pack(const struct pack_objects_args *args, + const char *destination, + const char *pack_prefix, + struct string_list *names, + struct string_list *existing_packs, + struct string_list *existing_kept_packs) +{ + struct child_process cmd = CHILD_PROCESS_INIT; + struct string_list_item *item; + FILE *in; + int ret; + const char *scratch; + int local = skip_prefix(destination, packdir, &scratch); + + /* We need to copy 'args' to modify it */ + struct pack_objects_args new_args = *args; + + /* No need to filter again */ + new_args.filter = NULL; + + prepare_pack_objects(&cmd, &new_args, destination); + + strvec_push(&cmd.args, "--stdin-packs"); + + cmd.in = -1; + + ret = start_command(&cmd); + if (ret) + return ret; + + /* + * names has a confusing double use: it both provides the list + * of just-written new packs, and accepts the name of the + * filtered pack we are writing. + * + * By the time it is read here, it contains only the pack(s) + * that were just written, which is exactly the set of packs we + * want to consider kept. + */ + in = xfdopen(cmd.in, "w"); + for_each_string_list_item(item, names) + fprintf(in, "^%s-%s.pack\n", pack_prefix, item->string); + for_each_string_list_item(item, existing_packs) + fprintf(in, "%s.pack\n", item->string); + for_each_string_list_item(item, existing_kept_packs) + fprintf(in, "^%s.pack\n", item->string); + fclose(in); + + return finish_pack_objects_cmd(&cmd, names, local); +} + static int write_cruft_pack(const struct pack_objects_args *args, const char *destination, const char *pack_prefix, @@ -866,6 +920,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) N_("limits the maximum number of threads")), OPT_STRING(0, "max-pack-size", &po_args.max_pack_size, N_("bytes"), N_("maximum size of each packfile")), + OPT_STRING(0, "filter", &po_args.filter, N_("args"), + N_("object filtering")), OPT_BOOL(0, "pack-kept-objects", &pack_kept_objects, N_("repack objects in packs marked with .keep")), OPT_STRING_LIST(0, "keep-pack", &keep_pack_list, N_("name"), @@ -1105,6 +1161,17 @@ int cmd_repack(int argc, const char **argv, const char *prefix) } } + if (po_args.filter) { + ret = write_filtered_pack(&po_args, + packtmp, + find_pack_prefix(), + &names, + &existing_nonkept_packs, + &existing_kept_packs); + if (ret) + goto cleanup; + } + string_list_sort(&names); close_object_store(the_repository->objects); diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index af79266c58..66589e4217 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -293,6 +293,22 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' test_must_be_empty actual ' +test_expect_success 'repacking with a filter works' ' + git -C bare.git repack -a -d && + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack && + git -C bare.git -c repack.writebitmaps=false repack -a -d --filter=blob:none && + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack && + commit_pack=$(test-tool -C bare.git find-pack HEAD) && + test -n "$commit_pack" && + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) && + test -n "$blob_pack" && + test "$commit_pack" != "$blob_pack" && + tree_pack=$(test-tool -C bare.git find-pack HEAD^{tree}) && + test "$tree_pack" = "$commit_pack" && + blob_pack2=$(test-tool -C bare.git find-pack HEAD:file2) && + test "$blob_pack2" = "$blob_pack" +' + objdir=.git/objects midx=$objdir/pack/multi-pack-index From patchwork Wed Jul 5 06:08:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC488EB64DD for ; Wed, 5 Jul 2023 06:09:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231481AbjGEGI6 (ORCPT ); Wed, 5 Jul 2023 02:08:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231408AbjGEGIm (ORCPT ); Wed, 5 Jul 2023 02:08:42 -0400 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B0D3F3 for ; Tue, 4 Jul 2023 23:08:40 -0700 (PDT) Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-4f122ff663eso9852549e87.2 for ; Tue, 04 Jul 2023 23:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537318; x=1691129318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CvWVo8fhmCKUJdwdhCMkduSb3K45TkxWIrngoL3x54s=; b=O3zzKvgf2XmnKhPS4/jSYU9e6rRy/F4kuoJ9B0ibb+Jm9DS57gftAf6rU/zN/ms10V IGsgcpEWz7/6+VH+Rwtgyto6czn5OEEP11Hy+uJcSuqpdmONzAt5s9NKSoVaHPAQyCsk 30k6zJknJpnzyoojzDCbCVule2EgRFte1OLjpQR9DsMS5ZR21SwitoUVjQASr/nDTz9r 5eM1/Nu4fKMZk4dOs+0DgPQ7OALOcBYUIvg2GwiPVw2hpSZZBOXZC33Oado6n+t/fL3b FWUuyUxxbsa+7wc1ZIa52RZW6hCENYj6Mcs32bl8JfGTJ6H85UU24AGFU+ZrdamkbX9e aQnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537318; x=1691129318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CvWVo8fhmCKUJdwdhCMkduSb3K45TkxWIrngoL3x54s=; b=cngLb7IBiabl8PYXYSG54IczU0Wh1GCloehJETpC5tG+LAFQYH21hXKN5VMzvsXygd SaTOvUnB5OaAd/gSfWz/b681inD1FNGIPNT20Hs2PpmxHBzj14faf4qmVuLEA526dFdY Q7gM3p9iE06I5G3CGRtrB0Nj2a7bd1Kl8BBAX82818J90FHMT+zxmAkfFCRDhuEtdQMF d6p4dsokQvJviZE9KASh/VAfAclh1gsa5CHCtcHNJ9KK6JZp0ojNrqkH+2P9hFNDuYP3 HXC5dRmvjIQIDjgezg/VLoVvbv3x4PvZ/FDwEqYXqVJzhiRsNGVDARnKxpT2z2rtSY7n LM9Q== X-Gm-Message-State: ABy/qLa/WAwi6icp4QP6mTp3pkqdaLIbf1TjBebFTfqvmVFIKZ5/mde+ sCwBT0KVaYi2+i5PcWCPS0R8sqqbt/I= X-Google-Smtp-Source: APBJJlH1/N31RSlQqeByBTVPGyxmn5exvuMgnk19VBb4BdjIn2CFkNbPNzLytqoxczyJUx1OMl9EAA== X-Received: by 2002:a05:6512:2250:b0:4fb:81f2:422b with SMTP id i16-20020a056512225000b004fb81f2422bmr11861649lfu.54.1688537318181; Tue, 04 Jul 2023 23:08:38 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:37 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 6/8] gc: add `gc.repackFilter` config option Date: Wed, 5 Jul 2023 08:08:10 +0200 Message-ID: <20230705060812.2865188-7-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A previous commit has implemented `git repack --filter=` to allow users to filter out some objects from the main pack and move them into a new different pack. Users might want to perform such a cleanup regularly at the same time as they perform other repacks and cleanups, so as part of `git gc`. Let's allow them to configure a for that purpose using a new gc.repackFilter config option. Now when `git gc` will perform a repack with a configured through this option and not empty, the repack process will be passed a corresponding `--filter=` argument. Signed-off-by: Christian Couder --- Documentation/config/gc.txt | 5 +++++ builtin/gc.c | 6 ++++++ t/t6500-gc.sh | 12 ++++++++++++ 3 files changed, 23 insertions(+) diff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt index ca47eb2008..2153bde7ac 100644 --- a/Documentation/config/gc.txt +++ b/Documentation/config/gc.txt @@ -145,6 +145,11 @@ Multiple hooks are supported, but all must exit successfully, else the operation (either generating a cruft pack or unpacking unreachable objects) will be halted. +gc.repackFilter:: + When repacking, use the specified filter to move certain + objects into a separate packfile. See the + `--filter=` option of linkgit:git-repack[1]. + gc.rerereResolved:: Records of conflicted merge you resolved earlier are kept for this many days when 'git rerere gc' is run. diff --git a/builtin/gc.c b/builtin/gc.c index 91eec7703a..046147fdcc 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -62,6 +62,7 @@ static timestamp_t gc_log_expire_time; static const char *gc_log_expire = "1.day.ago"; static const char *prune_expire = "2.weeks.ago"; static const char *prune_worktrees_expire = "3.months.ago"; +static char *repack_filter; static unsigned long big_pack_threshold; static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE; @@ -171,6 +172,8 @@ static void gc_config(void) git_config_get_ulong("gc.bigpackthreshold", &big_pack_threshold); git_config_get_ulong("pack.deltacachesize", &max_delta_cache_size); + git_config_get_string("gc.repackfilter", &repack_filter); + git_config(git_default_config, NULL); } @@ -356,6 +359,9 @@ static void add_repack_all_option(struct string_list *keep_pack) if (keep_pack) for_each_string_list(keep_pack, keep_one_pack, NULL); + + if (repack_filter && *repack_filter) + strvec_pushf(&repack, "--filter=%s", repack_filter); } static void add_repack_incremental_option(void) diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 69509d0c11..5b89faf505 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -202,6 +202,18 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip "e grep -E "^trace: (built-in|exec|run_command): git reflog expire --" trace.out ' +test_expect_success 'gc.repackFilter launches repack with a filter' ' + test_when_finished "rm -rf bare.git" && + git clone --no-local --bare . bare.git && + + git -C bare.git -c gc.cruftPacks=false gc && + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack && + + GIT_TRACE=$(pwd)/trace.out git -C bare.git -c gc.repackFilter=blob:none -c repack.writeBitmaps=false -c gc.cruftPacks=false gc && + test_stdout_line_count = 2 ls bare.git/objects/pack/*.pack && + grep -E "^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*" trace.out +' + prepare_cruft_history () { test_commit base && From patchwork Wed Jul 5 06:08:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A008C001B3 for ; Wed, 5 Jul 2023 06:09:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231441AbjGEGJE (ORCPT ); Wed, 5 Jul 2023 02:09:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231422AbjGEGIn (ORCPT ); Wed, 5 Jul 2023 02:08:43 -0400 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B71A10C8 for ; Tue, 4 Jul 2023 23:08:42 -0700 (PDT) Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-4fafe87c6fbso9794769e87.3 for ; Tue, 04 Jul 2023 23:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537320; x=1691129320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rtt/W9X/PYzk7qi7rcYewaNiWJiEssDMzazzYdPc7vg=; b=Mkd1UJ9mgjF0ac1Utj8fH0PFEV3BDmVBG8OQ8X/uST5J/rFx1rkdQe+VLR+zEmQ052 WxvN9YtTXosMMxPmIqh+ZyGwoM1KJzTTSX0x5MP6NIKuZx45WbnjiG//sY3TYhLdayIP +C1Gh8enMAQyW2Sdje5ouxfkG6QPwoJBhpCW92+c7kGxSrnl735lFJ3kUvP4uOKkDTvW bTQtCbpick29mTeo/098soQplOhqpHzXllZZiLYJju/9jvAjkFIxCM01dEKsTVUoVwqX v8mZJsRKouCN0mYTgu6spD4lmb4G8jmCvRdtOxbg9qBhkveHVxXonA/pYhyPn2CjcvjL UMBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537320; x=1691129320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rtt/W9X/PYzk7qi7rcYewaNiWJiEssDMzazzYdPc7vg=; b=BpdMihcWA82O91yBY+RV8qpNa0yjLaMGh8LQueGFMGnoMz04h0M1nHBOT0b4Os0R/9 PHlXFfGw+oax/9ISgFgvHLB6s+yY29LK4jiN60wm6cnSZRI55kH+8wnvGlg+s23BanWw 6uKd7aOCyC1ovcmXNcOc+qEZemPzSY6MbyasGmtsiuvZUkE0VVJJJ9o7Yp2CUY5VVX31 X12PDfUDGcFRmrumnErdyvAXlSZnY1LZ8U8jRlCPTgM/rNc6PTm4AAfTf8q9ujDbTPhQ Kq3NBg3/OtAllG59rYCB09dUbFeQ72M5aqXZZWlzs3HCB2ekGK8TT36/zXmHy2rz2FQa sZjg== X-Gm-Message-State: ABy/qLZTUku478tnNT45eioGRqRa7T1MFtP8a8MOvS7tbqCOIBmU4NQK 46/6FnOaPx7NrtY45kk2e72t+/FGx7Y= X-Google-Smtp-Source: APBJJlGEeQuTgQPtdnX9o1n2i+1PwdkOjcDqKKNgqErBE1J/mU1EwehtER6S1OV8M+Zwq1Sr57MG7Q== X-Received: by 2002:a19:e010:0:b0:4fb:89b3:3374 with SMTP id x16-20020a19e010000000b004fb89b33374mr9091805lfg.54.1688537319548; Tue, 04 Jul 2023 23:08:39 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:38 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 7/8] repack: implement `--filter-to` for storing filtered out objects Date: Wed, 5 Jul 2023 08:08:11 +0200 Message-ID: <20230705060812.2865188-8-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A previous commit has implemented `git repack --filter=` to allow users to filter out some objects from the main pack and move them into a new different pack. It would be nice if this new different pack could be created in a different directory than the regular pack. This would make it possible to move large blobs into a pack on a different kind of storage, for example cheaper storage. Even in a different directory this pack can be accessible if, for example, the Git alternates mechanism is used to point to it. While at it, as an example to show that `--filter` and `--filter-to` work well with other options, let's also add a test to check that these options work well with `--max-pack-size`. Signed-off-by: Christian Couder repack: add test with --max-pack-size --- Documentation/git-repack.txt | 6 ++++ builtin/repack.c | 11 +++++- t/t7700-repack.sh | 66 ++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 1 deletion(-) diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt index d702553033..396a91b9ac 100644 --- a/Documentation/git-repack.txt +++ b/Documentation/git-repack.txt @@ -152,6 +152,12 @@ depth is 4095. this option. See linkgit:git-rev-list[1] for valid `` forms. +--filter-to=:: + Write the pack containing filtered out objects to the + directory ``. This can be used for putting the pack on a + separate object directory that is accessed through the Git + alternates mechanism. Only useful with `--filter`. + -b:: --write-bitmap-index:: Write a reachability bitmap index as part of the repack. This diff --git a/builtin/repack.c b/builtin/repack.c index e2661b956c..5695f9734d 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -879,6 +879,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) int write_midx = 0; const char *cruft_expiration = NULL; const char *expire_to = NULL; + const char *filter_to = NULL; struct option builtin_repack_options[] = { OPT_BIT('a', NULL, &pack_everything, @@ -932,6 +933,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) N_("write a multi-pack index of the resulting packs")), OPT_STRING(0, "expire-to", &expire_to, N_("dir"), N_("pack prefix to store a pack containing pruned objects")), + OPT_STRING(0, "filter-to", &filter_to, N_("dir"), + N_("pack prefix to store a pack containing filtered out objects")), OPT_END() }; @@ -1075,6 +1078,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix) strvec_push(&cmd.args, "--incremental"); } + if (filter_to && !po_args.filter) + die(_("option '%s' can only be used along with '%s'"), "--filter-to", "--filter"); + if (geometry) cmd.in = -1; else @@ -1162,8 +1168,11 @@ int cmd_repack(int argc, const char **argv, const char *prefix) } if (po_args.filter) { + if (!filter_to) + filter_to = packtmp; + ret = write_filtered_pack(&po_args, - packtmp, + filter_to, find_pack_prefix(), &names, &existing_nonkept_packs, diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 66589e4217..a96c1635b2 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -309,6 +309,72 @@ test_expect_success 'repacking with a filter works' ' test "$blob_pack2" = "$blob_pack" ' +test_expect_success '--filter-to stores filtered out objects' ' + git -C bare.git repack -a -d && + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack && + + git init --bare filtered.git && + git -C bare.git -c repack.writebitmaps=false repack -a -d \ + --filter=blob:none \ + --filter-to=../filtered.git/objects/pack/pack && + test_stdout_line_count = 1 ls bare.git/objects/pack/pack-*.pack && + test_stdout_line_count = 1 ls filtered.git/objects/pack/pack-*.pack && + + commit_pack=$(test-tool -C bare.git find-pack HEAD) && + test -n "$commit_pack" && + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) && + test -z "$blob_pack" && + blob_hash=$(git -C bare.git rev-parse HEAD:file1) && + test -n "$blob_hash" && + blob_pack=$(test-tool -C filtered.git find-pack $blob_hash) && + test -n "$blob_pack" && + + echo $(pwd)/filtered.git/objects >bare.git/objects/info/alternates && + blob_pack=$(test-tool -C bare.git find-pack HEAD:file1) && + test -n "$blob_pack" && + blob_content=$(git -C bare.git show $blob_hash) && + test "$blob_content" = "content1" +' + +test_expect_success '--filter works with --max-pack-size' ' + rm -rf filtered.git && + git init --bare filtered.git && + git init max-pack-size && + ( + cd max-pack-size && + test_commit base && + # two blobs which exceed the maximum pack size + test-tool genrandom foo 1048576 >foo && + git hash-object -w foo && + test-tool genrandom bar 1048576 >bar && + git hash-object -w bar && + git add foo bar && + git commit -m "adding foo and bar" + ) && + git clone --no-local --bare max-pack-size max-pack-size.git && + ( + cd max-pack-size.git && + git -c repack.writebitmaps=false repack -a -d --filter=blob:none \ + --max-pack-size=1M \ + --filter-to=../filtered.git/objects/pack/pack && + echo $(cd .. && pwd)/filtered.git/objects >objects/info/alternates && + + # Check that the 3 blobs are in different packfiles in filtered.git + test_stdout_line_count = 3 ls ../filtered.git/objects/pack/pack-*.pack && + test_stdout_line_count = 1 ls objects/pack/pack-*.pack && + foo_pack=$(test-tool find-pack HEAD:foo) && + bar_pack=$(test-tool find-pack HEAD:bar) && + base_pack=$(test-tool find-pack HEAD:base.t) && + test "$foo_pack" != "$bar_pack" && + test "$foo_pack" != "$base_pack" && + test "$bar_pack" != "$base_pack" && + for pack in "$foo_pack" "$bar_pack" "$base_pack" + do + case "$foo_pack" in */filtered.git/objects/pack/*) true ;; *) return 1 ;; esac + done + ) +' + objdir=.git/objects midx=$objdir/pack/multi-pack-index From patchwork Wed Jul 5 06:08:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 13301688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F6F2C0015E for ; Wed, 5 Jul 2023 06:09:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231482AbjGEGJB (ORCPT ); Wed, 5 Jul 2023 02:09:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231421AbjGEGIn (ORCPT ); Wed, 5 Jul 2023 02:08:43 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 349E210E3 for ; Tue, 4 Jul 2023 23:08:42 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-3fbac8b01b3so2330995e9.1 for ; Tue, 04 Jul 2023 23:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688537320; x=1691129320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5eCxC6VmngybrcGM5jlxofbrLSLE9kKwNVVAU4IdBnw=; b=H3Kw2MOIDxMSI9kwfMgfPSgnCCuJPkEXoXUCu879ckXOZY//szzliTGV+lcwEcbUvb XgcMRtcLNrlmgcrIvgX6JsOX4IbLJ3ADjiouLhEjJIXs2R9AikiEPcpBcG9tUm3ZDMY2 BlMes2pkWUYW24wkmTdAHXmdTacFhi/yzn++ed1bE7Vec9kMoYBzdokV78KTF6fSxKxA TF28VPloyuU4RQz8QjK0YfbZ1GBGmHCkk2n4eTET98wyDFqU2JDD5G2qPhJedzIVc75V /HVcFM3vx0s5bgZzY+41rxQbFmu/BusPNRpT2nsEaBI1c4Gz0xngspKV92VDyx564JwG GB/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688537320; x=1691129320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5eCxC6VmngybrcGM5jlxofbrLSLE9kKwNVVAU4IdBnw=; b=NaUEXAd144uP1XdJBMnqpK5SRcSCwwEmF1JSyiSBU0L44gN5iXDY56zrkXi+APXt/1 p1KwpZcO4EApVGLdTjF96H31/vb3vw/V2E/XJlcM/swGQlnb6s5rELxsUf2/2IErrGXa 9Lv8UtAYFK7/l3Z9pLj4DK19thR/C9yTkuPTp1GvpAlhSlK5bRr23tMdBuQ6kTogbAki aWg7f9mxeYKAZM5x/IwpyUNU34uhKlf5xVEnELDe2IBtDVXsw+4qYsjzai4Qt05W0WMF pY1NdjLa0+AX6vif/bcWd7BZd3XM7Q6bNN6jhqPKytTdWPwo7A6BilPlJ7hGd2tgvxtj fRgw== X-Gm-Message-State: ABy/qLZf1p7Cy1ZYF3pqqlXWf1cPu9SpY6aKv3rjEJD0NOaEDA/Y6XTV jMYFE+LaslR3eu08uZ4zmnHkVNkpjzo= X-Google-Smtp-Source: APBJJlEyN+zHPVQa3eobi9kVvy/PF3MEvmsy7ulLO2s08YJkOcrroz5hT7bBtX3RFkjbpz2mS26B9w== X-Received: by 2002:a7b:ce8c:0:b0:3f9:fd12:a8b0 with SMTP id q12-20020a7bce8c000000b003f9fd12a8b0mr1006929wmj.20.1688537320163; Tue, 04 Jul 2023 23:08:40 -0700 (PDT) Received: from christian-Precision-5550.lan ([2001:861:3f04:7ca0:f6a2:3135:7895:378b]) by smtp.gmail.com with ESMTPSA id k15-20020a7bc40f000000b003fbaf9abf2fsm1087762wmi.23.2023.07.04.23.08.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Jul 2023 23:08:39 -0700 (PDT) From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Jonathan Tan , Jonathan Nieder , Taylor Blau , Derrick Stolee , Patrick Steinhardt , Christian Couder , Christian Couder Subject: [PATCH v2 8/8] gc: add `gc.repackFilterTo` config option Date: Wed, 5 Jul 2023 08:08:12 +0200 Message-ID: <20230705060812.2865188-9-christian.couder@gmail.com> X-Mailer: git-send-email 2.41.0.244.g8cb3faa74c In-Reply-To: <20230705060812.2865188-1-christian.couder@gmail.com> References: <20230614192541.1599256-1-christian.couder@gmail.com> <20230705060812.2865188-1-christian.couder@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A previous commit implemented the `gc.repackFilter` config option to specify a filter that should be used by `git gc` when performing repacks. Another previous commit has implemented `git repack --filter-to=` to specify the location of the packfile containing filtered out objects when using a filter. Let's implement the `gc.repackFilterTo` config option to specify that location in the config when `gc.repackFilter` is used. Now when `git gc` will perform a repack with a configured through this option and not empty, the repack process will be passed a corresponding `--filter-to=` argument. Signed-off-by: Christian Couder --- Documentation/config/gc.txt | 6 ++++++ builtin/gc.c | 4 ++++ t/t6500-gc.sh | 13 ++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt index 2153bde7ac..0e32007502 100644 --- a/Documentation/config/gc.txt +++ b/Documentation/config/gc.txt @@ -150,6 +150,12 @@ gc.repackFilter:: objects into a separate packfile. See the `--filter=` option of linkgit:git-repack[1]. +gc.repackFilterTo:: + When repacking and using a filter, see `gc.repackFilter`, the + specified location will be used to create the packfile + containing the filtered out objects. See the + `--filter-to=` option of linkgit:git-repack[1]. + gc.rerereResolved:: Records of conflicted merge you resolved earlier are kept for this many days when 'git rerere gc' is run. diff --git a/builtin/gc.c b/builtin/gc.c index 046147fdcc..f31167be6d 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -63,6 +63,7 @@ static const char *gc_log_expire = "1.day.ago"; static const char *prune_expire = "2.weeks.ago"; static const char *prune_worktrees_expire = "3.months.ago"; static char *repack_filter; +static char *repack_filter_to; static unsigned long big_pack_threshold; static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE; @@ -173,6 +174,7 @@ static void gc_config(void) git_config_get_ulong("pack.deltacachesize", &max_delta_cache_size); git_config_get_string("gc.repackfilter", &repack_filter); + git_config_get_string("gc.repackfilterto", &repack_filter_to); git_config(git_default_config, NULL); } @@ -362,6 +364,8 @@ static void add_repack_all_option(struct string_list *keep_pack) if (repack_filter && *repack_filter) strvec_pushf(&repack, "--filter=%s", repack_filter); + if (repack_filter_to && *repack_filter_to) + strvec_pushf(&repack, "--filter-to=%s", repack_filter_to); } static void add_repack_incremental_option(void) diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 5b89faf505..37056a824b 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -203,7 +203,6 @@ test_expect_success 'one of gc.reflogExpire{Unreachable,}=never does not skip "e ' test_expect_success 'gc.repackFilter launches repack with a filter' ' - test_when_finished "rm -rf bare.git" && git clone --no-local --bare . bare.git && git -C bare.git -c gc.cruftPacks=false gc && @@ -214,6 +213,18 @@ test_expect_success 'gc.repackFilter launches repack with a filter' ' grep -E "^trace: (built-in|exec|run_command): git repack .* --filter=blob:none ?.*" trace.out ' +test_expect_success 'gc.repackFilterTo store filtered out objects' ' + test_when_finished "rm -rf bare.git filtered.git" && + + git init --bare filtered.git && + git -C bare.git -c gc.repackFilter=blob:none \ + -c gc.repackFilterTo=../filtered.git/objects/pack/pack \ + -c repack.writeBitmaps=false -c gc.cruftPacks=false gc && + + test_stdout_line_count = 1 ls bare.git/objects/pack/*.pack && + test_stdout_line_count = 1 ls filtered.git/objects/pack/*.pack +' + prepare_cruft_history () { test_commit base &&