From patchwork Wed Apr 12 10:22:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F238C7619A for ; Wed, 12 Apr 2023 10:22:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231252AbjDLKWm (ORCPT ); Wed, 12 Apr 2023 06:22:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231127AbjDLKWi (ORCPT ); Wed, 12 Apr 2023 06:22:38 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2C146E86 for ; Wed, 12 Apr 2023 03:22:36 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 1A9283200979; Wed, 12 Apr 2023 06:22:36 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 12 Apr 2023 06:22:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294955; x=1681381355; bh=ZO ulI9eTvPJ9mLVAQ9N04IyJwCVTvNCP0mGWIoNXvNk=; b=HRewMT62A3UEACzJPh 9T+IRuZLATbZfbY8DITDeOyv5LlQwhDDNh118jn/PbFrRv+5SufebCuDgpdKCqiA 1cnFNwa2iEw/zSFvEtDCyH2Wt3RqbqkFM+kHiLzRuZWDNlNdpwFMcZo9xBVuFour q13C/fCSDLk2vxKkMq0l9J3kwncOdyHcd2G3qoGO1YMuHBbHhk1wMhHHi2soS0Ms V4mqu0stW9lFhMtaSB5BUoajazUDtGi/5ZfNYH9ZncQ3DC8NoPm2D7bvgjOu2lXG ldcrtVZ+7gKh/0xcDswqB7YG0hp0iq0jqy0Y6l83gUSAXbcmDPuXdTLaacWPFesh PWTQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294955; x=1681381355; bh=ZOulI9eTvPJ9m LVAQ9N04IyJwCVTvNCP0mGWIoNXvNk=; b=MtZDz8Wxw/rZhqwpnXq4e31PFELPJ QGV1H0DgdQX2Y4vmvhss/pHXvLWBRZZGBoFGg8xIY8vQd+4zjVTPM7FlzOlL2SmZ 2ynWDKYwApKaWuu/oZLtcZmzsxVhSw0CxyNDyw/nA6+BzBwh/+QQDEPTSnpQhmwr soBYXZ/bo0iMPR3wYZEeNBOpXn2zW8rJqvstvRG6C+iqDpSGALXVzE3lbeaqYUaL QT8vr2XnXsKpBNKNmhR9lBuWUp+bC/S1SEEGdVU7IPNFHMVFpyDrPR/Yn0jVFJHm wD6HqelzyxofDe75P2yvgXdo5CFs8v3cZCYd9whWXVjSwoGcMD0g7GT/A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:34 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id f124c3ef (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:12 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:31 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 1/8] midx: fix segfault with no packs and invalid preferred pack Message-ID: <5ecad306b42441fa7d4f50bdfb9c09ccce22b6c9.1681294715.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When asked to write a multi-pack-index the user can specify a preferred pack that is used as a tie breaker when multiple packs contain the same objects. When this packfile cannot be found, we just pick the first pack that is getting tracked by the newly written multi-pack-index as a fallback. Picking the fallback can fail in the case where we're asked to write a multi-pack-index with no packfiles at all: picking the fallback value will cause a segfault as we blindly index into the array of packfiles, which would be empty. Fix this bug by exiting early in case we have determined that the MIDX wouldn't have any packfiles to index. While the request itself does not make much sense anyway, it is still preferable to exit gracefully than to abort. Signed-off-by: Patrick Steinhardt --- midx.c | 6 ++++++ t/t5319-multi-pack-index.sh | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/midx.c b/midx.c index 9af3e5de88..22ea7ffb75 100644 --- a/midx.c +++ b/midx.c @@ -1307,6 +1307,12 @@ static int write_midx_internal(const char *object_dir, for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); + if (!ctx.nr) { + error(_("no pack files to index.")); + result = 1; + goto cleanup; + } + if ((ctx.m && ctx.nr == ctx.m->num_packs) && !(packs_to_include || packs_to_drop)) { struct bitmap_index *bitmap_git; diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 499d5d4c78..be7f3c1e1f 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -183,6 +183,17 @@ test_expect_success 'write midx with --stdin-packs' ' compare_results_with_midx "mixed mode (one pack + extra)" +test_expect_success 'write with no objects and preferred pack' ' + test_when_finished "rm -rf empty" && + git init empty && + test_must_fail git -C empty multi-pack-index write \ + --stdin-packs --preferred-pack=does-not-exist err && + cat >expect <<-EOF && + error: no pack files to index. + EOF + test_cmp expect err +' + test_expect_success 'write progress off for redirected stderr' ' git multi-pack-index --object-dir=$objdir write 2>err && test_line_count = 0 err From patchwork Wed Apr 12 10:22:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F8D0C77B6E for ; Wed, 12 Apr 2023 10:22:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231464AbjDLKWs (ORCPT ); Wed, 12 Apr 2023 06:22:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230345AbjDLKWm (ORCPT ); Wed, 12 Apr 2023 06:22:42 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57F907283 for ; Wed, 12 Apr 2023 03:22:40 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 73638320097C; Wed, 12 Apr 2023 06:22:39 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 12 Apr 2023 06:22:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294959; x=1681381359; bh=LQ 70rqnUoiQi4wFNiMWsX2f8kMeTMwBL42W0t9o6pvY=; b=SrL7PIHWlJMoOheAt+ JY9fCikZNW1fG29UDT+aVNEvem5cNi+X2xxOCY/e8m/sWVFQCfiRmk3iBq2X5f7O g7ps9T7SC5362kg3rFvdwy/iKhvPfTqCRUTy8DviRujn/G4eO5zO6+R2NEST7dU1 /twm++Nir4+d79LUa2FbuAtnWKCoPYLwfceuxonX/wijEwkEQEUO06+i/urjfXtz M1Iljt4uyJDO4vUmE+4t+rTQIucSXdCn0xse6Y223vpKboYf+rDuv4JWJSnHjJRT bqDaJy7Ct4d/g6g5kvrX27bukd4FrJzzACM9muvQdZ9qMsAoLLwz5mofzr1cdZfM K7Zw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294959; x=1681381359; bh=LQ70rqnUoiQi4 wFNiMWsX2f8kMeTMwBL42W0t9o6pvY=; b=Pc+vWFI1tU/3Uu+gNr46Yftojhx1+ /unqJBOrKhycli/nD2R3M9YCGOZ+zz9WEts0xvN4j0LuSwjD469xvJdlb4qxmP/8 kLnPA7IBA6cmV3PUo0A8pxb4dCCNRXtfjnA3EI/WTKzJTnKH2sVmsE6F34sRjiG2 nCSTSYhsO21VVrRSjYBk5BCLIlThPc0j7uua9rmycbutFteM7OHK6RahGqf5nKFU i8OofoBcxVdN5SoiDoK6unto8gv945YIsyAMn/m1hVHHWuemckvLdEG6B2NlYkUy O0Z51OF7AeUcsE8SxfWoqTs6++kFlCQ7lGS70370RC94aQMJ0xhOI5FCQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:37 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id e730a906 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:17 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:35 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 2/8] repack: fix trying to use preferred pack in alternates Message-ID: <011b08f3b64f264e3abbe8b49ee5338c221badb9.1681294715.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When doing a geometric repack with multi-pack-indices, then we ask git-multi-pack-index(1) to use the largest packfile as the preferred pack. It can happen though that the largest packfile is not part of the main object database, but instead part of an alternate object database. The result is that git-multi-pack-index(1) will not be able to find the preferred pack and print a warning. It then falls back to use the first packfile that the multi-pack-index shall reference. Fix this bug by only considering packfiles as preferred pack that are local. This is the right thing to do given that a multi-pack-index should never reference packfiles borrowed from an alternate. While at it, rename the function `get_largest_active_packfile()` to `get_preferred_pack()` to better document its intent. Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 21 ++++++++++++++++----- t/t7703-repack-geometric.sh | 31 +++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index df4d8e0f0b..9d36dc8b84 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -448,8 +448,10 @@ static void split_pack_geometry(struct pack_geometry *geometry, int factor) geometry->split = split; } -static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry) +static struct packed_git *get_preferred_pack(struct pack_geometry *geometry) { + uint32_t i; + if (!geometry) { /* * No geometry means either an all-into-one repack (in which @@ -464,7 +466,16 @@ static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry } if (geometry->split == geometry->pack_nr) return NULL; - return geometry->pack[geometry->pack_nr - 1]; + + for (i = geometry->pack_nr; i > 0; i--) + /* + * A pack that is not local would never be included in a + * multi-pack index. We thus skip over any non-local packs. + */ + if (geometry->pack[i - 1]->pack_local) + return geometry->pack[i - 1]; + + return NULL; } static void clear_pack_geometry(struct pack_geometry *geometry) @@ -591,7 +602,7 @@ static int write_midx_included_packs(struct string_list *include, { struct child_process cmd = CHILD_PROCESS_INIT; struct string_list_item *item; - struct packed_git *largest = get_largest_active_pack(geometry); + struct packed_git *preferred = get_preferred_pack(geometry); FILE *in; int ret; @@ -612,9 +623,9 @@ static int write_midx_included_packs(struct string_list *include, if (write_bitmaps) strvec_push(&cmd.args, "--bitmap"); - if (largest) + if (preferred) strvec_pushf(&cmd.args, "--preferred-pack=%s", - pack_basename(largest)); + pack_basename(preferred)); if (refs_snapshot) strvec_pushf(&cmd.args, "--refs-snapshot=%s", refs_snapshot); diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 8821fbd2dd..92a1aaa754 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -281,4 +281,35 @@ test_expect_success '--geometric with pack.packSizeLimit' ' ) ' +test_expect_success '--geometric --write-midx with packfiles in main and alternate ODB' ' + test_when_finished "rm -fr shared member" && + + # Create a shared repository that will serve as the alternate object + # database for the member linked to it. It has got some objects on its + # own that are packed into a single packfile. + git init shared && + test_commit -C shared common-object && + git -C shared repack -ad && + + # We create member so that its alternates file points to the shared + # repository. We then create a commit in it so that git-repack(1) has + # something to repack. + # of the shared object database. + git clone --shared shared member && + test_commit -C member unique-object && + git -C member repack --geometric=2 --write-midx 2>err && + test_must_be_empty err && + + # We should see that a new packfile was generated. + find shared/.git/objects/pack -type f -name "*.pack" | sort >packs && + test $(wc -l X-Patchwork-Id: 13208729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA3F5C7619A for ; Wed, 12 Apr 2023 10:22:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231499AbjDLKWy (ORCPT ); Wed, 12 Apr 2023 06:22:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231270AbjDLKWp (ORCPT ); Wed, 12 Apr 2023 06:22:45 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E5DC6EBD for ; Wed, 12 Apr 2023 03:22:44 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id DD8DD3200981; Wed, 12 Apr 2023 06:22:43 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 12 Apr 2023 06:22:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294963; x=1681381363; bh=yp c49vVdqlAZMXiFXzWnvH1mbc2r5EThPDLy6Wx/pxE=; b=Bwbotga8iDyaUzmWz5 H/v10PGUdr3X+vrn6ZxNLKX67NW/mdREfo59N4JGSyNFzc3CjPGmBcFJqa7ZVQaH NRxEy1MX+9AlgAb+noAaDZNCTqHZ7wVb+YCuEitLDu+G6fICYXjByPSQQ7ZJLmA1 QmDZCuuzRY9w1OwondgJ8yzygipOnQvJknZWfWwFvyB/GjDXR1sGlof/R20Vq/Ji PkSP78cTk3pzrqp9G8qyqGw6NUCCunod5H1gdMqyl5jHiG33/rvXcQdvF7D56J1k CHD4LvSDRffILc4EsCkFzmUi9uBoh22yUTfEFxUpxlksNSEOk+9KYzwA69p4gUWC 3jQQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294963; x=1681381363; bh=ypc49vVdqlAZM XiFXzWnvH1mbc2r5EThPDLy6Wx/pxE=; b=fKEXZrBMCGAza3BYI6C47q1iHykX1 JFdw/8tASemOIBoynIhLDqpYzHNOOPr9B+WMonUEAi+V/o7TkFw15Dg9bjJRyAHD HHdItTKLLavWHwQYzhpOuGJ7wY6B1wN91Ldcbwntsq8IVg1fXEtCZeLzdAx+9TXl kSwmhO3ZuLgY48u2Gnz9z4MBVlP/4Ddi+a66IHFbZS2w3Fg3WbwIYDnEumRZHWqM Snv+zHopdQAHYCxfn0e+xAQl/WmZ24xgqRI52qSGxK2MRn0K4ATAqfaHH6eG4FpP dSpKoP8DmsodfksnsoD1HxyAMNSaYrpudV7Nl4WEMRSfshaNtaFmz1qfw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:42 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 2bbc1714 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:21 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:39 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 3/8] repack: fix generating multi-pack-index with only non-local packs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing the multi-pack-index with geometric repacking we will add all packfiles to the index that are part of the geometric sequence. This can potentially also include packfiles borrowed from an alternate object directory. But given that a multi-pack-index can only ever include packs that are part of the main object database this does not make much sense whatsoever. In the edge case where all packfiles are contained in the alternate object database and the local repository has none itself this bug can cause us to invoke git-multi-pack-index(1) with only non-local packfiles that it ultimately cannot find. This causes it to return an error and thus causes the geometric repack to fail. Fix the code to skip non-local packfiles. Co-authored-by: Taylor Blau Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 11 +++++++++++ t/t7703-repack-geometric.sh | 24 ++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/builtin/repack.c b/builtin/repack.c index 9d36dc8b84..80d4e813c8 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -569,6 +569,17 @@ static void midx_included_packs(struct string_list *include, for (i = geometry->split; i < geometry->pack_nr; i++) { struct packed_git *p = geometry->pack[i]; + /* + * The multi-pack index never refers to packfiles part + * of an alternate object database, so we skip these. + * While git-multi-pack-index(1) would silently ignore + * them anyway, this allows us to skip executing the + * command completely when we have only non-local + * packfiles. + */ + if (!p->pack_local) + continue; + strbuf_addstr(&buf, pack_basename(p)); strbuf_strip_suffix(&buf, ".pack"); strbuf_addstr(&buf, ".idx"); diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 92a1aaa754..9e95350cbf 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -312,4 +312,28 @@ test_expect_success '--geometric --write-midx with packfiles in main and alterna git -C member multi-pack-index verify ' +test_expect_success '--geometric --with-midx with no local objects' ' + test_when_finished "rm -fr shared member" && + + # Create a repository with a single packfile that acts as alternate + # object database. + git init shared && + test_commit -C shared "shared-objects" && + git -C shared repack -ad && + + # Create a second repository linked to the first one and perform a + # geometric repack on it. + git clone --shared shared member && + git -C member repack --geometric 2 --write-midx 2>err && + test_must_be_empty err && + + # Assert that we wrote neither a new packfile nor a multi-pack-index. + # We should not have a packfile because the single packfile in the + # alternate object database does not invalidate the geometric sequence. + # And we should not have a multi-pack-index because these only index + # local packfiles, and there are none. + find member/.git/objects/pack -type f >actual && + test_must_be_empty actual +' + test_done From patchwork Wed Apr 12 10:22:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D129C7619A for ; Wed, 12 Apr 2023 10:23:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231480AbjDLKW7 (ORCPT ); Wed, 12 Apr 2023 06:22:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230099AbjDLKWx (ORCPT ); Wed, 12 Apr 2023 06:22:53 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30EE87A9B for ; Wed, 12 Apr 2023 03:22:48 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 40C2D3200979; Wed, 12 Apr 2023 06:22:48 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 12 Apr 2023 06:22:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294967; x=1681381367; bh=WW xzsL+aogjS+iPsHi7EjPHY6gJwOSprOj5oHJjX0tA=; b=YDUWb7GhNTdlUxldhJ XR6m9jKtDYZhTvOd/rP8yOb8+Wm1UWfcmTPmAyJtrRznD67hACtVlOT8v+ZxI7Ue sy6vOgtmTkQNUi1VtBI9inJeWFGwIFK0FEiRG3hGvs19+khRNlVKBV5iTEMYEUF+ B0h0l8pDI1Oreun4tp9yoIIcvUX3fKwZaOklOW8LnemdP0bp4WRxRM+iYzsVKHD7 GCxiDUdErgU3oruJmueMD7l1yQFkvz6036zmMeZJ1rJKieFDbejZJqGip7gO3GYs pEZFbCWbhoOJCKl8Jy/1RLL45pF8Y320ZkT9tcdYKQM6SYmZnwlVRwSqf2MgtY2t BbHw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294967; x=1681381367; bh=WWxzsL+aogjS+ iPsHi7EjPHY6gJwOSprOj5oHJjX0tA=; b=SWEq7ww1ud4GSItsWebbM+Sirg4ND jezfrea+/zegSgNmi1HKvDDM4xCgeeIdxcQlDKUdVcuQcwVcdK2FAPBFqy5dRTL0 F8kVp7VbVh1bT2MbwmK6op8iRm3V9EDtGXeg49ANoccs6DdaxGKWg/0WPEKRFpde KY9ajTZoZBBr6x0soV4ea2lE/D2Sa3MSG9Hurb9miMKe45TtGSFGt+1nLPE87z6Q ZeLKd5fwdehBQxlm0V2/QTrsmmjtv/PgskUisGZYWy3+M2blaZ0aLiJZMhaFFNzl WBx8fAFgApy46wC/SIbgo8vQTkDHRbmgmGJPzK1PbFWB/dMpblhCOSmFQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedvnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:46 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id b78cfa58 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:25 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:44 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 4/8] pack-objects: fix error when packing same pack twice Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When passed the same packfile twice via `--stdin-packs` we return an error that the packfile supposedly was not found. This is because when reading packs into the list of included or excluded packfiles, we will happily re-add packfiles even if they are part of the lists already. And while the list can now contain duplicates, we will only set the `util` pointer of the first list entry to the `packed_git` structure. We notice that at a later point when checking that all list entries have their `util` pointer set and die with an error. While this is kind of a nonsensical request, this scenario can be hit when doing geometric repacks. When a repository is connected to an alternate object directory and both have the exact same packfile then both would get added to the geometric sequence. And when we then decide to perform the repack, we will invoke git-pack-objects(1) with the same packfile twice. Fix this bug by removing any duplicates from both the included and excluded packs. Signed-off-by: Patrick Steinhardt --- builtin/pack-objects.c | 2 ++ t/t5331-pack-objects-stdin.sh | 29 +++++++++++++++++++++++++++++ t/t7703-repack-geometric.sh | 25 +++++++++++++++++++++++++ 3 files changed, 56 insertions(+) create mode 100755 t/t5331-pack-objects-stdin.sh diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 77d88f85b0..fdf3f440be 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3359,7 +3359,9 @@ static void read_packs_list_from_stdin(void) } string_list_sort(&include_packs); + string_list_remove_duplicates(&include_packs, 0); string_list_sort(&exclude_packs); + string_list_remove_duplicates(&exclude_packs, 0); for (p = get_all_packs(the_repository); p; p = p->next) { const char *pack_name = pack_basename(p); diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh new file mode 100755 index 0000000000..ab34cfc729 --- /dev/null +++ b/t/t5331-pack-objects-stdin.sh @@ -0,0 +1,29 @@ +#!/bin/sh + +test_description='pack-objects --stdin' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +test_expect_success 'pack-objects --stdin with duplicate packfile' ' + test_when_finished "rm -fr repo" && + + git init repo && + ( + cd repo && + test_commit "commit" && + git repack -ad && + + ( + basename .git/objects/pack/pack-*.pack && + basename .git/objects/pack/pack-*.pack + ) >packfiles && + + git pack-objects --stdin-packs generated-pack member/.git/objects/info/alternates && + find shared/.git/objects -type f >expected-files && + + # Verify that we can repack objects as expected without observing any + # error. Having the same packfile in both ODBs used to cause an error + # in git-pack-objects(1). + git -C member repack --geometric 2 2>err && + test_must_be_empty err && + # Nothing should have changed. + find shared/.git/objects -type f >actual-files && + test_cmp expected-files actual-files +' + test_done From patchwork Wed Apr 12 10:22:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A80BCC77B6E for ; Wed, 12 Apr 2023 10:23:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231588AbjDLKXL (ORCPT ); Wed, 12 Apr 2023 06:23:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230345AbjDLKW5 (ORCPT ); Wed, 12 Apr 2023 06:22:57 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73EC27DB2 for ; Wed, 12 Apr 2023 03:22:53 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 85EB83200977; Wed, 12 Apr 2023 06:22:52 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 12 Apr 2023 06:22:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294972; x=1681381372; bh=eq SV+lgEORpVzqcRPJ6Glj7EarT1imzXLG3qQ8h96M0=; b=Ug1IzJHGSnYrNB/QSW 2OvgyS1Oqg4YIV2gmsPGt0jbXkcfQXzbGx8gpwZjUHmR4uXmRRKXUzPaCJpITtc5 f46tPM99kQNgkdtyrbh+HkCQOM4b0STaBU6u49esZ5qmilT1zIx5aLcSv/TtBwSx 5LVwixNq5Wk0BoKUqcVZKVOQxg43h5N8POHXi3n9mgE2L8Nysua2Rwau4XOLhNVm EAgnX7CZnR2O9jkGVwfUEmEEKLh5oQJkvxaop5Nrty0aviJNQjy81p97NCxkcOgX cmjddrzQ2QFZefkwH8MWab3LrYd/NLFC0SNnW6T+Fpf7/4Kk3BoXC9ALRO4Dr3rf uTzQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294972; x=1681381372; bh=eqSV+lgEORpVz qcRPJ6Glj7EarT1imzXLG3qQ8h96M0=; b=FywyGKlr14zwlOdMqumoYyX3EAxwz CRPsU8Kqu0SutASOPh5k7gN6969LgJ6bb4CTqz9haU3Oh0s9Wmeskf1mTvyH12CC +BoJ/sDLhZ8PY5kPemyKyk8sRrQKdkAcOv6tUXfG/fsxWA0Df6pxBrZsL3zAO5qi 42CRrm2f9uhR85pfTTOWr2nd80h5dHVH3yFe3ArHF8UOvBSPzNavWE01kRt50mm9 j1/eQcAvHU7E1mMeDAByZEeVMiTYEhmNa7llr2pDnl9/F6vRSqmMuU5UxguuFmlO tV26xkVO3iOx/QaXjaa/P+1fGhtaGfmphM/4gUdEWMcTtl6vVRXpKpODA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:50 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id bb389869 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:29 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:48 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 5/8] pack-objects: fix error when same packfile is included and excluded Message-ID: <9b278a4c91a5631d1b7b11bf56ab560c6bb58645.1681294715.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When passing the same packfile both as included and excluded via the `--stdin-packs` option, then we will return an error because the excluded packfile cannot be found. This is because we will only set the `util` pointer for the included packfile list if it was found, so that we later die when we notice that it's in fact not set for the excluded packfile list. Fix this bug by always setting the `util` pointer for both the included and excluded list entries. Signed-off-by: Patrick Steinhardt --- builtin/pack-objects.c | 8 +++----- t/t5331-pack-objects-stdin.sh | 26 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index fdf3f440be..522eb4dd31 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3366,11 +3366,9 @@ static void read_packs_list_from_stdin(void) for (p = get_all_packs(the_repository); p; p = p->next) { const char *pack_name = pack_basename(p); - item = string_list_lookup(&include_packs, pack_name); - if (!item) - item = string_list_lookup(&exclude_packs, pack_name); - - if (item) + if ((item = string_list_lookup(&include_packs, pack_name))) + item->util = p; + if ((item = string_list_lookup(&exclude_packs, pack_name))) item->util = p; } diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index ab34cfc729..f389a78b38 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -7,6 +7,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh +packed_objects() { + git show-index <"$1" >tmp-object-list && + cut -d' ' -f2 tmp-object-list && + rm tmp-object-list + } + test_expect_success 'pack-objects --stdin with duplicate packfile' ' test_when_finished "rm -fr repo" && @@ -26,4 +32,24 @@ test_expect_success 'pack-objects --stdin with duplicate packfile' ' ) ' +test_expect_success 'pack-objects --stdin with same packfile excluded and included' ' + test_when_finished "rm -fr repo" && + + git init repo && + ( + cd repo && + test_commit "commit" && + git repack -ad && + + ( + basename .git/objects/pack/pack-*.pack && + printf "^%s\n" "$(basename .git/objects/pack/pack-*.pack)" + ) >packfiles && + + git pack-objects --stdin-packs generated-pack packed-objects && + test_must_be_empty packed-objects + ) +' + test_done From patchwork Wed Apr 12 10:22:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3CBEC7619A for ; Wed, 12 Apr 2023 10:23:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231473AbjDLKXW (ORCPT ); Wed, 12 Apr 2023 06:23:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231538AbjDLKXD (ORCPT ); Wed, 12 Apr 2023 06:23:03 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA53F188 for ; Wed, 12 Apr 2023 03:22:57 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id DDDD93200985; Wed, 12 Apr 2023 06:22:56 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Wed, 12 Apr 2023 06:22:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294975; x=1681381375; bh=3e zOkmRLBC+RPoRa9A7gr7+1JZYuo4xi/8eOeH9y89I=; b=SLS0IyenYGDvTfHrCo sKn3xDCvuvLb/2reDfhqscdfKEAqxGgYvjlHUjVPf99+xLcP6cmHtQnRqsXTg9tB 1moQ9kx5tybQHxjJem82uTBsJU/+MM7MUUGLExEWBaxv7OAuzbYVNgiOZkJ9k5i7 mhwG2/nPimWuehcPrIEB1iQ2Mah2haEKUGrMpXiBpGsFRKAqT1Ho6oes4GTRI4cH wBgopQ0RtvZLQnE+42DOmoPn3yx/2zjkOv1hm65kXS5vjGFL4s6lkNkS+FDWKyPR 1vvUeN6Od+sSGQHKXpigr+u5vdU1X4sJ+EVPiEK1iY1hXJQv2FQh8RXImba0CEwD p8aQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294975; x=1681381375; bh=3ezOkmRLBC+RP oRa9A7gr7+1JZYuo4xi/8eOeH9y89I=; b=kS6TvsrtcT273WVN6DJTCLflx1Xms zXDlSxAWhi4Ka9htbFjGrxcrVihyzHjbz8ih/xhLaGiCF567Kw5EiOCHLNVEo7cc Ua0E85KWEdrxuWyAr6LinFrd3R3vg1LPxovinW/jCKwgknNsqz+Y2UOaQ+ZBs4e8 AIdVjKF2NqvyteZynCPZd5a3JMKpXtfkIckTdCjDezZrAl4Ht7EwVu57Fy97xLR5 QOKlmPFO7rl8gKW8I0Q4SDGz4uYbF7eaJffhSAdBwc/8DcmNIOkS2d4szVuFBwBL GEkEHxluquNJWXvCqdgKQWFpMhoxaUkmQk2q+xUVbx32Y3ikZGGA1g+dg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:54 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id beea924c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:33 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:52 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 6/8] pack-objects: extend test coverage of `--stdin-packs` with alternates Message-ID: <3a7a2015f7ced01dc738099f18b0fce6537af05c.1681294715.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We don't have any tests that verify that git-pack-objects(1) works with `--stdin-packs` when combined with alternate object directories. Add some to make sure that the basic functionality works as expected. Signed-off-by: Patrick Steinhardt --- t/t5331-pack-objects-stdin.sh | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index f389a78b38..8bd1d26caf 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -52,4 +52,52 @@ test_expect_success 'pack-objects --stdin with same packfile excluded and includ ) ' +test_expect_success 'pack-objects --stdin with packfiles from alternate object database' ' + test_when_finished "rm -fr shared member" && + + # Set up a shared repository with a single packfile. + git init shared && + test_commit -C shared "shared-objects" && + git -C shared repack -ad && + basename shared/.git/objects/pack/pack-*.pack >packfile && + + # Set up a repository that is connected to the shared repository. This + # repository has no objects on its own, but we still expect to be able + # to pack objects from its alternate. + git clone --shared shared member && + git -C member pack-objects --stdin-packs generated-pack packfiles && + + ( + packed_objects shared/.git/objects/pack/pack-*.idx && + packed_objects member/.git/objects/pack/pack-*.idx + ) | sort >expected-objects && + + git -C member pack-objects --stdin-packs generated-pack actual-objects && + test_cmp expected-objects actual-objects +' + test_done From patchwork Wed Apr 12 10:22:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDE26C77B71 for ; Wed, 12 Apr 2023 10:23:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231485AbjDLKXY (ORCPT ); Wed, 12 Apr 2023 06:23:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231580AbjDLKXF (ORCPT ); Wed, 12 Apr 2023 06:23:05 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09E096E87 for ; Wed, 12 Apr 2023 03:23:00 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 43DB43200981; Wed, 12 Apr 2023 06:23:00 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 12 Apr 2023 06:23:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294979; x=1681381379; bh=cL C7rR08BvrCD4BZ30X2sSsFUcBMmeOu+5gFUCuK0cs=; b=cAvpse6HS9kSVeRSRN 3lTIG8Gckk/+/80ruVxxVGd5UtVqGJDJgUuOhDmG001aW7/U2Oz3I/f8k37IjoYt bFxvpEYY12v9tS9C6eU674InYTlTOJ6CGyiRcivDxBGiqK1RO/rEMbd20vL2wCrn FuQ6CP+i2MnZd/SPK4GTkU0FJ69aY6dg6YJcjyg4YAkUpxCteQ6NcbjsmRFUFrGk l4wahs0wm8UtVJCRNpnIonuSix66k1Zy0/SM1Tkip8eKN9mRUqoLHmMklKn+tjTU DMbQW5Xnj8XwGHsN7roywEAGKMqe93BfdDAOTB9fJItjvg8uf/3W5URRbdCPwhoD J0QQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294979; x=1681381379; bh=cLC7rR08BvrCD 4BZ30X2sSsFUcBMmeOu+5gFUCuK0cs=; b=WvY3nKDagb/lBcrGqGdqLPiKjXh+K HV3tcJqAmQciTFwRYR0XiCvzTeSC3WJVLlQLjf6zED678ajeQMbUEr2IUPhoTkw+ kVBOenIkPqGUPR60kq3BIrPTMsfJ/us2Xi9EUQuyy9caxOGOFAAJkpdONlbtWpVB gC7rl4cnOkzxAZOvmS/gncqm/DoDVRWV4b7bL6bwDOwGDIUjvx3oIGwY4O9hby5p FAEuK/ELQPG2DpPUecJaNZ9uvVnMl5d2auFBVMzre803R0bdhIIT6x29MhDxE6q5 ckUo3bh92YtrBGz4rECkoW12x+Jf+2M5JpHJZS3KFkyML4uMBpec1uaPw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpeefnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:22:58 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 31f0d03e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:38 +0000 (UTC) Date: Wed, 12 Apr 2023 12:22:56 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 7/8] repack: honor `-l` when calculating pack geometry Message-ID: <608dde4ad52c28ef42845b6bfdcb168e252bd29b.1681294715.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When the user passes `-l` to git-repack(1), then they essentially ask us to only repack objects part of the local object database while ignoring any packfiles part of an alternate object database. And we in fact honor this bit when doing a geometric repack as the resulting packfile will only ever contain local objects. What we're missing though is that we don't take locality of packfiles into account when computing whether the geometric sequence is intact or not. So even though we would only ever roll up local packfiles anyway, we could end up trying to repack because of non-local packfiles. This does not make much sense, and in the worst case it can cause us to try and do the geometric repack over and over again because we're never able to restore the geometric sequence. Fix this bug by honoring whether the user has passed `-l`. If so, we skip adding any non-local packfiles to the pack geometry. Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 13 ++++++-- t/t7703-repack-geometric.sh | 59 +++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 80d4e813c8..f57869f14a 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -325,7 +325,8 @@ static int geometry_cmp(const void *va, const void *vb) } static void init_pack_geometry(struct pack_geometry **geometry_p, - struct string_list *existing_kept_packs) + struct string_list *existing_kept_packs, + const struct pack_objects_args *args) { struct packed_git *p; struct pack_geometry *geometry; @@ -335,6 +336,14 @@ static void init_pack_geometry(struct pack_geometry **geometry_p, geometry = *geometry_p; for (p = get_all_packs(the_repository); p; p = p->next) { + if (args->local && !p->pack_local) + /* + * When asked to only repack local packfiles we skip + * over any packfiles that are borrowed from alternate + * object directories. + */ + continue; + if (!pack_kept_objects) { /* * Any pack that has its pack_keep bit set will appear @@ -897,7 +906,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (geometric_factor) { if (pack_everything) die(_("options '%s' and '%s' cannot be used together"), "--geometric", "-A/-a"); - init_pack_geometry(&geometry, &existing_kept_packs); + init_pack_geometry(&geometry, &existing_kept_packs, &po_args); split_pack_geometry(geometry, geometric_factor); } diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 0a2f2bd46c..96c8d4cdfa 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -10,6 +10,12 @@ objdir=.git/objects packdir=$objdir/pack midx=$objdir/pack/multi-pack-index +packed_objects() { + git show-index <"$1" >tmp-object-list && + cut -d' ' -f2 tmp-object-list && + rm tmp-object-list + } + test_expect_success '--geometric with no packs' ' git init geometric && test_when_finished "rm -fr geometric" && @@ -361,4 +367,57 @@ test_expect_success '--geometric with same pack in main and alternate ODB' ' test_cmp expected-files actual-files ' +test_expect_success '--geometric -l with non-intact geometric sequence across ODBs' ' + test_when_finished "rm -fr shared member" && + + git init shared && + test_commit_bulk -C shared --start=1 1 && + + git clone --shared shared member && + test_commit_bulk -C member --start=2 1 && + + # Verify that our assumptions actually hold: both generated packfiles + # should have three objects and should be non-equal. + packed_objects shared/.git/objects/pack/pack-*.idx >packed-objects && + test_line_count = 3 packed-objects && + packed_objects member/.git/objects/pack/pack-*.idx >packed-objetcs && + test_line_count = 3 packed-objects && + test "$(basename member/.git/objects/pack/pack-*.pack)" != "$(basename shared/.git/objects/pack/pack-*.pack)" && + + # Perform the geometric repack. With `-l`, we should only see the local + # packfile and thus arrive at the conclusion that the geometric + # sequence is intact. We thus expect no changes. + # + # Note that we are using stat(1) to verify idempotence to also verify + # that the mtime did not change. This is done in order to detect the + # case where we do repack objects, but the resulting packfile is the + # same. + stat member/.git/objects/pack/* >expected-member-packs && + git -C member repack --geometric=2 -l -d && + stat member/.git/objects/pack/* >actual-member-packs && + test_cmp expected-member-packs actual-member-packs && + + ( + packed_objects shared/.git/objects/pack/pack-*.idx && + packed_objects member/.git/objects/pack/pack-*.idx + ) | sort >expected-objects && + + # On the other hand, when doing a non-local geometric repack we should + # see both packfiles and thus repack them. We expect that the shared + # object database was not changed. + stat shared/.git/objects/pack/* >expected-shared-packs && + git -C member repack --geometric=2 -d && + stat shared/.git/objects/pack/* >actual-shared-packs && + test_cmp expected-shared-packs actual-shared-packs && + + # Furthermore, we expect that the member repository now has a single + # packfile that contains the combined shared and non-shared objects. + ls member/.git/objects/pack/pack-*.idx >actual && + test_line_count = 1 actual && + packed_objects member/.git/objects/pack/pack-*.idx >actual-objects && + test_line_count = 6 actual-objects && + sort actual-objects.sorted && + test_cmp expected-objects actual-objects.sorted +' + test_done From patchwork Wed Apr 12 10:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13208734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40695C77B71 for ; Wed, 12 Apr 2023 10:23:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231598AbjDLKXf (ORCPT ); Wed, 12 Apr 2023 06:23:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231262AbjDLKXI (ORCPT ); Wed, 12 Apr 2023 06:23:08 -0400 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85CF27D89 for ; Wed, 12 Apr 2023 03:23:05 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id A8A1D3200984; Wed, 12 Apr 2023 06:23:04 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 12 Apr 2023 06:23:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681294984; x=1681381384; bh=br 9prARjMcDuS9r+1lDr24rpMi1ZT2vNvswKW1/UsUU=; b=H70dU9eGWrcKnBXkWi FnejIh7WTDQVa1zg8uxfFbcypvAQ/Y2cyI80ovsoRlh7q0/mONPZQf2tu6e3uJ06 73tNx4WKQ8Xl7BsYl83jB36RF8o+QHSBJIz+5HHnNGlKecmED3Am2jQRs3Ktg9YB YygwE1D0pjqbxsSfKcYbbGYKxpUBN0BZJiBNb26nRdUdI6XwH/G2xR6SomuvH66t f9I2Ln8YRcTlkvOX8C1k3GpDvqdEBL5/CgGb1JqwnILhB7uet8YFsWzL6IipJCX+ Gq0dSm7tlmM+eS4VLS3ulpQPVz7R1AMhxaP2t6dnlgG5aFneJ+N7hGBYLvH/yTjm YzEQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681294984; x=1681381384; bh=br9prARjMcDuS 9r+1lDr24rpMi1ZT2vNvswKW1/UsUU=; b=AlbL334gC6HzJ/iq5vpZetb6Wd0CE IwGNfZy5QNpDFPQ2xMeQ9jptGTZwjv4q+w72nmE9Nm02qBdX0TW3MwIkFuymESNq xW7Cu+zIq/bCr+nM2fg+UTHbgcIXClPEzi6pLbmXQtpTXpy29LHzbHSIheFOeNjd Nlb3Cqszow1Hna2w4ism2mrHGL7dkSbnipTaXlNBw1w3jJHiX8SlgTa6rUKUpZhu 2/jqlytI5HzbLkw+TOcU+SNV796FrgzdINNPmR1WR0t6R1eoc1CkDxtTNP9UBPb4 6VgHugx7fBxgZXyBbin7F/RmJOy/0vIOUWAX9yBU5TnWcctQI7qjLd4PQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekiedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 12 Apr 2023 06:23:02 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id a4cc6bbc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 12 Apr 2023 10:22:42 +0000 (UTC) Date: Wed, 12 Apr 2023 12:23:01 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v2 8/8] repack: disable writing bitmaps when doing a local geometric repack Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In order to write a bitmap, we need to have full coverage of all objects that are about to be packed. In the traditional non-multi-pack-index world this meant we need to do a full repack of all objects into a single packfile. But in the new multi-pack-index world we can get away with writing bitmaps when we have multiple packfiles as long as the multi-pack-index covers all objects. This is not always the case though. When writing multi-pack-indices in a repository that is connected to an alternate object directory we may end up writing a multi-pack-index that only has partial coverage of objects. The end result is that writing the bitmap will fail: $ git multi-pack-index write --stdin-packs --bitmap --- builtin/repack.c | 20 ++++++++++++++++++++ t/t7703-repack-geometric.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/builtin/repack.c b/builtin/repack.c index f57869f14a..07d92fdf87 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -881,6 +881,26 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps > 0 && !write_midx; + if (write_midx && write_bitmaps && geometric_factor && po_args.local) { + struct packed_git *p; + + for (p = get_all_packs(the_repository); p; p = p->next) { + if (p->pack_local) + continue; + + /* + * When asked to do a local repack, but we have + * packfiles that are inherited from an alternate, then + * we cannot guarantee that the multi-pack-index would + * have full coverage of all objects. We thus disable + * writing bitmaps in that case. + */ + warning(_("disabling bitmap writing, as some objects are not being packed")); + write_bitmaps = 0; + break; + } + } + if (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx) die(_(incremental_bitmap_conflict_error)); diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 96c8d4cdfa..0aaec9f853 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -420,4 +420,31 @@ test_expect_success '--geometric -l with non-intact geometric sequence across OD test_cmp expected-objects actual-objects.sorted ' +test_expect_success '--geometric -l disables writing bitmaps with non-local packfiles' ' + test_when_finished "rm -fr shared member" && + + git init shared && + test_commit_bulk -C shared --start=1 1 && + + git clone --shared shared member && + test_commit_bulk -C member --start=2 1 && + + # When performing a geometric repack with `-l` while connecting to an + # alternate object database that has a packfile we do not have full + # coverage of objects. As a result, we expect that writing the bitmap + # will be disabled. + git -C member repack -l --geometric=2 --write-midx --write-bitmap-index 2>err && + cat >expect <<-EOF && + warning: disabling bitmap writing, as some objects are not being packed + EOF + test_cmp expect err && + test ! -f member/.git/objects/pack/multi-pack-index-*.bitmap && + + # On the other hand, when we repack without `-l`, we should see that + # the bitmap gets created. + git -C member repack --geometric=2 --write-midx --write-bitmap-index 2>err && + test_must_be_empty err && + test -f member/.git/objects/pack/multi-pack-index-*.bitmap +' + test_done