From patchwork Tue Sep 14 22:06:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAE25C433F5 for ; Tue, 14 Sep 2021 22:06:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B8F661175 for ; Tue, 14 Sep 2021 22:06:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234706AbhINWHW (ORCPT ); Tue, 14 Sep 2021 18:07:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232561AbhINWHV (ORCPT ); Tue, 14 Sep 2021 18:07:21 -0400 Received: from mail-il1-x12e.google.com (mail-il1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2A2AC061574 for ; Tue, 14 Sep 2021 15:06:03 -0700 (PDT) Received: by mail-il1-x12e.google.com with SMTP id b15so684394ils.10 for ; Tue, 14 Sep 2021 15:06:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=j/AWH0Z0pKHAMhkAG1FRjnnJyX3QbRZqaE5juAMfac0=; b=ZjkWq6wfwNaKeYmxh00z7f1iTqBeQWUJTsBFEkZrqkN6L7e+Dnh9UEFNM5pp+evNCe rkZj3DZvf8Ac8g2ArpJUgUnDSMsGIsYG36jZjWoSZ+k5Pt53dU/vfQk/iefhkDXAWjKG EelOiIgPdoHj5SwYtAJGnh1DS6wxDKJm/4fE2BzJmT0KatfdTUEcYo+fsbKfnh0sMxXE f8so9X9isruZ0ZQa3U2UfDTPKoq6OEFOFuEg1RkmWX1MPff49oQtW20PBEB8+fr/qSlm 4zfpKxvETwC4WMjfpK8AWqcZPsUHZkXH/WXj3v6cKrLHbeP8F/xfCWHH8sNhn5FKAXdg t8xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=j/AWH0Z0pKHAMhkAG1FRjnnJyX3QbRZqaE5juAMfac0=; b=Lx9eXPyaj1vESenPyGw9xTkQD2uDJcx+RsJTT6kQwWISRIk8z/XIAN4fnx7FdeyEsq enL13V55czG47IYb70uqbcHrEuUqXIWVjnuxmcF8WjavnqZsTuCxoGSOxNH9TATWeIL6 LKI3TfVtAXrZsX1WV0UhARERrtSa+LLdqBnsSWPShYIefwUFlJepw//fO/cDqcvOsRep bZHSXjkb8aSa95d657pJQthWMiHBnADE2Ma+Kabdod0UHJgbMmHGYYIrD0jQc3RALl0R qzc2uDa/itxqP/OsSK390UWOl1+RqDdDms3qLQScbDjcCQWvZxU9eukp6XylKsehtBfd T19w== X-Gm-Message-State: AOAM532L9liQ1JlprOILvXHBHHEv/tj2QT6nYhrAgCQBcPnaw4CBWdTz 94M99IIkwO615+Q1/kA5j3MvJ/7v06EBlYrE X-Google-Smtp-Source: ABdhPJzTaZ6ndkBw/oSk7B49e3AeQlKdaYnSbTJgR55OReihy1BzQzFnCGO/aU8K3zFK5buIaMJqzg== X-Received: by 2002:a05:6e02:1144:: with SMTP id o4mr12170304ill.264.1631657162952; Tue, 14 Sep 2021 15:06:02 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id n4sm7780789ilo.76.2021.09.14.15.06.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:02 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:02 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 1/7] t/helper/test-bitmap.c: add 'dump-hashes' mode Message-ID: <4f2b8d95303d54a50b13117d771da690c4700d3e.1631657157.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The pack-bitmap writer code is about to learn how to propagate values from an existing hash-cache. To prepare, teach the test-bitmap helper to dump the values from a bitmap's hash-cache extension in order to test those changes. Signed-off-by: Taylor Blau --- pack-bitmap.c | 27 +++++++++++++++++++++++++++ pack-bitmap.h | 1 + t/helper/test-bitmap.c | 10 +++++++++- 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 8504110a4d..04de387318 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1742,6 +1742,33 @@ int test_bitmap_commits(struct repository *r) return 0; } +int test_bitmap_hashes(struct repository *r) +{ + struct bitmap_index *bitmap_git = prepare_bitmap_git(r); + struct object_id oid; + uint32_t i, index_pos; + + if (!bitmap_git->hashes) + goto cleanup; + + for (i = 0; i < bitmap_num_objects(bitmap_git); i++) { + if (bitmap_is_midx(bitmap_git)) + index_pos = pack_pos_to_midx(bitmap_git->midx, i); + else + index_pos = pack_pos_to_index(bitmap_git->pack, i); + + nth_bitmap_object_oid(bitmap_git, &oid, index_pos); + + printf("%s %"PRIu32"\n", + oid_to_hex(&oid), get_be32(bitmap_git->hashes + index_pos)); + } + +cleanup: + free_bitmap_index(bitmap_git); + + return 0; +} + int rebuild_bitmap(const uint32_t *reposition, struct ewah_bitmap *source, struct bitmap *dest) diff --git a/pack-bitmap.h b/pack-bitmap.h index 469090bad2..ed46d27077 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -52,6 +52,7 @@ void traverse_bitmap_commit_list(struct bitmap_index *, show_reachable_fn show_reachable); void test_bitmap_walk(struct rev_info *revs); int test_bitmap_commits(struct repository *r); +int test_bitmap_hashes(struct repository *r); struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, struct list_objects_filter_options *filter, int filter_provided_objects); diff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c index 134a1e9d76..ff35f5999b 100644 --- a/t/helper/test-bitmap.c +++ b/t/helper/test-bitmap.c @@ -7,6 +7,11 @@ static int bitmap_list_commits(void) return test_bitmap_commits(the_repository); } +static int bitmap_dump_hashes(void) +{ + return test_bitmap_hashes(the_repository); +} + int cmd__bitmap(int argc, const char **argv) { setup_git_directory(); @@ -16,9 +21,12 @@ int cmd__bitmap(int argc, const char **argv) if (!strcmp(argv[1], "list-commits")) return bitmap_list_commits(); + if (!strcmp(argv[1], "dump-hashes")) + return bitmap_dump_hashes(); usage: - usage("\ttest-tool bitmap list-commits"); + usage("\ttest-tool bitmap list-commits\n" + "\ttest-tool bitmap dump-hashes"); return -1; } From patchwork Tue Sep 14 22:06:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4925C433EF for ; Tue, 14 Sep 2021 22:06:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AB871611ED for ; Tue, 14 Sep 2021 22:06:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235235AbhINWH0 (ORCPT ); Tue, 14 Sep 2021 18:07:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235116AbhINWHY (ORCPT ); Tue, 14 Sep 2021 18:07:24 -0400 Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E96DC061767 for ; Tue, 14 Sep 2021 15:06:06 -0700 (PDT) Received: by mail-il1-x130.google.com with SMTP id h29so741802ila.2 for ; Tue, 14 Sep 2021 15:06:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=cuLvF0o8RqwzkFaFHWHASqoMkWK041CGb/7BF49RNi4=; b=Ud7diljsPiVraWmQYAK/5Nn3NQ3bU6Kh4BhZIqgxUyP/fZ1ALQzLl8aZcUAFw9cCrZ TfZ+0a6Tsxc1KPE6DVjtu968dl2drU1wxNMBiMfU1fdUZ+LKUMorZTmwO6qlU794CL23 LO4mgaMn7kbH/yPoZ8gw592fJNswYlKYvmqto4Kh0JHPQC2LfZHWmqUsrYs1PIxwPOWl 3Od9CHJM4j9W4UQ0X+8df5lPpENb4iNPU+fFgIoHyOmluezPuFSUH+/QrS55eNi2QN52 AAid+3TtLQBcNZtWkgmiZ5ZI7e1MyHMwldBbGRpedYVlHpVGifWK47ElpII9gGKrxJ3/ OkFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=cuLvF0o8RqwzkFaFHWHASqoMkWK041CGb/7BF49RNi4=; b=2qGS0bUHjcO+IWwOEhULGY9Pq6ALilmY3M6LObYP99xFRJxhTZ189p2mnQ5Xlpo8IJ iklrQHucFKX3CssEAlaaAo79m8fWTL/mQjxH1HJVdH2K3VY+Yof3ykS0oMzhtg9aG37+ /zJT6Zp8IaKEFjBH4Oiyf2kxgkR4r7eKZ5ANWT6+u1c7Zlre2ozie7Jh0kc8EGa8X7XZ KkFJdM42mabksWETu9Fspz7FE+dyF7OJeO2AhTYRtOI78vwU9EFfhYIoKUpvsRezc0DO Y0kkx/EUXdfLzqM8mYadDBtkeH6CNTGe+zmo/WiIvjNPF/U0eTfJcVUKFxZ+At7IO7sM DPmQ== X-Gm-Message-State: AOAM531/JeL7B+5wNwyizIcXvTCr77R2A7dDbGE17uMtwfOSPw3HGgxS uS9aiLZZ0mMc/Wcw/CoCkDFzwbUiCG7t2Wwx X-Google-Smtp-Source: ABdhPJxcNiJGu2vrzz+fxeABSPAyyB/TtK4eWQ0DbZTfMODWLQZOGlb4HShRMQ5AUPkCalLwnbaikg== X-Received: by 2002:a05:6e02:eb2:: with SMTP id u18mr14112542ilj.138.1631657165355; Tue, 14 Sep 2021 15:06:05 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id g19sm8092685ilb.84.2021.09.14.15.06.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:05 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:04 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 2/7] pack-bitmap.c: propagate namehash values from existing bitmaps Message-ID: <2cd2f3aa5e74a3aefd6924c528cbfb578a510d31.1631657157.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When an old bitmap exists while writing a new one, we load it and build a "reposition" table which maps bit positions of objects from the old bitmap to their respective positions in the new bitmap. This can help when we encounter a commit which was selected in both the old and new bitmap, since we only need to permute its bit (not recompute it from scratch). We do not, however, repurpose existing namehash values in the case of the hash-cache extension. There has been thus far no good reason to do so, since all of the namehash values for objects in the new bitmap would be populated during the traversal that was just performed by pack-objects when generating single-pack reachability bitmaps. But this isn't the case for multi-pack bitmaps, which are written via `git multi-pack-index write --bitmap` and do not perform any traversal. In this case all namehash values are set to zero, but we don't even bother to check the `pack.writeBitmapHashcache` option anyway, so it fails to matter. There are two approaches we could take to fill in non-zero hash-cache values: - have either the multi-pack-index builtin run its own traversal to attempt to fill in some values, or let a hypothetical caller (like `pack-objects` when `repack` eventually drives the `multi-pack-index` builtin) fill in the values they found during their traversal - or copy any existing namehash values that were stored in an existing bitmap to their corresponding positions in the new bitmap In a system where a repository is generally repacked with `git repack --geometric=` and occasionally repacked with `git repack -a`, the hash-cache coverage will tend towards all objects. Since populating the hash-cache is additive (i.e., doing so only helps our delta search), any intermediate lack of full coverage is just fine. So let's start by just propagating any values from the existing hash-cache if we see one. The next patch will respect the `pack.writeBitmapHashcache` option while writing MIDX bitmaps, and then test this new behavior. Signed-off-by: Taylor Blau --- pack-bitmap.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 04de387318..33a3732992 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1818,18 +1818,20 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, for (i = 0; i < num_objects; ++i) { struct object_id oid; struct object_entry *oe; + uint32_t index_pos; if (bitmap_is_midx(bitmap_git)) - nth_midxed_object_oid(&oid, - bitmap_git->midx, - pack_pos_to_midx(bitmap_git->midx, i)); + index_pos = pack_pos_to_midx(bitmap_git->midx, i); else - nth_packed_object_id(&oid, bitmap_git->pack, - pack_pos_to_index(bitmap_git->pack, i)); + index_pos = pack_pos_to_index(bitmap_git->pack, i); + nth_bitmap_object_oid(bitmap_git, &oid, index_pos); oe = packlist_find(mapping, &oid); - if (oe) + if (oe) { reposition[i] = oe_in_pack_pos(mapping, oe) + 1; + if (bitmap_git->hashes && !oe->hash) + oe->hash = get_be32(bitmap_git->hashes + index_pos); + } } return reposition; From patchwork Tue Sep 14 22:06:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F5BDC433F5 for ; Tue, 14 Sep 2021 22:06:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5955F61157 for ; Tue, 14 Sep 2021 22:06:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234843AbhINWH1 (ORCPT ); Tue, 14 Sep 2021 18:07:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232561AbhINWH0 (ORCPT ); Tue, 14 Sep 2021 18:07:26 -0400 Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A1AAC061764 for ; Tue, 14 Sep 2021 15:06:08 -0700 (PDT) Received: by mail-io1-xd34.google.com with SMTP id n128so769169iod.5 for ; Tue, 14 Sep 2021 15:06:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=vfhOVs8nbYxdCqob3CH1UqT6XmsPZ60KFKOr7k9nZC4=; b=S0a7mYv45iArycOHQRjhMtp88PhaxMnLL41/MxEOw3ae+0U7EetsJnhnqhm9Uz1xHt SeapC1WouKJJab/UN99oEJWoQUqyLe3grZnwKVyuZTFXBARHYvgP0coTuwyRI5ZAAdwN J9Fi+JgZR7iY/nYP02BVQZq9xKDDuLKX5zQhaHUxcwCHllPw/4Nk+fY5spJN8pn65HL5 wKv+mQqqwrEHqtw7fMhHfWETiil79uN/j4TGmDFXNhsyrP8wHZoYXiL0JMIfPSQUrENH o28/Wo9ACFuTDFyvE6MCLg+BHQVHBkkPzHpKXO+2lfK18Vp1Utm5b7S3yaD8BXCMLIC4 B5Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=vfhOVs8nbYxdCqob3CH1UqT6XmsPZ60KFKOr7k9nZC4=; b=O4xUbdTmogCJaEC/Y51TkotblRB/FiNc5BDBCbnQkaX02ZYeWj8LkrvXMPhQbaOM09 cfvS/kW6WVrrfuZil2nmsS5AS7zDh/onk+Eq6sa5QiKieoi9BQdaSE4T0p6xhyy98Oic sO4l6grBGWKwvgNJNkPQ194mtoU4u1hn+b4PhikV8FU1qGXa+IrqpWQOR0Qw++RpeKiA BDkHT4LmcBUCJBBu5/PNG+kIm45WkElEF6gZjkfHLLLp+aXc7BwVhPOLXP+VSHlYn3C8 lBLV1AOj8ntauWRnzz194K45v4UigmBezvyiBrmkbBwNUssz+0w3nv2VXhudZBxXLONY T1Ww== X-Gm-Message-State: AOAM533DXBcbKVrEqmnCctexCBoXVa7/RsZq1m6cTivVqo6tfKMpSa+5 yaOFMC9JJQ36ljpTLSzjFQ69Aghqm3y5vIB5 X-Google-Smtp-Source: ABdhPJyHBMJiisK3QTyMwO2R0ibhcX41+O0h1v88J3Bxj/edq+tvu+yNmmPXcer6kw9yn5bJ28cQlA== X-Received: by 2002:a6b:f114:: with SMTP id e20mr15569425iog.41.1631657167732; Tue, 14 Sep 2021 15:06:07 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id c11sm6852264ile.22.2021.09.14.15.06.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:07 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:06 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 3/7] midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In the previous commit, the bitmap writing code learned to propagate values from an existing hash-cache extension into the bitmap that it is writing. Now that that functionality exists, let's expose it by teaching the 'git multi-pack-index' builtin to respect the `pack.writeBitmapHashCache` option so that the hash-cache may be written at all. Two minor points worth noting here: - The 'git multi-pack-index write' sub-command didn't previously read any configuration (instead this is handled in the base command). A separate handler is added here to respect this write-specific config option. - I briefly considered adding a 'bitmap_flags' field to the static options struct, but decided against it since it would require plumbing through a new parameter to the write_midx_file() function. Instead, a new MIDX-specific flag is added, which is translated to the corresponding bitmap one. Signed-off-by: Taylor Blau --- Documentation/config/pack.txt | 4 ++++ builtin/multi-pack-index.c | 21 +++++++++++++++++++++ midx.c | 6 +++++- midx.h | 1 + 4 files changed, 31 insertions(+), 1 deletion(-) diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt index 763f7af7c4..ad7f73a1ea 100644 --- a/Documentation/config/pack.txt +++ b/Documentation/config/pack.txt @@ -159,6 +159,10 @@ pack.writeBitmapHashCache:: between an older, bitmapped pack and objects that have been pushed since the last gc). The downside is that it consumes 4 bytes per object of disk space. Defaults to true. ++ +When writing a multi-pack reachability bitmap, no new namehashes are +computed; instead, any namehashes stored in an existing bitmap are +permuted into their appropriate location when writing a new bitmap. pack.writeReverseIndex:: When true, git will write a corresponding .rev file (see: diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 73c0113b48..578ffea6e8 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -61,6 +61,23 @@ static struct option *add_common_options(struct option *prev) return parse_options_concat(common_opts, prev); } +static int git_multi_pack_index_write_config(const char *var, const char *value, + void *cb) +{ + if (!strcmp(var, "pack.writebitmaphashcache")) { + if (git_config_bool(var, value)) + opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE; + else + opts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE; + } + + /* + * We should never make a fall-back call to 'git_default_config', since + * this was already called in 'cmd_multi_pack_index()'. + */ + return 0; +} + static int cmd_multi_pack_index_write(int argc, const char **argv) { struct option *options; @@ -73,6 +90,10 @@ static int cmd_multi_pack_index_write(int argc, const char **argv) OPT_END(), }; + opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE; + + git_config(git_multi_pack_index_write_config, NULL); + options = add_common_options(builtin_multi_pack_index_write_options); trace2_cmd_mode(argv[0]); diff --git a/midx.c b/midx.c index 864034a6ad..38c8600458 100644 --- a/midx.c +++ b/midx.c @@ -1008,9 +1008,13 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash, struct pack_idx_entry **index; struct commit **commits = NULL; uint32_t i, commits_nr; + uint16_t options = 0; char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name, hash_to_hex(midx_hash)); int ret; + if (flags & MIDX_WRITE_BITMAP_HASH_CACHE) + options |= BITMAP_OPT_HASH_CACHE; + prepare_midx_packing_data(&pdata, ctx); commits = find_commits_for_midx_bitmap(&commits_nr, ctx); @@ -1049,7 +1053,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash, goto cleanup; bitmap_writer_set_checksum(midx_hash); - bitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0); + bitmap_writer_finish(index, pdata.nr_objects, bitmap_name, options); cleanup: free(index); diff --git a/midx.h b/midx.h index aa3da557bb..541d9ac728 100644 --- a/midx.h +++ b/midx.h @@ -44,6 +44,7 @@ struct multi_pack_index { #define MIDX_PROGRESS (1 << 0) #define MIDX_WRITE_REV_INDEX (1 << 1) #define MIDX_WRITE_BITMAP (1 << 2) +#define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) const unsigned char *get_midx_checksum(struct multi_pack_index *m); char *get_midx_filename(const char *object_dir); From patchwork Tue Sep 14 22:06:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38B80C433EF for ; Tue, 14 Sep 2021 22:06:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1FD7061157 for ; Tue, 14 Sep 2021 22:06:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235252AbhINWHi (ORCPT ); Tue, 14 Sep 2021 18:07:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235205AbhINWH2 (ORCPT ); Tue, 14 Sep 2021 18:07:28 -0400 Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAD34C061764 for ; Tue, 14 Sep 2021 15:06:10 -0700 (PDT) Received: by mail-il1-x12a.google.com with SMTP id w1so744691ilv.1 for ; Tue, 14 Sep 2021 15:06:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=u4HKy7NV11ULIZEUfMYUHNPx8hTvspNv2zuQZqYUx3A=; b=d8n/UR8lpiY3juv3WXR63f9NssWwhU0DEIpiESuowBJKFITfVyB1AA6EQ/CdI9OVNH j0CQJikuODASqmh0bVFazKK4W/Rgi9GPpyGOhBQBsaATbblHePdAO3NJE17hEZDFnzNa PuObzJsguWKxD6rLzQnJ3e4dXC76dmR0W5cbCoVRxgs4tVSY29L13jcda9Rg/CCNqGxZ JtT96r9VT3yVEGY6PEk9UBJGVoSHgaHp704AgBuFJJUwRWGOKWikUfnYeOAHx7XKb2/c osZP+N85FQzuvI+QYzH3YxTYIOdEmlWGtpN+SJ+m/JO1QmiXV/HrgLDTZ03i/W5ZgsHw 5ljw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=u4HKy7NV11ULIZEUfMYUHNPx8hTvspNv2zuQZqYUx3A=; b=cFbxCClAC7FSXqBTv+NUl6ec9UWK/DWk7DdmIreu8GokS38aZO6txrd5i11IPFVOfL 9bE6YPUB9b3twNQY0J5cjYIMk72607SWn14j7j05jBlQ0z0pUXTaOjtUb3262KJ1ZzpL 3aCAiRHn3YQRv3RWHJzh+TNVRbDnAqfP8aVRel6LAqJg/PPn8IZRdPCl4DaHo7Lgicog gM0FnbOlMqg9lf7psjiQVQmBq6F7lEOoLcOr9fSr4uc8fMPvwKS17Egp3tb+p0TLtLRu 95C7rAYDYRDGGr8a/vLDx1ahCmjc0wicS9y68wrSdNcyhuzRl8LagT2/z0UkYQh6snLZ VLQQ== X-Gm-Message-State: AOAM530crg2fhfPWAdIczDoeaVdL3rbEFziE1V1b1kd3o+ayL9BIJPJL aUZTFu0kL5PJgLBX0xE/bEWkpexsGwwo9dL8 X-Google-Smtp-Source: ABdhPJyISUsaU91gbdP1gRy/McL5ZAcxCraQQcCVOzHvmNHOoccf1uF7kJONA5iDEYIzTeyBbLzQDA== X-Received: by 2002:a92:c744:: with SMTP id y4mr13246635ilp.288.1631657170067; Tue, 14 Sep 2021 15:06:10 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id w1sm5387135ilj.55.2021.09.14.15.06.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:09 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:09 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 4/7] p5326: create missing 'perf-tag' tag Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Some of the tests in test_full_bitmap rely on having a tag named perf-tag in place. We could create it in test_full_bitmap(), but we want to have it in place before the repack starts. Signed-off-by: Taylor Blau --- t/perf/p5326-multi-pack-bitmaps.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh index 5845109ac7..51b5636259 100755 --- a/t/perf/p5326-multi-pack-bitmaps.sh +++ b/t/perf/p5326-multi-pack-bitmaps.sh @@ -10,6 +10,12 @@ test_expect_success 'enable multi-pack index' ' git config core.multiPackIndex true ' +# we need to create the tag up front such that it is covered by the repack and +# thus by generated bitmaps. +test_expect_success 'create tags' ' + git tag --message="tag pointing to HEAD" perf-tag HEAD +' + test_perf 'setup multi-pack index' ' git repack -ad && git multi-pack-index write --bitmap From patchwork Tue Sep 14 22:06:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2F99C433FE for ; Tue, 14 Sep 2021 22:06:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8C23761157 for ; Tue, 14 Sep 2021 22:06:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235255AbhINWHj (ORCPT ); Tue, 14 Sep 2021 18:07:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235350AbhINWHh (ORCPT ); Tue, 14 Sep 2021 18:07:37 -0400 Received: from mail-il1-x12c.google.com (mail-il1-x12c.google.com [IPv6:2607:f8b0:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 486D9C0613E1 for ; Tue, 14 Sep 2021 15:06:13 -0700 (PDT) Received: by mail-il1-x12c.google.com with SMTP id a20so706608ilq.7 for ; Tue, 14 Sep 2021 15:06:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=dViVSMM9wxtKjXJwwl3RE2HtW0ysTkcAVhFYfSvCLx8=; b=kFa6OhapByq9+1aD1mKjKvkO1iYesAmoPNm9DTwOQ0PHw0aN4wlsq10gWexd/HtGx7 iGwjsWu4xlcLU+W9TsMevPU353ObsokUpgR78sOLcsEHLbtCKWeLYpdJF558LRNrhUWM Vpd5GVOvjm15WIAVpJp/79FcO5p3+qR8COVIxxtzi1UyYmHtms0cRr2GqP6Z5GY0jqOX I8PSp+K65IZcaN9BIZ6aSE6dFex2Cw8JO96kMclCBL9bmhlI7z9Li9zjVAa8OEfqqnYT /Gsx8Y5MQkZqmwvPH3HCtMcXTMJm031gHqjz0vFDrkvvy5YixkEwfwoWdIo3WfpQidA4 zOZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=dViVSMM9wxtKjXJwwl3RE2HtW0ysTkcAVhFYfSvCLx8=; b=BP8dwOBYv83jvF7J9qJZxJzHAgmmVVzsgXi2sYu3fH+zB8scgZ/8CHUpQeSznEEM3d KnNG6dP1AOcIY9c5U6cFtyWpu+N1Iybbu/e7aWGLn9T9RAZccnOWF+JA2R7G3C6x5J6v fdDIL5sP0g1HK0IlYum1CqHbaUhD7yT8axwAVG/y1TLavI6kv2RSm4GFlQSadBDSpAk1 minrvY5XloAwV+PF+06ffrSF7SGC58eXxS0xf37u2+IGRNF0ssKlb+IMYiliXHGqZffq 2Mixrs/Q68AKqEQfBCopIJTtQDtwLKD7raqc1P50ScKQnroFshxU1zyFsfavk3iopSJ4 hXCw== X-Gm-Message-State: AOAM530CZmG9u/yBDUjT+VvxxAOwnhRQ0xh4PJrz3qPgp40EaO7nI2wy O5DWBrruncHIUU0MHgfuefgsaImZFx1fSIBX X-Google-Smtp-Source: ABdhPJw1dOae6t6AXCn/qnLPuFFxeFJiXp96oaIjzJJYPFFnQhGcg0q+07U0ujXAGBFAev2KBLOI7Q== X-Received: by 2002:a92:870b:: with SMTP id m11mr13904531ild.132.1631657172484; Tue, 14 Sep 2021 15:06:12 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id y11sm1680656ilh.73.2021.09.14.15.06.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:12 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:11 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 5/7] p5326: don't set core.multiPackIndex unnecessarily Message-ID: <191922c8f28c338ab3aa4faa57d75770a9eccd02.1631657157.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When this performance test was originally written, `core.multiPackIndex` was not the default and thus had to be enabled. But now that we have 18e449f86b (midx: enable core.multiPackIndex by default, 2020-09-25), we no longer need this. Drop the unnecessary setup (even though it's not hurting anything, it is unnecessary at best and confusing at worst). Signed-off-by: Taylor Blau --- t/perf/p5326-multi-pack-bitmaps.sh | 4 ---- 1 file changed, 4 deletions(-) diff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh index 51b5636259..a9c5499537 100755 --- a/t/perf/p5326-multi-pack-bitmaps.sh +++ b/t/perf/p5326-multi-pack-bitmaps.sh @@ -6,10 +6,6 @@ test_description='Tests performance using midx bitmaps' test_perf_large_repo -test_expect_success 'enable multi-pack index' ' - git config core.multiPackIndex true -' - # we need to create the tag up front such that it is covered by the repack and # thus by generated bitmaps. test_expect_success 'create tags' ' From patchwork Tue Sep 14 22:06:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D504C433EF for ; Tue, 14 Sep 2021 22:06:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4322061157 for ; Tue, 14 Sep 2021 22:06:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235299AbhINWHk (ORCPT ); Tue, 14 Sep 2021 18:07:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235205AbhINWHi (ORCPT ); Tue, 14 Sep 2021 18:07:38 -0400 Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83DFBC0613E3 for ; Tue, 14 Sep 2021 15:06:15 -0700 (PDT) Received: by mail-io1-xd2f.google.com with SMTP id b7so776428iob.4 for ; Tue, 14 Sep 2021 15:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ghVbd1GBYARFJUxpuM6vPGFayOKq7w19PilPKv0KbVM=; b=pzZcc+yywbzNtq1oUeuUHmBxv1NfucUEr3yGrFdaa79WztnjfX6aUSc0U0pl/dcoCM vBKntqFRywnTSBwMUv1IVCJdJMoONW6ZWi8PiHVT4udsZh3DYgM4Pa8Jm70LGzWIGvaS RLpvqFiM12sKFYLEalp7vwj+xaM47m9XUKL/+b/0awUJjD6GF6rmUhaZMK6qLjlWgEJD 6nBSwA3gCknJW5nHn3lqs37h2IB8rIWWVCNxQnHd5a1RwmZmA4Y37y6fpiOjXtbwCI3t uzTTprcY2LqDVtPdT3NORP44LPbuyzdvIN2cSI5KlVORzQhJpX8TsdD/PIkHJ+QN6yvx dv2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ghVbd1GBYARFJUxpuM6vPGFayOKq7w19PilPKv0KbVM=; b=Vmndy4uRKVfA/39WlE+FBygltp60ZTtRTtSy+VMnNFxrgKHgYcY8cfdcrHcxp5MQAZ kBufNgBdSYHl3GrJT4Xj12R92pGM1qX3asL9+ECpkseUmgdnliY1DeXB56SSMBXYcaQy iJ2fyem1yAjWe90g3XbaJz57wjUkQ6dLl7eb0Tl6Fyb8oaA/M0ufrZNFoSqwkkNAuk0b CmEDmULmHkyeNduf0dlPoBfhqtDp7egUn7mnuuFQDC71yKaip/hSsg0I9ID5PMoqhAq2 jlfAg86RPlUQXqvnsmAoTOjAFkKvy26LQHVBbfOrfpI5deZcEi0vSs58F/0zMnv2sTCX P/1w== X-Gm-Message-State: AOAM530vaIJIzyBTwtXxpvYE+9reizSWMovwzv1iBk3BydqKhl3pRaMG oHHXhF+AhUXP2ugAIsNweA7q8gww1x3FVUo+ X-Google-Smtp-Source: ABdhPJyGnuWloC9gWOweQ3runa4j9T9Nk+Lb+N+Rqt0sljpiJ6dxa1XwkXZ1PsjGVI9pDWW15VmyOA== X-Received: by 2002:a5d:9c89:: with SMTP id p9mr15398843iop.13.1631657174824; Tue, 14 Sep 2021 15:06:14 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id x8sm7524944ilc.40.2021.09.14.15.06.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:14 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:14 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 6/7] p5326: generate pack bitmaps before writing the MIDX bitmap Message-ID: <040bb40548017bae807c1d349fa078c21ac46725.1631657157.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org To help test the performance of permuting the contents of the hash-cache when generating a MIDX bitmap, we need a bitmap which has its hash-cache populated. And since multi-pack bitmaps don't add *new* values to the hash-cache, we have to rely on a single-pack bitmap to generate those values for us. Therefore, generate a pack bitmap before the MIDX one in order to ensure that the MIDX bitmap has entries in its hash-cache. Signed-off-by: Taylor Blau --- t/perf/p5326-multi-pack-bitmaps.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh index a9c5499537..38557859b7 100755 --- a/t/perf/p5326-multi-pack-bitmaps.sh +++ b/t/perf/p5326-multi-pack-bitmaps.sh @@ -13,7 +13,7 @@ test_expect_success 'create tags' ' ' test_perf 'setup multi-pack index' ' - git repack -ad && + git repack -adb && git multi-pack-index write --bitmap ' From patchwork Tue Sep 14 22:06:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12494963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5B63C433FE for ; Tue, 14 Sep 2021 22:06:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 860CA6113B for ; Tue, 14 Sep 2021 22:06:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235274AbhINWHl (ORCPT ); Tue, 14 Sep 2021 18:07:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235254AbhINWHi (ORCPT ); Tue, 14 Sep 2021 18:07:38 -0400 Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E539CC0613E4 for ; Tue, 14 Sep 2021 15:06:17 -0700 (PDT) Received: by mail-il1-x131.google.com with SMTP id b6so789472ilv.0 for ; Tue, 14 Sep 2021 15:06:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=QqoO9mYcks/QS/cXaESVnH2zQ7d2jbHdYDnIhx9CVn0=; b=i7ymTNSgHHqM3ssZW6ydhIvAS4snXaCjqp3XURyq9e9e76pxHayhZ7nCyVJJNykMew 7vXKOCc5dCIr099HvR3yb9F8mk/do/H/jZJCmLgtas3v1UveF1mB5V4r6qci/kRa5E2l AiQE6h5n/cO69PDjwiK0v3DvLOrITRogsEOgUkI31YL3Wg8MyjVXBeGI51yf5y4b+u7d S2RCTzEdVgiXrZxL/S3avPKmDGRGwqr0DSNxWTD5Es1jhrO6+p/dTLM5oQ42X+L17KVp H1+H2wTSRkGBN1nhgOLrLrTQFHC7t6h8QOHUbs3+hT60cbE5/7uePdc5wGFeYEPhCZSm lhmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=QqoO9mYcks/QS/cXaESVnH2zQ7d2jbHdYDnIhx9CVn0=; b=eqx+lk48NErG6O3ExohUZeq4tux4iYaUVisAMt559HPU5U98t1PFE8jOHzRs+bB7+F cYHjagIbPoQNlj3G8KtEDtX/pc6eGX9Dj0jtMYgTMJtzSYVTPfepTuEc27B158Zr237F up/9f+PelLLi0DnC9q0MaWz7SUDVO2Uc5u9XiOd859vwg2xMQs5xIEw356sOURRQsy0S TxvCPxvNYOrS3Dhp1WooCDf96ZTDXhajbSV0FaaFMTl7fSZw2bckiJdwHFaF5SYHrBMb HP7olxgE3CBS4DdhJ6j5GoqLJuBTWXA8eSnIe6cG9nLiFCYuBNJN+fd/gD4/BJCNq0iE xQ+w== X-Gm-Message-State: AOAM5316KvA2hUj64VavynDiY99AtsI3akvPWTFj3TbMtnqK4ZAly8RV v/5LyLzeSkkZXEFh/saStzT+TzIZW4QzuMDu X-Google-Smtp-Source: ABdhPJx43V8+HF21Y3wgclOWjmu6CF3YcDF6an0mRPZ/VPpT1tF4AQ2cvsT19YvhFNcEjZwXxiuDng== X-Received: by 2002:a05:6e02:20e7:: with SMTP id q7mr14399863ilv.212.1631657177202; Tue, 14 Sep 2021 15:06:17 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id s10sm7466316iom.40.2021.09.14.15.06.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 15:06:16 -0700 (PDT) Date: Tue, 14 Sep 2021 18:06:16 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, gitster@pobox.com, avarab@gmail.com Subject: [PATCH v2 7/7] t5326: test propagating hashcache values Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Now that we both can propagate values from the hashcache, and respect the configuration to enable the hashcache at all, test that both of these function correctly by hardening their behavior with a test. Like the hash-cache in classic single-pack bitmaps, this helps more proportionally the more up-to-date your bitmap coverage is. When our bitmap coverage is out-of-date with the ref tips, we spend more time proportionally traversing, and all of that traversal gets the name-hash filled in. But for the up-to-date bitmaps, this helps quite a bit. These numbers are on git.git, with `pack.threads=1` to help see the difference reflected in the overall runtime. Test origin/tb/multi-pack-bitmaps HEAD ------------------------------------------------------------------------------------- 5326.4: simulated clone 1.87(1.80+0.07) 1.46(1.42+0.03) -21.9% 5326.5: simulated fetch 2.66(2.61+0.04) 1.47(1.43+0.04) -44.7% 5326.6: pack to file (bitmap) 2.74(2.62+0.12) 1.89(1.82+0.07) -31.0% Signed-off-by: Taylor Blau --- t/t5326-multi-pack-bitmaps.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4ad7c2c969..24148ca35b 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -283,4 +283,36 @@ test_expect_success 'pack.preferBitmapTips' ' ) ' +test_expect_success 'hash-cache values are propagated from pack bitmaps' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + git config pack.writeBitmapHashCache true && + + test_commit base && + test_commit base2 && + git repack -adb && + + test-tool bitmap dump-hashes >pack.raw && + test_file_not_empty pack.raw && + sort pack.raw >pack.hashes && + + test_commit new && + git repack && + git multi-pack-index write --bitmap && + + test-tool bitmap dump-hashes >midx.raw && + sort midx.raw >midx.hashes && + + # ensure that every namehash in the pack bitmap can be found in + # the midx bitmap (i.e., that there are no oid-namehash pairs + # unique to the pack bitmap). + comm -23 pack.hashes midx.hashes >dropped.hashes && + test_must_be_empty dropped.hashes + ) +' + test_done