From patchwork Fri Apr 9 18:10:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D6CC433B4 for ; Fri, 9 Apr 2021 18:10:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AD2CC61007 for ; Fri, 9 Apr 2021 18:10:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234481AbhDISK5 (ORCPT ); Fri, 9 Apr 2021 14:10:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233332AbhDISKz (ORCPT ); Fri, 9 Apr 2021 14:10:55 -0400 Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2E7BC061761 for ; Fri, 9 Apr 2021 11:10:42 -0700 (PDT) Received: by mail-il1-x129.google.com with SMTP id 6so5436789ilt.9 for ; Fri, 09 Apr 2021 11:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=W75OvUzV1Ne5gy9b3QjGrVR36zju90e4pT7ovIYRPkg=; b=Et0SkSaDZJUl/PJimgsyRpv1cjtLKBtjvlz2dQgU07u3NM0fIZQ4XiJ/pCv9ybETbq B9+NWU8EKzl3OxxnRmG4iYkZX2t93wYhOOkv4l3YXfoTHfeHgc3CBE2DnaTYQHBbba0C dW3nSHnHS9zktmBq4pTDA7xhakyRCJpiBisdmhM2OugcRVUGy8V0Pq8rMFk9WlX89w5F vpVAL8ErWaTV40VkA/sQxl0Z5LQXPKa2nsBImgOv9IiJVQ67AwvV4Cr9LRZoleA+4/BW YOuPRGKQm+0PwA1P3J8BtJejelVvMJ6AGa9JzVUQ9SR+oesM+y23xaRS9ptnrmCcLPZV t0FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=W75OvUzV1Ne5gy9b3QjGrVR36zju90e4pT7ovIYRPkg=; b=pyueVD4ykLyyX+G4wK09PYA7XZenoFWBRIE2A/qZeq2xQVDHKIUZdkGvDpjya2IKiL sX/SbT6+JSzEOWS6VH5qYI0t/VRnnNRPWsctN6j1cW8gMAd5D8Amz3YpBPONsdwELKPB RTJRBptnrf5bn4+fGiXxGH9u7FFLJlGvut8eInzL0mPLF978kxcKiueCOrAcTndOPkyV 9hD2j6EyOmlc5edmtNO90kivcmMuNLO8tNKc0WImCoFnUJj9QUO2N9mFsjKTR4p5n3nq zTBWYZpcLJZvKTwkjTdfV42aAt7ykNMDtEtNEEbKu/eXYLIlaiV/m/YVpTSN+BN7Sm03 dSQg== X-Gm-Message-State: AOAM531KLyk6i6IUlYJgV/jX0LuxQU8dYIDXmRm7FXsvK22pyoZgfmmk qk3XkdRE7DS1VJfDTxt4zf6WnVWEVv7BAP8O X-Google-Smtp-Source: ABdhPJw5p7gIyFh0eqLvqrrUIHQ4XlCS/HEb5VnGBYkPu03Dv0/1nqhMsvfwRubOMLCve6mvYhVTFQ== X-Received: by 2002:a05:6e02:198f:: with SMTP id g15mr12255895ilf.200.1617991841930; Fri, 09 Apr 2021 11:10:41 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id q21sm1594917ioh.41.2021.04.09.11.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:10:41 -0700 (PDT) Date: Fri, 9 Apr 2021 14:10:39 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 01/22] pack-bitmap.c: harden 'test_bitmap_walk()' to check type bitmaps Message-ID: <2d1c6ccab5e2feaa4db1bcd08178a698b3e00241.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The special `--test-bitmap` mode of `git rev-list` is used to compare the result of an object traversal with a bitmap to check its integrity. This mode does not, however, assert that the types of reachable objects are stored correctly. Harden this mode by teaching it to also check that each time an object's bit is marked, the corresponding bit should be set in exactly one of the type bitmaps (whose type matches the object's true type). Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- pack-bitmap.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/pack-bitmap.c b/pack-bitmap.c index 3ed15431cd..d45e91db1e 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1263,10 +1263,52 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git, struct bitmap_test_data { struct bitmap_index *bitmap_git; struct bitmap *base; + struct bitmap *commits; + struct bitmap *trees; + struct bitmap *blobs; + struct bitmap *tags; struct progress *prg; size_t seen; }; +static void test_bitmap_type(struct bitmap_test_data *tdata, + struct object *obj, int pos) +{ + enum object_type bitmap_type = OBJ_NONE; + int bitmaps_nr = 0; + + if (bitmap_get(tdata->commits, pos)) { + bitmap_type = OBJ_COMMIT; + bitmaps_nr++; + } + if (bitmap_get(tdata->trees, pos)) { + bitmap_type = OBJ_TREE; + bitmaps_nr++; + } + if (bitmap_get(tdata->blobs, pos)) { + bitmap_type = OBJ_BLOB; + bitmaps_nr++; + } + if (bitmap_get(tdata->tags, pos)) { + bitmap_type = OBJ_TAG; + bitmaps_nr++; + } + + if (!bitmap_type) + die("object %s not found in type bitmaps", + oid_to_hex(&obj->oid)); + + if (bitmaps_nr > 1) + die("object %s does not have a unique type", + oid_to_hex(&obj->oid)); + + if (bitmap_type != obj->type) + die("object %s: real type %s, expected: %s", + oid_to_hex(&obj->oid), + type_name(obj->type), + type_name(bitmap_type)); +} + static void test_show_object(struct object *object, const char *name, void *data) { @@ -1276,6 +1318,7 @@ static void test_show_object(struct object *object, const char *name, bitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid); if (bitmap_pos < 0) die("Object not in bitmap: %s\n", oid_to_hex(&object->oid)); + test_bitmap_type(tdata, object, bitmap_pos); bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); @@ -1290,6 +1333,7 @@ static void test_show_commit(struct commit *commit, void *data) &commit->object.oid); if (bitmap_pos < 0) die("Object not in bitmap: %s\n", oid_to_hex(&commit->object.oid)); + test_bitmap_type(tdata, &commit->object, bitmap_pos); bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); @@ -1337,6 +1381,10 @@ void test_bitmap_walk(struct rev_info *revs) tdata.bitmap_git = bitmap_git; tdata.base = bitmap_new(); + tdata.commits = ewah_to_bitmap(bitmap_git->commits); + tdata.trees = ewah_to_bitmap(bitmap_git->trees); + tdata.blobs = ewah_to_bitmap(bitmap_git->blobs); + tdata.tags = ewah_to_bitmap(bitmap_git->tags); tdata.prg = start_progress("Verifying bitmap entries", result_popcnt); tdata.seen = 0; From patchwork Fri Apr 9 18:10:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEDC5C43460 for ; Fri, 9 Apr 2021 18:10:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A992061007 for ; Fri, 9 Apr 2021 18:10:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234512AbhDISLD (ORCPT ); Fri, 9 Apr 2021 14:11:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234499AbhDISLC (ORCPT ); Fri, 9 Apr 2021 14:11:02 -0400 Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6084CC061764 for ; Fri, 9 Apr 2021 11:10:48 -0700 (PDT) Received: by mail-qk1-x730.google.com with SMTP id q26so6725174qkm.6 for ; Fri, 09 Apr 2021 11:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=R0zebXQCWK+dh07tKQkqnIsSb1QbSNRZWJ8lqTQRcPU=; b=l8IeaBXsKNP08/jY9NaN/c4hEp9V7vOHeorOhVXZMmihGK4Qbnkk1/wWh2RECHz+Qb 14zstD6cKBR9PZM2ZVMLizKCsE0UdC6h81gNDkfQsnyutx/rVyZi/2DoX0iZFZOQhmhc +CiUwx7kVLyXymioiLCRbu73nupjbpVeT7S/RNVhAe8675YWyckRUMKbJ46bpl9jj3ht baSXz+MvMyGYKRhXRuNoqgB1Xr9mfAnd8CA4vmUSTzclzldSOn87G3sonue6iXyqC+jE mt7yZOq51nWAXEnLO1ADxl6HVkVU9b/6bVKwWSZzlDkOM2C4PzBMKw+TXuqhLnB7nMm/ p1XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=R0zebXQCWK+dh07tKQkqnIsSb1QbSNRZWJ8lqTQRcPU=; b=lO0b95LeZELGUekMrta35DqHXAL6ruA0AY8uhGdr40qdk+qATvU7rP4DOQOO2yMd/D 0Ihg7VVWPUJwa/JoirzQ1CRtE8HyhtxwI/Aq0jbhuv6CnbrO2CQZ2bU9lEPzNNUlVR1Y fdWh/vlE1YnNZ7up8dtvDjdapikY5QCz+/WgKUQL3QFr3csHyI1QAsLnCyJ+RAzfNox9 JhCToHZwqfrKqD+pFu9lNlwlIEMpGzrIJwjEV4ocKBoBMoMzMK7jjFYmcG4TZPByilnV HrSHkHVOufGbYgfMtrFiI1WeElOQcCs2iO29+HRdc3Nv8OdAMSDTnZLA+MwMMRlH7CIW QNqw== X-Gm-Message-State: AOAM530IEuE2Hd7mM59gBnp9g3v0w4RTF2yrxvQINyWan5Flu3QtgHjW 7SWqSEOKY8hlCkkCZ1vW3O4Pt4H+Z7ThwD10 X-Google-Smtp-Source: ABdhPJzMDCiPBiwecReZn7xblvshOZAb2Go41AVYkaIPDXvYmTfUB+Aw2vQVAvHIsILBUNA2YC/IOw== X-Received: by 2002:a37:43c2:: with SMTP id q185mr9085997qka.224.1617991847109; Fri, 09 Apr 2021 11:10:47 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id 1sm2529863qtu.96.2021.04.09.11.10.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:10:46 -0700 (PDT) Date: Fri, 9 Apr 2021 14:10:44 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 02/22] pack-bitmap-write.c: gracefully fail to write non-closed bitmaps Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The set of objects covered by a bitmap must be closed under reachability, since it must be the case that there is a valid bit position assigned for every possible reachable object (otherwise the bitmaps would be incomplete). Pack bitmaps are never written from 'git repack' unless repacking all-into-one, and so we never write non-closed bitmaps. But multi-pack bitmaps change this, since it isn't known whether the set of objects in the MIDX is closed under reachability until walking them. Plumb through a bit that is set when a reachable object isn't found. As soon as a reachable object isn't found in the set of objects to include in the bitmap, bitmap_writer_build() knows that the set is not closed, and so it now fails gracefully. (The new conditional in builtin/pack-objects.c:bitmap_writer_build() guards against other failure modes, but is never triggered here, because of the all-into-one detail above. This return value will be important to check from the multi-pack index caller.) Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 3 +- pack-bitmap-write.c | 76 +++++++++++++++++++++++++++++------------- pack-bitmap.h | 2 +- 3 files changed, 56 insertions(+), 25 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index a1e33d7507..5205dde2e1 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1116,7 +1116,8 @@ static void write_pack_file(void) bitmap_writer_show_progress(progress); bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1); - bitmap_writer_build(&to_pack); + if (bitmap_writer_build(&to_pack) < 0) + die(_("failed to write bitmap index")); bitmap_writer_finish(written_list, nr_written, tmpname.buf, write_bitmap_options); write_bitmap_index = 0; diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 88d9e696a5..e829c46649 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -125,15 +125,20 @@ static inline void push_bitmapped_commit(struct commit *commit) writer.selected_nr++; } -static uint32_t find_object_pos(const struct object_id *oid) +static uint32_t find_object_pos(const struct object_id *oid, int *found) { struct object_entry *entry = packlist_find(writer.to_pack, oid); if (!entry) { - die("Failed to write bitmap index. Packfile doesn't have full closure " + if (found) + *found = 0; + warning("Failed to write bitmap index. Packfile doesn't have full closure " "(object %s is missing)", oid_to_hex(oid)); + return 0; } + if (found) + *found = 1; return oe_in_pack_pos(writer.to_pack, entry); } @@ -331,9 +336,10 @@ static void bitmap_builder_clear(struct bitmap_builder *bb) bb->commits_nr = bb->commits_alloc = 0; } -static void fill_bitmap_tree(struct bitmap *bitmap, - struct tree *tree) +static int fill_bitmap_tree(struct bitmap *bitmap, + struct tree *tree) { + int found; uint32_t pos; struct tree_desc desc; struct name_entry entry; @@ -342,9 +348,11 @@ static void fill_bitmap_tree(struct bitmap *bitmap, * If our bit is already set, then there is nothing to do. Both this * tree and all of its children will be set. */ - pos = find_object_pos(&tree->object.oid); + pos = find_object_pos(&tree->object.oid, &found); + if (!found) + return -1; if (bitmap_get(bitmap, pos)) - return; + return 0; bitmap_set(bitmap, pos); if (parse_tree(tree) < 0) @@ -355,11 +363,15 @@ static void fill_bitmap_tree(struct bitmap *bitmap, while (tree_entry(&desc, &entry)) { switch (object_type(entry.mode)) { case OBJ_TREE: - fill_bitmap_tree(bitmap, - lookup_tree(the_repository, &entry.oid)); + if (fill_bitmap_tree(bitmap, + lookup_tree(the_repository, &entry.oid)) < 0) + return -1; break; case OBJ_BLOB: - bitmap_set(bitmap, find_object_pos(&entry.oid)); + pos = find_object_pos(&entry.oid, &found); + if (!found) + return -1; + bitmap_set(bitmap, pos); break; default: /* Gitlink, etc; not reachable */ @@ -368,15 +380,18 @@ static void fill_bitmap_tree(struct bitmap *bitmap, } free_tree_buffer(tree); + return 0; } -static void fill_bitmap_commit(struct bb_commit *ent, - struct commit *commit, - struct prio_queue *queue, - struct prio_queue *tree_queue, - struct bitmap_index *old_bitmap, - const uint32_t *mapping) +static int fill_bitmap_commit(struct bb_commit *ent, + struct commit *commit, + struct prio_queue *queue, + struct prio_queue *tree_queue, + struct bitmap_index *old_bitmap, + const uint32_t *mapping) { + int found; + uint32_t pos; if (!ent->bitmap) ent->bitmap = bitmap_new(); @@ -401,11 +416,16 @@ static void fill_bitmap_commit(struct bb_commit *ent, * Mark ourselves and queue our tree. The commit * walk ensures we cover all parents. */ - bitmap_set(ent->bitmap, find_object_pos(&c->object.oid)); + pos = find_object_pos(&c->object.oid, &found); + if (!found) + return -1; + bitmap_set(ent->bitmap, pos); prio_queue_put(tree_queue, get_commit_tree(c)); for (p = c->parents; p; p = p->next) { - int pos = find_object_pos(&p->item->object.oid); + pos = find_object_pos(&p->item->object.oid, &found); + if (!found) + return -1; if (!bitmap_get(ent->bitmap, pos)) { bitmap_set(ent->bitmap, pos); prio_queue_put(queue, p->item); @@ -413,8 +433,12 @@ static void fill_bitmap_commit(struct bb_commit *ent, } } - while (tree_queue->nr) - fill_bitmap_tree(ent->bitmap, prio_queue_get(tree_queue)); + while (tree_queue->nr) { + if (fill_bitmap_tree(ent->bitmap, + prio_queue_get(tree_queue)) < 0) + return -1; + } + return 0; } static void store_selected(struct bb_commit *ent, struct commit *commit) @@ -432,7 +456,7 @@ static void store_selected(struct bb_commit *ent, struct commit *commit) kh_value(writer.bitmaps, hash_pos) = stored; } -void bitmap_writer_build(struct packing_data *to_pack) +int bitmap_writer_build(struct packing_data *to_pack) { struct bitmap_builder bb; size_t i; @@ -441,6 +465,7 @@ void bitmap_writer_build(struct packing_data *to_pack) struct prio_queue tree_queue = { NULL }; struct bitmap_index *old_bitmap; uint32_t *mapping; + int closed = 1; /* until proven otherwise */ writer.bitmaps = kh_init_oid_map(); writer.to_pack = to_pack; @@ -463,8 +488,11 @@ void bitmap_writer_build(struct packing_data *to_pack) struct commit *child; int reused = 0; - fill_bitmap_commit(ent, commit, &queue, &tree_queue, - old_bitmap, mapping); + if (fill_bitmap_commit(ent, commit, &queue, &tree_queue, + old_bitmap, mapping) < 0) { + closed = 0; + break; + } if (ent->selected) { store_selected(ent, commit); @@ -499,7 +527,9 @@ void bitmap_writer_build(struct packing_data *to_pack) stop_progress(&writer.progress); - compute_xor_offsets(); + if (closed) + compute_xor_offsets(); + return closed; } /** diff --git a/pack-bitmap.h b/pack-bitmap.h index 78f2b3ff79..988ed3a30d 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -86,7 +86,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, struct commit *commit); void bitmap_writer_select_commits(struct commit **indexed_commits, unsigned int indexed_commits_nr, int max_bitmaps); -void bitmap_writer_build(struct packing_data *to_pack); +int bitmap_writer_build(struct packing_data *to_pack); void bitmap_writer_finish(struct pack_idx_entry **index, uint32_t index_nr, const char *filename, From patchwork Fri Apr 9 18:10:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE74EC433ED for ; Fri, 9 Apr 2021 18:11:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87B7661181 for ; Fri, 9 Apr 2021 18:11:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234597AbhDISLP (ORCPT ); Fri, 9 Apr 2021 14:11:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234579AbhDISLL (ORCPT ); Fri, 9 Apr 2021 14:11:11 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 517C6C061764 for ; Fri, 9 Apr 2021 11:10:58 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id e8so6827415iok.5 for ; Fri, 09 Apr 2021 11:10:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BupGobCfypnzxGBp5+C0pQPe2do9EyXqzE5iYIKyKII=; b=lFVxaCRoDoaBG38mJ8l/fJI02L7Cl83kziuV2Fj+eFauImSRUFT7HBfO2YdwbTufEJ rb5+yj8si0yiOAb97krbEtdd5XI29UGbYqFjwtHG7SVmbl5JSm8TldoQrxuGw9zOslsI B2gq24M6FTVYjQElABpvvRy73EA82q2XZkPOc1waYzAOm82fO4G6titjDN5brBlGmdWw WTFsCVZNRDB4psawoX7Z7JbObe26lGSzW+WFmZFpMtLQJH3hhk9w32qwlfztSK5xUtFy Bg4mwwND/cXbIL9PapGXbDtx1Kl0Fg2s4/06Km3Slc1MBVhWuTZdjgkrIdIR4pmi8eB2 E3hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BupGobCfypnzxGBp5+C0pQPe2do9EyXqzE5iYIKyKII=; b=URRQ+KScnSfJ99qNNi1WEUvE29n06870GeqQDIt6bMBCmc/rllcHOedMKTerlu3GhJ WrVvPvzp776F5uQaLFiyGhhNXwHC5BznAaAmBrIP6hNmzwUdZVU3y6CmvBw8OhJCqB7f ix+Z9CejNcv0jGSgKvSprzKbvA/gKmadFyg7z8V/wZCTYgvK7yF35q0iAVXOfRbVLymr Tyykv+KIBpdsE8bbHIGfFxsxwyB9bGBolB5oyIm2MlP5VdL+2SzuD8WLN4h8BPUvAuZM WC5vstYBAAFBf1ghKwfLvP3/rO9LCrzJmLSQZ6gUTrla1UfVmx+so1pKP/EtSHMSEDlU Oh9w== X-Gm-Message-State: AOAM532G13jFbEIcTiIaB+hvyJKy/wLJDjje+DknkmpNyfynwOqCoi4k OU/dz71xh88DmkwgbXSme5ueVPB0oU1+iRkD X-Google-Smtp-Source: ABdhPJzA+MvnYlvJibxhrGTM2KSAEVAUOUY8d3Md2ZsNjSpPngZdW0Rokazhdl8JrmAMwmkp7cEyKQ== X-Received: by 2002:a5e:dd03:: with SMTP id t3mr30792iop.199.1617991857599; Fri, 09 Apr 2021 11:10:57 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id n11sm1434411ioz.25.2021.04.09.11.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:10:57 -0700 (PDT) Date: Fri, 9 Apr 2021 14:10:48 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 03/22] pack-bitmap-write.c: free existing bitmaps Message-ID: <014c18b8965fda2c5feb4ec5c4fe9d82f213a744.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing a new bitmap, the bitmap writer code attempts to read the existing bitmap (if one is present). This is done in order to quickly permute the bits of any bitmaps for commits which appear in the existing bitmap, and were also selected for the new bitmap. But since this code was added in 341fa34887 (pack-bitmap-write: use existing bitmaps, 2020-12-08), the resources associated with opening an existing bitmap were never released. It's fine to ignore this, but it's bad hygiene. It will also cause a problem for the multi-pack-index builtin, which will be responsible not only for writing bitmaps, but also for expiring any old multi-pack bitmaps. If an existing bitmap was reused here, it will also be expired. That will cause a problem on platforms which require file resources to be closed before unlinking them, like Windows. Avoid this by ensuring we close reused bitmaps with free_bitmap_index() before removing them. Signed-off-by: Taylor Blau --- pack-bitmap-write.c | 1 + 1 file changed, 1 insertion(+) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index e829c46649..f90e100e3e 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -520,6 +520,7 @@ int bitmap_writer_build(struct packing_data *to_pack) clear_prio_queue(&queue); clear_prio_queue(&tree_queue); bitmap_builder_clear(&bb); + free_bitmap_index(old_bitmap); free(mapping); trace2_region_leave("pack-bitmap-write", "building_bitmaps_total", From patchwork Fri Apr 9 18:10:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23E46C433ED for ; Fri, 9 Apr 2021 18:11:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F256961007 for ; Fri, 9 Apr 2021 18:11:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234544AbhDISLY (ORCPT ); Fri, 9 Apr 2021 14:11:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234615AbhDISLT (ORCPT ); Fri, 9 Apr 2021 14:11:19 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE3AEC061761 for ; Fri, 9 Apr 2021 11:11:02 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id es6so3137834qvb.0 for ; Fri, 09 Apr 2021 11:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ePeiUgb5oEGMA/x3BFIaUoin53uoN8y235q0g4j7KVY=; b=KlcmcUJN3DrlboHseaPh1+xi2+wXKLH5Bj3k5tZahlfX7Nf6OY0VJ7O5uRDg9Gk9+2 u8OpQNNP6eDITNMAMblUSmUGyghWtbiDK4H6jhcEpL3+ZyJKLdkQ+o3d195tF/iD3MbV 0XT3w/lrJ06jwmfydHWRccY2PZhweWjIt9Lqql7aPRCA7SK3QyNeevf8svxXCMml4nb1 8QYwOasX5Mtvwva7EAGAIHL743eC+47xDder+toaYAZ6pIGPuFbWLwtjR5fA1eNxiu/S ouaeqQSRhkKabtkzc32er3lYLwWtj7EctOE4vat1Lk1PrguoQ6gq8NI2kJx3vPWjoOjC 20qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ePeiUgb5oEGMA/x3BFIaUoin53uoN8y235q0g4j7KVY=; b=hTLxcCE9JYgVtbpj2qWShsFn2rR0SBTm38W6GSKTYlHl96cqTXimOClfu6lw+Kn3In T2xHnIJLWBLN/p21dQoso2xFpqw+aJ9wBi7F8OZp7+5hNg8R1Bc2HDrbUf7JVGBNY5WE SCO/Ea6OVNc2sEI48HjJ+/J8tNyu80FgL31MFvlVqqKZJpxXDHitaNjPZpAAtM1RO4Pq vsuVcgNP4kEUPyQalQcJDu9lX6fQqT0bg+RRzCrhG8Q9P6mSa9YCQmrPRDIaXtk1gewT hy9PNXgHwU2NRC2NTJfDYnv5Hpc/Dk+w5d8x5hkAIB8nCYKu/GfLWb7L1fPAsAuk03n6 kfEQ== X-Gm-Message-State: AOAM533mfR6Yz8nvbtOaCkaD7I3IOnj9pqzsU2G9mr9zIPiuk6z+BVKT ugBdNL84Vq3/a7WiHuakR0GFnaBpk7iof1C4 X-Google-Smtp-Source: ABdhPJwU7MrwMlcioBRXz98yLzlmy4ZFTjKYQaReTqgZ43nnjpwtDsz0ycTF+t6Lkn254kCEDNzFKg== X-Received: by 2002:a0c:8148:: with SMTP id 66mr15523552qvc.55.1617991861971; Fri, 09 Apr 2021 11:11:01 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id i12sm2215988qkl.49.2021.04.09.11.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:01 -0700 (PDT) Date: Fri, 9 Apr 2021 14:10:59 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 04/22] Documentation: build 'technical/bitmap-format' by default Message-ID: <46de889cd2d5314cede0cb2bf6d570638015235b.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Even though the 'TECH_DOCS' variable was introduced all the way back in 5e00439f0a (Documentation: build html for all files in technical and howto, 2012-10-23), the 'bitmap-format' document was never added to that list when it was created. Prepare for changes to this file by including it in the list of technical documentation that 'make doc' will build by default. Signed-off-by: Taylor Blau --- Documentation/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/Makefile b/Documentation/Makefile index 874a01d7a8..6d60c8c165 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -83,6 +83,7 @@ SP_ARTICLES += $(API_DOCS) TECH_DOCS += MyFirstContribution TECH_DOCS += MyFirstObjectWalk TECH_DOCS += SubmittingPatches +TECH_DOCS += technical/bitmap-format TECH_DOCS += technical/hash-function-transition TECH_DOCS += technical/http-protocol TECH_DOCS += technical/index-format From patchwork Fri Apr 9 18:11:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 761CBC433ED for ; Fri, 9 Apr 2021 18:11:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5438C61007 for ; Fri, 9 Apr 2021 18:11:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234629AbhDISLY (ORCPT ); Fri, 9 Apr 2021 14:11:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234618AbhDISLU (ORCPT ); Fri, 9 Apr 2021 14:11:20 -0400 Received: from mail-il1-x12e.google.com (mail-il1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AB2FC061764 for ; Fri, 9 Apr 2021 11:11:07 -0700 (PDT) Received: by mail-il1-x12e.google.com with SMTP id l19so1572357ilk.13 for ; Fri, 09 Apr 2021 11:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=E1ukGpQl2b4/B9Wy3ah+SELJbNqBJxQ0kTbWxrKWFkE=; b=1QNRaYPuDGBpCFZProWKsQCumvBWxsfgXn4Lx/FxBI/xW444VPms6VYl+fh+2oseGE x0hYMdqmJJh+d2urSbo7kqPiTPudoojEKVWlFGSyv/GRximC80+ED9EzwY67D3G3yt06 jbQIOgJ2LRxzziV6KSeefLtZIrDMhl0Ygy/ILjpNn5NRltDUzEAKaBVgXvYnav4lwikw 6P1ACmYxJ2p1FG0KxFwsdRVA+7hj5eSTPXh2Fy4YeESd9C04rUWx24hfj2fxWNakChxH 5xDtwZo0pmctSsm5w2gegRgNY6WeZDhZme7QsVNCDr1wOkj8l72vrEm521vSlbHzyf1W GIgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=E1ukGpQl2b4/B9Wy3ah+SELJbNqBJxQ0kTbWxrKWFkE=; b=TocFz4eptNbK8ZiNJwrrL87P685rfwP4YQiGGJYp+nQXSiOF5kNnohOlTFDYPwxFzN 25QSabWJXv+0ct8KO51q8M/u/2gjwYc6uUSdU/IDqN36ERXpC/QMVhwKyZy1h4sDJr58 NOaqyCbMRsCsJqERDRi8XQ5Z34tagQND+9PF8GOOQ++aAAs0qgrSUzxza9l1/LrN8s4G Dj1r3OVAFsXMu7dKMayzZ5OoUyNAedd5BJDwAwQbdmuHSCPdN1lrRDqVXhtBPhCrBuRD aMbZyBJEYTgCdwj4JkFvRzcdnQ6sLsTE30LuAacHSwRk04mrGhkuDYkJOaCnRIgD0Re1 QwIQ== X-Gm-Message-State: AOAM532s8eMXl6QX9alAECOQ233MPmUM0A2ndG2kilNx6Kr3Eb1yZPD2 MSlzdU2ITyTHHS91W7vLSuop6JHv+4rR7qgR X-Google-Smtp-Source: ABdhPJwBGF/3vtRhRiIfFZqkYKLZYyvsGS1uMu5t/+pnzWLOvkKqPKcerrP2jL2+QxvYxzxEhjlF4g== X-Received: by 2002:a05:6e02:501:: with SMTP id d1mr5483542ils.76.1617991866236; Fri, 09 Apr 2021 11:11:06 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id k17sm1591556ioa.44.2021.04.09.11.11.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:05 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:03 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 05/22] Documentation: describe MIDX-based bitmaps Message-ID: <0d4822a64e3b70cc315ff572145e5d5e95b958d9.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Update the technical documentation to describe the multi-pack bitmap format. This patch merely introduces the new format, and describes its high-level ideas. Git does not yet know how to read nor write these multi-pack variants, and so the subsequent patches will: - Introduce code to interpret multi-pack bitmaps, according to this document. - Then, introduce code to write multi-pack bitmaps from the 'git multi-pack-index write' sub-command. Finally, the implementation will gain tests in subsequent patches (as opposed to inline with the patch teaching Git how to write multi-pack bitmaps) to avoid a cyclic dependency. Signed-off-by: Taylor Blau --- Documentation/technical/bitmap-format.txt | 72 ++++++++++++++++---- Documentation/technical/multi-pack-index.txt | 10 +-- 2 files changed, 61 insertions(+), 21 deletions(-) diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt index f8c18a0f7a..25221c7ec8 100644 --- a/Documentation/technical/bitmap-format.txt +++ b/Documentation/technical/bitmap-format.txt @@ -1,6 +1,45 @@ GIT bitmap v1 format ==================== +== Pack and multi-pack bitmaps + +Bitmaps store reachability information about the set of objects in a packfile, +or a multi-pack index (MIDX). The former is defined obviously, and the latter is +defined as the union of objects in packs contained in the MIDX. + +A bitmap may belong to either one pack, or the repository's multi-pack index (if +it exists). A repository may have at most one bitmap. + +An object is uniquely described by its bit position within a bitmap: + + - If the bitmap belongs to a packfile, the __n__th bit corresponds to + the __n__th object in pack order. For a function `offset` which maps + objects to their byte offset within a pack, pack order is defined as + follows: + + o1 <= o2 <==> offset(o1) <= offset(o2) + + - If the bitmap belongs to a MIDX, the __n__th bit corresponds to the + __n__th object in MIDX order. With an additional function `pack` which + maps objects to the pack they were selected from by the MIDX, MIDX order + is defined as follows: + + o1 <= o2 <==> pack(o1) <= pack(o2) /\ offset(o1) <= offset(o2) + + The ordering between packs is done lexicographically by the pack name, + with the exception of the preferred pack, which sorts ahead of all other + packs. + +The on-disk representation (described below) of a bitmap is the same regardless +of whether or not that bitmap belongs to a packfile or a MIDX. The only +difference is the interpretation of the bits, which is described above. + +Certain bitmap extensions are supported (see: Appendix B). No extensions are +required for bitmaps corresponding to packfiles. For bitmaps that correspond to +MIDXs, both the bit-cache and rev-cache extensions are required. + +== On-disk format + - A header appears at the beginning: 4-byte signature: {'B', 'I', 'T', 'M'} @@ -14,17 +53,19 @@ GIT bitmap v1 format The following flags are supported: - BITMAP_OPT_FULL_DAG (0x1) REQUIRED - This flag must always be present. It implies that the bitmap - index has been generated for a packfile with full closure - (i.e. where every single object in the packfile can find - its parent links inside the same packfile). This is a - requirement for the bitmap index format, also present in JGit, - that greatly reduces the complexity of the implementation. + This flag must always be present. It implies that the + bitmap index has been generated for a packfile or + multi-pack index (MIDX) with full closure (i.e. where + every single object in the packfile/MIDX can find its + parent links inside the same packfile/MIDX). This is a + requirement for the bitmap index format, also present in + JGit, that greatly reduces the complexity of the + implementation. - BITMAP_OPT_HASH_CACHE (0x4) If present, the end of the bitmap file contains `N` 32-bit name-hash values, one per object in the - pack. The format and meaning of the name-hash is + pack/MIDX. The format and meaning of the name-hash is described below. 4-byte entry count (network byte order) @@ -33,7 +74,8 @@ GIT bitmap v1 format 20-byte checksum - The SHA1 checksum of the pack this bitmap index belongs to. + The SHA1 checksum of the pack/MIDX this bitmap index + belongs to. - 4 EWAH bitmaps that act as type indexes @@ -50,7 +92,7 @@ GIT bitmap v1 format - Tags In each bitmap, the `n`th bit is set to true if the `n`th object - in the packfile is of that type. + in the packfile or multi-pack index is of that type. The obvious consequence is that the OR of all 4 bitmaps will result in a full set (all bits set), and the AND of all 4 bitmaps will @@ -62,8 +104,9 @@ GIT bitmap v1 format Each entry contains the following: - 4-byte object position (network byte order) - The position **in the index for the packfile** where the - bitmap for this commit is found. + The position **in the index for the packfile or + multi-pack index** where the bitmap for this commit is + found. - 1-byte XOR-offset The xor offset used to compress this bitmap. For an entry @@ -146,10 +189,11 @@ Name-hash cache --------------- If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains -a cache of 32-bit values, one per object in the pack. The value at +a cache of 32-bit values, one per object in the pack/MIDX. The value at position `i` is the hash of the pathname at which the `i`th object -(counting in index order) in the pack can be found. This can be fed -into the delta heuristics to compare objects with similar pathnames. +(counting in index or multi-pack index order) in the pack/MIDX can be found. +This can be fed into the delta heuristics to compare objects with similar +pathnames. The hash algorithm used is: diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt index fb688976c4..1a73c3ee20 100644 --- a/Documentation/technical/multi-pack-index.txt +++ b/Documentation/technical/multi-pack-index.txt @@ -71,14 +71,10 @@ Future Work still reducing the number of binary searches required for object lookups. -- The reachability bitmap is currently paired directly with a single - packfile, using the pack-order as the object order to hopefully - compress the bitmaps well using run-length encoding. This could be - extended to pair a reachability bitmap with a multi-pack-index. If - the multi-pack-index is extended to store a "stable object order" +- If the multi-pack-index is extended to store a "stable object order" (a function Order(hash) = integer that is constant for a given hash, - even as the multi-pack-index is updated) then a reachability bitmap - could point to a multi-pack-index and be updated independently. + even as the multi-pack-index is updated) then MIDX bitmaps could be + updated independently of the MIDX. - Packfiles can be marked as "special" using empty files that share the initial name but replace ".pack" with ".keep" or ".promisor". From patchwork Fri Apr 9 18:11:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C109C43461 for ; Fri, 9 Apr 2021 18:11:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 653F061181 for ; Fri, 9 Apr 2021 18:11:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234609AbhDISL0 (ORCPT ); Fri, 9 Apr 2021 14:11:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234507AbhDISLY (ORCPT ); Fri, 9 Apr 2021 14:11:24 -0400 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45331C061761 for ; Fri, 9 Apr 2021 11:11:11 -0700 (PDT) Received: by mail-qk1-x72d.google.com with SMTP id q3so6683405qkq.12 for ; Fri, 09 Apr 2021 11:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=NITf+oxH1I2YlyZzr2LDI7tPEwqBuAVwTNUkVA1ihwQ=; b=D2Mk8zB9Ji+LF7/1vlx/G5Big2bMO0Ec0+vf51V8XK0GFvLi1MYN1zeyLzjJf9/f+A Yi9IKR18pp213c9plQtCYpKzT124UqAkCxVqIaqzMx86NUv/kd1u1Qmx4qCH8reJvauI I/nLsSRFa3QMKoOesrSGiJJ9a72lvWXiLjrWQI733K9PQAMDThEuBaLG+PEHzzGnnGOs Py1/bN8BzGPXIq8T7XALCp7ESWst7g9KdF77PvoapttDSjDOFq/uzZZlxVylfHSwZ8we UOFOKmDYFqZfRxADxwaHkbxLYdnFAM/dv9u949SMSMRrrZX6awv04Sbr7yb5W77kybxb WFsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=NITf+oxH1I2YlyZzr2LDI7tPEwqBuAVwTNUkVA1ihwQ=; b=j1ghxi1Q2CIrYW/I2YBJ4Ia5Z/MtmCcWHGfAOJnvm3KSsUE7+AiypF03f/BMpdtISV myd3zatDo2s9rQTO8jf/cmw2KBI/lgCS7xXI+F661JEgI8BHhiCKcf3oQ8K2cdEa1zUg vrwa8Y4QFYJDJhbXERbrSn7DtUqeZ0oyDeSr604jMYePEbv11b6VjXX7bJJk3LonrMXS /ohmB+lZ52GKIz1h6ypaLEi/rnbJLY/5e16k0EbPsHRWmZM+g5Gbs2xdt3CQi1/zQdKm 5nwiqFxwUGStBXnKuRRCd0gwuO+qgbLg2tWZ1rj+ErIBPYt2FICQOhPeq4jXeRo0h42F RK2w== X-Gm-Message-State: AOAM531B0eoIrb6v5MONznICCP7fZhN0nPbeTOU/hNJ3y0DyswjoTLnh 4Luq94LMu45I9ES1m2fyeNY+GHTmWFSBqoPW X-Google-Smtp-Source: ABdhPJxbdL89q+N59b1yJbzp5CgR7YtXwSn3TpgcxTKt4LNDGrQYF1E3RchTLhEmDwzYU32xsJEXsQ== X-Received: by 2002:a37:9c4b:: with SMTP id f72mr7423568qke.237.1617991870331; Fri, 09 Apr 2021 11:11:10 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id z188sm2369343qkb.40.2021.04.09.11.11.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:10 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:08 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 06/22] midx: make a number of functions non-static Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org These functions will be called from outside of midx.c in a subsequent patch. Signed-off-by: Taylor Blau --- midx.c | 4 ++-- midx.h | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 9e86583172..5249802326 100644 --- a/midx.c +++ b/midx.c @@ -48,12 +48,12 @@ static uint8_t oid_version(void) } } -static const unsigned char *get_midx_checksum(struct multi_pack_index *m) +const unsigned char *get_midx_checksum(struct multi_pack_index *m) { return m->data + m->data_len - the_hash_algo->rawsz; } -static char *get_midx_filename(const char *object_dir) +char *get_midx_filename(const char *object_dir) { return xstrfmt("%s/pack/multi-pack-index", object_dir); } diff --git a/midx.h b/midx.h index 8684cf0fef..1172df1a71 100644 --- a/midx.h +++ b/midx.h @@ -42,6 +42,8 @@ struct multi_pack_index { #define MIDX_PROGRESS (1 << 0) #define MIDX_WRITE_REV_INDEX (1 << 1) +const unsigned char *get_midx_checksum(struct multi_pack_index *m); +char *get_midx_filename(const char *object_dir); char *get_midx_rev_filename(struct multi_pack_index *m); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); From patchwork Fri Apr 9 18:11:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 386BAC43460 for ; Fri, 9 Apr 2021 18:11:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1614A61165 for ; Fri, 9 Apr 2021 18:11:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234521AbhDISLq (ORCPT ); Fri, 9 Apr 2021 14:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234641AbhDISLi (ORCPT ); Fri, 9 Apr 2021 14:11:38 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48B55C061761 for ; Fri, 9 Apr 2021 11:11:24 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id v70so6712395qkb.8 for ; Fri, 09 Apr 2021 11:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=bdA8LxFUWNagk/RIX9FVsJyfYnGnBv/1R9IfVNDS8JE=; b=GKHOqUmSih9O6LVsI5YzBS9XM7HqXEWoz6I7rwS4Vn8G77woPXCAJYaXVqdBGyd0fu oETeIkdFziLWVzlv9prhgdWIWG9/NNZKQP03/pbm7bxsm2fbkXvg954jR0cm4kICGjEY pUYBGY5Zl0Qgrt0I4h8iosA3Ce+4xXiV8T+xsef6g59+PP+A2+5APjxUKoYh2WUT4QAN alxHywkOd+0LiD6zEVWwt8pF8uENwy2lsRA4qTmYRK3CqKt3WfVwGsBCJHlnCw1WjeRr kXpT4q53HcSwnY1wucSwcbW44C0C/2B8Qp5GrNidp4RwfPVydGW3+dZZ01cftFTFU36a 7+XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=bdA8LxFUWNagk/RIX9FVsJyfYnGnBv/1R9IfVNDS8JE=; b=OaDoBoc5Ic9+J4p238s3APopUfZlPU3oTdPM/cw3gPL+oFzvBR3cYFu6qR1DF9cnRr Q6AEcqEIkQ/57QS01AC4ZeQ9KtZl0iCko9VQ7Q+GAN4djYB12okmZg8MFPvaex9BKnDK yi+3uVSlpGGEFmUd5Nh0OZ28t1vmhrRVRUTESwYwFFlPlYbrdjdQ7/V6xL6bFa77PxlV j3ge6mnEeY21xrgvOFAlUThqoRiQPlIcmkVjdyUhAJZomds0F3tf30MAB2qkPRiTtIqb djeA0uLjwM27ZUfGP+md/xxtLP4kwz/SkhtdLP2dV5KaX09TJg8D3YdOjxrkowabvtkY qtew== X-Gm-Message-State: AOAM532HVOOyG/ElDC/goEzOd5zf8D5Pbvr+FGCWH8kxy3HRm5TTjkoS /xSVIwTutI9asxg9QMt29fOJGpTaW9V5Iceb X-Google-Smtp-Source: ABdhPJxotrLHFKwIyfi73JqlXmE7B393K1SVjcWwDIjrw3POdjqqyWKBczZ+BV52EmwCA7Q3ZvCjdw== X-Received: by 2002:a05:620a:16b0:: with SMTP id s16mr14832824qkj.115.1617991883316; Fri, 09 Apr 2021 11:11:23 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id j26sm2374836qtp.30.2021.04.09.11.11.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:23 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:21 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 07/22] midx: clear auxiliary .rev after replacing the MIDX Message-ID: <26c3a312f9b2d073b0f50c44b78a7f3eba204eda.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing a new multi-pack index, write_midx_internal() attempts to clean up any auxiliary files (currently just the MIDX's `.rev` file, but soon to include a `.bitmap`, too) corresponding to the MIDX it's replacing. This step should happen after the new MIDX is written into place, since doing so beforehand means that the old MIDX could be read without its corresponding .rev file. Signed-off-by: Taylor Blau --- midx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 5249802326..a24c36968d 100644 --- a/midx.c +++ b/midx.c @@ -1076,10 +1076,11 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * if (flags & MIDX_WRITE_REV_INDEX) write_midx_reverse_index(midx_name, midx_hash, &ctx); - clear_midx_files_ext(the_repository, ".rev", midx_hash); commit_lock_file(&lk); + clear_midx_files_ext(the_repository, ".rev", midx_hash); + cleanup: for (i = 0; i < ctx.nr; i++) { if (ctx.info[i].p) { From patchwork Fri Apr 9 18:11:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB820C433ED for ; Fri, 9 Apr 2021 18:11:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C56B7610A6 for ; Fri, 9 Apr 2021 18:11:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234552AbhDISLu (ORCPT ); Fri, 9 Apr 2021 14:11:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234657AbhDISLo (ORCPT ); Fri, 9 Apr 2021 14:11:44 -0400 Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32BFCC0613D7 for ; Fri, 9 Apr 2021 11:11:28 -0700 (PDT) Received: by mail-io1-xd35.google.com with SMTP id v26so6807104iox.11 for ; Fri, 09 Apr 2021 11:11:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Go+L+Yv0YAe8sLVQCQ1HzOKhlRDJel+G+r6ct2W0Grk=; b=D7hlf61dOXPNL8Kv4rgFyX+pROW85d2Ih8jCToE9FfV+/NkptyWm5eSKNxyGGA43dY Bw2a0VDUKZonhnQVqNjUcPqMit/069mqdUwzPDbagauGK3Pb5GOCYnwvEp96k58X2uTg fTR/JdrNgjpBPiBMsB7luMUt+77sRf8hmAhHnxMuJ9qEWRNNmZDvCKDq3Rk+su2+t5UB SvxTNU3cj/irert1FoiRAythqkMAvHx1gy8QtmFwWr6YCa6pcd8s4zrBubhrvhZdsl7o 50uUUC3ypkZHcX0q3rWOQE/0OYxcaPuLQrYMb4q5astUatW9PrOiE7nDVEXC9s73wQw1 v2vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Go+L+Yv0YAe8sLVQCQ1HzOKhlRDJel+G+r6ct2W0Grk=; b=Y8xrxSnYyrwvbzc7JDSOUeK6KQqfbB2ht4vN0OxV8//OfNDJ+q/nIu2JBspSJBPNkK NmFBwF/iVqJ1oSkSVQ3fOYXw4ZTPIbeSMaAkScWegArbe6cTZtwaCGmnnYgVSrhctJhk jYsuh+F9t8z9QyXBWuHNW9ORNNHQj9+tTfPfDBiGKmoNNzWpQSC7jW1aynfDpKUB4hUk QJAmiXRrylRYIbjcwxRdHoJg7/guM05KQcy3ip/FitmdrGW4+r/QN8e8ELZKVAJazJTd E7U+ynF2pkwrXk3nps8/yIWcxfK0EXH1HOdcymnpywou5lmEd2mEQMvBmnItdr3uOiok rBpw== X-Gm-Message-State: AOAM533soQY7k6F35wvWDa/ElcMIn1Dek4L7Ngo0whUylyl5FrYBMRU3 JueJjGdki+IYHWxWcvlrk4soFzJE1oQe+1fq X-Google-Smtp-Source: ABdhPJz11P9On5swrLHkresU/KOaaz2Q4jkOXkWrh8Iecrcaj5tcn9rUs5nZ98bWbMKeHnY43S4+bQ== X-Received: by 2002:a5e:de0d:: with SMTP id e13mr12431067iok.208.1617991887357; Fri, 09 Apr 2021 11:11:27 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id o8sm1549146ilt.4.2021.04.09.11.11.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:27 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:25 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 08/22] midx: respect 'core.multiPackIndex' when writing Message-ID: <8643174a67812f347856bef274d6b971b15dbffc.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing a new multi-pack index, write_midx_internal() attempts to load any existing one to fill in some pieces of information. But it uses load_multi_pack_index(), which ignores the configuration "core.multiPackIndex", which indicates whether or not Git is allowed to read an existing multi-pack-index. Replace this with a routine that does respect that setting, to avoid reading multi-pack-index files when told not to. This avoids a problem that would arise in subsequent patches due to the combination of 'git repack' reopening the object store in-process and the multi-pack index code not checking whether a pack already exists in the object store when calling add_pack_to_midx(). This would ultimately lead to a cycle being created along the 'packed_git' struct's '->next' pointer. That is obviously bad, but it has hard-to-debug downstream effects like saying a bitmap can't be loaded for a pack because one already exists (for the same pack). Signed-off-by: Taylor Blau --- midx.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index a24c36968d..567cdf0fcf 100644 --- a/midx.c +++ b/midx.c @@ -908,8 +908,18 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * if (m) ctx.m = m; - else - ctx.m = load_multi_pack_index(object_dir, 1); + else { + struct multi_pack_index *cur; + + prepare_multi_pack_index_one(the_repository, object_dir, 1); + + ctx.m = NULL; + for (cur = the_repository->objects->multi_pack_index; cur; + cur = cur->next) { + if (!strcmp(object_dir, cur->object_dir)) + ctx.m = cur; + } + } ctx.nr = 0; ctx.alloc = ctx.m ? ctx.m->num_packs : 16; From patchwork Fri Apr 9 18:11:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9413C433B4 for ; Fri, 9 Apr 2021 18:11:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B996F610A6 for ; Fri, 9 Apr 2021 18:11:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234632AbhDISLu (ORCPT ); Fri, 9 Apr 2021 14:11:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234640AbhDISLp (ORCPT ); Fri, 9 Apr 2021 14:11:45 -0400 Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F2C0C061761 for ; Fri, 9 Apr 2021 11:11:32 -0700 (PDT) Received: by mail-il1-x131.google.com with SMTP id d2so5431942ilm.10 for ; Fri, 09 Apr 2021 11:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=HQJpJBcl+ocwZEOaMQu2s4dVVytvNTme32SUj0y6IZI=; b=TG6QhqhSrV67ykvwg4X68+TSKQzALl06iumw5y+v38/fMsxtb5xQn5VPdg3twxodiL j/1LGRuA4q9++3Y9TR2Agsf/WDx6QW8cTw16FDM61xHcom98W/XmjfGFtzJUmzC2BH5Q awvdHD5aPBpQ54w8EJOSIdbu2LnGpb7DZUPXKKZP0/wgMQL7QQ9JHSBNp9G0n3FXS9aG HWmwaeuhnLqzhtmAi0RDbocS8RVGBF49tuznv0gDKSPjkvEOT4mgHs9p90AI/+tV/Wvo 8XrWFEC+CvkU0JxeJ9ijCbDYr65XGkwmv1czrFESariL980QQhgMPwuYQubLWzLEjm9a 7JGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=HQJpJBcl+ocwZEOaMQu2s4dVVytvNTme32SUj0y6IZI=; b=WSO5jSWnP6drB0SNROt0X06CTKzHc/BfNL/OkbJW040s/Czv3fSZm+OtedWbymWjgB 9uLOvP4x1iXRsZ3oi69rHZK7TvGApQQbcDn/MC55uYmi9T/ZBLXBedU8G1ct7zpaxa5p moBtCaLVf39D7qkkiCCgWvn60ROBiaieYL7nozfa4GkgtrBE6u8x0yOS64Q5y7kt3nCt ospO+Jhyf++YBLQpZC93hlJ/v9i9kFYB8cm8UGw3zJLjtAnEIGwipG22K1gYqetpl6TR qKmWiXmAHTFEdm+i+T2kPuLigMV7TYLbGMlwLm7x3KdncphbGmwXYAltY28j3eiSg/Ja GHMw== X-Gm-Message-State: AOAM533rWy/v7sCItfe+RH7Qh3tFB1mC+kdQr7ce8LkyssGafkG0TN4D PzxzbY9j5kNryGcJ5W/9JG/EoHdkyAjtY+2z X-Google-Smtp-Source: ABdhPJxIvqMvbd+rx/nG2gFcCBSnEO2z0xTT4vF2GfVr3yDWrK/bNPvDib6ItLXxYOz4Os9lSV8TaQ== X-Received: by 2002:a92:c214:: with SMTP id j20mr7270761ilo.178.1617991891668; Fri, 09 Apr 2021 11:11:31 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id e6sm1565576ioq.13.2021.04.09.11.11.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:31 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:29 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 09/22] pack-bitmap.c: introduce 'bitmap_num_objects()' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to return how many objects are contained in a bitmap. Signed-off-by: Taylor Blau --- pack-bitmap.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index d45e91db1e..a6c616aa3e 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -136,6 +136,11 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) return b; } +static uint32_t bitmap_num_objects(struct bitmap_index *index) +{ + return index->pack->num_objects; +} + static int load_bitmap_header(struct bitmap_index *index) { struct bitmap_disk_header *header = (void *)index->map; @@ -154,7 +159,7 @@ static int load_bitmap_header(struct bitmap_index *index) /* Parse known bitmap format options */ { uint32_t flags = ntohs(header->options); - size_t cache_size = st_mult(index->pack->num_objects, sizeof(uint32_t)); + size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t)); unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz; if ((flags & BITMAP_OPT_FULL_DAG) == 0) @@ -399,7 +404,7 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git, if (pos < kh_end(positions)) { int bitmap_pos = kh_value(positions, pos); - return bitmap_pos + bitmap_git->pack->num_objects; + return bitmap_pos + bitmap_num_objects(bitmap_git); } return -1; @@ -451,7 +456,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git, bitmap_pos = kh_value(eindex->positions, hash_pos); } - return bitmap_pos + bitmap_git->pack->num_objects; + return bitmap_pos + bitmap_num_objects(bitmap_git); } struct bitmap_show_data { @@ -647,7 +652,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; ++i) { struct object *obj; - if (!bitmap_get(objects, bitmap_git->pack->num_objects + i)) + if (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) continue; obj = eindex->objects[i]; @@ -808,7 +813,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, * individually. */ for (i = 0; i < eindex->count; i++) { - uint32_t pos = i + bitmap_git->pack->num_objects; + uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == type && bitmap_get(to_filter, pos) && !bitmap_get(tips, pos)) @@ -835,7 +840,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, oi.sizep = &size; - if (pos < pack->num_objects) { + if (pos < bitmap_num_objects(bitmap_git)) { off_t ofs = pack_pos_to_offset(pack, pos); if (packed_object_info(the_repository, pack, ofs, &oi) < 0) { struct object_id oid; @@ -845,7 +850,7 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, } } else { struct eindex *eindex = &bitmap_git->ext_index; - struct object *obj = eindex->objects[pos - pack->num_objects]; + struct object *obj = eindex->objects[pos - bitmap_num_objects(bitmap_git)]; if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0) die(_("unable to get size of %s"), oid_to_hex(&obj->oid)); } @@ -887,7 +892,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, } for (i = 0; i < eindex->count; i++) { - uint32_t pos = i + bitmap_git->pack->num_objects; + uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == OBJ_BLOB && bitmap_get(to_filter, pos) && !bitmap_get(tips, pos) && @@ -1075,8 +1080,8 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git, enum object_type type; unsigned long size; - if (pos >= bitmap_git->pack->num_objects) - return; /* not actually in the pack */ + if (pos >= bitmap_num_objects(bitmap_git)) + return; /* not actually in the pack or MIDX */ offset = header = pack_pos_to_offset(bitmap_git->pack, pos); type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size); @@ -1142,6 +1147,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, struct pack_window *w_curs = NULL; size_t i = 0; uint32_t offset; + uint32_t objects_nr = bitmap_num_objects(bitmap_git); assert(result); @@ -1149,8 +1155,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, i++; /* Don't mark objects not in the packfile */ - if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD) - i = bitmap_git->pack->num_objects / BITS_IN_EWORD; + if (i > objects_nr / BITS_IN_EWORD) + i = objects_nr / BITS_IN_EWORD; reuse = bitmap_word_alloc(i); memset(reuse->words, 0xFF, i * sizeof(eword_t)); @@ -1234,7 +1240,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; ++i) { if (eindex->objects[i]->type == type && - bitmap_get(objects, bitmap_git->pack->num_objects + i)) + bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) count++; } @@ -1455,7 +1461,7 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, uint32_t i, num_objects; uint32_t *reposition; - num_objects = bitmap_git->pack->num_objects; + num_objects = bitmap_num_objects(bitmap_git); CALLOC_ARRAY(reposition, num_objects); for (i = 0; i < num_objects; ++i) { @@ -1538,7 +1544,6 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git) { struct bitmap *result = bitmap_git->result; - struct packed_git *pack = bitmap_git->pack; struct eindex *eindex = &bitmap_git->ext_index; off_t total = 0; struct object_info oi = OBJECT_INFO_INIT; @@ -1550,7 +1555,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git) for (i = 0; i < eindex->count; i++) { struct object *obj = eindex->objects[i]; - if (!bitmap_get(result, pack->num_objects + i)) + if (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i)) continue; if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0) From patchwork Fri Apr 9 18:11:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40EFCC43460 for ; Fri, 9 Apr 2021 18:11:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E528610CD for ; Fri, 9 Apr 2021 18:11:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234645AbhDISLv (ORCPT ); Fri, 9 Apr 2021 14:11:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234648AbhDISLt (ORCPT ); Fri, 9 Apr 2021 14:11:49 -0400 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C0D7C061762 for ; Fri, 9 Apr 2021 11:11:36 -0700 (PDT) Received: by mail-il1-x12d.google.com with SMTP id 6so5439009ilt.9 for ; Fri, 09 Apr 2021 11:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=2bntr2nvQj0B9IO4AfZjCtG7ckldbI/jOp0eVzXWjCk=; b=pggiFvMcsvQQlY520UiP/lY5cwL5iAICQk9ZhN9xRWMkfw/Yr7hXfJVqPualMCqHbm GS3VxBzKhS29Z1k11EGqodY5HeZEQy8jf2FXAswxJMMktl1xnv6fftoOhf9Yf8yXNiiH UrhcQkPqPSmuDvRZzBMIAJtkOP6MShdobs2LYPOhdBnr6OnDyMdiVi5GLeC0Zt+hekbW +lhFNTCxW2HOVlm1wiL1Fv8qq9HlacVZpdpbH4t1auJpOl3jKOlyDp6A0AlNylr/bgTB oZMn72g5A83HUH7tk8/EPClB5oJo02EQOjebckbt+4g8HG1+P/iptOS/OQzD7MWlfRPz ThEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=2bntr2nvQj0B9IO4AfZjCtG7ckldbI/jOp0eVzXWjCk=; b=kyZt0V6uzU5EbDjtjPC+tyel5NYMFfvMxlVdWNHl/pYDVWfhxylRjjxNERDmQ+rZij j+sRPkCuv0DygDgUdb/zZeQWG4jF0Ab2qAsjMV0z1pJFvjbV8cCnC0AqAEFe2aWX/Svg n+MaVDhwUeZ0jKXo4KlGxWqgwOMXvwEq9yg1GL+MLpFfzQ4nOLdMDoxDf15yGUGy/qtW X6Pj4J5LtxIu0VH6sDdzMgcDnF74NVQ7eGb5GWzFp1UQh/Xe4wx9BCd1JORY0kQVZNwH CHcenD2VqVKrzd1FHj1LUDuU09jV7LUvajtJwD7Q9UKvFLCJJuSE542j1px14WFj1BjS 5iOg== X-Gm-Message-State: AOAM530HzfY/RJdhweaFnqG8nZB2acAW5qg8EF24ZfoHY3yoURDsDfY5 jxpd0SJa4ulhTw286AFxGf7aziC8N3Er+a31 X-Google-Smtp-Source: ABdhPJwckRNgBPorfVtfSCSM4WH61fOMGWVDPg+EaHfYv6RF/63qzee9CKSHv+NPqhopamNwm3/CDQ== X-Received: by 2002:a05:6e02:1989:: with SMTP id g9mr12332420ilf.40.1617991895672; Fri, 09 Apr 2021 11:11:35 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id b4sm1579541iog.15.2021.04.09.11.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:35 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:33 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 10/22] pack-bitmap.c: introduce 'nth_bitmap_object_oid()' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A subsequent patch to support reading MIDX bitmaps will be less noisy after extracting a generic function to fetch the nth OID contained in the bitmap. Signed-off-by: Taylor Blau --- pack-bitmap.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index a6c616aa3e..97ee2d331d 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -223,6 +223,13 @@ static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos) #define MAX_XOR_OFFSET 160 +static void nth_bitmap_object_oid(struct bitmap_index *index, + struct object_id *oid, + uint32_t n) +{ + nth_packed_object_id(oid, index->pack, n); +} + static int load_bitmap_entries_v1(struct bitmap_index *index) { uint32_t i; @@ -242,9 +249,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index) xor_offset = read_u8(index->map, &index->map_pos); flags = read_u8(index->map, &index->map_pos); - if (nth_packed_object_id(&oid, index->pack, commit_idx_pos) < 0) - return error("corrupt ewah bitmap: commit index %u out of range", - (unsigned)commit_idx_pos); + nth_bitmap_object_oid(index, &oid, commit_idx_pos); bitmap = read_bitmap_1(index); if (!bitmap) @@ -844,8 +849,8 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, off_t ofs = pack_pos_to_offset(pack, pos); if (packed_object_info(the_repository, pack, ofs, &oi) < 0) { struct object_id oid; - nth_packed_object_id(&oid, pack, - pack_pos_to_index(pack, pos)); + nth_bitmap_object_oid(bitmap_git, &oid, + pack_pos_to_index(pack, pos)); die(_("unable to get size of %s"), oid_to_hex(&oid)); } } else { From patchwork Fri Apr 9 18:11:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEDABC433ED for ; Fri, 9 Apr 2021 18:11:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A83FA610C8 for ; Fri, 9 Apr 2021 18:11:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234648AbhDISLz (ORCPT ); Fri, 9 Apr 2021 14:11:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234647AbhDISLy (ORCPT ); Fri, 9 Apr 2021 14:11:54 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C05B9C061762 for ; Fri, 9 Apr 2021 11:11:40 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id g14so729253qtu.8 for ; Fri, 09 Apr 2021 11:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=J53tdVoNb6hBZ5473C9fSuiXjh6lbniFFpei3psdk5M=; b=MZKhTD3Pmdy1z6/ZqDLdFHlDXnGEwh1o4MWoxnETNvPhZFSviDfRShk6l+HxEh3ihV uqM3I7sL1CqW95Gn429l6ETO2Q7+VMuJ/NYKOj1RTh8kSdOxzYXC4gnmNcci69TRFzFS HpTqdwdZUx/XpCjNeeXxh45957vVoZwq3NiqF0FYTlKdscihC/N/TqWngeYkwaEz16iY 4MC4OlwWA51yWqHQIXkD1yW9GNIQg+eaP1h5d0QtW8HPs5wcFindxssoxsW9OAJrcEwL EzU+ch8bx0ATUVjHPkt3K6aifZGd6YTgwb4kFIq6IHiUqn8pUhxNn7hF52OnJdHYmueP zjLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=J53tdVoNb6hBZ5473C9fSuiXjh6lbniFFpei3psdk5M=; b=eZYuoMqH6xCrmwuMYHIsYPc3mH+PETucTDXAOCx2t/Nb7x0vhIprBjWOEO92olxHxZ rpy9CURjxhTLUAHNK4Z/WxvxmCksP7Z7VhSSL5bpNCAGP9GGhH8SREjCmmp2O8aU/gBZ GxGBCriFunyAGqbFv/H8xsySysdXE7FEVmRc6UiPYEl31BdVnHyyeRS9GmhnOkxkSIPK P+q8UKQ2hsrB4FseoRF0T5L8tdBc6kuWAafWtUSIRuGwzrZFQ+7bLPlfkLFzWcii3A82 g1IU9tA3F14K2t7y7a21j104Sej0uSx2MmaZKtIreRLZAYjFOywFREZoIskik7Sk/ABh 6JgA== X-Gm-Message-State: AOAM532hn/8+8tDnuJpQRgqveg/VN8se5YTYnF08iw4ui4bsRzIMjh2i XQ1BinL3VAicA7SMQy49YvyTlPxqcSJ4G3k/ X-Google-Smtp-Source: ABdhPJxYvDSMge5ria+vXLvfO3mL0lpPnFYq2DcIKyuGSpU0fPyK5dNkVwCmMybVuuFe9t+bNzFtOw== X-Received: by 2002:a05:622a:1c5:: with SMTP id t5mr13318810qtw.49.1617991899816; Fri, 09 Apr 2021 11:11:39 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id t63sm902998qkh.6.2021.04.09.11.11.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:39 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:37 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 11/22] pack-bitmap.c: introduce 'bitmap_is_preferred_refname()' Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a recent commit, pack-objects learned support for the 'pack.preferBitmapTips' configuration. This patch prepares the multi-pack bitmap code to respect this configuration, too. Since the multi-pack bitmap code already does a traversal of all references (in order to discover the set of reachable commits in the multi-pack index), it is more efficient to check whether or not each reference is a suffix of any value of 'pack.preferBitmapTips' rather than do an additional traversal. Implement a function 'bitmap_is_preferred_refname()' which does just that. The caller will be added in a subsequent patch. Signed-off-by: Taylor Blau --- pack-bitmap.c | 16 ++++++++++++++++ pack-bitmap.h | 1 + 2 files changed, 17 insertions(+) diff --git a/pack-bitmap.c b/pack-bitmap.c index 97ee2d331d..be52570b0f 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1594,3 +1594,19 @@ const struct string_list *bitmap_preferred_tips(struct repository *r) { return repo_config_get_value_multi(r, "pack.preferbitmaptips"); } + +int bitmap_is_preferred_refname(struct repository *r, const char *refname) +{ + const struct string_list *preferred_tips = bitmap_preferred_tips(r); + struct string_list_item *item; + + if (!preferred_tips) + return 0; + + for_each_string_list_item(item, preferred_tips) { + if (starts_with(refname, item->string)) + return 1; + } + + return 0; +} diff --git a/pack-bitmap.h b/pack-bitmap.h index 988ed3a30d..0bf75ff2a7 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -93,5 +93,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index, uint16_t options); const struct string_list *bitmap_preferred_tips(struct repository *r); +int bitmap_is_preferred_refname(struct repository *r, const char *refname); #endif From patchwork Fri Apr 9 18:11:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08B89C433ED for ; Fri, 9 Apr 2021 18:11:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D2F8F610A6 for ; Fri, 9 Apr 2021 18:11:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234658AbhDISMC (ORCPT ); Fri, 9 Apr 2021 14:12:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234394AbhDISMA (ORCPT ); Fri, 9 Apr 2021 14:12:00 -0400 Received: from mail-il1-x12f.google.com (mail-il1-x12f.google.com [IPv6:2607:f8b0:4864:20::12f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5799C061761 for ; Fri, 9 Apr 2021 11:11:47 -0700 (PDT) Received: by mail-il1-x12f.google.com with SMTP id b17so5455747ilh.6 for ; Fri, 09 Apr 2021 11:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=T54RqKagbgC8VyzwQE2celhWYWYnr72WXfewtIcTOtA=; b=KCAKb3v+m0yZaAV84uJntF2HobCy/m4G2cLCZH08IFBwG803hhFqp5rHIInmc+qDe0 3SqtCiM9cx9PQFN9ah0CljVr3RYOqPyAdZ5+8uWeHWS33bvhXqeVVuiUHDQj/4vHZtSI o1T+8J5VnQO8rnd3B+psix0NmZSMqn3RLwNElCaBQ90zQDXe2gnblXrlqYeelxibW/ec 6hnGZ0xSRaloOWAcxhR3K4q0JPy9oWp0uiMDRBL1bD5ACaT4/Pp8g3TIbBYcucDVfB9T jc6YWuk1ROXnRfU8VV4COxvXmsonn8srhsoiAmVW1qiyqPwQNrdPDm7knn/7DH+VOxjp D+Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=T54RqKagbgC8VyzwQE2celhWYWYnr72WXfewtIcTOtA=; b=N5y4vGunRUMAMgF8yv4qAM2qKa52z62Jrbpi2C4m9pSWRJ0Z+b5AG/B+au038Z2z4+ RlVBflCfWNcZJg6Gj660sGSdF+zAIFA/JBM0avfb/6Fc8W8n4cSVCBN/fTLSVfg1wz8+ wyToCl7F2MWvD+q6sEEfA9gnosXU7gPyy5IOPTVr/KOCM0rsSpABlkPtx/tF/KtpOHQG ryl71KajQzBkCYSb621bVLJTGz6u0Bc831NNsGery32+yXJhh6I7cqM8meOXp6N/yMuC KWR2ty5h7squMoXfSIl8Cb7WcKoXck+nOz4cdnTa2ze9w5YT4LkFXRLhHEwA+fyYjsfM IRxA== X-Gm-Message-State: AOAM530XvZomNgYj7/s7p6tydJo9kD7D6SZLzWM7NCtgoeq8RPhHx140 JMrHMOEQLtr7uqTHFVt+6rFogmKmh/k6E4P2 X-Google-Smtp-Source: ABdhPJxT47PsRoTORZLKswcWstr8/NeUCqDsfq9rhhywqrD99Tx9gpohdHmQzE+EqVWrXVLKqjYBpQ== X-Received: by 2002:a92:ad07:: with SMTP id w7mr10826716ilh.98.1617991906629; Fri, 09 Apr 2021 11:11:46 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id h2sm1552969ioj.30.2021.04.09.11.11.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:46 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:44 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 12/22] pack-bitmap: read multi-pack bitmaps Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This prepares the code in pack-bitmap to interpret the new multi-pack bitmaps described in Documentation/technical/bitmap-format.txt, which mostly involves converting bit positions to accommodate looking them up in a MIDX. Note that there are currently no writers who write multi-pack bitmaps, and that this will be implemented in the subsequent commit. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 12 +- pack-bitmap-write.c | 2 +- pack-bitmap.c | 349 +++++++++++++++++++++++++++++++++++++---- pack-bitmap.h | 5 + packfile.c | 2 +- 5 files changed, 338 insertions(+), 32 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 5205dde2e1..a4e4e4ebcc 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -984,7 +984,17 @@ static void write_reused_pack(struct hashfile *f) break; offset += ewah_bit_ctz64(word >> offset); - write_reused_pack_one(pos + offset, f, &w_curs); + if (bitmap_is_midx(bitmap_git)) { + off_t pack_offs = bitmap_pack_offset(bitmap_git, + pos + offset); + uint32_t pos; + + if (offset_to_pack_pos(reuse_packfile, pack_offs, &pos) < 0) + die(_("write_reused_pack: could not locate %"PRIdMAX), + (intmax_t)pack_offs); + write_reused_pack_one(pos, f, &w_curs); + } else + write_reused_pack_one(pos + offset, f, &w_curs); display_progress(progress_state, ++written); } } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index f90e100e3e..020c1774c8 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -48,7 +48,7 @@ void bitmap_writer_show_progress(int show) } /** - * Build the initial type index for the packfile + * Build the initial type index for the packfile or multi-pack-index */ void bitmap_writer_build_type_index(struct packing_data *to_pack, struct pack_idx_entry **index, diff --git a/pack-bitmap.c b/pack-bitmap.c index be52570b0f..e41fce9675 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -13,6 +13,7 @@ #include "repository.h" #include "object-store.h" #include "list-objects-filter-options.h" +#include "midx.h" #include "config.h" /* @@ -35,8 +36,15 @@ struct stored_bitmap { * the active bitmap index is the largest one. */ struct bitmap_index { - /* Packfile to which this bitmap index belongs to */ + /* + * The pack or multi-pack index (MIDX) that this bitmap index belongs + * to. + * + * Exactly one of these must be non-NULL; this specifies the object + * order used to interpret this bitmap. + */ struct packed_git *pack; + struct multi_pack_index *midx; /* * Mark the first `reuse_objects` in the packfile as reused: @@ -71,6 +79,8 @@ struct bitmap_index { /* If not NULL, this is a name-hash cache pointing into map. */ uint32_t *hashes; + const unsigned char *checksum; + /* * Extended index. * @@ -138,6 +148,8 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) static uint32_t bitmap_num_objects(struct bitmap_index *index) { + if (index->midx) + return index->midx->num_objects; return index->pack->num_objects; } @@ -175,6 +187,7 @@ static int load_bitmap_header(struct bitmap_index *index) } index->entry_count = ntohl(header->entry_count); + index->checksum = header->checksum; index->map_pos += header_size; return 0; } @@ -227,7 +240,10 @@ static void nth_bitmap_object_oid(struct bitmap_index *index, struct object_id *oid, uint32_t n) { - nth_packed_object_id(oid, index->pack, n); + if (index->midx) + nth_midxed_object_oid(oid, index->midx, n); + else + nth_packed_object_id(oid, index->pack, n); } static int load_bitmap_entries_v1(struct bitmap_index *index) @@ -272,7 +288,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index) return 0; } -static char *pack_bitmap_filename(struct packed_git *p) +char *midx_bitmap_filename(struct multi_pack_index *midx) +{ + return xstrfmt("%s-%s.bitmap", + get_midx_filename(midx->object_dir), + hash_to_hex(get_midx_checksum(midx))); +} + +char *pack_bitmap_filename(struct packed_git *p) { size_t len; @@ -281,6 +304,54 @@ static char *pack_bitmap_filename(struct packed_git *p) return xstrfmt("%.*s.bitmap", (int)len, p->pack_name); } +static int open_midx_bitmap_1(struct bitmap_index *bitmap_git, + struct multi_pack_index *midx) +{ + struct stat st; + char *idx_name = midx_bitmap_filename(midx); + int fd = git_open(idx_name); + + free(idx_name); + + if (fd < 0) + return -1; + + if (fstat(fd, &st)) { + close(fd); + return -1; + } + + if (bitmap_git->pack || bitmap_git->midx) { + /* ignore extra bitmap file; we can only handle one */ + return -1; + } + + bitmap_git->midx = midx; + bitmap_git->map_size = xsize_t(st.st_size); + bitmap_git->map_pos = 0; + bitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, + MAP_PRIVATE, fd, 0); + close(fd); + + if (load_bitmap_header(bitmap_git) < 0) + goto cleanup; + + if (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum)) + goto cleanup; + + if (load_midx_revindex(bitmap_git->midx) < 0) { + warning(_("multi-pack bitmap is missing required reverse index")); + goto cleanup; + } + return 0; + +cleanup: + munmap(bitmap_git->map, bitmap_git->map_size); + bitmap_git->map_size = 0; + bitmap_git->map = NULL; + return -1; +} + static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git *packfile) { int fd; @@ -302,12 +373,18 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git return -1; } - if (bitmap_git->pack) { + if (bitmap_git->pack || bitmap_git->midx) { + /* ignore extra bitmap file; we can only handle one */ warning("ignoring extra bitmap file: %s", packfile->pack_name); close(fd); return -1; } + if (!is_pack_valid(packfile)) { + close(fd); + return -1; + } + bitmap_git->pack = packfile; bitmap_git->map_size = xsize_t(st.st_size); bitmap_git->map = xmmap(NULL, bitmap_git->map_size, PROT_READ, MAP_PRIVATE, fd, 0); @@ -324,13 +401,36 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git return 0; } -static int load_pack_bitmap(struct bitmap_index *bitmap_git) +static int load_reverse_index(struct bitmap_index *bitmap_git) +{ + if (bitmap_is_midx(bitmap_git)) { + uint32_t i; + int ret; + + ret = load_midx_revindex(bitmap_git->midx); + if (ret) + return ret; + + for (i = 0; i < bitmap_git->midx->num_packs; i++) { + if (prepare_midx_pack(the_repository, bitmap_git->midx, i)) + die(_("load_reverse_index: could not open pack")); + ret = load_pack_revindex(bitmap_git->midx->packs[i]); + if (ret) + return ret; + } + return 0; + } + return load_pack_revindex(bitmap_git->pack); +} + +static int load_bitmap(struct bitmap_index *bitmap_git) { assert(bitmap_git->map); bitmap_git->bitmaps = kh_init_oid_map(); bitmap_git->ext_index.positions = kh_init_oid_pos(); - if (load_pack_revindex(bitmap_git->pack)) + + if (load_reverse_index(bitmap_git)) goto failed; if (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) || @@ -374,11 +474,35 @@ static int open_pack_bitmap(struct repository *r, return ret; } +static int open_midx_bitmap(struct repository *r, + struct bitmap_index *bitmap_git) +{ + struct multi_pack_index *midx; + + assert(!bitmap_git->map); + + for (midx = get_multi_pack_index(r); midx; midx = midx->next) { + if (!open_midx_bitmap_1(bitmap_git, midx)) + return 0; + } + return -1; +} + +static int open_bitmap(struct repository *r, + struct bitmap_index *bitmap_git) +{ + assert(!bitmap_git->map); + + if (!open_midx_bitmap(r, bitmap_git)) + return 0; + return open_pack_bitmap(r, bitmap_git); +} + struct bitmap_index *prepare_bitmap_git(struct repository *r) { struct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git)); - if (!open_pack_bitmap(r, bitmap_git) && !load_pack_bitmap(bitmap_git)) + if (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git)) return bitmap_git; free_bitmap_index(bitmap_git); @@ -428,10 +552,26 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git, return pos; } +static int bitmap_position_midx(struct bitmap_index *bitmap_git, + const struct object_id *oid) +{ + uint32_t want, got; + if (!bsearch_midx(oid, bitmap_git->midx, &want)) + return -1; + + if (midx_to_pack_pos(bitmap_git->midx, want, &got) < 0) + return -1; + return got; +} + static int bitmap_position(struct bitmap_index *bitmap_git, const struct object_id *oid) { - int pos = bitmap_position_packfile(bitmap_git, oid); + int pos; + if (bitmap_is_midx(bitmap_git)) + pos = bitmap_position_midx(bitmap_git, oid); + else + pos = bitmap_position_packfile(bitmap_git, oid); return (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid); } @@ -721,6 +861,7 @@ static void show_objects_for_type( continue; for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + struct packed_git *pack; struct object_id oid; uint32_t hash = 0, index_pos; off_t ofs; @@ -730,14 +871,28 @@ static void show_objects_for_type( offset += ewah_bit_ctz64(word >> offset); - index_pos = pack_pos_to_index(bitmap_git->pack, pos + offset); - ofs = pack_pos_to_offset(bitmap_git->pack, pos + offset); - nth_packed_object_id(&oid, bitmap_git->pack, index_pos); + if (bitmap_is_midx(bitmap_git)) { + struct multi_pack_index *m = bitmap_git->midx; + uint32_t pack_id; + + index_pos = pack_pos_to_midx(m, pos + offset); + ofs = nth_midxed_offset(m, index_pos); + nth_midxed_object_oid(&oid, m, index_pos); + + pack_id = nth_midxed_pack_int_id(m, index_pos); + pack = bitmap_git->midx->packs[pack_id]; + } else { + index_pos = pack_pos_to_index(bitmap_git->pack, pos + offset); + ofs = pack_pos_to_offset(bitmap_git->pack, pos + offset); + nth_bitmap_object_oid(bitmap_git, &oid, index_pos); + + pack = bitmap_git->pack; + } if (bitmap_git->hashes) hash = get_be32(bitmap_git->hashes + index_pos); - show_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs); + show_reach(&oid, object_type, 0, hash, pack, ofs); } } } @@ -749,8 +904,13 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git, struct object *object = roots->item; roots = roots->next; - if (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0) - return 1; + if (bitmap_is_midx(bitmap_git)) { + if (bsearch_midx(&object->oid, bitmap_git->midx, NULL)) + return 1; + } else { + if (find_pack_entry_one(object->oid.hash, bitmap_git->pack) > 0) + return 1; + } } return 0; @@ -839,14 +999,26 @@ static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git, static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, uint32_t pos) { - struct packed_git *pack = bitmap_git->pack; unsigned long size; struct object_info oi = OBJECT_INFO_INIT; oi.sizep = &size; if (pos < bitmap_num_objects(bitmap_git)) { - off_t ofs = pack_pos_to_offset(pack, pos); + struct packed_git *pack; + off_t ofs; + + if (bitmap_is_midx(bitmap_git)) { + uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos); + uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos); + + pack = bitmap_git->midx->packs[pack_id]; + ofs = nth_midxed_offset(bitmap_git->midx, midx_pos); + } else { + pack = bitmap_git->pack; + ofs = pack_pos_to_offset(pack, pos); + } + if (packed_object_info(the_repository, pack, ofs, &oi) < 0) { struct object_id oid; nth_bitmap_object_oid(bitmap_git, &oid, @@ -990,7 +1162,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, /* try to open a bitmapped pack, but don't parse it yet * because we may not need to use it */ CALLOC_ARRAY(bitmap_git, 1); - if (open_pack_bitmap(revs->repo, bitmap_git) < 0) + if (open_bitmap(revs->repo, bitmap_git) < 0) goto cleanup; for (i = 0; i < revs->pending.nr; ++i) { @@ -1034,7 +1206,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, * from disk. this is the point of no return; after this the rev_list * becomes invalidated and we must perform the revwalk through bitmaps */ - if (load_pack_bitmap(bitmap_git) < 0) + if (load_bitmap(bitmap_git) < 0) goto cleanup; object_array_clear(&revs->pending); @@ -1081,15 +1253,29 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git, struct bitmap *reuse, struct pack_window **w_curs) { - off_t offset, header; + struct packed_git *pack; + off_t offset, delta_obj_offset; enum object_type type; unsigned long size; if (pos >= bitmap_num_objects(bitmap_git)) return; /* not actually in the pack or MIDX */ - offset = header = pack_pos_to_offset(bitmap_git->pack, pos); - type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size); + if (bitmap_is_midx(bitmap_git)) { + uint32_t pack_id, midx_pos; + + midx_pos = pack_pos_to_midx(bitmap_git->midx, pos); + pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos); + + pack = bitmap_git->midx->packs[pack_id]; + offset = nth_midxed_offset(bitmap_git->midx, midx_pos); + } else { + pack = bitmap_git->pack; + offset = pack_pos_to_offset(bitmap_git->pack, pos); + } + + delta_obj_offset = offset; + type = unpack_object_header(pack, w_curs, &offset, &size); if (type < 0) return; /* broken packfile, punt */ @@ -1105,11 +1291,11 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git, * and the normal slow path will complain about it in * more detail. */ - base_offset = get_delta_base(bitmap_git->pack, w_curs, - &offset, type, header); + base_offset = get_delta_base(pack, w_curs, &offset, type, + delta_obj_offset); if (!base_offset) return; - if (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0) + if (offset_to_pack_pos(pack, base_offset, &base_pos) < 0) return; /* @@ -1120,6 +1306,16 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git, * packs we write fresh, and OFS_DELTA is the default). But * let's double check to make sure the pack wasn't written with * odd parameters. + * + * Note that the base does not need to be repositioned, i.e., + * the MIDX is guaranteed to have selected the copy of "base" + * from the same pack, since this function is only ever called + * on the preferred pack (and all duplicate objects are resolved + * in favor of the preferred pack). + * + * This means that we can reuse base_pos when looking up the bit + * in the reuse bitmap, too, since bits corresponding to the + * preferred pack precede all bits from other packs. */ if (base_pos >= pos) return; @@ -1142,6 +1338,14 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git, bitmap_set(reuse, pos); } +static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git) +{ + struct multi_pack_index *m = bitmap_git->midx; + if (!m) + BUG("midx_preferred_pack: requires non-empty MIDX"); + return nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0)); +} + int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, struct packed_git **packfile_out, uint32_t *entries, @@ -1153,13 +1357,29 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, size_t i = 0; uint32_t offset; uint32_t objects_nr = bitmap_num_objects(bitmap_git); + uint32_t preferred_pack = 0; assert(result); + load_reverse_index(bitmap_git); + + if (bitmap_is_midx(bitmap_git)) { + preferred_pack = midx_preferred_pack(bitmap_git); + objects_nr = bitmap_git->midx->packs[preferred_pack]->num_objects; + } else + objects_nr = bitmap_git->pack->num_objects; + while (i < result->word_alloc && result->words[i] == (eword_t)~0) i++; - /* Don't mark objects not in the packfile */ + /* + * Don't mark objects not in the packfile or preferred pack. This bitmap + * marks objects eligible for reuse, but the pack-reuse code only + * understands how to reuse a single pack. Since the preferred pack is + * guaranteed to have all bases for its deltas (in a multi-pack bitmap), + * we use it instead of another pack. In single-pack bitmaps, the choice + * is made for us. + */ if (i > objects_nr / BITS_IN_EWORD) i = objects_nr / BITS_IN_EWORD; @@ -1175,6 +1395,14 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, break; offset += ewah_bit_ctz64(word >> offset); + if (bitmap_is_midx(bitmap_git)) { + /* + * Can't reuse from a non-preferred pack (see + * above). + */ + if (pos + offset >= objects_nr) + continue; + } try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs); } } @@ -1192,7 +1420,9 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, * need to be handled separately. */ bitmap_and_not(result, reuse); - *packfile_out = bitmap_git->pack; + *packfile_out = bitmap_git->pack ? + bitmap_git->pack : + bitmap_git->midx->packs[preferred_pack]; *reuse_out = reuse; return 0; } @@ -1466,6 +1696,12 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, uint32_t i, num_objects; uint32_t *reposition; + if (!bitmap_is_midx(bitmap_git)) + load_reverse_index(bitmap_git); + else if (load_midx_revindex(bitmap_git->midx) < 0) + BUG("rebuild_existing_bitmaps: missing required rev-cache " + "extension"); + num_objects = bitmap_num_objects(bitmap_git); CALLOC_ARRAY(reposition, num_objects); @@ -1473,8 +1709,13 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, struct object_id oid; struct object_entry *oe; - nth_packed_object_id(&oid, bitmap_git->pack, - pack_pos_to_index(bitmap_git->pack, i)); + if (bitmap_is_midx(bitmap_git)) + nth_midxed_object_oid(&oid, + bitmap_git->midx, + pack_pos_to_midx(bitmap_git->midx, i)); + else + nth_packed_object_id(&oid, bitmap_git->pack, + pack_pos_to_index(bitmap_git->pack, i)); oe = packlist_find(mapping, &oid); if (oe) @@ -1500,6 +1741,19 @@ void free_bitmap_index(struct bitmap_index *b) free(b->ext_index.hashes); bitmap_free(b->result); bitmap_free(b->haves); + if (bitmap_is_midx(b)) { + /* + * Multi-pack bitmaps need to have resources associated with + * their on-disk reverse indexes unmapped so that stale .rev and + * .bitmap files can be removed. + * + * Unlike pack-based bitmaps, multi-pack bitmaps can be read and + * written in the same 'git multi-pack-index write --bitmap' + * process. Close resources so they can be removed safely on + * platforms like Windows. + */ + close_midx_revindex(b->midx); + } free(b); } @@ -1514,7 +1768,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, enum object_type object_type) { struct bitmap *result = bitmap_git->result; - struct packed_git *pack = bitmap_git->pack; + struct packed_git *pack; off_t total = 0; struct ewah_iterator it; eword_t filter; @@ -1538,6 +1792,29 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, offset += ewah_bit_ctz64(word >> offset); pos = base + offset; + + if (bitmap_is_midx(bitmap_git)) { + uint32_t pack_pos; + uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, pos); + uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos); + off_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos); + + pack = bitmap_git->midx->packs[pack_id]; + + if (offset_to_pack_pos(pack, offset, &pack_pos) < 0) { + struct object_id oid; + nth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos); + + die(_("could not find %s in pack #%"PRIu32" at offset %"PRIuMAX), + oid_to_hex(&oid), + pack_id, + (uintmax_t)offset); + } + + pos = pack_pos; + } else + pack = bitmap_git->pack; + total += pack_pos_to_offset(pack, pos + 1) - pack_pos_to_offset(pack, pos); } @@ -1590,6 +1867,20 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *bitmap_git, return total; } +int bitmap_is_midx(struct bitmap_index *bitmap_git) +{ + return !!bitmap_git->midx; +} + +off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos) +{ + if (bitmap_is_midx(bitmap_git)) + return nth_midxed_offset(bitmap_git->midx, + pack_pos_to_midx(bitmap_git->midx, pos)); + return nth_packed_object_offset(bitmap_git->pack, + pack_pos_to_index(bitmap_git->pack, pos)); +} + const struct string_list *bitmap_preferred_tips(struct repository *r) { return repo_config_get_value_multi(r, "pack.preferbitmaptips"); diff --git a/pack-bitmap.h b/pack-bitmap.h index 0bf75ff2a7..0dc6f7a7e4 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -91,6 +91,11 @@ void bitmap_writer_finish(struct pack_idx_entry **index, uint32_t index_nr, const char *filename, uint16_t options); +char *midx_bitmap_filename(struct multi_pack_index *midx); +char *pack_bitmap_filename(struct packed_git *p); + +int bitmap_is_midx(struct bitmap_index *bitmap_git); +off_t bitmap_pack_offset(struct bitmap_index *bitmap_git, uint32_t pos); const struct string_list *bitmap_preferred_tips(struct repository *r); int bitmap_is_preferred_refname(struct repository *r, const char *refname); diff --git a/packfile.c b/packfile.c index 8668345d93..c444e365a3 100644 --- a/packfile.c +++ b/packfile.c @@ -863,7 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len, if (!strcmp(file_name, "multi-pack-index")) return; if (starts_with(file_name, "multi-pack-index") && - ends_with(file_name, ".rev")) + (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev"))) return; if (ends_with(file_name, ".idx") || ends_with(file_name, ".rev") || From patchwork Fri Apr 9 18:11:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2CA6C433ED for ; Fri, 9 Apr 2021 18:12:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BCF2561007 for ; Fri, 9 Apr 2021 18:12:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234562AbhDISMR (ORCPT ); Fri, 9 Apr 2021 14:12:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234394AbhDISMG (ORCPT ); Fri, 9 Apr 2021 14:12:06 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFCDBC061761 for ; Fri, 9 Apr 2021 11:11:51 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id bs7so2622453qvb.12 for ; Fri, 09 Apr 2021 11:11:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ZiG1NnlzlmLSJHzO32tUvfpX78lDOrTGm4HUo1MIDto=; b=UszBudJfUE1IMHhDqVfzRZaZdT9ZcMzDSfa00/O90EtvEPRBJxHyfKnAOvhn0rEmdi DkTcnJjxqpulWtKHVhmhV7nkC4dPiPAGG2IhHMQ/NW4Y5t5UaeYCfW8/03gO6uvDZ5b+ UHLQLkt2rCfkqLSZ4cETetma1POus8DF9+mNEh5Hhi5FA6S7eRx3YtBWfvghmqj8B4B6 qsi8RXWtL8b6pIvBTZc6QqcdsJksEElXOePKvEastG3pl2uf7BC7t1cV5HSciWmTESdh Elq5t0KRpZ/W0916XSLNImGCfbBGe/+yIeJ1HNvhdO9sNNfBJQkwF1/oe9z/rO2XZjdR gDqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ZiG1NnlzlmLSJHzO32tUvfpX78lDOrTGm4HUo1MIDto=; b=AL6ebCJPXzDdOgiR7sQldvImAtHPh4RLiwILlmrZ+HXnbHVW8SWO+5kxXIoZ5eGGGe e5HxQa6N4uDQEwWFMc/E1Qatv9C0SZWku/7lSUuzzcVFsW7MZVnHZ/w9nl1axeROdCN9 IyEDTSFaBSZNs9qQhvOMIqM9SuW9UlZIhHB7h7YAL2D2yEVU8QtJiJn4yBt+eqmKbwXZ qNvveBBO5/SsNwkdSp4aDen+iny6uVNC6qeLQOPyoPBZqjjWVwZlhepAXRu+pYVgt2jh tBly/2CYlg1HODn2pJJZbJng837kccWrUC6ul+NvgrfUdFaxy4rDBkz+peOmMTHtyfhL HW/w== X-Gm-Message-State: AOAM531NcRLDaP9ikMlaosUJO0j/xl7fgU+d375Sp9GMsqBTtoyjhYTF yyDgx/AegljrgxV9QlpQ5A6SfzvpH4iEmyfW X-Google-Smtp-Source: ABdhPJxRC18BzNPlCTeymzO21DWDwLcjQ6bP8S9MgI17NQZfiApK3LphTVOZY174rmUSZKzj155ytg== X-Received: by 2002:ad4:576a:: with SMTP id r10mr15575648qvx.12.1617991910618; Fri, 09 Apr 2021 11:11:50 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id q4sm2274060qkq.134.2021.04.09.11.11.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:50 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:48 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 13/22] pack-bitmap: write multi-pack bitmaps Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Write multi-pack bitmaps in the format described by Documentation/technical/bitmap-format.txt, inferring their presence with the absence of '--bitmap'. To write a multi-pack bitmap, this patch attempts to reuse as much of the existing machinery from pack-objects as possible. Specifically, the MIDX code prepares a packing_data struct that pretends as if a single packfile has been generated containing all of the objects contained within the MIDX. Signed-off-by: Taylor Blau --- Documentation/git-multi-pack-index.txt | 12 +- builtin/multi-pack-index.c | 2 + midx.c | 195 ++++++++++++++++++++++++- midx.h | 1 + 4 files changed, 202 insertions(+), 8 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index ffd601bc17..ada14deb2c 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -10,7 +10,7 @@ SYNOPSIS -------- [verse] 'git multi-pack-index' [--object-dir=] [--[no-]progress] - [--preferred-pack=] + [--preferred-pack=] [--[no-]bitmap] DESCRIPTION ----------- @@ -40,6 +40,9 @@ write:: multiple packs contain the same object. If not given, ties are broken in favor of the pack with the lowest mtime. + + --[no-]bitmap:: + Control whether or not a multi-pack bitmap is written. -- verify:: @@ -81,6 +84,13 @@ EXAMPLES $ git multi-pack-index write ----------------------------------------------- +* Write a MIDX file for the packfiles in the current .git folder with a +corresponding bitmap. ++ +------------------------------------------------------------- +$ git multi-pack-index write --preferred-pack --bitmap +------------------------------------------------------------- + * Write a MIDX file for the packfiles in an alternate object store. + ----------------------------------------------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 5d3ea445fd..bf6fa982e3 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -68,6 +68,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv) OPT_STRING(0, "preferred-pack", &opts.preferred_pack, N_("preferred-pack"), N_("pack for reuse when computing a multi-pack bitmap")), + OPT_BIT(0, "bitmap", &opts.flags, N_("write multi-pack bitmap"), + MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_END(), }; diff --git a/midx.c b/midx.c index 567cdf0fcf..32d7d184c0 100644 --- a/midx.c +++ b/midx.c @@ -13,6 +13,10 @@ #include "repository.h" #include "chunk-format.h" #include "pack.h" +#include "pack-bitmap.h" +#include "refs.h" +#include "revision.h" +#include "list-objects.h" #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */ #define MIDX_VERSION 1 @@ -885,6 +889,145 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash, static void clear_midx_files_ext(struct repository *r, const char *ext, unsigned char *keep_hash); +static void prepare_midx_packing_data(struct packing_data *pdata, + struct write_midx_context *ctx) +{ + uint32_t i; + + memset(pdata, 0, sizeof(struct packing_data)); + prepare_packing_data(the_repository, pdata); + + for (i = 0; i < ctx->entries_nr; i++) { + struct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]]; + struct object_entry *to = packlist_alloc(pdata, &from->oid); + + oe_set_in_pack(pdata, to, + ctx->info[ctx->pack_perm[from->pack_int_id]].p); + } +} + +static int add_ref_to_pending(const char *refname, + const struct object_id *oid, + int flag, void *cb_data) +{ + struct rev_info *revs = (struct rev_info*)cb_data; + struct object *object; + + if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) { + warning("symbolic ref is dangling: %s", refname); + return 0; + } + + object = parse_object_or_die(oid, refname); + if (object->type != OBJ_COMMIT) + return 0; + + add_pending_object(revs, object, ""); + if (bitmap_is_preferred_refname(revs->repo, refname)) + object->flags |= NEEDS_BITMAP; + return 0; +} + +struct bitmap_commit_cb { + struct commit **commits; + size_t commits_nr, commits_alloc; + + struct write_midx_context *ctx; +}; + +static const struct object_id *bitmap_oid_access(size_t index, + const void *_entries) +{ + const struct pack_midx_entry *entries = _entries; + return &entries[index].oid; +} + +static void bitmap_show_commit(struct commit *commit, void *_data) +{ + struct bitmap_commit_cb *data = _data; + if (oid_pos(&commit->object.oid, data->ctx->entries, + data->ctx->entries_nr, + bitmap_oid_access) > -1) { + ALLOC_GROW(data->commits, data->commits_nr + 1, + data->commits_alloc); + data->commits[data->commits_nr++] = commit; + } +} + +static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p, + struct write_midx_context *ctx) +{ + struct rev_info revs; + struct bitmap_commit_cb cb; + + memset(&cb, 0, sizeof(struct bitmap_commit_cb)); + cb.ctx = ctx; + + repo_init_revisions(the_repository, &revs, NULL); + for_each_ref(add_ref_to_pending, &revs); + + fetch_if_missing = 0; + revs.exclude_promisor_objects = 1; + + if (prepare_revision_walk(&revs)) + die(_("revision walk setup failed")); + + traverse_commit_list(&revs, bitmap_show_commit, NULL, &cb); + if (indexed_commits_nr_p) + *indexed_commits_nr_p = cb.commits_nr; + + return cb.commits; +} + +static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash, + struct write_midx_context *ctx, + unsigned flags) +{ + struct packing_data pdata; + struct pack_idx_entry **index; + struct commit **commits = NULL; + uint32_t i, commits_nr; + char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name, hash_to_hex(midx_hash)); + int ret; + + prepare_midx_packing_data(&pdata, ctx); + + commits = find_commits_for_midx_bitmap(&commits_nr, ctx); + + /* + * Build the MIDX-order index based on pdata.objects (which is already + * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of + * this order). + */ + ALLOC_ARRAY(index, pdata.nr_objects); + for (i = 0; i < pdata.nr_objects; i++) + index[i] = (struct pack_idx_entry *)&pdata.objects[i]; + + bitmap_writer_show_progress(flags & MIDX_PROGRESS); + bitmap_writer_build_type_index(&pdata, index, pdata.nr_objects); + + /* + * bitmap_writer_select_commits expects objects in lex order, but + * pack_order gives us exactly that. use it directly instead of + * re-sorting the array + */ + for (i = 0; i < pdata.nr_objects; i++) + index[ctx->pack_order[i]] = (struct pack_idx_entry *)&pdata.objects[i]; + + bitmap_writer_select_commits(commits, commits_nr, -1); + ret = bitmap_writer_build(&pdata); + if (!ret) + goto cleanup; + + bitmap_writer_set_checksum(midx_hash); + bitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0); + +cleanup: + free(index); + free(bitmap_name); + return ret; +} + static int write_midx_internal(const char *object_dir, struct multi_pack_index *m, struct string_list *packs_to_drop, const char *preferred_pack_name, @@ -930,9 +1073,16 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * for (i = 0; i < ctx.m->num_packs; i++) { ALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc); + if (prepare_midx_pack(the_repository, ctx.m, i)) { + error(_("could not load pack %s"), + ctx.m->pack_names[i]); + result = 1; + goto cleanup; + } + ctx.info[ctx.nr].orig_pack_int_id = i; ctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]); - ctx.info[ctx.nr].p = NULL; + ctx.info[ctx.nr].p = ctx.m->packs[i]; ctx.info[ctx.nr].expired = 0; ctx.nr++; } @@ -947,8 +1097,26 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); - if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) - goto cleanup; + if (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) { + struct bitmap_index *bitmap_git; + int bitmap_exists; + int want_bitmap = flags & MIDX_WRITE_BITMAP; + + bitmap_git = prepare_bitmap_git(the_repository); + bitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git); + free_bitmap_index(bitmap_git); + + if (bitmap_exists || !want_bitmap) { + /* + * The correct MIDX already exists, and so does a + * corresponding bitmap (or one wasn't requested). + */ + if (!want_bitmap) + clear_midx_files_ext(the_repository, ".bitmap", + NULL); + goto cleanup; + } + } ctx.preferred_pack_idx = -1; if (preferred_pack_name) { @@ -1048,9 +1216,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR); f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); - if (ctx.m) - close_midx(ctx.m); - if (ctx.nr - dropped_packs == 0) { error(_("no pack files to index.")); result = 1; @@ -1081,14 +1246,17 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); - if (flags & MIDX_WRITE_REV_INDEX) + if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) ctx.pack_order = midx_pack_order(&ctx); if (flags & MIDX_WRITE_REV_INDEX) write_midx_reverse_index(midx_name, midx_hash, &ctx); + if (flags & MIDX_WRITE_BITMAP) + write_midx_bitmap(midx_name, midx_hash, &ctx, flags); commit_lock_file(&lk); + clear_midx_files_ext(the_repository, ".bitmap", midx_hash); clear_midx_files_ext(the_repository, ".rev", midx_hash); cleanup: @@ -1096,6 +1264,15 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * if (ctx.info[i].p) { close_pack(ctx.info[i].p); free(ctx.info[i].p); + if (ctx.m) { + /* + * Destroy a stale reference to the pack in + * 'ctx.m'. + */ + uint32_t orig = ctx.info[i].orig_pack_int_id; + if (orig < ctx.m->num_packs) + ctx.m->packs[orig] = NULL; + } } free(ctx.info[i].pack_name); } @@ -1105,6 +1282,9 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * free(ctx.pack_perm); free(ctx.pack_order); free(midx_name); + if (ctx.m) + close_midx(ctx.m); + return result; } @@ -1166,6 +1346,7 @@ void clear_midx_file(struct repository *r) if (remove_path(midx)) die(_("failed to clear multi-pack-index at %s"), midx); + clear_midx_files_ext(r, ".bitmap", NULL); clear_midx_files_ext(r, ".rev", NULL); free(midx); diff --git a/midx.h b/midx.h index 1172df1a71..350f4d0a7b 100644 --- a/midx.h +++ b/midx.h @@ -41,6 +41,7 @@ struct multi_pack_index { #define MIDX_PROGRESS (1 << 0) #define MIDX_WRITE_REV_INDEX (1 << 1) +#define MIDX_WRITE_BITMAP (1 << 2) const unsigned char *get_midx_checksum(struct multi_pack_index *m); char *get_midx_filename(const char *object_dir); From patchwork Fri Apr 9 18:11:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DFD6C433B4 for ; Fri, 9 Apr 2021 18:12:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D0FF1610CD for ; Fri, 9 Apr 2021 18:12:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234419AbhDISMT (ORCPT ); Fri, 9 Apr 2021 14:12:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234653AbhDISML (ORCPT ); Fri, 9 Apr 2021 14:12:11 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17E6BC061763 for ; Fri, 9 Apr 2021 11:11:56 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id e186so6808884iof.7 for ; Fri, 09 Apr 2021 11:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=OZ0l2LRhAv2TG2op0mypykW1kcDQNxN6GOoGtYGbeF4=; b=tQ1bcrekFnnuxUSgUdaTLjRhEYLQ5c74J50xh0DNzt/QdTlKq+ngya3TgacWxwGmN7 HpWuOu8PE8baedkvagG3xgFiZM6lr6SR8JRDaMjbmpiTUW4d/bCcIuC7UhFTRVDhgtxO wqjAW7vVD5L4d/jlqp0xbIK4kHrC9FRykTjlHIGh/LFMWXcXQqAE8BbE7+gjnn+j3Lnm tYuQzU5lnp9fWaW1IeKTH2mSC68hYEjlPy2q8fAjoBfmuraWK1noTL3IoL3+t5iLpmDG 3WEyPjGmzOaIeUojp8vnXj8eq2FdxeJQbCXS4HAb8Ok6VxYFFKKqj8XRhzL9AUDnsqyc 37bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=OZ0l2LRhAv2TG2op0mypykW1kcDQNxN6GOoGtYGbeF4=; b=O7sOZZ6jgQinfEChBj8Hny7gEOxIGym7MjUZ9HHcqIor7iy4W6HuoFXYJGeHQPlmFS 0MSmihqbsTc6ozZTIbFG0Ko6f/LJmJSKTycZYkFcTf17ET2fMuF9u1qRsN3Yq/thPQyj FqVa0Uy8dm23XeYeiIRjboHH4mK6F5rTsFEEWVWD43uDSYZTD5b9ynnOFo2NFNis2Mx2 hk/7cV6vbsmGSrwJ2iVeHMILhvAn8L5Xo3xTNRz+L7oIpX9c8wmJprsGFk2FWgehSQFA STa+qJ6FgzajUZBTqsNxBXXqThDd6mcswueIk0HojplrIuZD1z7Jul4rNrmB5CPghWUL m6Kw== X-Gm-Message-State: AOAM530F51u2Cm3WYz7Q5Lc5opqyKvQjHtj0RQMxSwONon81DtA1t01z cxrlQw804R9W4hvDH8yGwJ/5cB5ivIfzrjLt X-Google-Smtp-Source: ABdhPJyAEBIbXLeziEUyOt0RBLRzvSWbyKt547rN8fkqs0lzzwo+BVyxMr5ox2HWP4ZSHkg45PEOqg== X-Received: by 2002:a6b:e814:: with SMTP id f20mr3220304ioh.206.1617991915056; Fri, 09 Apr 2021 11:11:55 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id s15sm1449330ilv.13.2021.04.09.11.11.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:54 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:52 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 14/22] t5310: move some tests to lib-bitmap.sh Message-ID: <570e3de9ed9d597cb2b9dcc38490836e45998e58.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We'll soon be adding a test script that will cover many of the same bitmap concepts as t5310, but for MIDX bitmaps. Let's pull out as many of the applicable tests as we can so we don't have to rewrite them. There should be no functional change to t5310; we still run the same operations in the same order. Signed-off-by: Taylor Blau --- t/lib-bitmap.sh | 212 ++++++++++++++++++++++++++++++++++++++++ t/t5310-pack-bitmaps.sh | 204 +------------------------------------- 2 files changed, 216 insertions(+), 200 deletions(-) diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index fe3f98be24..c655a9bf35 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -1,3 +1,6 @@ +# Helpers for scripts testing bitamp functionality; see t5310 for +# example usage. + # Compare a file containing rev-list bitmap traversal output to its non-bitmap # counterpart. You can't just use test_cmp for this, because the two produce # subtly different output: @@ -24,3 +27,212 @@ test_bitmap_traversal () { test_cmp "$1.normalized" "$2.normalized" && rm -f "$1.normalized" "$2.normalized" } + +# To ensure the logic for "maximal commits" is exercised, make +# the repository a bit more complicated. +# +# other second +# * * +# (99 commits) (99 commits) +# * * +# |\ /| +# | * octo-other octo-second * | +# |/|\_________ ____________/|\| +# | \ \/ __________/ | +# | | ________/\ / | +# * |/ * merge-right * +# | _|__________/ \____________ | +# |/ | \| +# (l1) * * merge-left * (r1) +# | / \________________________ | +# |/ \| +# (l2) * * (r2) +# \___________________________ | +# \| +# * (base) +# +# We only push bits down the first-parent history, which +# makes some of these commits unimportant! +# +# The important part for the maximal commit algorithm is how +# the bitmasks are extended. Assuming starting bit positions +# for second (bit 0) and other (bit 1), the bitmasks at the +# end should be: +# +# second: 1 (maximal, selected) +# other: 01 (maximal, selected) +# (base): 11 (maximal) +# +# This complicated history was important for a previous +# version of the walk that guarantees never walking a +# commit multiple times. That goal might be important +# again, so preserve this complicated case. For now, this +# test will guarantee that the bitmaps are computed +# correctly, even with the repeat calculations. +setup_bitmap_history() { + test_expect_success 'setup repo with moderate-sized history' ' + test_commit_bulk --id=file 10 && + git branch -M second && + git checkout -b other HEAD~5 && + test_commit_bulk --id=side 10 && + + # add complicated history setup, including merges and + # ambiguous merge-bases + + git checkout -b merge-left other~2 && + git merge second~2 -m "merge-left" && + + git checkout -b merge-right second~1 && + git merge other~1 -m "merge-right" && + + git checkout -b octo-second second && + git merge merge-left merge-right -m "octopus-second" && + + git checkout -b octo-other other && + git merge merge-left merge-right -m "octopus-other" && + + git checkout other && + git merge octo-other -m "pull octopus" && + + git checkout second && + git merge octo-second -m "pull octopus" && + + # Remove these branches so they are not selected + # as bitmap tips + git branch -D merge-left && + git branch -D merge-right && + git branch -D octo-other && + git branch -D octo-second && + + # add padding to make these merges less interesting + # and avoid having them selected for bitmaps + test_commit_bulk --id=file 100 && + git checkout other && + test_commit_bulk --id=side 100 && + git checkout second && + + bitmaptip=$(git rev-parse second) && + blob=$(echo tagged-blob | git hash-object -w --stdin) && + git tag tagged-blob $blob + ' +} + +rev_list_tests_head () { + test_expect_success "counting commits via bitmap ($state, $branch)" ' + git rev-list --count $branch >expect && + git rev-list --use-bitmap-index --count $branch >actual && + test_cmp expect actual + ' + + test_expect_success "counting partial commits via bitmap ($state, $branch)" ' + git rev-list --count $branch~5..$branch >expect && + git rev-list --use-bitmap-index --count $branch~5..$branch >actual && + test_cmp expect actual + ' + + test_expect_success "counting commits with limit ($state, $branch)" ' + git rev-list --count -n 1 $branch >expect && + git rev-list --use-bitmap-index --count -n 1 $branch >actual && + test_cmp expect actual + ' + + test_expect_success "counting non-linear history ($state, $branch)" ' + git rev-list --count other...second >expect && + git rev-list --use-bitmap-index --count other...second >actual && + test_cmp expect actual + ' + + test_expect_success "counting commits with limiting ($state, $branch)" ' + git rev-list --count $branch -- 1.t >expect && + git rev-list --use-bitmap-index --count $branch -- 1.t >actual && + test_cmp expect actual + ' + + test_expect_success "counting objects via bitmap ($state, $branch)" ' + git rev-list --count --objects $branch >expect && + git rev-list --use-bitmap-index --count --objects $branch >actual && + test_cmp expect actual + ' + + test_expect_success "enumerate commits ($state, $branch)" ' + git rev-list --use-bitmap-index $branch >actual && + git rev-list $branch >expect && + test_bitmap_traversal --no-confirm-bitmaps expect actual + ' + + test_expect_success "enumerate --objects ($state, $branch)" ' + git rev-list --objects --use-bitmap-index $branch >actual && + git rev-list --objects $branch >expect && + test_bitmap_traversal expect actual + ' + + test_expect_success "bitmap --objects handles non-commit objects ($state, $branch)" ' + git rev-list --objects --use-bitmap-index $branch tagged-blob >actual && + grep $blob actual + ' +} + +rev_list_tests () { + state=$1 + + for branch in "second" "other" + do + rev_list_tests_head + done +} + +basic_bitmap_tests () { + tip="$1" + test_expect_success 'rev-list --test-bitmap verifies bitmaps' " + git rev-list --test-bitmap "${tip:-HEAD}" + " + + rev_list_tests 'full bitmap' + + test_expect_success 'clone from bitmapped repository' ' + rm -fr clone.git && + git clone --no-local --bare . clone.git && + git rev-parse HEAD >expect && + git --git-dir=clone.git rev-parse HEAD >actual && + test_cmp expect actual + ' + + test_expect_success 'partial clone from bitmapped repository' ' + test_config uploadpack.allowfilter true && + rm -fr partial-clone.git && + git clone --no-local --bare --filter=blob:none . partial-clone.git && + ( + cd partial-clone.git && + pack=$(echo objects/pack/*.pack) && + git verify-pack -v "$pack" >have && + awk "/blob/ { print \$1 }" blobs && + # we expect this single blob because of the direct ref + git rev-parse refs/tags/tagged-blob >expect && + test_cmp expect blobs + ) + ' + + test_expect_success 'setup further non-bitmapped commits' ' + test_commit_bulk --id=further 10 + ' + + rev_list_tests 'partial bitmap' + + test_expect_success 'fetch (partial bitmap)' ' + git --git-dir=clone.git fetch origin second:second && + git rev-parse HEAD >expect && + git --git-dir=clone.git rev-parse HEAD >actual && + test_cmp expect actual + ' +} + +# have_delta +# +# Note that because this relies on cat-file, it might find _any_ copy of an +# object in the repository. The caller is responsible for making sure +# there's only one (e.g., via "repack -ad", or having just fetched a copy). +have_delta () { + echo $2 >expect && + echo $1 | git cat-file --batch-check="%(deltabase)" >actual && + test_cmp expect actual +} diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index f53efc8229..4318f84d53 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -25,93 +25,10 @@ has_any () { grep -Ff "$1" "$2" } -# To ensure the logic for "maximal commits" is exercised, make -# the repository a bit more complicated. -# -# other second -# * * -# (99 commits) (99 commits) -# * * -# |\ /| -# | * octo-other octo-second * | -# |/|\_________ ____________/|\| -# | \ \/ __________/ | -# | | ________/\ / | -# * |/ * merge-right * -# | _|__________/ \____________ | -# |/ | \| -# (l1) * * merge-left * (r1) -# | / \________________________ | -# |/ \| -# (l2) * * (r2) -# \___________________________ | -# \| -# * (base) -# -# We only push bits down the first-parent history, which -# makes some of these commits unimportant! -# -# The important part for the maximal commit algorithm is how -# the bitmasks are extended. Assuming starting bit positions -# for second (bit 0) and other (bit 1), the bitmasks at the -# end should be: -# -# second: 1 (maximal, selected) -# other: 01 (maximal, selected) -# (base): 11 (maximal) -# -# This complicated history was important for a previous -# version of the walk that guarantees never walking a -# commit multiple times. That goal might be important -# again, so preserve this complicated case. For now, this -# test will guarantee that the bitmaps are computed -# correctly, even with the repeat calculations. +setup_bitmap_history -test_expect_success 'setup repo with moderate-sized history' ' - test_commit_bulk --id=file 10 && - git branch -M second && - git checkout -b other HEAD~5 && - test_commit_bulk --id=side 10 && - - # add complicated history setup, including merges and - # ambiguous merge-bases - - git checkout -b merge-left other~2 && - git merge second~2 -m "merge-left" && - - git checkout -b merge-right second~1 && - git merge other~1 -m "merge-right" && - - git checkout -b octo-second second && - git merge merge-left merge-right -m "octopus-second" && - - git checkout -b octo-other other && - git merge merge-left merge-right -m "octopus-other" && - - git checkout other && - git merge octo-other -m "pull octopus" && - - git checkout second && - git merge octo-second -m "pull octopus" && - - # Remove these branches so they are not selected - # as bitmap tips - git branch -D merge-left && - git branch -D merge-right && - git branch -D octo-other && - git branch -D octo-second && - - # add padding to make these merges less interesting - # and avoid having them selected for bitmaps - test_commit_bulk --id=file 100 && - git checkout other && - test_commit_bulk --id=side 100 && - git checkout second && - - bitmaptip=$(git rev-parse second) && - blob=$(echo tagged-blob | git hash-object -w --stdin) && - git tag tagged-blob $blob && - git config repack.writebitmaps true +test_expect_success 'setup writing bitmaps during repack' ' + git config repack.writeBitmaps true ' test_expect_success 'full repack creates bitmaps' ' @@ -123,109 +40,7 @@ test_expect_success 'full repack creates bitmaps' ' grep "\"key\":\"num_maximal_commits\",\"value\":\"107\"" trace ' -test_expect_success 'rev-list --test-bitmap verifies bitmaps' ' - git rev-list --test-bitmap HEAD -' - -rev_list_tests_head () { - test_expect_success "counting commits via bitmap ($state, $branch)" ' - git rev-list --count $branch >expect && - git rev-list --use-bitmap-index --count $branch >actual && - test_cmp expect actual - ' - - test_expect_success "counting partial commits via bitmap ($state, $branch)" ' - git rev-list --count $branch~5..$branch >expect && - git rev-list --use-bitmap-index --count $branch~5..$branch >actual && - test_cmp expect actual - ' - - test_expect_success "counting commits with limit ($state, $branch)" ' - git rev-list --count -n 1 $branch >expect && - git rev-list --use-bitmap-index --count -n 1 $branch >actual && - test_cmp expect actual - ' - - test_expect_success "counting non-linear history ($state, $branch)" ' - git rev-list --count other...second >expect && - git rev-list --use-bitmap-index --count other...second >actual && - test_cmp expect actual - ' - - test_expect_success "counting commits with limiting ($state, $branch)" ' - git rev-list --count $branch -- 1.t >expect && - git rev-list --use-bitmap-index --count $branch -- 1.t >actual && - test_cmp expect actual - ' - - test_expect_success "counting objects via bitmap ($state, $branch)" ' - git rev-list --count --objects $branch >expect && - git rev-list --use-bitmap-index --count --objects $branch >actual && - test_cmp expect actual - ' - - test_expect_success "enumerate commits ($state, $branch)" ' - git rev-list --use-bitmap-index $branch >actual && - git rev-list $branch >expect && - test_bitmap_traversal --no-confirm-bitmaps expect actual - ' - - test_expect_success "enumerate --objects ($state, $branch)" ' - git rev-list --objects --use-bitmap-index $branch >actual && - git rev-list --objects $branch >expect && - test_bitmap_traversal expect actual - ' - - test_expect_success "bitmap --objects handles non-commit objects ($state, $branch)" ' - git rev-list --objects --use-bitmap-index $branch tagged-blob >actual && - grep $blob actual - ' -} - -rev_list_tests () { - state=$1 - - for branch in "second" "other" - do - rev_list_tests_head - done -} - -rev_list_tests 'full bitmap' - -test_expect_success 'clone from bitmapped repository' ' - git clone --no-local --bare . clone.git && - git rev-parse HEAD >expect && - git --git-dir=clone.git rev-parse HEAD >actual && - test_cmp expect actual -' - -test_expect_success 'partial clone from bitmapped repository' ' - test_config uploadpack.allowfilter true && - git clone --no-local --bare --filter=blob:none . partial-clone.git && - ( - cd partial-clone.git && - pack=$(echo objects/pack/*.pack) && - git verify-pack -v "$pack" >have && - awk "/blob/ { print \$1 }" blobs && - # we expect this single blob because of the direct ref - git rev-parse refs/tags/tagged-blob >expect && - test_cmp expect blobs - ) -' - -test_expect_success 'setup further non-bitmapped commits' ' - test_commit_bulk --id=further 10 -' - -rev_list_tests 'partial bitmap' - -test_expect_success 'fetch (partial bitmap)' ' - git --git-dir=clone.git fetch origin second:second && - git rev-parse HEAD >expect && - git --git-dir=clone.git rev-parse HEAD >actual && - test_cmp expect actual -' +basic_bitmap_tests test_expect_success 'incremental repack fails when bitmaps are requested' ' test_commit more-1 && @@ -461,17 +276,6 @@ test_expect_success 'truncated bitmap fails gracefully (cache)' ' test_i18ngrep corrupted.bitmap.index stderr ' -# have_delta -# -# Note that because this relies on cat-file, it might find _any_ copy of an -# object in the repository. The caller is responsible for making sure -# there's only one (e.g., via "repack -ad", or having just fetched a copy). -have_delta () { - echo $2 >expect && - echo $1 | git cat-file --batch-check="%(deltabase)" >actual && - test_cmp expect actual -} - # Create a state of history with these properties: # # - refs that allow a client to fetch some new history, while sharing some old From patchwork Fri Apr 9 18:11:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 326B5C433B4 for ; Fri, 9 Apr 2021 18:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 080A5610CD for ; Fri, 9 Apr 2021 18:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234662AbhDISMY (ORCPT ); Fri, 9 Apr 2021 14:12:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234534AbhDISMO (ORCPT ); Fri, 9 Apr 2021 14:12:14 -0400 Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE8C1C061761 for ; Fri, 9 Apr 2021 11:11:59 -0700 (PDT) Received: by mail-il1-x133.google.com with SMTP id z9so5456340ilb.4 for ; Fri, 09 Apr 2021 11:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=SDGKy41XqkDSXhJ1++SeUTTUQsq4CUovBkrBgBCqmr0=; b=V8ya9Ou64l4OZSHYEl30Bn6aiolsGdFAuK4nTTgSpymswy4ObvM0zBTu2DJ3kBsw8K Sj0Bu3suchb6KG4zRHnYvrcV8xF78L9QvnUCpCrbBqa9gW1m4quOm655TrbgPb7XIA6c 9Vo5JMJ3oXTQ+/IESnh4upB0c+W4rMjy6sV7j7vem4KxnO1SNiHedHZk7XgkQmLymPj6 A1L4RL/IROkz2QHBSxx7jQDuBCtXZZvOHaZSh8BHXWGU7KKUctQuHMTklal38D73W5WV b0YjZ5qhhgUFYm9H2wd/itUaWpRExQIFBhR1/bAQET5rriGByIC2JUXSuS44KaAh2V88 Jm+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=SDGKy41XqkDSXhJ1++SeUTTUQsq4CUovBkrBgBCqmr0=; b=TMUktqYlB3YfPeng5uv2catM0CjU51XASM4uJpZDRdBq0CAKSOL5iWWeJJj7w8++S4 sKFJCr/d3C68umN716Wfv8UsnQMQtFOLRrbZVAEpp3X23So54wrZ7+u7EXUmgQhf9uTM NdQjeuLUeg/aOadj9EUOK3JE/lK2S7cnfSRTYF1TlDVFx5s5qYYd+axkqpAZ+wVq8b9f JP2OeoAVN05yABPc50lwRlrsd2e4WFPqISVLaXA1KanUgkclaTfvGw8KPz3j1K23mmZG t6fkMfUO/IMY2PKgf/jHqJo1xXuOfTb1U0S0f0b8SdycTwri10vE0oPcWCnnzBZROQcl +A6w== X-Gm-Message-State: AOAM533WeyYqCKq2iPHCexxamf/A/dyF/Bj8hrJEm3izUIN6oX5Y81Ss 97fF/uf17cQtM60jCQGmueMspz1NhTr2NyRF X-Google-Smtp-Source: ABdhPJzn/SVAwRfUEwnCHD27a7DirdgzmMJyU5FkTo3yxhhBGxnIMqpnp/iDAvX3wPkIQyF8gULWsQ== X-Received: by 2002:a05:6e02:4b1:: with SMTP id e17mr13199992ils.94.1617991919056; Fri, 09 Apr 2021 11:11:59 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id x20sm1532088ilc.88.2021.04.09.11.11.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:11:58 -0700 (PDT) Date: Fri, 9 Apr 2021 14:11:56 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 15/22] t/helper/test-read-midx.c: add --checksum mode Message-ID: <060ee427bebdb6f45766c173878a9c056f312baa.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Subsequent tests will want to check for the existence of a multi-pack bitmap which matches the multi-pack-index stored in the pack directory. The multi-pack bitmap includes the hex checksum of the MIDX it corresponds to in its filename (for example, '$packdir/multi-pack-index-.bitmap'). As a result, some tests want a way to learn what '' is. This helper addresses that need by printing the checksum of the repository's multi-pack-index. Signed-off-by: Taylor Blau --- t/helper/test-read-midx.c | 16 +++++++++++++++- t/lib-bitmap.sh | 4 ++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c index 7c2eb11a8e..cb0d27049a 100644 --- a/t/helper/test-read-midx.c +++ b/t/helper/test-read-midx.c @@ -60,12 +60,26 @@ static int read_midx_file(const char *object_dir, int show_objects) return 0; } +static int read_midx_checksum(const char *object_dir) +{ + struct multi_pack_index *m; + + setup_git_directory(); + m = load_multi_pack_index(object_dir, 1); + if (!m) + return 1; + printf("%s\n", hash_to_hex(get_midx_checksum(m))); + return 0; +} + int cmd__read_midx(int argc, const char **argv) { if (!(argc == 2 || argc == 3)) - usage("read-midx [--show-objects] "); + usage("read-midx [--show-objects|--checksum] "); if (!strcmp(argv[1], "--show-objects")) return read_midx_file(argv[2], 1); + else if (!strcmp(argv[1], "--checksum")) + return read_midx_checksum(argv[2]); return read_midx_file(argv[1], 0); } diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index c655a9bf35..a5ac8b41a7 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -236,3 +236,7 @@ have_delta () { echo $1 | git cat-file --batch-check="%(deltabase)" >actual && test_cmp expect actual } + +midx_checksum () { + test-tool read-midx --checksum "${1:-.git/objects}" +} From patchwork Fri Apr 9 18:12:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A59F5C433ED for ; Fri, 9 Apr 2021 18:12:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 734136115B for ; Fri, 9 Apr 2021 18:12:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234666AbhDISMW (ORCPT ); Fri, 9 Apr 2021 14:12:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234657AbhDISMR (ORCPT ); Fri, 9 Apr 2021 14:12:17 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51C9CC061762 for ; Fri, 9 Apr 2021 11:12:04 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id j26so6794727iog.13 for ; Fri, 09 Apr 2021 11:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=bnhu1WdSK2enzrNWsWRMgkG8Greqw8cXSJw8oJfXEGE=; b=u6x6vG8OTNricqXKTujNGq5PQ6hid98M0XZ9VvP6jl3fRWbHYXCy1xMFpbXi7UtQS6 wQM8Orq4V1+eeg/OkYSSIrWvMSC41QismsVBRZLp1BvHLws6HmyzdynJNSBZytPtUe6F l5vJK4oeyNNv+rb/n4jxwYRbbtbNpmPE7gdZ0IiA6Uq9l7IAfrvpkWyfQiopiPX8sFH7 YpfLJSEjO7iO2vOYXoUMBh3SvschSO5yw/qSI9OrVJNylfXb0sV3KqCSkRJwmBOMdns3 AqTmjhvtSw+GqHPfyYJ0o5pWdrDJlJe3M0rMXvOAxNFrazI/asXTsd3cSMaTSPEZOtbu FamA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=bnhu1WdSK2enzrNWsWRMgkG8Greqw8cXSJw8oJfXEGE=; b=gvvTY9i9E0WXszLRUCoVD7o7/xIUpdFGR7STbnh5ACY20sljqfQKTOACTJbRbNam3I udRLgsFnFP3vdD1wmM+v7tFydnOauQvEdAcC1WRj6Kk3p/RPu6TeksP5k4hV+hiWxK90 2ascdSUeI6RbxhldIqgEDIKL+4tCPciHMiW6RbOWnqtz+w9sFxi21OdzD2jknUWPRQZl cndOYEwr5TJ2Z4gztWqoxZW8LPjbOMabVn49yfk2+qNAnchzxAQFW98woFmll9wcyDbE ZWy4n5nDQNNXnKw8Or6M9usMB8Vs7w7me3ZJqLj7vgIc4CmtpGyBMOp4OBBw9yauQO1n +ooQ== X-Gm-Message-State: AOAM531E/xDmYzQ+/tbGygWrpftxmBfUiZDQ5VrbE8zTe/peJHbpkVOg /rP9bvBGG7AU0WWgbYSV7O9YLM9C2sFe2y3K X-Google-Smtp-Source: ABdhPJww4X1SQ5wu1WXRCIZannUFSdwggSLQAriEnUV23yegeEQ/cJvWcLPyVQQV4/QiRQdXkwN7rg== X-Received: by 2002:a05:6638:381c:: with SMTP id i28mr15605976jav.60.1617991923063; Fri, 09 Apr 2021 11:12:03 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id k14sm1512942iov.35.2021.04.09.11.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:02 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:00 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 16/22] t5326: test multi-pack bitmap behavior Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This patch introduces a new test, t5326, which tests the basic functionality of multi-pack bitmaps. Some trivial behavior is tested, such as: - Whether bitmaps can be generated with more than one pack. - Whether clones can be served with all objects in the bitmap. - Whether follow-up fetches can be served with some objects outside of the server's bitmap These use lib-bitmap's tests (which in turn were pulled from t5310), and we cover cases where the MIDX represents both a single pack and multiple packs. In addition, some non-trivial and MIDX-specific behavior is tested, too, including: - Whether multi-pack bitmaps behave correctly with respect to the pack-reuse machinery when the base for some object is selected from a different pack than the delta. - Whether multi-pack bitmaps correctly respect the pack.preferBitmapTips configuration. Signed-off-by: Taylor Blau --- t/t5326-multi-pack-bitmaps.sh | 278 ++++++++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100755 t/t5326-multi-pack-bitmaps.sh diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh new file mode 100755 index 0000000000..51c4f9ad78 --- /dev/null +++ b/t/t5326-multi-pack-bitmaps.sh @@ -0,0 +1,278 @@ +#!/bin/sh + +test_description='exercise basic multi-pack bitmap functionality' +. ./test-lib.sh +. "${TEST_DIRECTORY}/lib-bitmap.sh" + +# We'll be writing our own midx and bitmaps, so avoid getting confused by the +# automatic ones. +GIT_TEST_MULTI_PACK_INDEX=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 + +objdir=.git/objects +midx=$objdir/pack/multi-pack-index + +# midx_pack_source +midx_pack_source () { + test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2 +} + +setup_bitmap_history + +test_expect_success 'enable core.multiPackIndex' ' + git config core.multiPackIndex true +' + +test_expect_success 'create single-pack midx with bitmaps' ' + git repack -ad && + git multi-pack-index write --bitmap && + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap +' + +basic_bitmap_tests + +test_expect_success 'create new additional packs' ' + for i in $(test_seq 1 16) + do + test_commit "$i" && + git repack -d + done && + + git checkout -b other2 HEAD~8 && + for i in $(test_seq 1 8) + do + test_commit "side-$i" && + git repack -d + done && + git checkout second +' + +test_expect_success 'create multi-pack midx with bitmaps' ' + git multi-pack-index write --bitmap && + + ls $objdir/pack/pack-*.pack >packs && + test_line_count = 26 packs && + + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap +' + +basic_bitmap_tests + +test_expect_success '--no-bitmap is respected when bitmaps exist' ' + git multi-pack-index write --bitmap && + + test_commit respect--no-bitmap && + GIT_TEST_MULTI_PACK_INDEX=0 git repack -d && + + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + + git multi-pack-index write --no-bitmap && + + test_path_is_file $midx && + test_path_is_missing $midx-$(midx_checksum $objdir).bitmap +' + +test_expect_success 'setup midx with base from later pack' ' + # Write a and b so that "a" is a delta on top of base "b", since Git + # prefers to delete contents out of a base rather than add to a shorter + # object. + test_seq 1 128 >a && + test_seq 1 130 >b && + + git add a b && + git commit -m "initial commit" && + + a=$(git rev-parse HEAD:a) && + b=$(git rev-parse HEAD:b) && + + # In the first pack, "a" is stored as a delta to "b". + p1=$(git pack-objects .git/objects/pack/pack <<-EOF + $a + $b + EOF + ) && + + # In the second pack, "a" is missing, and "b" is not a delta nor base to + # any other object. + p2=$(git pack-objects .git/objects/pack/pack <<-EOF + $b + $(git rev-parse HEAD) + $(git rev-parse HEAD^{tree}) + EOF + ) && + + git prune-packed && + # Use the second pack as the preferred source, so that "b" occurs + # earlier in the MIDX object order, rendering "a" unusable for pack + # reuse. + git multi-pack-index write --bitmap --preferred-pack=pack-$p2.idx && + + have_delta $a $b && + test $(midx_pack_source $a) != $(midx_pack_source $b) +' + +rev_list_tests 'full bitmap with backwards delta' + +test_expect_success 'clone with bitmaps enabled' ' + git clone --no-local --bare . clone-reverse-delta.git && + test_when_finished "rm -fr clone-reverse-delta.git" && + + git rev-parse HEAD >expect && + git --git-dir=clone-reverse-delta.git rev-parse HEAD >actual && + test_cmp expect actual +' + +bitmap_reuse_tests() { + from=$1 + to=$2 + + test_expect_success "setup pack reuse tests ($from -> $to)" ' + rm -fr repo && + git init repo && + ( + cd repo && + test_commit_bulk 16 && + git tag old-tip && + + git config core.multiPackIndex true && + if test "MIDX" = "$from" + then + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad && + git multi-pack-index write --bitmap + else + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb + fi + ) + ' + + test_expect_success "build bitmap from existing ($from -> $to)" ' + ( + cd repo && + test_commit_bulk --id=further 16 && + git tag new-tip && + + if test "MIDX" = "$to" + then + GIT_TEST_MULTI_PACK_INDEX=0 git repack -d && + git multi-pack-index write --bitmap + else + GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb + fi + ) + ' + + test_expect_success "verify resulting bitmaps ($from -> $to)" ' + ( + cd repo && + git for-each-ref && + git rev-list --test-bitmap refs/tags/old-tip && + git rev-list --test-bitmap refs/tags/new-tip + ) + ' +} + +bitmap_reuse_tests 'pack' 'MIDX' +bitmap_reuse_tests 'MIDX' 'pack' +bitmap_reuse_tests 'MIDX' 'MIDX' + +test_expect_success 'missing object closure fails gracefully' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + test_commit loose && + test_commit packed && + + # Do not pass "--revs"; we want a pack without the "loose" + # commit. + git pack-objects $objdir/pack/pack <<-EOF && + $(git rev-parse packed) + EOF + + git multi-pack-index write --bitmap 2>err && + grep "doesn.t have full closure" err && + test_path_is_file $midx && + test_path_is_missing $midx-$(midx_checksum $objdir).bitmap + ) +' + +test_expect_success 'setup partial bitmaps' ' + test_commit packed && + git repack && + test_commit loose && + git multi-pack-index write --bitmap 2>err && + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap +' + +basic_bitmap_tests HEAD~ + +test_expect_success 'removing a MIDX clears stale bitmaps' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + test_commit base && + git repack && + git multi-pack-index write --bitmap && + + # Write a MIDX and bitmap; remove the MIDX but leave the bitmap. + stale_bitmap=$midx-$(midx_checksum $objdir).bitmap && + rm $midx && + + # Then write a new MIDX. + test_commit new && + git repack && + git multi-pack-index write --bitmap && + + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + test_path_is_missing $stale_bitmap + ) +' + +test_expect_success 'pack.preferBitmapTips' ' + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + + test_commit_bulk --message="%s" 103 && + + git log --format="%H" >commits.raw && + sort commits && + + git log --format="create refs/tags/%s %H" HEAD >refs && + git update-ref --stdin bitmaps && + comm -13 bitmaps commits >before && + test_line_count = 1 before && + + perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \ + bitmaps && + comm -13 bitmaps commits >after && + + ! test_cmp before after + ) +' + +test_done From patchwork Fri Apr 9 18:12:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DB2BC43460 for ; Fri, 9 Apr 2021 18:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEA3A61178 for ; Fri, 9 Apr 2021 18:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234408AbhDISM0 (ORCPT ); Fri, 9 Apr 2021 14:12:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234594AbhDISMW (ORCPT ); Fri, 9 Apr 2021 14:12:22 -0400 Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1867AC061762 for ; Fri, 9 Apr 2021 11:12:08 -0700 (PDT) Received: by mail-io1-xd35.google.com with SMTP id x17so6827317iog.2 for ; Fri, 09 Apr 2021 11:12:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=C5Yl6mIwi9zLmDdBydnpT2kQELkZPYxenRWJSBxoMY8=; b=VoZVf3tnBvzAYm0NhNA5kOAe3FMOqSbel9SiYhDlaUMsz5K6UQn6FM1otTNj8Dij6J +5sOe5pFmZk76DOhApIfiBxlAhfR3I5AeCY/datEg5aX9TYnAx709sSI5/hKI5nOyrHn 24Cw/8sFWeWI8eX7hpJsBJdu37Vk9asmm+L6yMbkHvtF2/gH1iyzG0VQapiInL7YD5Hq dM3/dLb9x6dZHuyXW4K5AqU1tcnyc8DZRhQbUpz7EmrwP7nYz8Tna7cQ3m6w+HzWH1Pn u8tQanzfiGrxrOMS17pbe7ELesBlNd70G7662Wj7QAm5UkEGjKGF9HfJtclK+j0gKEtp 0QYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=C5Yl6mIwi9zLmDdBydnpT2kQELkZPYxenRWJSBxoMY8=; b=RtG9A7qrL8FCoe6onkzMcmYssmU1F5vE/V8xWYgF6+9G8LUXkgCWg2Moax0NFCqgHk ydrHZ8YbS9WVBLoYteISdCvT84gE/IhaaWcjA1R1p9XssTTAxCv1aKc3gzEZw7cSo39Z 8MQFFU1kbzzhkn9RBOJw20a+JQJyHs1Hu1f4O03NjnMZjzzURMJR1E8+6S8ptFKYqDzD zm0crV6lc5j5XYjfTkeH1B6JB7VbgNhxBVZgrRzingRd+msasCAlfxKVR5mf0E9cz8Ey vZNYslc+MSPO1sjxBpJbQaG2oDwzs070/tsNjVy1PUvk8PMS4lJ2IDJSdHheYwnpCVSD xfxQ== X-Gm-Message-State: AOAM533qDFxUhNFt1qt7RyIcqmycVZAkWAd2urIKTNro8DaYq5TbLB2Y fBXivn6/KttnyBW86ix6uh7/BusoAPsKEjH1 X-Google-Smtp-Source: ABdhPJwzFTVm8+kPUcjwp442F0DgKwzJ9bd9gVwy7R9uVA4kgMcYewlb+VaOCo7UAWmwdpctV3kl4Q== X-Received: by 2002:a05:6638:d4e:: with SMTP id d14mr16059466jak.103.1617991927333; Fri, 09 Apr 2021 11:12:07 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id c19sm1505680ilk.42.2021.04.09.11.12.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:06 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:04 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 17/22] t5310: disable GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP Message-ID: <8f328bb5bc46149bd05860c29be032fe219951a4.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Generating a MIDX bitmap confuses many of the tests in t5310, which expect to control whether and how bitmaps are written. Since the relevant MIDX-bitmap tests here are covered already in t5326, let's just disable the flag for the whole t5310 script. Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- t/t5310-pack-bitmaps.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index 4318f84d53..673baa5c3c 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -8,6 +8,10 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . "$TEST_DIRECTORY"/lib-bundle.sh . "$TEST_DIRECTORY"/lib-bitmap.sh +# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in +# their place. +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 + objpath () { echo ".git/objects/$(echo "$1" | sed -e 's|\(..\)|\1/|')" } From patchwork Fri Apr 9 18:12:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AD1FC433B4 for ; Fri, 9 Apr 2021 18:12:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62644610A6 for ; Fri, 9 Apr 2021 18:12:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234663AbhDISMd (ORCPT ); Fri, 9 Apr 2021 14:12:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234674AbhDISMZ (ORCPT ); Fri, 9 Apr 2021 14:12:25 -0400 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B4BCC061762 for ; Fri, 9 Apr 2021 11:12:12 -0700 (PDT) Received: by mail-qk1-x72f.google.com with SMTP id q26so6729464qkm.6 for ; Fri, 09 Apr 2021 11:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=urkpKRCVUcnkeA5xm5AkC500A1RReChHW2zksdFNMEU=; b=Y32qPnFrY2GnjdmoBhdubkfBSn+fR3PTUBebny2ITdo4wTNFD0AINA481inRYPwhqW fwd/eW8+ucbb+8GLvkWuAWnRztbxCvKWUfLKE3i5xk1KzOyhT4OzUpdfcJP/x3xwup/P 3yprkYaxeeNzA5I0EPXGpwxij1zGV0Sy8ywpeb4kOhnchNKSPra2aLGuE+mAsYGqXDtT evg8/9FQPWNO9oIhv4LeZB0vQ8mhUbZNl3GdZy14XBgzuZAtUqymRqDfd9ezGLgyQgL0 KnOaxDjVNRV8hJpP/1e74HLaIGECzumWd0lzFy40vc2N9cmPxIjjjdzpf+z2NFtCzosp IlJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=urkpKRCVUcnkeA5xm5AkC500A1RReChHW2zksdFNMEU=; b=SRgR4qZuWLh9zl/4HhbHfjuyP+ykWW29g9GNjRObJQLqL8HfZpQzAjxXC+qeHXmw+v XqyyY93vtgonghVILYkgZF8kqI9DG7CtDyhsOdzPpSSPkovDk/uSbWkPwz6Bva7BT2ZE jrM3xPVOYYKIDpCSXPVynuN/fCaJHhEuff7J3q2Pnl6GmHQBwkXyUZAHsisQlTVNLXT2 rvvZGOjAdeztqCVsNBt8S2ypmsCqNLPMNicJBzdEt79DiuUnCb4kMqN7/7Yhkd6xxc+E qKyJIi+0WcbmGJyKvX0zIgRuqK5XVSmoqyO/U0yLT3qz47iekOpfB0zMqXrolJEi2/vv h8+A== X-Gm-Message-State: AOAM532mlFo04AFIO3wW14uI1Djep61VpX1B/OksbU1GBqFN4xgSn/rg P9xGepXNJYYzOCJZ8g29Exv322FYEKjxeF7f X-Google-Smtp-Source: ABdhPJwk2ggzhaFQhlxP7Nic3kD4lC/2LYO3genSm2+lfHgDkyzSmup9WnVxvn1fUQYmSfKtebmwvg== X-Received: by 2002:a37:40d5:: with SMTP id n204mr14934161qka.79.1617991931428; Fri, 09 Apr 2021 11:12:11 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id y9sm2315699qtf.58.2021.04.09.11.12.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:11 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:09 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 18/22] t5319: don't write MIDX bitmaps in t5319 Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This test is specifically about generating a midx still respecting a pack-based bitmap file. Generating a MIDX bitmap would confuse the test. Let's override the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' variable to make sure we don't do so. Signed-off-by: Taylor Blau --- t/t5319-multi-pack-index.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 5641d158df..69f1c815aa 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -474,7 +474,8 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' ' compare_results_with_midx "after repack" test_expect_success 'multi-pack-index and pack-bitmap' ' - git -c repack.writeBitmaps=true repack -ad && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -c repack.writeBitmaps=true repack -ad && git multi-pack-index write && git rev-list --test-bitmap HEAD ' From patchwork Fri Apr 9 18:12:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E50DC433B4 for ; Fri, 9 Apr 2021 18:12:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1C26961007 for ; Fri, 9 Apr 2021 18:12:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234657AbhDISMg (ORCPT ); Fri, 9 Apr 2021 14:12:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234665AbhDISM3 (ORCPT ); Fri, 9 Apr 2021 14:12:29 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B513C061764 for ; Fri, 9 Apr 2021 11:12:16 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id d12so2685923iod.12 for ; Fri, 09 Apr 2021 11:12:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=P9XhA576oXjIKXlNHfQ5aXRwRaVis24OihjlRxxBAZs=; b=U4WQ+N1qd+HJtmsCdtzy/3mf0mquDM+3mTr32e+0bIAg0XEtCSvDcSh5Rij0iqkFPl IdECOnrohgJtULYUEiH6V835kZEaxQjwJ/4CJBwcbncvLgS11UpUAzz9UoHXgD/eacXc 0Vg4btfazENKj+vGiKJFL0L6e6DK1TsBhxZeBWqJAL3p0+DkCm3wWKua5atTNuJ2iHIR j+Zq1cQbuMds0rNAbKTq4BSiPbRtx+oJkpjyBDSh5wEUPcylzlxBe0Py55xSjpCQu2fK F9w76JttRMBpI2RKTuube8S93d5ubZOBD8zpSLfNCp261so5um3DVTtQb5w0CEDY6ChN nVHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=P9XhA576oXjIKXlNHfQ5aXRwRaVis24OihjlRxxBAZs=; b=M0zuFxkYGdU5NBG94Uk3UwQqLXB4MVbqghZo4+M+Km6zWFREPsRliIcQhjslNtk0vQ oZcyiDFf7VDd4b3GMwxPG5oo5ycY3ifVayZgIC1t9A4f/UTt8n/sFeUE+Mz8t4TBeKTV GKzS07yQ2v5+r4+A1SygxWf1jy7Y83y3OQKVdUUckzWNe7wArXesGVk/bA0FyFr5kBop aNSq9C/89dKutic2Y9KY1kl6rYnlxn7Adh5st7+O2EBB5xZMOpPO8nYmSSx3lHJlJwKV JZAStbdBLB2Ud885XuP1GRKw+Y4kiC7NbasqJaTVAZt65bn9XjHZilEmJDY+fvkmGZn0 077w== X-Gm-Message-State: AOAM531qBBd4030KPGD4zZXQQN+mU5XwhRbNuDbebP+7MLsxvUxPKZ1E 6ltpWLpidxnoDBqV77nGnWmc9AXTIbcsUhMA X-Google-Smtp-Source: ABdhPJyKnfI3hPuCPZyva18RBeJEKCIsk7pjmXN/FC64fnqv6qoz7QoCOV+urtOl/R5mp20bdJeSEA== X-Received: by 2002:a05:6638:22a:: with SMTP id f10mr16080798jaq.38.1617991935470; Fri, 09 Apr 2021 11:12:15 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id y15sm1368212ilv.70.2021.04.09.11.12.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:15 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:13 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 19/22] t7700: update to work with MIDX bitmap test knob Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau --- t/t7700-repack.sh | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 25b235c063..98eda3bfeb 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -63,13 +63,14 @@ test_expect_success 'objects in packs marked .keep are not repacked' ' test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - git repack -Adbl && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl && test_has_duplicate_object true ' test_expect_success 'writing bitmaps via config can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - git -c repack.writebitmaps=true repack -Adl && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -c repack.writebitmaps=true repack -Adl && test_has_duplicate_object true ' @@ -189,7 +190,9 @@ test_expect_success 'repack --keep-pack' ' test_expect_success 'bitmaps are created by default in bare repos' ' git clone --bare .git bare.git && - git -C bare.git repack -ad && + rm -f bare.git/objects/pack/*.bitmap && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap) && test_path_is_file "$bitmap" ' @@ -200,7 +203,8 @@ test_expect_success 'incremental repack does not complain' ' ' test_expect_success 'bitmaps can be disabled on bare repos' ' - git -c repack.writeBitmaps=false -C bare.git repack -ad && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -c repack.writeBitmaps=false -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap || :) && test -z "$bitmap" ' @@ -211,7 +215,8 @@ test_expect_success 'no bitmaps created if .keep files present' ' keep=${pack%.pack}.keep && test_when_finished "rm -f \"\$keep\"" && >"$keep" && - git -C bare.git repack -ad 2>stderr && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack/ -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -222,7 +227,8 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' blob=$(test-tool genrandom big $((1024*1024)) | git -C bare.git hash-object -w --stdin) && git -C bare.git update-ref refs/tags/big $blob && - git -C bare.git repack -ad 2>stderr && + GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack -type f -name "*.bitmap" >actual && test_must_be_empty actual From patchwork Fri Apr 9 18:12:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23AE2C433ED for ; Fri, 9 Apr 2021 18:12:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E8C98610CD for ; Fri, 9 Apr 2021 18:12:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234378AbhDISMr (ORCPT ); Fri, 9 Apr 2021 14:12:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234670AbhDISMe (ORCPT ); Fri, 9 Apr 2021 14:12:34 -0400 Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9512BC061762 for ; Fri, 9 Apr 2021 11:12:20 -0700 (PDT) Received: by mail-il1-x129.google.com with SMTP id t14so5454112ilu.3 for ; Fri, 09 Apr 2021 11:12:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=S1ySFHRG+npC90ooRQ0ysx7tyG5oQfPYJznoh4OAs70=; b=iK0kkzDZISCcRhWOTKku2/3PPnkOHQW36S2H9tyCCXxCbtNj+4xyir8jDgo9X228yy aitf2HxCG9QQ9XU9lTlBNmhrRmGUJzeYPMSdhhv33ODsEMZhdqGU38wod08FYNwkKFqZ 4dBt8q6Yo3OOgjvIgKW4qz6RFi9yHtVoLDxXPyM+h2h5NcKFV2hRRsIIfNdxQRmi+eqE nDjUO9cZ0ns/lUeiqy+9f5ciB3tVKd/kuSQIPDrEUGVM4byr5QpL8GSioHtlQCxcVozC D2yVXePd0V+Ch9zTylRpzb9M+RshR7PBB+Nd/uFvwjwTAVEMBVtcUb28rDOG7zFiJnCq VqUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=S1ySFHRG+npC90ooRQ0ysx7tyG5oQfPYJznoh4OAs70=; b=bxPM7sSBe+FoCFa/BqDMMRtPfpBEHYY/J+LAAeMvD7DoXG8r5s16LDm3oYSO0PnMvj 2tvcClPjdCWuu6x/2Hr0ps7QpuJ8wV7PUYblOYor79o0sFLzLiRzthEK05KbW8Rnvjlb CqLnWE8yn9DqhOY4au6pyxj02IpEkt9wvcjEqdMmblNkMC4F4E1gZSyyXBbmCFGYVcU1 N+kBzvfNuD7FwGjW6uK9AQomYTfOtf+449QxVT5stmb4L1Kw5Ss7RZjtoqGB3omcdsBb jC5OciomBzsxx5L1fasUvp5q8kPCK8ZLmeMaCaUiYXe/2YasNrtLw7lFRnmqtIbRuuvQ c1Ag== X-Gm-Message-State: AOAM532+P6QMm+iMjcCvcMpYvQXDtZ0bevFASYwgXekc1VAQOnyxIkN1 1j3rwnl7txN2FfU41MfqBXf31Td3ec4pQe/u X-Google-Smtp-Source: ABdhPJxnJsn0nEhO97VW6ZDyp/0ME/41mBtTyQiFwKmw03DSzQnt07cU/x9Iw/U1SWrwLGJr1elPcA== X-Received: by 2002:a92:3647:: with SMTP id d7mr13129503ilf.264.1617991939794; Fri, 09 Apr 2021 11:12:19 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id m1sm1485583ilh.69.2021.04.09.11.12.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:19 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:17 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 20/22] midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' Message-ID: <83614f928486577892f9345f6a91a6240b47e173.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Introduce a new 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable to also write a multi-pack bitmap when 'GIT_TEST_MULTI_PACK_INDEX' is set. Signed-off-by: Taylor Blau --- builtin/repack.c | 13 ++++++++++--- ci/run-build-and-tests.sh | 1 + midx.h | 2 ++ t/README | 4 ++++ 4 files changed, 17 insertions(+), 3 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 2847fdfbab..3cb843fb59 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -515,7 +515,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()) write_bitmaps = 0; - } + } else if (write_bitmaps && + git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) && + git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) + write_bitmaps = 0; if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps > 0; @@ -720,8 +723,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) update_server_info(0); remove_temporary_files(); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) - write_midx_file(get_object_directory(), NULL, 0); + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { + unsigned flags = 0; + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) + flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX; + write_midx_file(get_object_directory(), NULL, flags); + } string_list_clear(&names, 0); string_list_clear(&rollback, 0); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index a66b5e8c75..7c55a5033e 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -22,6 +22,7 @@ linux-gcc) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 + export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 export GIT_TEST_ADD_I_USE_BUILTIN=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_WRITE_REV_INDEX=1 diff --git a/midx.h b/midx.h index 350f4d0a7b..aa3da557bb 100644 --- a/midx.h +++ b/midx.h @@ -8,6 +8,8 @@ struct pack_entry; struct repository; #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" +#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \ + "GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP" struct multi_pack_index { struct multi_pack_index *next; diff --git a/t/README b/t/README index fd9375b146..956731da44 100644 --- a/t/README +++ b/t/README @@ -420,6 +420,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the +'--bitmap' option on all invocations of 'git multi-pack-index write', +and ignores pack-objects' '--write-bitmap-index'. + GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises From patchwork Fri Apr 9 18:12:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2133FC43460 for ; Fri, 9 Apr 2021 18:12:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 03335610A6 for ; Fri, 9 Apr 2021 18:12:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234669AbhDISMp (ORCPT ); Fri, 9 Apr 2021 14:12:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234665AbhDISMi (ORCPT ); Fri, 9 Apr 2021 14:12:38 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08892C061762 for ; Fri, 9 Apr 2021 11:12:25 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id 7so6720069qka.7 for ; Fri, 09 Apr 2021 11:12:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=sAWq7ApHqrUs88UHszHQDp+raK9BhywjCqAJGMbjwdk=; b=s88Fv3l8pVvVdO5FuBbtEmJxlKx+Az0zT8ozzOnM5TMClUUjnGWrZkHDpUkoVr4qbn MIVQSfm6tF/SzFrQQjRlBzLKE6BICQfA6XEgqv9Uajdv+vfxDqVwjDRkNvNSfyELFwa6 tofrtH8weSOdVkcYFCSWGF4UZHCQi21Z6f8l6h1DqvkzCttkzvwU9+6JETkuM/2w1YEv bKdSni69tMpNYJdQ7/gNr2ngRIpQi3iMMHmlKFKyYcubQVUifKKOagid92I9LUhOT0FD MpwtfseuAL7Cs3blkoShgScz0j6osp7tI+EjYixlZIPJcmjESU/2cUk5tyHgzK3xqCjq X6eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=sAWq7ApHqrUs88UHszHQDp+raK9BhywjCqAJGMbjwdk=; b=M4mKMMvvhmLEeeml+JYsyv9sdsrsoaZu79ZnaHHxQndexUcJYBetq/ZcOMH2er86jz I1iSj8B1Gr37JSiI2Y/0DrPe2xSShHuP2yuv8O9py8gZahUO21m6xmtOPbf0xQiRA1mb mJr/9Sv5VkDqbQ93Qk1K21+rnwPHw7aOI4JDcucAsPxWhAcU4QZqX1F+pjsrQ/QWAJod CuenL4thdpCXlBTZTYdU6G9SCdOn/QcQMTArho9FbSCyMHGtN9jaE13TCmlaD5WkmkQP 8faMnDtFICa7nH1RvGdhWMEWw8r3PE73hCHebJRyKR8PPkX4j0zvZiUf1lCIXV/UYJ/7 noQg== X-Gm-Message-State: AOAM533Zl4UcPyx1O9efFYhR1pzq+o+5sNhyzq6Ceo59QU8EnxKjAWiv F3AidsZ3pGBmHW0VQTR2gS7Ryyps4+e5txjM X-Google-Smtp-Source: ABdhPJwEx9uLqt8iX+Ezl+j/p7jIqHOBtjAf+z9Y34HWYDZB0Ky/A9D5WzZvl9nkWRUEV4yC0muBBw== X-Received: by 2002:a37:9e4e:: with SMTP id h75mr14979295qke.180.1617991943994; Fri, 09 Apr 2021 11:12:23 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id 130sm2378522qkg.22.2021.04.09.11.12.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:23 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:21 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 21/22] p5310: extract full and partial bitmap tests Message-ID: <2f3836bf2e2e7dafc6ad3c1cb20e00de49f638ef.1617991824.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A new p5326 introduced by the next patch will want these same tests, interjecting its own setup in between. Move them out so that both perf tests can reuse them. Signed-off-by: Taylor Blau --- t/perf/lib-bitmap.sh | 69 ++++++++++++++++++++++++++++++++++++ t/perf/p5310-pack-bitmaps.sh | 65 ++------------------------------- 2 files changed, 72 insertions(+), 62 deletions(-) create mode 100644 t/perf/lib-bitmap.sh diff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh new file mode 100644 index 0000000000..63d3bc7cec --- /dev/null +++ b/t/perf/lib-bitmap.sh @@ -0,0 +1,69 @@ +# Helper functions for testing bitmap performance; see p5310. + +test_full_bitmap () { + test_perf 'simulated clone' ' + git pack-objects --stdout --all /dev/null + ' + + test_perf 'simulated fetch' ' + have=$(git rev-list HEAD~100 -1) && + { + echo HEAD && + echo ^$have + } | git pack-objects --revs --stdout >/dev/null + ' + + test_perf 'pack to file (bitmap)' ' + git pack-objects --use-bitmap-index --all pack1b /dev/null + ' + + test_perf 'rev-list (commits)' ' + git rev-list --all --use-bitmap-index >/dev/null + ' + + test_perf 'rev-list (objects)' ' + git rev-list --all --use-bitmap-index --objects >/dev/null + ' + + test_perf 'rev-list with tag negated via --not --all (objects)' ' + git rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null + ' + + test_perf 'rev-list with negative tag (objects)' ' + git rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null + ' + + test_perf 'rev-list count with blob:none' ' + git rev-list --use-bitmap-index --count --objects --all \ + --filter=blob:none >/dev/null + ' + + test_perf 'rev-list count with blob:limit=1k' ' + git rev-list --use-bitmap-index --count --objects --all \ + --filter=blob:limit=1k >/dev/null + ' + + test_perf 'rev-list count with tree:0' ' + git rev-list --use-bitmap-index --count --objects --all \ + --filter=tree:0 >/dev/null + ' + + test_perf 'simulated partial clone' ' + git pack-objects --stdout --all --filter=blob:none /dev/null + ' +} + +test_partial_bitmap () { + test_perf 'clone (partial bitmap)' ' + git pack-objects --stdout --all /dev/null + ' + + test_perf 'pack to file (partial bitmap)' ' + git pack-objects --use-bitmap-index --all pack2b /dev/null + ' + + test_perf 'rev-list with tree filter (partial bitmap)' ' + git rev-list --use-bitmap-index --count --objects --all \ + --filter=tree:0 >/dev/null + ' +} diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh index 452be01056..7ad4f237bc 100755 --- a/t/perf/p5310-pack-bitmaps.sh +++ b/t/perf/p5310-pack-bitmaps.sh @@ -2,6 +2,7 @@ test_description='Tests pack performance using bitmaps' . ./perf-lib.sh +. "${TEST_DIRECTORY}/perf/lib-bitmap.sh" test_perf_large_repo @@ -25,56 +26,7 @@ test_perf 'repack to disk' ' git repack -ad ' -test_perf 'simulated clone' ' - git pack-objects --stdout --all /dev/null -' - -test_perf 'simulated fetch' ' - have=$(git rev-list HEAD~100 -1) && - { - echo HEAD && - echo ^$have - } | git pack-objects --revs --stdout >/dev/null -' - -test_perf 'pack to file (bitmap)' ' - git pack-objects --use-bitmap-index --all pack1b /dev/null -' - -test_perf 'rev-list (commits)' ' - git rev-list --all --use-bitmap-index >/dev/null -' - -test_perf 'rev-list (objects)' ' - git rev-list --all --use-bitmap-index --objects >/dev/null -' - -test_perf 'rev-list with tag negated via --not --all (objects)' ' - git rev-list perf-tag --not --all --use-bitmap-index --objects >/dev/null -' - -test_perf 'rev-list with negative tag (objects)' ' - git rev-list HEAD --not perf-tag --use-bitmap-index --objects >/dev/null -' - -test_perf 'rev-list count with blob:none' ' - git rev-list --use-bitmap-index --count --objects --all \ - --filter=blob:none >/dev/null -' - -test_perf 'rev-list count with blob:limit=1k' ' - git rev-list --use-bitmap-index --count --objects --all \ - --filter=blob:limit=1k >/dev/null -' - -test_perf 'rev-list count with tree:0' ' - git rev-list --use-bitmap-index --count --objects --all \ - --filter=tree:0 >/dev/null -' - -test_perf 'simulated partial clone' ' - git pack-objects --stdout --all --filter=blob:none /dev/null -' +test_full_bitmap test_expect_success 'create partial bitmap state' ' # pick a commit to represent the repo tip in the past @@ -97,17 +49,6 @@ test_expect_success 'create partial bitmap state' ' git update-ref HEAD $orig_tip ' -test_perf 'clone (partial bitmap)' ' - git pack-objects --stdout --all /dev/null -' - -test_perf 'pack to file (partial bitmap)' ' - git pack-objects --use-bitmap-index --all pack2b /dev/null -' - -test_perf 'rev-list with tree filter (partial bitmap)' ' - git rev-list --use-bitmap-index --count --objects --all \ - --filter=tree:0 >/dev/null -' +test_partial_bitmap test_done From patchwork Fri Apr 9 18:12:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 12194755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35687C433B4 for ; Fri, 9 Apr 2021 18:12:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 152E461007 for ; Fri, 9 Apr 2021 18:12:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234049AbhDISMs (ORCPT ); Fri, 9 Apr 2021 14:12:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234671AbhDISMp (ORCPT ); Fri, 9 Apr 2021 14:12:45 -0400 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E160CC061761 for ; Fri, 9 Apr 2021 11:12:28 -0700 (PDT) Received: by mail-il1-x12d.google.com with SMTP id c18so5448486iln.7 for ; Fri, 09 Apr 2021 11:12:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=4z4LT0lv5k9umGSR/ooyeHTuJXpCfSVmHmMOmakBiFE=; b=05AAIlsrSNWhrFoB/ePjtYTOR8WeZ4n6IkxvYT/rt9knilvhikFBd+zNd+AuUNvBQE GBls0EuNX4TZa3P3T6wiBswq1UtIT8XH1mjnwpjuuWLyWfNiEYB/Iy/FqRCzNnEuCNyi lYScuVRxcG02WtJpuhB8G9GVj17zsOGMQR626vpl/KRBG+RuuwU/BW++i7r5tdwoMewl g0IMuzAb2kiFzb3rVNHZq/2mi2O5+NVobj5pEdmsWV+j40XgBdWw81V+PXBDkJp8gmfv n3wDtvL2yCN223iBvZCoWZ/AesEVzZtjVYZn0mwQQDUGI2cFB2jC2X4ICj9C5/CbX9fo QfEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=4z4LT0lv5k9umGSR/ooyeHTuJXpCfSVmHmMOmakBiFE=; b=iG/473HLklgIgBNu1lP/nIpNLEGx/mHa6cZ4qVvLMOdyMjgTuUrWC0l14ZXf9jGeww vFpkN06YDkjFtuSPJqrNcLDnuRU5gBDhH8dIZDpz32IMS/jKeJAwgQfecvBDfaOrZg7C w8OGBFMBRW0nPvMNHYW6jStq8WThpLRxLBMRExs+J03tjN7/Q5ihbv5sufR3HDlQCOjb WDqJ1MRkMa6R/tl/3so5eV+UvJE8E2vFvF0TmbkFjAfuNJdNZsgXn+Qd92N2rRuhPH8F PQFkSQmecpuQu7KyK/q4juqCXktHb9fRllFiCounqLCQnhOTf2YY6U0VD5gfd7w2lHaz hFew== X-Gm-Message-State: AOAM53171fhhQwiZy1OCNOHq4iHncqp1RqEpUTaU0WqKibLaVmmYLHti SBMSiFlCjRAcKKshxyUgBzHBYlwScA3rzR5x X-Google-Smtp-Source: ABdhPJx5nTZoUIDykwiVRoN0e/2+6PnJDy1woQOwM750ES2p7ZlRnIFqB983ZNy659SW5bEX/3B86g== X-Received: by 2002:a92:8752:: with SMTP id d18mr12086930ilm.22.1617991948062; Fri, 09 Apr 2021 11:12:28 -0700 (PDT) Received: from localhost ([2605:9480:22e:ff10:64f1:fbc0:31eb:386a]) by smtp.gmail.com with ESMTPSA id g6sm303271ioq.6.2021.04.09.11.12.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Apr 2021 11:12:27 -0700 (PDT) Date: Fri, 9 Apr 2021 14:12:25 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, dstolee@microsoft.com, gitster@pobox.com, jonathantanmy@google.com Subject: [PATCH 22/22] p5326: perf tests for MIDX bitmaps Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org These new performance tests demonstrate effectively the same behavior as p5310, but use a multi-pack bitmap instead of a single-pack one. Notably, p5326 does not create a MIDX bitmap with multiple packs. This is so we can measure a direct comparison between it and p5310. Any difference between the two is measuring just the overhead of using MIDX bitmaps. Here are the results of p5310 and p5326 together, measured at the same time and on the same machine (using a Xenon W-2255 CPU): Test HEAD ------------------------------------------------------------------------ 5310.2: repack to disk 96.78(93.39+11.33) 5310.3: simulated clone 9.98(9.79+0.19) 5310.4: simulated fetch 1.75(4.26+0.19) 5310.5: pack to file (bitmap) 28.20(27.87+8.70) 5310.6: rev-list (commits) 0.41(0.36+0.05) 5310.7: rev-list (objects) 1.61(1.54+0.07) 5310.8: rev-list count with blob:none 0.25(0.21+0.04) 5310.9: rev-list count with blob:limit=1k 2.65(2.54+0.10) 5310.10: rev-list count with tree:0 0.23(0.19+0.04) 5310.11: simulated partial clone 4.34(4.21+0.12) 5310.13: clone (partial bitmap) 11.05(12.21+0.48) 5310.14: pack to file (partial bitmap) 31.25(34.22+3.70) 5310.15: rev-list with tree filter (partial bitmap) 0.26(0.22+0.04) versus the same tests (this time using a multi-pack index): Test HEAD ------------------------------------------------------------------------ 5326.2: setup multi-pack index 78.99(75.29+11.58) 5326.3: simulated clone 11.78(11.56+0.22) 5326.4: simulated fetch 1.70(4.49+0.13) 5326.5: pack to file (bitmap) 28.02(27.72+8.76) 5326.6: rev-list (commits) 0.42(0.36+0.06) 5326.7: rev-list (objects) 1.65(1.58+0.06) 5326.8: rev-list count with blob:none 0.26(0.21+0.05) 5326.9: rev-list count with blob:limit=1k 2.97(2.86+0.10) 5326.10: rev-list count with tree:0 0.25(0.20+0.04) 5326.11: simulated partial clone 5.65(5.49+0.16) 5326.13: clone (partial bitmap) 12.22(13.43+0.38) 5326.14: pack to file (partial bitmap) 30.05(31.57+7.25) 5326.15: rev-list with tree filter (partial bitmap) 0.24(0.20+0.04) There is slight overhead in "simulated clone", "simulated partial clone", and "clone (partial bitmap)". Unsurprisingly, that overhead is due to using the MIDX's reverse index to map between bit positions and MIDX positions. This can be reproduced by running "git repack -adb" along with "git multi-pack-index write --bitmap" in a large-ish repository. Then run: $ perf record -o pack.perf git -c core.multiPackIndex=false \ pack-objects --all --stdout >/dev/null /dev/null --- t/perf/p5326-multi-pack-bitmaps.sh | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 t/perf/p5326-multi-pack-bitmaps.sh diff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh new file mode 100755 index 0000000000..5845109ac7 --- /dev/null +++ b/t/perf/p5326-multi-pack-bitmaps.sh @@ -0,0 +1,43 @@ +#!/bin/sh + +test_description='Tests performance using midx bitmaps' +. ./perf-lib.sh +. "${TEST_DIRECTORY}/perf/lib-bitmap.sh" + +test_perf_large_repo + +test_expect_success 'enable multi-pack index' ' + git config core.multiPackIndex true +' + +test_perf 'setup multi-pack index' ' + git repack -ad && + git multi-pack-index write --bitmap +' + +test_full_bitmap + +test_expect_success 'create partial bitmap state' ' + # pick a commit to represent the repo tip in the past + cutoff=$(git rev-list HEAD~100 -1) && + orig_tip=$(git rev-parse HEAD) && + + # now pretend we have just one tip + rm -rf .git/logs .git/refs/* .git/packed-refs && + git update-ref HEAD $cutoff && + + # and then repack, which will leave us with a nice + # big bitmap pack of the "old" history, and all of + # the new history will be loose, as if it had been pushed + # up incrementally and exploded via unpack-objects + git repack -Ad && + git multi-pack-index write --bitmap && + + # and now restore our original tip, as if the pushes + # had happened + git update-ref HEAD $orig_tip +' + +test_partial_bitmap + +test_done