| Message ID | 2520abf24a8a194b3f7040e218f878dc88a740a0.1731518931.git.me@ttaylorr.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | pack-objects: more brown-paper-bag multi-pack reuse fixes | expand |
On Wed, Nov 13, 2024 at 12:32:58PM -0500, Taylor Blau wrote: > Instead, we can only safely perform the whole-word reuse optimization on > the preferred pack, where we know with certainty that no gaps exist in > that region of the bitmap. We can still reuse objects from non-preferred > packs, but we have to inspect them individually in write_reused_pack() > to ensure that any gaps that may exist are accounted for. Yep. With the disclaimer that I'm biased because I helped a little with debugging, this change is obviously correct. Forgetting the bug you saw in the real world, we know this function cannot work as-is because of the potential for those gaps. > This allows us to simplify the implementation of > write_reused_pack_verbatim() back to almost its pre-multi-pack reuse > form, since we can now assume that the beginning of the pack appears at > the beginning of the bitmap, meaning that we don't have to account for > any bits up to the first word boundary (like we had to special case in > ca0fd69e37). > > The only significant changes from the pre-ca0fd69e37 implementation are: > [...] Thanks for this section. My first instinct was to go back and look at the diff to the pre-midx version of the function, and this nicely explains the hunks I see there. So this patch looks good to me. I was able to follow your explanation in the commit message, but that may not count for much since I'm probably the only other person with deep knowledge of the verbatim-reuse code in the first place. ;) I do think the explanation in the message of the first commit would be a lot simpler if it were simply combined into this patch. With them split you effectively have to explain the problem twice. I don't feel that strongly about changing it, though. -Peff
On Wed, Nov 13, 2024 at 07:25:04PM -0500, Jeff King wrote: > On Wed, Nov 13, 2024 at 12:32:58PM -0500, Taylor Blau wrote: > > > Instead, we can only safely perform the whole-word reuse optimization on > > the preferred pack, where we know with certainty that no gaps exist in > > that region of the bitmap. We can still reuse objects from non-preferred > > packs, but we have to inspect them individually in write_reused_pack() > > to ensure that any gaps that may exist are accounted for. > > Yep. With the disclaimer that I'm biased because I helped a little with > debugging, this change is obviously correct. Forgetting the bug you saw > in the real world, we know this function cannot work as-is because of > the potential for those gaps. Yep, and thanks again for your help ;-). > > This allows us to simplify the implementation of > > write_reused_pack_verbatim() back to almost its pre-multi-pack reuse > > form, since we can now assume that the beginning of the pack appears at > > the beginning of the bitmap, meaning that we don't have to account for > > any bits up to the first word boundary (like we had to special case in > > ca0fd69e37). > > > > The only significant changes from the pre-ca0fd69e37 implementation are: > > [...] > > Thanks for this section. My first instinct was to go back and look at > the diff to the pre-midx version of the function, and this nicely > explains the hunks I see there. > > So this patch looks good to me. I was able to follow your explanation in > the commit message, but that may not count for much since I'm probably > the only other person with deep knowledge of the verbatim-reuse code in > the first place. ;) Heh. > I do think the explanation in the message of the first commit would be a > lot simpler if it were simply combined into this patch. With them split > you effectively have to explain the problem twice. I don't feel that > strongly about changing it, though. I always seem to go back and forth on that. I feel somewhat strongly that for complicated regression fixes that we should demonstrate the existing failure mode in a separate commit with a test_expect_failure. That forces the author to ensure they really understand the bug and can produce a minimal (or close to it) reproduction. It also makes it easier to demonstrate that the fix actually does what it says, instead of assuming that the test fails without the fix applied (and passes with it applied). That does force the author to potentially explain the bug twice. In my experience, I tend to keep the explanation in the first patch relatively brief, hinting at details that will appear in the subsequent patch instead of explaining them in full detail. So I dunno. It's a tradeoff for sure, but I think having an explicit point in the log that demonstrates the existing bug is valuable. Thanks, Taylor
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 08007142671..f413344e90c 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1100,78 +1100,64 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile, static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile, struct hashfile *out, - off_t pack_start, struct pack_window **w_curs) { - size_t pos = reuse_packfile->bitmap_pos; + size_t pos = 0; size_t end; - if (pos % BITS_IN_EWORD) { - size_t word_pos = (pos / BITS_IN_EWORD); - size_t offset = pos % BITS_IN_EWORD; - size_t last; - eword_t word = reuse_packfile_bitmap->words[word_pos]; - - if (offset + reuse_packfile->bitmap_nr < BITS_IN_EWORD) - last = offset + reuse_packfile->bitmap_nr; - else - last = BITS_IN_EWORD; - - for (; offset < last; offset++) { - if (word >> offset == 0) - return word_pos; - if (!bitmap_get(reuse_packfile_bitmap, - word_pos * BITS_IN_EWORD + offset)) - return word_pos; - } - - pos += BITS_IN_EWORD - (pos % BITS_IN_EWORD); - } - - /* - * Now we're going to copy as many whole eword_t's as possible. - * "end" is the index of the last whole eword_t we copy, but - * there may be additional bits to process. Those are handled - * individually by write_reused_pack(). - * - * Begin by advancing to the first word boundary in range of the - * bit positions occupied by objects in "reuse_packfile". Then - * pick the last word boundary in the same range. If we have at - * least one word's worth of bits to process, continue on. - */ - end = reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr; - if (end % BITS_IN_EWORD) - end -= end % BITS_IN_EWORD; - if (pos >= end) + if (reuse_packfile->bitmap_pos) { + /* + * We can't reuse whole chunks verbatim out of + * non-preferred packs since we can't guarantee that + * all duplicate objects were resolved in favor of + * that pack. + * + * Even if we have a whole eword_t worth of bits that + * could be reused, there may be objects between the + * objects corresponding to the first and last bit of + * that word which were selected from a different + * pack, causing us to send duplicate or unwanted + * objects. + * + * Handle non-preferred packs from within + * write_reused_pack(), which inspects and reuses + * individual bits. + */ return reuse_packfile->bitmap_pos / BITS_IN_EWORD; + } - while (pos < end && - reuse_packfile_bitmap->words[pos / BITS_IN_EWORD] == (eword_t)~0) - pos += BITS_IN_EWORD; + /* + * Only read through the last word whose bits all correspond + * to objects in the given packfile, since we must stop at a + * word boundary. + * + * If there is no whole word to read (i.e. the packfile + * contains fewer than BITS_IN_EWORD objects), then we'll + * inspect bits one-by-one in write_reused_pack(). + */ + end = reuse_packfile->bitmap_nr / BITS_IN_EWORD; + if (reuse_packfile_bitmap->word_alloc < end) + BUG("fewer words than expected in reuse_packfile_bitmap"); - if (pos > end) - pos = end; + while (pos < end && reuse_packfile_bitmap->words[pos] == (eword_t)~0) + pos++; - if (reuse_packfile->bitmap_pos < pos) { - off_t pack_start_off = pack_pos_to_offset(reuse_packfile->p, 0); - off_t pack_end_off = pack_pos_to_offset(reuse_packfile->p, - pos - reuse_packfile->bitmap_pos); + if (pos) { + off_t to_write; - written += pos - reuse_packfile->bitmap_pos; + written = (pos * BITS_IN_EWORD); + to_write = pack_pos_to_offset(reuse_packfile->p, written) + - sizeof(struct pack_header); /* We're recording one chunk, not one object. */ - record_reused_object(pack_start_off, - pack_start_off - (hashfile_total(out) - pack_start)); + record_reused_object(sizeof(struct pack_header), 0); hashflush(out); copy_pack_data(out, reuse_packfile->p, w_curs, - pack_start_off, pack_end_off - pack_start_off); + sizeof(struct pack_header), to_write); display_progress(progress_state, written); } - if (pos % BITS_IN_EWORD) - BUG("attempted to jump past a word boundary to %"PRIuMAX, - (uintmax_t)pos); - return pos / BITS_IN_EWORD; + return pos; } static void write_reused_pack(struct bitmapped_pack *reuse_packfile, @@ -1183,8 +1169,7 @@ static void write_reused_pack(struct bitmapped_pack *reuse_packfile, struct pack_window *w_curs = NULL; if (allow_ofs_delta) - i = write_reused_pack_verbatim(reuse_packfile, f, pack_start, - &w_curs); + i = write_reused_pack_verbatim(reuse_packfile, f, &w_curs); for (; i < reuse_packfile_bitmap->word_alloc; ++i) { eword_t word = reuse_packfile_bitmap->words[i]; diff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh index 8f403d9fdaa..06836a4206c 100755 --- a/t/t5332-multi-pack-reuse.sh +++ b/t/t5332-multi-pack-reuse.sh @@ -259,7 +259,7 @@ test_expect_success 'duplicate objects' ' ) ' -test_expect_failure 'duplicate objects with verbatim reuse' ' +test_expect_success 'duplicate objects with verbatim reuse' ' git init duplicate-objects-verbatim && ( cd duplicate-objects-verbatim &&
When reusing objects from source pack(s), write_reused_pack_verbatim() is responsible for reusing objects whole eword_t's at a time. It works by taking the longest continuous run of objects from the beginning of each source pack that the caller wants, and reuses the entirety of that section from each pack. This is based on the assumption that we don't have any gaps within the region. This assumption relieves us from having to patch any OFS_DELTAs, since we know that there aren't any gaps between any delta and its base in that region. To illustrate why this assumption is necessary, suppose we have some pack P, which has objects X, Y, and Z. If the MIDX's copy of Y was selected from a pack other than P, then the bit corresponding to object Y will appear earlier in the bitmap than the bits corresponding to X and Z. If pack-objects already has or will use the copy of Y from the pack it was selected from in the MIDX, then it is an error to reuse all objects between X and Z in the source pack. Doing so will cause us to reuse Y from a different pack than the one which represents Y in the MIDX, causing us to either: - include the object twice, assuming that the caller wants Y in the pack, or - include the object once, resulting in us packing more objects than necessary. This regression comes from ca0fd69e37 (pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse, 2023-12-14), which incorrectly assumed that there would be no gaps in reusable regions of non-preferred packs. Instead, we can only safely perform the whole-word reuse optimization on the preferred pack, where we know with certainty that no gaps exist in that region of the bitmap. We can still reuse objects from non-preferred packs, but we have to inspect them individually in write_reused_pack() to ensure that any gaps that may exist are accounted for. This allows us to simplify the implementation of write_reused_pack_verbatim() back to almost its pre-multi-pack reuse form, since we can now assume that the beginning of the pack appears at the beginning of the bitmap, meaning that we don't have to account for any bits up to the first word boundary (like we had to special case in ca0fd69e37). The only significant changes from the pre-ca0fd69e37 implementation are: - that we can no longer inspect words up to the end of reuse_packfile_bitmap->word_alloc, since we only want to look at words whose bits all correspond to objects in the given packfile, and - that we return early when given a reuse_packfile which is not preferred, making the call a noop. In the future, it might be possible to restore this optimization if we could guarantee that some reuse packs don't contain any gaps by construction (similar to the "disjoint packs" idea in very early versions of multi-pack reuse). Helped-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> --- builtin/pack-objects.c | 101 +++++++++++++++--------------------- t/t5332-multi-pack-reuse.sh | 2 +- 2 files changed, 44 insertions(+), 59 deletions(-)