| Message ID | cover.1660944574.git.me@ttaylorr.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | midx: permit changing the preferred pack when reusing the MIDX | expand |
On 8/19/2022 5:30 PM, Taylor Blau wrote: > This series resolves a bug that was reported[1] by Johannes, and > investigated by him, Abhradeep, and Stolee in that same sub-thread. > > The crux of the issue is that a MIDX bitmap can enter a corrupt state > when changing the preferred pack from its value in an existing MIDX in > certain circumstances as described in the first and final patches. > > This series is structured as follows: > > - The first patch of this series adds a test which demonstrates the > problem. (This is an improvement from the debugging in [1], where we > only noticed the problem racily in an existing test, and only in > SHA-256 mode). > > - The next small handful of patches refactor midx.c's > `get_sorted_entries()` function to make fixing this bug more > straightforward. > > - The final patch resolves the bug and updates the test to no longer > expect failure. Thanks for putting this together. Definitely not an easy bug to find and fix. I mostly have nitpicks, but the overall structure is sound. Thanks, -Stolee
On Mon, Aug 22, 2022 at 01:04:29PM -0400, Derrick Stolee wrote: > Thanks for putting this together. Definitely not an easy bug to find > and fix. > > I mostly have nitpicks, but the overall structure is sound. Thanks very much for reviewing (both to you and Abhradeep). I have a new version that I'll send now, which incorporates your suggestions. It also adds a new patch on top that takes a slight optimization, by avoiding adding objects from the MIDX that show up in the (new) preferred pack, since we know we'll discard them anyway. Thanks, Taylor
This series resolves a bug that was reported[1] by Johannes, and investigated by him, Abhradeep, and Stolee in that same sub-thread. The crux of the issue is that a MIDX bitmap can enter a corrupt state when changing the preferred pack from its value in an existing MIDX in certain circumstances as described in the first and final patches. This series is structured as follows: - The first patch of this series adds a test which demonstrates the problem. (This is an improvement from the debugging in [1], where we only noticed the problem racily in an existing test, and only in SHA-256 mode). - The next small handful of patches refactor midx.c's `get_sorted_entries()` function to make fixing this bug more straightforward. - The final patch resolves the bug and updates the test to no longer expect failure. A couple of meta-notes: - This bug has existed since the introduction of MIDX bitmaps, but probably wasn't noticed until now since it is only triggerable when the `--stdin-packs` mode *isn't* passed, so it never occurs when invoked via `git repack`. - We could likely change the behavior of 56d863e979 (midx: expose `write_midx_file_only()` publicly, 2021-09-28), which explicitly disables reusing the existing MIDX (by avoiding loading it altogether) when `--stdin-packs` is given. The rationale in the comment added by 56d863e979 is somewhat unclear, but I have a vague recollection of running into a bug that was squashed by avoiding reusing an existing MIDX to write one with bitmaps. This was likely that bug. Thanks in advance for your review, and thanks to Johannes, Abhradeep, and Stolee for investigating this bug while I was on vacation. [1]: https://lore.kernel.org/git/p3r70610-8n52-s8q0-n641-onp4ps01330n@tzk.qr/ Taylor Blau (6): t5326: demonstrate potential bitmap corruption t/lib-bitmap.sh: avoid silencing stderr midx.c: extract `struct midx_fanout` midx.c: extract `midx_fanout_add_midx_fanout()` midx.c: extract `midx_fanout_add_pack_fanout()` midx.c: include preferred pack correctly with existing MIDX midx.c | 128 ++++++++++++++++++++++------------ t/lib-bitmap.sh | 2 +- t/t5326-multi-pack-bitmaps.sh | 43 ++++++++++++ 3 files changed, 127 insertions(+), 46 deletions(-)