| Message ID | 20200312173520.2401776-1-damien.olivier.robert+git@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v2,1/1] midx.c: fix an integer overflow | expand |
From Damien Robert, Thu 12 Mar 2020 at 18:35:20 (+0100) : > When verifying a midx index with 0 objects, the > m->num_objects - 1 > overflows to 4294967295. > > Fix this both by checking that the midx contains at least one oid, > and also that we don't write any midx when there is no packfiles. I forgot to add: previously I was wondering about a warning when in 'write' when we only index one pack file, but this could make sense in the case of a 'midx repack'. So I only warn if there is no objects.
On 3/12/2020 1:35 PM, Damien Robert wrote: > When verifying a midx index with 0 objects, the > m->num_objects - 1 > overflows to 4294967295. > > Fix this both by checking that the midx contains at least one oid, > and also that we don't write any midx when there is no packfiles. > > Signed-off-by: Damien Robert <damien.olivier.robert+git@gmail.com> > --- > Should I add a test? It is a bit troublesome to generate a zero object midx > file since this patch prevents it from using 'midx write'... I'm glad that your patch makes it impossible to generate a zero-object multi-pack-index, and that makes a test hard to implement. I'm not sure what history Git has for storing explicit binary content into the test suite. There really is only one "empty" multi-pack-index, but it is unfortunately still a bit big for a test case to write explicitly due to the 256-word fanout table. I _think_ the t/tXXXX directories are used for this kind of data storage, so you could generate an empty multi-pack-index from an older version of Git then store it there. Please wait for someone else on-list to say that this is a good idea, though. It may not be worth the pain of a binary file in the patch. > midx.c | 35 +++++++++++++++++++++++------------ > 1 file changed, 23 insertions(+), 12 deletions(-) > > diff --git a/midx.c b/midx.c > index 1527e464a7..2cece7f9ea 100644 > --- a/midx.c > +++ b/midx.c > @@ -923,6 +923,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * > cur_chunk = 0; > num_chunks = large_offsets_needed ? 5 : 4; > > + if (packs.nr - dropped_packs == 0) { > + error(_("no pack files to index.")); nit: I would use "pack-files" here. Second best is "packfiles". > + result = 1; > + goto cleanup; > + } > + > written = write_midx_header(f, num_chunks, packs.nr - dropped_packs); > > chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES; > @@ -1124,22 +1130,27 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag > i, oid_fanout1, oid_fanout2, i + 1); > } > > - if (flags & MIDX_PROGRESS) > - progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), > - m->num_objects - 1); > - for (i = 0; i < m->num_objects - 1; i++) { > - struct object_id oid1, oid2; > + if (m->num_objects == 0) > + midx_report(_("Warning: the midx contains no oid.")); Should this "Warning: " be here? The other calls to midx_report() do not have such prefix. It could be valuable to add "warning: %s\n" to the fprintf inside midx_report(), but that should be done as its own patch. Also, it may be valuable to return from this block so you do not need to put the block below in a tabbed block, reducing the complexity of this patch. > + else > + { > + if (flags & MIDX_PROGRESS) > + progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), > + m->num_objects - 1); > + for (i = 0; i < m->num_objects - 1; i++) { > + struct object_id oid1, oid2; > > - nth_midxed_object_oid(&oid1, m, i); > - nth_midxed_object_oid(&oid2, m, i + 1); > + nth_midxed_object_oid(&oid1, m, i); > + nth_midxed_object_oid(&oid2, m, i + 1); > > - if (oidcmp(&oid1, &oid2) >= 0) > - midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), > - i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); > + if (oidcmp(&oid1, &oid2) >= 0) > + midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), > + i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); > > - midx_display_sparse_progress(progress, i + 1); > + midx_display_sparse_progress(progress, i + 1); > + } > + stop_progress(&progress); > } > - stop_progress(&progress); > > /* > * Create an array mapping each object to its packfile id. Sort it > Thanks for digging into this! -Stolee
From Derrick Stolee, Thu 12 Mar 2020 at 14:28:11 (-0400) : > I _think_ the t/tXXXX directories are used for this kind of data storage, > so you could generate an empty multi-pack-index from an older version of > Git then store it there. Yes I anticipated that and have one available on hand :) It weights 1116 characters. > > - if (flags & MIDX_PROGRESS) > > - progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), > > - m->num_objects - 1); > > - for (i = 0; i < m->num_objects - 1; i++) { > > - struct object_id oid1, oid2; > > + if (m->num_objects == 0) > > + midx_report(_("Warning: the midx contains no oid.")); > > Should this "Warning: " be here? The other calls to midx_report() do not have such prefix. Right, I agree it should not. > Also, it may be valuable to return from this block so you do not need to put the block below in a tabbed block, reducing the complexity of this patch. Agreed: we don't want to run the other checks anyway if we don't have any objects. That'll be for v3 once I get advice on what to do for tests.
diff --git a/midx.c b/midx.c index 1527e464a7..2cece7f9ea 100644 --- a/midx.c +++ b/midx.c @@ -923,6 +923,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index * cur_chunk = 0; num_chunks = large_offsets_needed ? 5 : 4; + if (packs.nr - dropped_packs == 0) { + error(_("no pack files to index.")); + result = 1; + goto cleanup; + } + written = write_midx_header(f, num_chunks, packs.nr - dropped_packs); chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES; @@ -1124,22 +1130,27 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag i, oid_fanout1, oid_fanout2, i + 1); } - if (flags & MIDX_PROGRESS) - progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), - m->num_objects - 1); - for (i = 0; i < m->num_objects - 1; i++) { - struct object_id oid1, oid2; + if (m->num_objects == 0) + midx_report(_("Warning: the midx contains no oid.")); + else + { + if (flags & MIDX_PROGRESS) + progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), + m->num_objects - 1); + for (i = 0; i < m->num_objects - 1; i++) { + struct object_id oid1, oid2; - nth_midxed_object_oid(&oid1, m, i); - nth_midxed_object_oid(&oid2, m, i + 1); + nth_midxed_object_oid(&oid1, m, i); + nth_midxed_object_oid(&oid2, m, i + 1); - if (oidcmp(&oid1, &oid2) >= 0) - midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), - i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + if (oidcmp(&oid1, &oid2) >= 0) + midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), + i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); - midx_display_sparse_progress(progress, i + 1); + midx_display_sparse_progress(progress, i + 1); + } + stop_progress(&progress); } - stop_progress(&progress); /* * Create an array mapping each object to its packfile id. Sort it
When verifying a midx index with 0 objects, the m->num_objects - 1 overflows to 4294967295. Fix this both by checking that the midx contains at least one oid, and also that we don't write any midx when there is no packfiles. Signed-off-by: Damien Robert <damien.olivier.robert+git@gmail.com> --- Should I add a test? It is a bit troublesome to generate a zero object midx file since this patch prevents it from using 'midx write'... midx.c | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)