| Message ID | 20230808191536.GA4033224@coredump.intra.peff.net (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RFC/PATCH] commit-graph: verify swapped zero/non-zero generation cases | expand |
On Tue, Aug 08, 2023 at 03:15:36PM -0400, Jeff King wrote: > This is marked as RFC because I'm still confused about a lot of things. > For one, my explanation above about what the code is doing is mostly a > guess. It _looks_ to me like that's what the existing check is trying to > do. But if so, then why is the generation_zero flag defined outside the > loop over each object? I'd think it would be a per-object thing. I thought the same thing initially, but looking back at 1373e547f7 (commit-graph: verify generation number, 2018-06-27), I think the scope of generation_zero is correct. This is an artifact from when commit-graphs were written with all commit generation numbers equal to zero. So I think the logic is something like: - If the commit-graph has a generation number of 0 for some commit, but we saw a non-zero value from any another commit, report it. - Otherwise, if the commit-graph had a non-zero value for the commit's generation number, and we had previously seen a generation number of zero for some other commit, report it. IOW, I think we expect to see either all zeros, or all non-zero values in a single commit-graph's set of generation numbers. Earlier in your message, you wrote: > There's a matching GENERATION_NUMBER_EXISTS value, which in theory would > be used to find the case that we see the entries in the opposite order: > > 1. When we see an entry with a non-zero generation, we set the > generation_zero flag to GENERATION_NUMBER_EXISTS. > > 2. When we later see an entry with a zero generation, we complain if > the flag is GENERATION_NUMBER_EXISTS. > > But that doesn't work; step 2 is implemented, but there is no step 1. We > never use NUMBER_EXISTS at all, and Coverity rightly complains that step > 2 is dead code. So I think the missing part is setting GENERATION_NUMBER_EXISTS when we have a non-zero generation number from the commit-graph, but have generation_zero set to GENERATION_ZERO_EXISTS (IOW, we have seen at least one commit with generation number 0). --- 8< --- diff --git a/commit-graph.c b/commit-graph.c index 0aa1640d15..935bc15440 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -2676,9 +2676,11 @@ static int verify_one_commit_graph(struct repository *r, graph_report(_("commit-graph has generation number zero for commit %s, but non-zero elsewhere"), oid_to_hex(&cur_oid)); generation_zero = GENERATION_ZERO_EXISTS; - } else if (generation_zero == GENERATION_ZERO_EXISTS) + } else if (generation_zero == GENERATION_ZERO_EXISTS) { graph_report(_("commit-graph has non-zero generation number for commit %s, but zero elsewhere"), oid_to_hex(&cur_oid)); + generation_zero = GENERATION_NUMBER_EXISTS; + } if (generation_zero == GENERATION_ZERO_EXISTS) continue; --- >8 --- > So I kind of wonder if there's something I'm not getting here. Coverity > is definitely right that our "step 2" is dead code (because we never set > NUMBER_EXISTS). But I'm not sure if we should be deleting it, or trying > to fix an underlying bug. I think that above is correct in that we should be fixing an underlying bug. But the fact that this isn't caught by our existing tests indicates that there is a gap in coverage. Let me see if I can find a test case that highlights this bug... Thanks, Taylor
On Thu, Aug 10, 2023 at 12:00:43PM -0400, Taylor Blau wrote: > > There's a matching GENERATION_NUMBER_EXISTS value, which in theory would > > be used to find the case that we see the entries in the opposite order: > > > > 1. When we see an entry with a non-zero generation, we set the > > generation_zero flag to GENERATION_NUMBER_EXISTS. > > > > 2. When we later see an entry with a zero generation, we complain if > > the flag is GENERATION_NUMBER_EXISTS. > > > > But that doesn't work; step 2 is implemented, but there is no step 1. We > > never use NUMBER_EXISTS at all, and Coverity rightly complains that step > > 2 is dead code. > > So I think the missing part is setting GENERATION_NUMBER_EXISTS when we > have a non-zero generation number from the commit-graph, but have > generation_zero set to GENERATION_ZERO_EXISTS (IOW, we have seen at > least one commit with generation number 0). > > --- 8< --- > diff --git a/commit-graph.c b/commit-graph.c > index 0aa1640d15..935bc15440 100644 > --- a/commit-graph.c > +++ b/commit-graph.c > @@ -2676,9 +2676,11 @@ static int verify_one_commit_graph(struct repository *r, > graph_report(_("commit-graph has generation number zero for commit %s, but non-zero elsewhere"), > oid_to_hex(&cur_oid)); > generation_zero = GENERATION_ZERO_EXISTS; > - } else if (generation_zero == GENERATION_ZERO_EXISTS) > + } else if (generation_zero == GENERATION_ZERO_EXISTS) { > graph_report(_("commit-graph has non-zero generation number for commit %s, but zero elsewhere"), > oid_to_hex(&cur_oid)); > + generation_zero = GENERATION_NUMBER_EXISTS; > + } > > if (generation_zero == GENERATION_ZERO_EXISTS) > continue; > --- >8 --- OK, I investigated this a little bit more and now I think I understand fully what's going on here. There are a couple of things wrong with the diff that I posted above. First, it has a logic error that we should set GENERATION_NUMBER_EXISTS when we have a non-zero generation number from the graph, regardless of whether or not GENERATION_ZERO_EXISTS is set (like how it is done in your patch). But more importantly, we'll never end up in the first arm of that conditional as-is (the one that fires for when we see a generation number of zero) as a consequence of 2ee11f7261 (commit-graph: return generation from memory, 2023-03-20), which only returns non-zero generation numbers (or GENERATION_NUMBER_INFINITY, which is also non-zero). I think you want something like `commit_graph_generation()` that returns whatever is in `data->generation` regardless of whether or not it is zero valued. You'd then want to use that function instead of calling commit_graph_generation() directly. > > So I kind of wonder if there's something I'm not getting here. Coverity > > is definitely right that our "step 2" is dead code (because we never set > > NUMBER_EXISTS). But I'm not sure if we should be deleting it, or trying > > to fix an underlying bug. > > I think that above is correct in that we should be fixing an underlying > bug. But the fact that this isn't caught by our existing tests indicates > that there is a gap in coverage. Let me see if I can find a test case > that highlights this bug... Doing the above allows me to write these two tests on top of your patch, which both pass: --- &< --- diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 4df76173a8..8e96471b34 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -450,14 +450,15 @@ GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255)) GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256)) GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8)) GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10)) +GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16)) GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS)) GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN)) GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4)) GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3)) GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11)) +GRAPH_BYTE_COMMIT_GENERATION_LAST=$(($GRAPH_BYTE_COMMIT_GENERATION + $(($NUM_COMMITS - 1)) * $GRAPH_COMMIT_DATA_WIDTH)) GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12)) -GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16)) GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS)) GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) @@ -596,11 +597,6 @@ test_expect_success 'detect incorrect generation number' ' "generation for commit" ' -test_expect_success 'detect incorrect generation number' ' - corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION "\01" \ - "commit-graph generation for commit" -' - test_expect_success 'detect incorrect commit date' ' corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE "\01" \ "commit date" @@ -622,6 +618,16 @@ test_expect_success 'detect incorrect chunk count' ' $GRAPH_CHUNK_LOOKUP_OFFSET ' +test_expect_success 'detect mixed generation numbers (non-zero to zero)' ' + corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION_LAST "\0\0\0\0" \ + "but non-zero elsewhere" +' + +test_expect_success 'detect mixed generation numbers (zero to non-zero)' ' + corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION "\0\0\0\0" \ + "but zero elsewhere" +' + test_expect_success 'git fsck (checks commit-graph when config set to true)' ' git -C full fsck && corrupt_graph_and_verify $GRAPH_BYTE_FOOTER "\00" \ --- >8 --- Note that we remove the duplicate "detect incorrect generation number" test, which was originally introduced in 1373e547f7 (commit-graph: verify generation number, 2018-06-27), but was modified in 2ee11f7261. That test is replaced by the latter "non-zero to zero" variant. Thanks, Taylor
diff --git a/commit-graph.c b/commit-graph.c index 0aa1640d15..40cd55eb15 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -2676,9 +2676,13 @@ static int verify_one_commit_graph(struct repository *r, graph_report(_("commit-graph has generation number zero for commit %s, but non-zero elsewhere"), oid_to_hex(&cur_oid)); generation_zero = GENERATION_ZERO_EXISTS; - } else if (generation_zero == GENERATION_ZERO_EXISTS) - graph_report(_("commit-graph has non-zero generation number for commit %s, but zero elsewhere"), + } else { + if (generation_zero == GENERATION_ZERO_EXISTS) + graph_report(_("commit-graph has non-zero generation number for commit %s, but zero elsewhere"), oid_to_hex(&cur_oid)); + else + generation_zero = GENERATION_NUMBER_EXISTS; + } if (generation_zero == GENERATION_ZERO_EXISTS) continue;
In verify_one_commit_graph(), we have code that complains when a commit is found with a generation number of zero, and then later with a non-zero number. It works like this: 1. When we see an entry with generation zero, we set the generation_zero flag to GENERATION_ZERO_EXISTS. 2. When we later see an entry with a non-zero generation, we complain if the flag is GENERATION_ZERO_EXISTS. There's a matching GENERATION_NUMBER_EXISTS value, which in theory would be used to find the case that we see the entries in the opposite order: 1. When we see an entry with a non-zero generation, we set the generation_zero flag to GENERATION_NUMBER_EXISTS. 2. When we later see an entry with a zero generation, we complain if the flag is GENERATION_NUMBER_EXISTS. But that doesn't work; step 2 is implemented, but there is no step 1. We never use NUMBER_EXISTS at all, and Coverity rightly complains that step 2 is dead code. We can fix that by implementing that step 1. Signed-off-by: Jeff King <peff@peff.net> --- This is marked as RFC because I'm still confused about a lot of things. For one, my explanation above about what the code is doing is mostly a guess. It _looks_ to me like that's what the existing check is trying to do. But if so, then why is the generation_zero flag defined outside the loop over each object? I'd think it would be a per-object thing. Likewise, just below this code, we check: if (generation_zero == GENERATION_ZERO_EXISTS) continue; Is the intent here "if this is the zero-th generation, we can skip the rest of the loop because there are no more parents to look at"? If so, then would it make more sense to check commit_graph_generation() directly? I took care to preserve the existing behavior by pushing the set of NUMBER_EXISTS into an "else", but it seems like a weird use of the flag to me. So I kind of wonder if there's something I'm not getting here. Coverity is definitely right that our "step 2" is dead code (because we never set NUMBER_EXISTS). But I'm not sure if we should be deleting it, or trying to fix an underlying bug. commit-graph.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)