| Message ID | patch-11.15-26c838f0560-20220302T171755Z-avarab@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | tests: don't ignore "git" exit codes | expand |
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > Change a couple of uses of "test_expect_failure" to use a > "test_expect_success" to positively assert the current behavior, and > replace the intent of "test_expect_failure" with a "TODO" comment int > the description. > > As noted in [1] the "test_expect_failure" feature is overly eager to And noted in [2], it is not a good idea to abuse "test_expect_success" for this purpose, either, though. [2] https://lore.kernel.org/git/xmqq4k9kj15p.fsf@gitster.g/
On Wed, Mar 02 2022, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> Change a couple of uses of "test_expect_failure" to use a >> "test_expect_success" to positively assert the current behavior, and >> replace the intent of "test_expect_failure" with a "TODO" comment int >> the description. >> >> As noted in [1] the "test_expect_failure" feature is overly eager to > > And noted in [2], it is not a good idea to abuse "test_expect_success" > for this purpose, either, though. > > [2] https://lore.kernel.org/git/xmqq4k9kj15p.fsf@gitster.g/ As noted I do have a "test_todo" (or "test_expect_todo") replacement for "test_expect_failure" which I think I think will address your concern there. But do you mind if this is left like this for now? Due to the semantics of "test_expect_failure" we can't use it in conjunction with "test_must_fail" currently and not hide segfaults or abort(). So having it marked as "ok ... # TODO" v.s. "not ok ... # TODO" isn't ideal, but certainly better than silently hiding abort() and segfaults.
diff --git a/t/t6102-rev-list-unexpected-objects.sh b/t/t6102-rev-list-unexpected-objects.sh index 6f0902b8638..cf0195e8263 100755 --- a/t/t6102-rev-list-unexpected-objects.sh +++ b/t/t6102-rev-list-unexpected-objects.sh @@ -17,8 +17,13 @@ test_expect_success 'setup unexpected non-blob entry' ' broken_tree="$(git hash-object -w --literally -t tree broken-tree)" ' -test_expect_failure 'traverse unexpected non-blob entry (lone)' ' - test_must_fail git rev-list --objects $broken_tree +test_expect_success !SANITIZE_LEAK 'TODO (should fail!): traverse unexpected non-blob entry (lone)' ' + sed "s/Z$//" >expect <<-EOF && + $broken_tree Z + $tree foo + EOF + git rev-list --objects $broken_tree >actual && + test_cmp expect actual ' test_expect_success 'traverse unexpected non-blob entry (seen)' ' @@ -116,8 +121,8 @@ test_expect_success 'setup unexpected non-blob tag' ' tag=$(git hash-object -w --literally -t tag broken-tag) ' -test_expect_failure 'traverse unexpected non-blob tag (lone)' ' - test_must_fail git rev-list --objects $tag +test_expect_success !SANITIZE_LEAK 'TODO (should fail!): traverse unexpected non-blob tag (lone)' ' + git rev-list --objects $tag ' test_expect_success 'traverse unexpected non-blob tag (seen)' '
Change a couple of uses of "test_expect_failure" to use a "test_expect_success" to positively assert the current behavior, and replace the intent of "test_expect_failure" with a "TODO" comment int the description. As noted in [1] the "test_expect_failure" feature is overly eager to accept any failure as OK, and thus by design hides segfaults, abort() etc. Because of that I didn't notice in dd9cede9136 (leak tests: mark some rev-list tests as passing with SANITIZE=leak, 2021-10-31) that this test leaks memory under SANITIZE=leak. I have some larger local changes to add a better "test_expect_failure", which would work just like "test_expect_success", but would allow us say "test_todo" here (and "success" would emit a "not ok [...] # TODO", not "ok [...]". In lieu of those larger changes let's more narrowly fix the problem at hand here and stop conflating the current "success" with actual SANITIZE=leak failures. 1. https://lore.kernel.org/git/87tuhmk19c.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> --- t/t6102-rev-list-unexpected-objects.sh | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)