[kvm-unit-tests,RFC,0/1] Detecting crashes in TAP output

Message

Nico Boehr Feb. 23, 2022, 10:34 a.m. UTC

Hi,

I recently had a test on s390x which caused an exception in the guest. As
expected, run_tests.sh reported it as failing:

    FAIL sometest (1 tests)

However, when I turn on the TAP output everything looks like it is fine:

    TAP version 13
    ok 1 - sometest: sometest: some report
    1..1

There is no way to see from the TAP output the test actually had an unexpected
exception. Our internal scripts rely on the TAP output and will thus believe
everything is fine, even though it really isn't.

In the logfile, one can at least see the exception backtrace, but if something
exits silently, there is no indication something went wrong there either.

TAP provides the test plan (the "1..1") as a solution to this problem. It
gives the expected number of test lines if all tests would run. The harness can
count test lines in TAP output and compare this to the number of tests in the
plan and report an error if it doesn't match.

This won't work with kvm-unit-tests, since there really isn't any way to know
how many report()s and thus test lines you will have without actually running
the tests.

That's why I came up with an alternative approach in the patch below. It
adds an additional test line to the TAP output and the logfiles which
states the overall result of a test.

With the attached patch, the TAP output of the test above will now look like
this:

    TAP version 13
    ok 1 - sometest: sometest: some report
    not ok 2 - sometest: (1 tests)
    1..2

This shows something went wrong.

Your feedback or alternative solutions to this problem are welcome.

Nico Boehr (1):
  scripts/runtime: add test result to log and TAP output

 scripts/runtime.bash | 2 ++
 1 file changed, 2 insertions(+)