| Message ID | 1550852864-56308-1-git-send-email-pbonzini@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [kvm-unit-tests,v3] tap13: list testcases individually | expand |
On 22.02.19 17:27, Paolo Bonzini wrote: > From: Stefan Raspl <raspl@linux.ibm.com> > > The current tap13 support summarizes testcases on a rather coarse > granularity. Which sort of defeats the purpose, especially in CI > environments, where having results for individual tests to pinpoint > problems helps a lot. > > To improve things, this patch makes TAP mode bypass print_result and extract > results for individual testcases as they are sent to the log file. It then > introduces test case numbers and writes the plan at the end. > > One could argue that post-processing the logs is a rather fragile approach. > Which is true - but apparently already the case, see e.g. extract_summary > in scripts/runtime.bash. Plus this is quite cheap, while a proper solution > would require to modify the kernels, so we can e.g. pass in a counter for > the test case number. And we would probably have to come up with reasonable > test case names, while this approach simply derives them from each test > output. > > Before: > $ ./run_tests.sh -t > TAP version 13 > ok selftest-setup > ok intercept > ok emulator > ok sieve > ok sthyi > ok skey > ok diag10 > ok pfmf > ok cmm > ok vector > ok gs # SKIP > ok iep # SKIP > 1..12 > > After: > $ ./run_tests.sh -t > TAP version 13 > ok 1 - cmm: privileged: Program interrupt: expected(2) > ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6) > ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6) > ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6) > ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6) > ok 6 - diag10: start/end: end < start: Program interrupt: expected(6) > ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6) > ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6) > ok 9 - diag10: privileged: Program interrupt: expected(2) > ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero > ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged > ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied > ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero > 1..180 > [...] > > Signed-off-by: Stefan Raspl <raspl@linux.ibm.com> > [Almost rewritten, but still giving a lot of credit to Stefan. - Paolo] > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > --- > run_tests.sh | 59 +++++++++++++++++++++++++++++++++++++++++++--------- > scripts/runtime.bash | 19 +++-------------- > 2 files changed, 52 insertions(+), 26 deletions(-) > > diff --git a/run_tests.sh b/run_tests.sh > index 6b0af19..edfd8a4 100755 > --- a/run_tests.sh > +++ b/run_tests.sh > @@ -69,12 +69,50 @@ shift $((OPTIND - 1)) > only_tests="$*" > > # RUNTIME_log_file will be configured later > -RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; } > +if [[ $tap_output == "no" ]]; then > + process_test_output() { cat >> $RUNTIME_log_file; } > + postprocess_suite_output() { cat; } > +else > + process_test_output() { > + CR=$'\r' > + while read -r line; do > + line="${line%$CR}" > + case "${line:0:4}" in > + PASS) > + echo "ok TEST_NUMBER - ${line#??????}" >&3 > + ;; > + FAIL) > + echo "not ok TEST_NUMBER - ${line#??????}" >&3 > + ;; > + SKIP) > + echo "ok TEST_NUMBER - ${line#??????} # skip" >&3 > + ;; Nice! However, you re-use the output of each test in full, e.g. ok 104 - intercept: stpx: Program interrupt: expected(4) == received(4) But that's not a stable testcase name: The "received(4)" might change in case of an error, that's why I was truncating everything from " ==" onwards. And it would be nice in CI environments to have stable testcase names, makes things a bit easier when comparing with previous output. Could you fix that? Otherwise, thanks for the credit - would be bizarr if I gave a rev'd & tested-by, though :O Ciao, Stefan
diff --git a/run_tests.sh b/run_tests.sh index 6b0af19..edfd8a4 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -69,12 +69,50 @@ shift $((OPTIND - 1)) only_tests="$*" # RUNTIME_log_file will be configured later -RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; } +if [[ $tap_output == "no" ]]; then + process_test_output() { cat >> $RUNTIME_log_file; } + postprocess_suite_output() { cat; } +else + process_test_output() { + CR=$'\r' + while read -r line; do + line="${line%$CR}" + case "${line:0:4}" in + PASS) + echo "ok TEST_NUMBER - ${line#??????}" >&3 + ;; + FAIL) + echo "not ok TEST_NUMBER - ${line#??????}" >&3 + ;; + SKIP) + echo "ok TEST_NUMBER - ${line#??????} # skip" >&3 + ;; + *) + ;; + esac + echo "${line}" + done >> $RUNTIME_log_file + } + postprocess_suite_output() { + test_number=0 + while read -r line; do + case "${line}" in + ok*|"not ok"*) + (( test_number++ )) + echo "${line/TEST_NUMBER/${test_number}}" ;; + *) echo "${line}" ;; + esac + done + echo "1..$test_number" + } +fi + +RUNTIME_log_stderr () { process_test_output; } RUNTIME_log_stdout () { if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then - ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file + ./scripts/pretty_print_stacks.py $1 | process_test_output else - cat >> $RUNTIME_log_file + process_test_output fi } @@ -84,7 +122,6 @@ function run_task() if [ -z "$testname" ]; then return fi - test_number=$((test_number+1)) while (( $(jobs | wc -l) == $unittest_run_queues )); do # wait for any background test to finish @@ -110,15 +147,17 @@ mkdir $unittest_log_dir || exit 2 echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY if [[ $tap_output == "yes" ]]; then - test_number=0 echo "TAP version 13" fi + trap "wait; exit 130" SIGINT -for_each_unittest $config run_task + +( + # preserve stdout so that process_test_output output can write TAP to it + exec 3>&1 + test "$tap_output" == "yes" && exec > /dev/null + for_each_unittest $config run_task +) | postprocess_suite_output # wait until all tasks finish wait - -if [[ $tap_output == "yes" ]]; then - echo "1..$test_number" -fi diff --git a/scripts/runtime.bash b/scripts/runtime.bash index 2f8026d..200d5b6 100644 --- a/scripts/runtime.bash +++ b/scripts/runtime.bash @@ -61,23 +61,10 @@ function print_result() local summary="$3" local reason="$4" - if [[ $tap_output == "no" ]]; then - if [ -z "$reason" ]; then - echo "`$status` $testname $summary" - else - echo "`$status` $testname ($reason)" - fi - return - fi - - if [[ $status == "FAIL" ]]; then - echo "not ok $testname $reason" - elif [[ $status == "PASS" ]]; then - echo "ok $testname" - elif [[ $status == "SKIP" ]]; then - echo "ok $testname # SKIP $reason" + if [ -z "$reason" ]; then + echo "`$status` $testname $summary" else - echo "not ok # TODO unknown test status" + echo "`$status` $testname ($reason)" fi }