| Message ID | cover.1687156559.git.falcon@tinylab.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | selftests/nolibc: improve test report support | expand |
Hi Zhangjin, On Mon, Jun 19, 2023 at 02:52:31PM +0800, Zhangjin Wu wrote: > Hi, Willy > > Here is the v2 of our old patchset about test report [1]. > > The trailing '\r' fixup has been merged, so, here only resend the left > parts with an additional patch to restore the failed tests print. > > This patchset is rebased on the dev.2023.06.14a branch of linux-rcu [2]. > > Tests have passed for 'x86 run': > > 138 test(s) passed, 0 skipped, 0 failed. > See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out (...) > 2. selftests/nolibc: always print the path to test log file > > Always print the path to test log file, but move it to a new line to > avoid annoying people when the test pass without any failures. I'm still really missing the (s+f > 0) test I added which was a time saver for me, because I could trivially check in the output reports which ones were totally OK and which ones required attention. Sure I could also start to grep for "passed," | grep -v " 0 skipped, 0 failed" but that's quite a pain, really. I'm going to merge your series anyway otherwise we'll continue to bikeshed for many weeks and I know how annoying it is to keep unmerged series. But I would like that we find a solution that satisfies everyone. Maybe one possibility would be to add a "status" at the end of the line that emits "success", "warning", "failure" depending on the highest level reached like this: 138 test(s) passed, 0 skipped, 0 failed => status: success 136 test(s) passed, 2 skipped, 0 failed => status: warning 136 test(s) passed, 1 skipped, 1 failed => status: failure This way it's easy to grep -v "status: success" or grep "status: failure" to instantly get the corresponding details and also grep for them from multiple files. Thanks! Willy
> Hi Zhangjin, > > On Mon, Jun 19, 2023 at 02:52:31PM +0800, Zhangjin Wu wrote: > > Hi, Willy > > > > Here is the v2 of our old patchset about test report [1]. > > > > The trailing '\r' fixup has been merged, so, here only resend the left > > parts with an additional patch to restore the failed tests print. > > > > This patchset is rebased on the dev.2023.06.14a branch of linux-rcu [2]. > > > > Tests have passed for 'x86 run': > > > > 138 test(s) passed, 0 skipped, 0 failed. > > See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out > (...) > > 2. selftests/nolibc: always print the path to test log file > > > > Always print the path to test log file, but move it to a new line to > > avoid annoying people when the test pass without any failures. > > I'm still really missing the (s+f > 0) test I added which was a time saver > for me, because I could trivially check in the output reports which ones > were totally OK and which ones required attention. Sure I could also start > to grep for "passed," | grep -v " 0 skipped, 0 failed" but that's quite a > pain, really. > > I'm going to merge your series anyway otherwise we'll continue to bikeshed > for many weeks and I know how annoying it is to keep unmerged series. But > I would like that we find a solution that satisfies everyone. > > Maybe one possibility would be to add a "status" at the end of the line > that emits "success", "warning", "failure" depending on the highest level > reached like this: > > 138 test(s) passed, 0 skipped, 0 failed => status: success > 136 test(s) passed, 2 skipped, 0 failed => status: warning > 136 test(s) passed, 1 skipped, 1 failed => status: failure > > This way it's easy to grep -v "status: success" or grep "status: failure" > to instantly get the corresponding details and also grep for them from > multiple files. Ok, it will be a further step, based on your new awk script, it is not that hard. This should work: REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++;print} /\[SKIPPED\][\r]*$$/{s++} \ END{ printf("\n%d test(s): %d passed, %d skipped, %d failed => status: ", p+s+f, p, s, f); \ if (f) printf("failure\n"); else if (s) printf("warning\n"); else printf("success\n");; \ printf("\nSee all results in %s\n", ARGV[1]); }' It reports something like this: ... Total number of errors: 0 Exiting with status 0 143 test(s): 140 passed, 3 skipped, 0 failed => status: warning See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/build/x86/run.out Two newlines are added around the summary line, so, it is not crowded as before. If this is ok, will send a new patch on your new branch. Thanks very much. Best regards, Zhangjin > > Thanks! > Willy
Hi, Willy Here is the v2 of our old patchset about test report [1]. The trailing '\r' fixup has been merged, so, here only resend the left parts with an additional patch to restore the failed tests print. This patchset is rebased on the dev.2023.06.14a branch of linux-rcu [2]. Tests have passed for 'x86 run': 138 test(s) passed, 0 skipped, 0 failed. See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out Also did 'run-user' for x86, mips and arm64. Changes from v1 -> v2: 1. selftests/nolibc: add a standalone test report macro As Willy pointed out, the old method with additional test-report target not work in 'make -j'. A new macro is added to share the same report logic among the run-user, run and rerun targets, the path to test log file is 2. selftests/nolibc: always print the path to test log file Always print the path to test log file, but move it to a new line to avoid annoying people when the test pass without any failures. 3. selftests/nolibc: restore the failed tests print Restore printing of the failed tests to avoid manually opening the test log file when there are really failues. Best regards, Zhangjin --- [1]: https://lore.kernel.org/lkml/cover.1685936428.git.falcon@tinylab.org/ [2]: https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/ Zhangjin Wu (3): selftests/nolibc: add a standalone test report macro selftests/nolibc: always print the path to test log file selftests/nolibc: restore the failed tests print tools/testing/selftests/nolibc/Makefile | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)