| Message ID | 20210827225812.3247919-1-rmoar@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | selftests: tool: update format of kselftest TAP results to improve compatibility with KTAP | expand |
On Fri, Aug 27, 2021 at 10:58:10PM +0000, Rae Moar wrote: > This series of patches updates the format of kselftest TAP results to improve > compatibility with the proposed KTAP specification > (https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/). > > Three changes: > - Change from "# " to " " for indentation of nested tests This seems like subtests would need to have a known global nesting tracking again. That was removed to simplified various aspects of kselftest. > - Add subtest header line at start of tests with subtests. Line format > is "# Subtest: [name of test]". That would mean subtests are no longer individually parseable by existing TAP parsers (i.e. when LAVA splits up kselftest output, suddenly there is no TAP header). > - Remove TAP header in nested tests What benefit does this provide? This just makes things harder to machine parse now. > Standardizing TAP results would not only allow for clearer documentation and ease of reading but by standardizing the format across different testing frameworks, we could also share the use of tools. > > As an example: > > This is a truncated version of TAP results from the kselftest ptrace with the new format changes: > > TAP version 13 > 1..1 > # selftests: ptrace: get_syscall_info > # Subtest: selftests: ptrace: get_syscall_info > 1..1 > # Starting 1 tests from 1 test cases. > # RUN global.get_syscall_info ... > # OK global.get_syscall_info > ok 1 global.get_syscall_info > # PASSED: 1 / 1 tests passed. > # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 > ok 1 selftests: ptrace: get_syscall_info But this would mean changing every TAP parser to notice "# Subtest:" instead of "TAP version ..." Why bother doing that? A parser aware of subtests just needs to examine indentation level currently. > > With the new patch to update the KUnit parser to improve compatibility with the proposed KTAP specification, (https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@google.com/) the above TAP results would be parsed as the following: > > [20:46:09] ============================================================ > [20:46:09] ===== selftests: ptrace: get_syscall_info (1 subtest) ====== > [20:46:09] [PASSED] global.get_syscall_info > [20:46:09] ======= [PASSED] selftests: ptrace: get_syscall_info ======= > [20:46:09] ============================================================ > [20:46:09] Testing complete. Passed: 1, Failed: 0, Crashed: 0, Skipped: 0, Errors: 0 > > Thus, the kunit parser could become a useful tool for kselftest users. > > Rae Moar (2): > selftests: tool: Add subtest header line and change indentation format > in TAP results > Revert "selftests: Remove KSFT_TAP_LEVEL" > > tools/testing/selftests/Makefile | 6 ++++++ > tools/testing/selftests/kselftest/prefix.pl | 2 +- > tools/testing/selftests/kselftest/runner.sh | 7 ++++--- > 3 files changed, 11 insertions(+), 4 deletions(-) > > -- > 2.33.0.259.gc128427fd7-goog >
These changes to the format of TAP results for kselftests are based on this proposed specification of KTAP: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/ However, there has been considerable discussion on this proposal recently. It is great to see this discourse on the specification KTAP. However, the changes to the format of TAP results for kselftests in this series of patches should be paused until further agreement on the exact specification of KTAP. - Rae On Sat, Aug 28, 2021 at 2:10 AM Kees Cook <keescook@chromium.org> wrote: > > On Fri, Aug 27, 2021 at 10:58:10PM +0000, Rae Moar wrote: > > This series of patches updates the format of kselftest TAP results to improve > > compatibility with the proposed KTAP specification > > (https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/). > > > > Three changes: > > - Change from "# " to " " for indentation of nested tests > > This seems like subtests would need to have a known global nesting > tracking again. That was removed to simplified various aspects of > kselftest. > > > - Add subtest header line at start of tests with subtests. Line format > > is "# Subtest: [name of test]". > > That would mean subtests are no longer individually parseable by > existing TAP parsers (i.e. when LAVA splits up kselftest output, > suddenly there is no TAP header). > > > - Remove TAP header in nested tests > > What benefit does this provide? This just makes things harder to machine > parse now. > > > Standardizing TAP results would not only allow for clearer documentation and ease of reading but by standardizing the format across different testing frameworks, we could also share the use of tools. > > > > As an example: > > > > This is a truncated version of TAP results from the kselftest ptrace with the new format changes: > > > > TAP version 13 > > 1..1 > > # selftests: ptrace: get_syscall_info > > # Subtest: selftests: ptrace: get_syscall_info > > 1..1 > > # Starting 1 tests from 1 test cases. > > # RUN global.get_syscall_info ... > > # OK global.get_syscall_info > > ok 1 global.get_syscall_info > > # PASSED: 1 / 1 tests passed. > > # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 > > ok 1 selftests: ptrace: get_syscall_info > > But this would mean changing every TAP parser to notice "# Subtest:" > instead of "TAP version ..." Why bother doing that? A parser aware of > subtests just needs to examine indentation level currently. > > > > > With the new patch to update the KUnit parser to improve compatibility with the proposed KTAP specification, (https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@google.com/) the above TAP results would be parsed as the following: > > > > [20:46:09] ============================================================ > > [20:46:09] ===== selftests: ptrace: get_syscall_info (1 subtest) ====== > > [20:46:09] [PASSED] global.get_syscall_info > > [20:46:09] ======= [PASSED] selftests: ptrace: get_syscall_info ======= > > [20:46:09] ============================================================ > > [20:46:09] Testing complete. Passed: 1, Failed: 0, Crashed: 0, Skipped: 0, Errors: 0 > > > > Thus, the kunit parser could become a useful tool for kselftest users. > > > > Rae Moar (2): > > selftests: tool: Add subtest header line and change indentation format > > in TAP results > > Revert "selftests: Remove KSFT_TAP_LEVEL" > > > > tools/testing/selftests/Makefile | 6 ++++++ > > tools/testing/selftests/kselftest/prefix.pl | 2 +- > > tools/testing/selftests/kselftest/runner.sh | 7 ++++--- > > 3 files changed, 11 insertions(+), 4 deletions(-) > > > > -- > > 2.33.0.259.gc128427fd7-goog > > > > -- > Kees Cook
This series of patches updates the format of kselftest TAP results to improve compatibility with the proposed KTAP specification (https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/). Three changes: - Change from "# " to " " for indentation of nested tests - Add subtest header line at start of tests with subtests. Line format is "# Subtest: [name of test]". - Remove TAP header in nested tests Standardizing TAP results would not only allow for clearer documentation and ease of reading but by standardizing the format across different testing frameworks, we could also share the use of tools. As an example: This is a truncated version of TAP results from the kselftest ptrace with the new format changes: TAP version 13 1..1 # selftests: ptrace: get_syscall_info # Subtest: selftests: ptrace: get_syscall_info 1..1 # Starting 1 tests from 1 test cases. # RUN global.get_syscall_info ... # OK global.get_syscall_info ok 1 global.get_syscall_info # PASSED: 1 / 1 tests passed. # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 ok 1 selftests: ptrace: get_syscall_info With the new patch to update the KUnit parser to improve compatibility with the proposed KTAP specification, (https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@google.com/) the above TAP results would be parsed as the following: [20:46:09] ============================================================ [20:46:09] ===== selftests: ptrace: get_syscall_info (1 subtest) ====== [20:46:09] [PASSED] global.get_syscall_info [20:46:09] ======= [PASSED] selftests: ptrace: get_syscall_info ======= [20:46:09] ============================================================ [20:46:09] Testing complete. Passed: 1, Failed: 0, Crashed: 0, Skipped: 0, Errors: 0 Thus, the kunit parser could become a useful tool for kselftest users. Rae Moar (2): selftests: tool: Add subtest header line and change indentation format in TAP results Revert "selftests: Remove KSFT_TAP_LEVEL" tools/testing/selftests/Makefile | 6 ++++++ tools/testing/selftests/kselftest/prefix.pl | 2 +- tools/testing/selftests/kselftest/runner.sh | 7 ++++--- 3 files changed, 11 insertions(+), 4 deletions(-)