[net-next,0/4] selftests: kselftest_harness: support using xfail

Message

Jakub Kicinski Feb. 13, 2024, 3:44 p.m. UTC

Hi!

When running selftests for our subsystem in our CI we'd like all
tests to pass. Currently some tests use SKIP for cases they
expect to fail, because the kselftest_harness limits the return
codes to pass/fail/skip.

Clean up and support the use of the full range of ksft exit codes
under kselftest_harness.

To avoid conflicts and get the functionality into the networking
tree ASAP I'd like to put the patches on shared branch so that
both linux-kselftest and net-next can pull it in. Shuah, please
LMK if that'd work for you, and if so which -rc should I base
the branch on. Or is merging directly into net-next okay?

Jakub Kicinski (4):
  selftests: kselftest_harness: pass step via shared memory
  selftests: kselftest_harness: use KSFT_* exit codes
  selftests: kselftest_harness: support using xfail
  selftests: ip_local_port_range: use XFAIL instead of SKIP

 tools/testing/selftests/kselftest_harness.h   | 67 ++++++++++++++-----
 .../selftests/net/ip_local_port_range.c       |  2 +-
 2 files changed, 52 insertions(+), 17 deletions(-)

Comments

Kees Cook Feb. 13, 2024, 6 p.m. UTC | #1

On Tue, Feb 13, 2024 at 07:44:12AM -0800, Jakub Kicinski wrote:
> Hi!
> 
> When running selftests for our subsystem in our CI we'd like all
> tests to pass. Currently some tests use SKIP for cases they
> expect to fail, because the kselftest_harness limits the return
> codes to pass/fail/skip.
> 
> Clean up and support the use of the full range of ksft exit codes
> under kselftest_harness.
> 
> To avoid conflicts and get the functionality into the networking
> tree ASAP I'd like to put the patches on shared branch so that
> both linux-kselftest and net-next can pull it in. Shuah, please
> LMK if that'd work for you, and if so which -rc should I base
> the branch on. Or is merging directly into net-next okay?

I would use XFAIL for seccomp selftests too, but I can wait for the next
release. i.e. I don't need a shared branch -- it'd be fine in net-next.
But I defer to Shuah as far as the selftest tree is concerned. (FWIW, I
don't see any current conflicts.)

Jakub Sitnicki Feb. 14, 2024, 10:09 a.m. UTC | #2

On Tue, Feb 13, 2024 at 07:44 AM -08, Jakub Kicinski wrote:
> Hi!
>
> When running selftests for our subsystem in our CI we'd like all
> tests to pass. Currently some tests use SKIP for cases they
> expect to fail, because the kselftest_harness limits the return
> codes to pass/fail/skip.
>
> Clean up and support the use of the full range of ksft exit codes
> under kselftest_harness.
>
> To avoid conflicts and get the functionality into the networking
> tree ASAP I'd like to put the patches on shared branch so that
> both linux-kselftest and net-next can pull it in. Shuah, please
> LMK if that'd work for you, and if so which -rc should I base
> the branch on. Or is merging directly into net-next okay?
>
> Jakub Kicinski (4):
>   selftests: kselftest_harness: pass step via shared memory
>   selftests: kselftest_harness: use KSFT_* exit codes
>   selftests: kselftest_harness: support using xfail
>   selftests: ip_local_port_range: use XFAIL instead of SKIP
>
>  tools/testing/selftests/kselftest_harness.h   | 67 ++++++++++++++-----
>  .../selftests/net/ip_local_port_range.c       |  2 +-
>  2 files changed, 52 insertions(+), 17 deletions(-)

Nice!
We've been ignoring skipped tests in our internal CI.
But this is the wrong approach, as you point out.

For the series:

Tested-by: Jakub Sitnicki <[e-mail]>