[mptcp-net,v2,00/37] selftests: mptcp: skip tests when features are not supported

Message

Matthieu Baerts May 22, 2023, 4:37 p.m. UTC

Selftests are supposed to run on any kernels, including the old ones not
supporting all MPTCP features.

These different patches allow the different MPTCP selftests to be
executed on older kernels.

I tried to minimise the modifications and do the checks from the bash
script not to have to detect the feature when being used in a .c file,
then pass the info to the bash script, skip the test, clean the
environment, etc. As you can see, even with this minimalistic way, a lot
of modifications have been done to support old kernels.

Note that if we expect to have all features available,
SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var can be set to 1 to mark
the tests as failed instead of skipped and track regressions just in
case one of these checks is wrong. The public CI already set this env
var to 1.

This is a big series impacting new development. Probably best to apply
it "quickly" in our tree and fix issues later to avoid multiple rebases.

Closes: https://github.com/multipath-tcp/mptcp_net-next/issues/368
Signed-off-by: Matthieu Baerts <[e-mail]>
---
Changes in v2:
- The first patches are almost the same as in v1, the main modifications
  are in mptcp_lib.sh (and in the commit messages): new helpers but also
  the use of SELFTESTS_MPTCP_LIB_EXPECT_ALL_FEATURES env var.
- Link to v1: https://lore.kernel.org/r/[e-mail]

---
Matthieu Baerts (37):
      selftests: mptcp: connect: skip if MPTCP is not supported
      selftests: mptcp: pm nl: skip if MPTCP is not supported
      selftests: mptcp: join: skip if MPTCP is not supported
      selftests: mptcp: diag: skip if MPTCP is not supported
      selftests: mptcp: simult flows: skip if MPTCP is not supported
      selftests: mptcp: sockopt: skip if MPTCP is not supported
      selftests: mptcp: userspace pm: skip if MPTCP is not supported
      selftests: mptcp: lib: skip if missing symbol
      selftests: mptcp: lib: skip if not below kernel version
      selftests: mptcp: connect: skip transp tests if not supported
      selftests: mptcp: connect: skip disconnect tests if not supported
      selftests: mptcp: connect: skip TFO tests if not supported
      selftests: mptcp: diag: skip listen tests if not supported
      selftests: mptcp: diag: skip inuse tests if not supported
      selftests: mptcp: pm nl: remove hardcoded default limits
      selftests: mptcp: pm nl: skip fullmesh flag checks if not supported
      selftests: mptcp: sockopt: relax expected returned size
      selftests: mptcp: sockopt: skip getsockopt checks if not supported
      selftests: mptcp: sockopt: skip TCP_INQ checks if not supported
      selftests: mptcp: userspace pm: skip if 'ip' tool is unavailable
      selftests: mptcp: userspace pm: skip if not supported
      selftests: mptcp: userspace pm: skip PM listener events tests if unavailable
      selftests: mptcp: join: use 'iptables-legacy' if available
      selftests: mptcp: join: helpers to skip tests
      selftests: mptcp: join: skip check if MIB counter not supported
      selftests: mptcp: join: skip test if iptables/tc cmds fail
      selftests: mptcp: join: support local endpoint being tracked or not
      selftests: mptcp: join: skip Fastclose tests if not supported
      selftests: mptcp: join: support RM_ADDR for used endpoints or not
      selftests: mptcp: join: skip implicit tests if not supported
      selftests: mptcp: join: skip backup if set flag on ID not supported
      selftests: mptcp: join: skip fullmesh flag tests if not supported
      selftests: mptcp: join: skip userspace PM tests if not supported
      selftests: mptcp: join: skip fail tests if not supported
      selftests: mptcp: join: skip MPC backups tests if not supported
      selftests: mptcp: join: skip PM listener tests if not supported
      selftests: mptcp: join: skip mixed tests if not supported

 tools/testing/selftests/net/mptcp/Makefile         |   2 +-
 tools/testing/selftests/net/mptcp/config           |   1 +
 tools/testing/selftests/net/mptcp/diag.sh          |  46 +-
 tools/testing/selftests/net/mptcp/mptcp_connect.sh |  24 +
 tools/testing/selftests/net/mptcp/mptcp_join.sh    | 510 +++++++++++++--------
 tools/testing/selftests/net/mptcp/mptcp_lib.sh     |  99 ++++
 tools/testing/selftests/net/mptcp/mptcp_sockopt.c  |  12 +-
 tools/testing/selftests/net/mptcp/mptcp_sockopt.sh |  24 +-
 tools/testing/selftests/net/mptcp/pm_netlink.sh    |  31 +-
 tools/testing/selftests/net/mptcp/simult_flows.sh  |   4 +
 tools/testing/selftests/net/mptcp/userspace_pm.sh  |  17 +-
 11 files changed, 546 insertions(+), 224 deletions(-)
---
base-commit: 194dd0efe579cb5d3a746d248b3476f4b3fc0b48
change-id: 20230406-mptcp-issue-368-selftests-old-kernels-fffd69d67d91

Best regards,

Comments

Matthieu Baerts May 24, 2023, 5:04 p.m. UTC | #1

Hi Paolo,

(and anybody interested by this ;-) )

On 22/05/2023 18:37, Matthieu Baerts wrote:
> Selftests are supposed to run on any kernels, including the old ones not
> supporting all MPTCP features.

We discussed about this series at the meeting yesterday but I don't
think the next steps are clear, at least not to me :-)

Here is a short recap of the current situation:

- The first patches skipping MPTCP if it is not supported have been applied.

- check kernel version: what about adding an env var that can be set to
skip the versions check? So it works if a new feature is backported to
an older kernel.

- get_counter() to get the MIB counters: I can help the stable team with
the backports that will not be difficult I think but will cause conflicts

- check nf/tc issues: looks safer just in case a kconfig is missing or
an issue with nf/tc but can be removed if you prefer (still I don't
think we should exit in the middle of a test)

- one check per block of tests around the same theme: it would indeed
avoid re-doing the same check 2 or 3 times but it would break the IDs (+
maybe less obvious a test might be skipped when reviewing code)

- backports: at the end, we just need all these new checks to be
backported on the last stable kernel (6.3) because that's what is used
by the different CIs. It is fine if it is not backported after. (but
still I can help with the backports, I'm used to :) )

WDYT?

Cheers,
Matt