| Message ID | 20240516152513.1115270-1-kuba@kernel.org (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | fe56d6e4a99a40f50e64d5a8043f1fa838b1f7a1 |
| Headers | show |
| Series | [net,v3] selftests: net: local_termination: annotate the expected failures | expand |
On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote: > Vladimir said when adding this test: > > The bridge driver fares particularly badly [...] mainly because > it does not implement IFF_UNICAST_FLT. > > See commit 90b9566aa5cd ("selftests: forwarding: add a test for > local_termination.sh"). > > We don't want to hide the known gaps, but having a test which > always fails prevents us from catching regressions. Report > the cases we know may fail as XFAIL. > > Signed-off-by: Jakub Kicinski <kuba@kernel.org> I agree that XFAIL is appropriate for cases like this - the test is expected to fail and this can act like a TODO. I also looked over how xfail_on_veth works and this usage seems correct to me. Reviewed-by: Simon Horman <horms@kernel.org>
On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote: > Vladimir said when adding this test: > > The bridge driver fares particularly badly [...] mainly because > it does not implement IFF_UNICAST_FLT. > > See commit 90b9566aa5cd ("selftests: forwarding: add a test for > local_termination.sh"). > > We don't want to hide the known gaps, but having a test which > always fails prevents us from catching regressions. Report > the cases we know may fail as XFAIL. > > Signed-off-by: Jakub Kicinski <kuba@kernel.org> > --- > CC: liuhangbin@gmail.com > CC: shuah@kernel.org > CC: linux-kselftest@vger.kernel.org > CC: Petr Machata <petrm@nvidia.com> > CC: vladimir.oltean@nxp.com > > v3: > - use xfail_on_veth correctly as a "prefix" call > - dropping Vladimir's tags since the code is quite different now > v2: https://lore.kernel.org/r/20240509235553.5740-1-kuba@kernel.org/ > - remove duplicated log_test_xfail > v1: https://lore.kernel.org/all/20240509235553.5740-1-kuba@kernel.org/ > --- > .../net/forwarding/local_termination.sh | 30 +++++++++++-------- > 1 file changed, 18 insertions(+), 12 deletions(-) > > diff --git a/tools/testing/selftests/net/forwarding/local_termination.sh b/tools/testing/selftests/net/forwarding/local_termination.sh > index c5b0cbc85b3e..4b364cdf3ef0 100755 > --- a/tools/testing/selftests/net/forwarding/local_termination.sh > +++ b/tools/testing/selftests/net/forwarding/local_termination.sh > @@ -155,25 +155,30 @@ run_test() > "$smac > $MACVLAN_ADDR, ethertype IPv4 (0x0800)" \ > true > > - check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \ > - "$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \ > - false > + xfail_on_veth $h1 \ > + check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \ > + "$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \ > + false > > check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, promisc" \ > "$smac > $UNKNOWN_UC_ADDR2, ethertype IPv4 (0x0800)" \ > true > > - check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, allmulti" \ > - "$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \ > - false > + xfail_on_veth $h1 \ > + check_rcv $rcv_if_name \ > + "Unicast IPv4 to unknown MAC address, allmulti" \ > + "$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \ > + false > > check_rcv $rcv_if_name "Multicast IPv4 to joined group" \ > "$smac > $JOINED_MACV4_MC_ADDR, ethertype IPv4 (0x0800)" \ > true > > - check_rcv $rcv_if_name "Multicast IPv4 to unknown group" \ > - "$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \ > - false > + xfail_on_veth $h1 \ > + check_rcv $rcv_if_name \ > + "Multicast IPv4 to unknown group" \ > + "$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \ > + false > > check_rcv $rcv_if_name "Multicast IPv4 to unknown group, promisc" \ > "$smac > $UNKNOWN_MACV4_MC_ADDR2, ethertype IPv4 (0x0800)" \ > @@ -187,9 +192,10 @@ run_test() > "$smac > $JOINED_MACV6_MC_ADDR, ethertype IPv6 (0x86dd)" \ > true > > - check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \ > - "$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \ > - false > + xfail_on_veth $h1 \ > + check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \ > + "$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \ > + false > > check_rcv $rcv_if_name "Multicast IPv6 to unknown group, promisc" \ > "$smac > $UNKNOWN_MACV6_MC_ADDR2, ethertype IPv6 (0x86dd)" \ > -- > 2.45.0 > Reviewed-by: Hangbin Liu <liuhangbin@gmail.com>
Hangbin Liu <liuhangbin@gmail.com> writes: > On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote: >> Vladimir said when adding this test: >> >> The bridge driver fares particularly badly [...] mainly because >> it does not implement IFF_UNICAST_FLT. >> >> See commit 90b9566aa5cd ("selftests: forwarding: add a test for >> local_termination.sh"). >> >> We don't want to hide the known gaps, but having a test which >> always fails prevents us from catching regressions. Report >> the cases we know may fail as XFAIL. >> >> Signed-off-by: Jakub Kicinski <kuba@kernel.org> I'm still confused that the failure was shown for $rcv_if_name==bridge, yet we are testing $h1==veth. But mechanically the code is correct. Reviewed-by: Petr Machata <petrm@nvidia.com>
Hello: This patch was applied to netdev/net.git (main) by Jakub Kicinski <kuba@kernel.org>: On Thu, 16 May 2024 08:25:13 -0700 you wrote: > Vladimir said when adding this test: > > The bridge driver fares particularly badly [...] mainly because > it does not implement IFF_UNICAST_FLT. > > See commit 90b9566aa5cd ("selftests: forwarding: add a test for > local_termination.sh"). > > [...] Here is the summary with links: - [net,v3] selftests: net: local_termination: annotate the expected failures https://git.kernel.org/netdev/net/c/fe56d6e4a99a You are awesome, thank you!
diff --git a/tools/testing/selftests/net/forwarding/local_termination.sh b/tools/testing/selftests/net/forwarding/local_termination.sh index c5b0cbc85b3e..4b364cdf3ef0 100755 --- a/tools/testing/selftests/net/forwarding/local_termination.sh +++ b/tools/testing/selftests/net/forwarding/local_termination.sh @@ -155,25 +155,30 @@ run_test() "$smac > $MACVLAN_ADDR, ethertype IPv4 (0x0800)" \ true - check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \ - "$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \ - false + xfail_on_veth $h1 \ + check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \ + "$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \ + false check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, promisc" \ "$smac > $UNKNOWN_UC_ADDR2, ethertype IPv4 (0x0800)" \ true - check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, allmulti" \ - "$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \ - false + xfail_on_veth $h1 \ + check_rcv $rcv_if_name \ + "Unicast IPv4 to unknown MAC address, allmulti" \ + "$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \ + false check_rcv $rcv_if_name "Multicast IPv4 to joined group" \ "$smac > $JOINED_MACV4_MC_ADDR, ethertype IPv4 (0x0800)" \ true - check_rcv $rcv_if_name "Multicast IPv4 to unknown group" \ - "$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \ - false + xfail_on_veth $h1 \ + check_rcv $rcv_if_name \ + "Multicast IPv4 to unknown group" \ + "$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \ + false check_rcv $rcv_if_name "Multicast IPv4 to unknown group, promisc" \ "$smac > $UNKNOWN_MACV4_MC_ADDR2, ethertype IPv4 (0x0800)" \ @@ -187,9 +192,10 @@ run_test() "$smac > $JOINED_MACV6_MC_ADDR, ethertype IPv6 (0x86dd)" \ true - check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \ - "$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \ - false + xfail_on_veth $h1 \ + check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \ + "$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \ + false check_rcv $rcv_if_name "Multicast IPv6 to unknown group, promisc" \ "$smac > $UNKNOWN_MACV6_MC_ADDR2, ethertype IPv6 (0x86dd)" \
Vladimir said when adding this test: The bridge driver fares particularly badly [...] mainly because it does not implement IFF_UNICAST_FLT. See commit 90b9566aa5cd ("selftests: forwarding: add a test for local_termination.sh"). We don't want to hide the known gaps, but having a test which always fails prevents us from catching regressions. Report the cases we know may fail as XFAIL. Signed-off-by: Jakub Kicinski <kuba@kernel.org> --- CC: liuhangbin@gmail.com CC: shuah@kernel.org CC: linux-kselftest@vger.kernel.org CC: Petr Machata <petrm@nvidia.com> CC: vladimir.oltean@nxp.com v3: - use xfail_on_veth correctly as a "prefix" call - dropping Vladimir's tags since the code is quite different now v2: https://lore.kernel.org/r/20240509235553.5740-1-kuba@kernel.org/ - remove duplicated log_test_xfail v1: https://lore.kernel.org/all/20240509235553.5740-1-kuba@kernel.org/ --- .../net/forwarding/local_termination.sh | 30 +++++++++++-------- 1 file changed, 18 insertions(+), 12 deletions(-)