diff mbox series

[2/2] selftests: pmtu.sh: improve the test result processing

Message ID 20201105105051.64258-3-po-hsu.lin@canonical.com (mailing list archive)
State Superseded
Delegated to: Netdev Maintainers
Headers show
Series selftests: pmtu.sh: improve the test result processing | expand

Commit Message

Po-Hsu Lin Nov. 5, 2020, 10:50 a.m. UTC
This test will treat all non-zero return codes as failures, it will
make the pmtu.sh test script being marked as FAILED when some
sub-test got skipped.

Improve the result processing by
  * Only mark the whole test script as SKIP when all of the
    sub-tests were skipped
  * If the sub-tests were either passed or skipped, the overall
    result will be PASS
  * If any of them has failed, the overall result will be FAIL
  * Treat other return codes (e.g. 127 for command not found) as FAIL

Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
---
 tools/testing/selftests/net/pmtu.sh | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

Comments

Jakub Kicinski Nov. 7, 2020, 11:02 p.m. UTC | #1
On Thu,  5 Nov 2020 18:50:51 +0800 Po-Hsu Lin wrote:
> This test will treat all non-zero return codes as failures, it will
> make the pmtu.sh test script being marked as FAILED when some
> sub-test got skipped.
> 
> Improve the result processing by
>   * Only mark the whole test script as SKIP when all of the
>     sub-tests were skipped
>   * If the sub-tests were either passed or skipped, the overall
>     result will be PASS
>   * If any of them has failed, the overall result will be FAIL
>   * Treat other return codes (e.g. 127 for command not found) as FAIL
> 
> Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>

Patch 1 looks like a cleanup while patch 2 is more of a fix, can we
separate the two and apply the former to -next and latter to 5.10?
They shouldn't conflict, right?

> diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
> index fb53987..5c86fb1 100755
> --- a/tools/testing/selftests/net/pmtu.sh
> +++ b/tools/testing/selftests/net/pmtu.sh
> @@ -1652,7 +1652,23 @@ run_test() {
>  	return $ret
>  	)
>  	ret=$?
> -	[ $ret -ne 0 ] && exitcode=1
> +	case $ret in
> +		0)
> +			all_skipped=false
> +			[ $exitcode=$ksft_skip ] && exitcode=0
> +		;;
> +		1)
> +			all_skipped=false
> +			exitcode=1
> +		;;

Does it make sense to remove this case? The handling is identical to
the default case *).

> +		$ksft_skip)
> +			[ $all_skipped = true ] && exitcode=$ksft_skip
> +		;;
> +		*)
> +			all_skipped=false
> +			exitcode=1
> +		;;
> +	esac
>  
>  	return $ret
>  }
> @@ -1786,6 +1802,7 @@ usage() {
>  #
>  exitcode=0
>  desc=0
> +all_skipped=true
>  
>  while getopts :ptv o
>  do
Po-Hsu Lin Nov. 9, 2020, 3:42 a.m. UTC | #2
On Sun, Nov 8, 2020 at 7:02 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu,  5 Nov 2020 18:50:51 +0800 Po-Hsu Lin wrote:
> > This test will treat all non-zero return codes as failures, it will
> > make the pmtu.sh test script being marked as FAILED when some
> > sub-test got skipped.
> >
> > Improve the result processing by
> >   * Only mark the whole test script as SKIP when all of the
> >     sub-tests were skipped
> >   * If the sub-tests were either passed or skipped, the overall
> >     result will be PASS
> >   * If any of them has failed, the overall result will be FAIL
> >   * Treat other return codes (e.g. 127 for command not found) as FAIL
> >
> > Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
>
> Patch 1 looks like a cleanup while patch 2 is more of a fix, can we
> separate the two and apply the former to -next and latter to 5.10?
> They shouldn't conflict, right?
>

Hello Jakub,

Yes the first patch is just changing return code to $ksft_skip, the
real fix is the second one. However the second patch was based on the
first one, if we want to apply them separately we might need to change
this $ksft_skip handling part in the second patch.

What should I do to deal with this?
Resend the former for -next and rebase + resend the latter (plus the
fix to remove case 1) for 5.10 without the former patch?
Thanks!

> > diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
> > index fb53987..5c86fb1 100755
> > --- a/tools/testing/selftests/net/pmtu.sh
> > +++ b/tools/testing/selftests/net/pmtu.sh
> > @@ -1652,7 +1652,23 @@ run_test() {
> >       return $ret
> >       )
> >       ret=$?
> > -     [ $ret -ne 0 ] && exitcode=1
> > +     case $ret in
> > +             0)
> > +                     all_skipped=false
> > +                     [ $exitcode=$ksft_skip ] && exitcode=0
> > +             ;;
> > +             1)
> > +                     all_skipped=false
> > +                     exitcode=1
> > +             ;;
>
> Does it make sense to remove this case? The handling is identical to
> the default case *).
>

Yes you're right, we can remove this part.

> > +             $ksft_skip)
> > +                     [ $all_skipped = true ] && exitcode=$ksft_skip
> > +             ;;
> > +             *)
> > +                     all_skipped=false
> > +                     exitcode=1
> > +             ;;
> > +     esac
> >
> >       return $ret
> >  }
> > @@ -1786,6 +1802,7 @@ usage() {
> >  #
> >  exitcode=0
> >  desc=0
> > +all_skipped=true
> >
> >  while getopts :ptv o
> >  do
>
Jakub Kicinski Nov. 9, 2020, 6:09 p.m. UTC | #3
On Mon, 9 Nov 2020 11:42:33 +0800 Po-Hsu Lin wrote:
> On Sun, Nov 8, 2020 at 7:02 AM Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > On Thu,  5 Nov 2020 18:50:51 +0800 Po-Hsu Lin wrote:  
> > > This test will treat all non-zero return codes as failures, it will
> > > make the pmtu.sh test script being marked as FAILED when some
> > > sub-test got skipped.
> > >
> > > Improve the result processing by
> > >   * Only mark the whole test script as SKIP when all of the
> > >     sub-tests were skipped
> > >   * If the sub-tests were either passed or skipped, the overall
> > >     result will be PASS
> > >   * If any of them has failed, the overall result will be FAIL
> > >   * Treat other return codes (e.g. 127 for command not found) as FAIL
> > >
> > > Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>  
> >
> > Patch 1 looks like a cleanup while patch 2 is more of a fix, can we
> > separate the two and apply the former to -next and latter to 5.10?
> > They shouldn't conflict, right?
> >  
> 
> Hello Jakub,
> 
> Yes the first patch is just changing return code to $ksft_skip, the
> real fix is the second one. However the second patch was based on the
> first one, if we want to apply them separately we might need to change
> this $ksft_skip handling part in the second patch.

Ah, I misread the situation, ksft_skip is 4, not 2, so the patch is
more than just refactoring.

> What should I do to deal with this?
> Resend the former for -next and rebase + resend the latter (plus the
> fix to remove case 1) for 5.10 without the former patch?

Let's apply both of the patches to net-next if that's fine with you.
Indeed detangling them is may be more effort that it's worth.
Po-Hsu Lin Nov. 10, 2020, 1:36 a.m. UTC | #4
On Tue, Nov 10, 2020 at 2:09 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Mon, 9 Nov 2020 11:42:33 +0800 Po-Hsu Lin wrote:
> > On Sun, Nov 8, 2020 at 7:02 AM Jakub Kicinski <kuba@kernel.org> wrote:
> > >
> > > On Thu,  5 Nov 2020 18:50:51 +0800 Po-Hsu Lin wrote:
> > > > This test will treat all non-zero return codes as failures, it will
> > > > make the pmtu.sh test script being marked as FAILED when some
> > > > sub-test got skipped.
> > > >
> > > > Improve the result processing by
> > > >   * Only mark the whole test script as SKIP when all of the
> > > >     sub-tests were skipped
> > > >   * If the sub-tests were either passed or skipped, the overall
> > > >     result will be PASS
> > > >   * If any of them has failed, the overall result will be FAIL
> > > >   * Treat other return codes (e.g. 127 for command not found) as FAIL
> > > >
> > > > Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
> > >
> > > Patch 1 looks like a cleanup while patch 2 is more of a fix, can we
> > > separate the two and apply the former to -next and latter to 5.10?
> > > They shouldn't conflict, right?
> > >
> >
> > Hello Jakub,
> >
> > Yes the first patch is just changing return code to $ksft_skip, the
> > real fix is the second one. However the second patch was based on the
> > first one, if we want to apply them separately we might need to change
> > this $ksft_skip handling part in the second patch.
>
> Ah, I misread the situation, ksft_skip is 4, not 2, so the patch is
> more than just refactoring.
>
> > What should I do to deal with this?
> > Resend the former for -next and rebase + resend the latter (plus the
> > fix to remove case 1) for 5.10 without the former patch?
>
> Let's apply both of the patches to net-next if that's fine with you.
> Indeed detangling them is may be more effort that it's worth.

That would be great, but allow me to resend V2 to get rid of case 1 first.
Thanks!
diff mbox series

Patch

diff --git a/tools/testing/selftests/net/pmtu.sh b/tools/testing/selftests/net/pmtu.sh
index fb53987..5c86fb1 100755
--- a/tools/testing/selftests/net/pmtu.sh
+++ b/tools/testing/selftests/net/pmtu.sh
@@ -1652,7 +1652,23 @@  run_test() {
 	return $ret
 	)
 	ret=$?
-	[ $ret -ne 0 ] && exitcode=1
+	case $ret in
+		0)
+			all_skipped=false
+			[ $exitcode=$ksft_skip ] && exitcode=0
+		;;
+		1)
+			all_skipped=false
+			exitcode=1
+		;;
+		$ksft_skip)
+			[ $all_skipped = true ] && exitcode=$ksft_skip
+		;;
+		*)
+			all_skipped=false
+			exitcode=1
+		;;
+	esac
 
 	return $ret
 }
@@ -1786,6 +1802,7 @@  usage() {
 #
 exitcode=0
 desc=0
+all_skipped=true
 
 while getopts :ptv o
 do