| Message ID | 56a687e167af434fa54429558164ff7fee32c67d.1611823132.git.riteshh@linux.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | check: add CLI option to repeat and stop tests in case of failure | expand |
On Thu, Jan 28, 2021 at 02:19:41PM +0530, Ritesh Harjani wrote: > Currently with -i <n> option the test can run for many iterations, > but in case if we want to stop the iteration in case of a failure, > it is much easier to have such an option which could check the failed > status and stop the test from further proceeding. > > This patch adds such an option thereby extending the -i <n> option > functionality. > > Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com> > --- > check | 9 +++++++++ > 1 file changed, 9 insertions(+) > > diff --git a/check b/check > index c6ad1d6c0733..6f3a5d47e212 100755 > --- a/check > +++ b/check > @@ -15,6 +15,7 @@ sum_bad=0 > bad="" > n_notrun=0 > notrun="" > +tc_status="" > interrupt=true > diff="diff -u" > showme=false > @@ -27,6 +28,7 @@ brief_test_summary=false > do_report=false > DUMP_OUTPUT=false > iterations=1 > +istop=false > > # This is a global variable used to pass test failure text to reporting gunk > _err_msg="" > @@ -68,6 +70,7 @@ check options > -T output timestamps > -r randomize test order > -i <n> iterate the test list <n> times > + -istop <n> iterate the test list <n> times, but stops iterating further in case of any test failure Perhaps '-I' is a better option? > -d dump test output to stdout > -b brief test summary > -R fmt[,fmt] generate report in formats specified. Supported format: [xunit] > @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do > -n) showme=true ;; > -r) randomize=true ;; > -i) iterations=$2; shift ;; > + -istop) iterations=$2; istop=true; shift ;; Same here. > -T) timestamp=true ;; > -d) DUMP_OUTPUT=true ;; > -b) brief_test_summary=true;; > @@ -926,6 +930,11 @@ function run_section() > for ((iters = 0; iters < $iterations; iters++)) do > for section in $HOST_OPTIONS_SECTIONS; do > run_section $section > + if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then $tc_status only records the status of last test in the test list, and if the last test passed, the iteration continues. e.g. I tested generic/00[12], and generic/001 fails while generic/002 passes ./check -i 2 generic/001 generic/002 check still run 2 iterations as "tc_status" of generic/002 is not "fail". I think we could check $sum_bad is 0 or not. Thanks, Eryu > + interrupt=false > + status=`expr $sum_bad != 0` > + exit > + fi > done > done > > -- > 2.26.2
On 1/31/21 8:33 PM, Eryu Guan wrote: > On Thu, Jan 28, 2021 at 02:19:41PM +0530, Ritesh Harjani wrote: >> Currently with -i <n> option the test can run for many iterations, >> but in case if we want to stop the iteration in case of a failure, >> it is much easier to have such an option which could check the failed >> status and stop the test from further proceeding. >> >> This patch adds such an option thereby extending the -i <n> option >> functionality. >> >> Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com> >> --- >> check | 9 +++++++++ >> 1 file changed, 9 insertions(+) >> >> diff --git a/check b/check >> index c6ad1d6c0733..6f3a5d47e212 100755 >> --- a/check >> +++ b/check >> @@ -15,6 +15,7 @@ sum_bad=0 >> bad="" >> n_notrun=0 >> notrun="" >> +tc_status="" >> interrupt=true >> diff="diff -u" >> showme=false >> @@ -27,6 +28,7 @@ brief_test_summary=false >> do_report=false >> DUMP_OUTPUT=false >> iterations=1 >> +istop=false >> >> # This is a global variable used to pass test failure text to reporting gunk >> _err_msg="" >> @@ -68,6 +70,7 @@ check options >> -T output timestamps >> -r randomize test order >> -i <n> iterate the test list <n> times >> + -istop <n> iterate the test list <n> times, but stops iterating further in case of any test failure > > Perhaps '-I' is a better option? Sure, will change it. > >> -d dump test output to stdout >> -b brief test summary >> -R fmt[,fmt] generate report in formats specified. Supported format: [xunit] >> @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do >> -n) showme=true ;; >> -r) randomize=true ;; >> -i) iterations=$2; shift ;; >> + -istop) iterations=$2; istop=true; shift ;; > > Same here. yes. > >> -T) timestamp=true ;; >> -d) DUMP_OUTPUT=true ;; >> -b) brief_test_summary=true;; >> @@ -926,6 +930,11 @@ function run_section() >> for ((iters = 0; iters < $iterations; iters++)) do >> for section in $HOST_OPTIONS_SECTIONS; do >> run_section $section >> + if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then > > $tc_status only records the status of last test in the test list, and if > the last test passed, the iteration continues. e.g. I tested > generic/00[12], and generic/001 fails while generic/002 passes > > ./check -i 2 generic/001 generic/002 > > check still run 2 iterations as "tc_status" of generic/002 is not > "fail". > > I think we could check $sum_bad is 0 or not. Yes, right. Thanks for pointing it out. I will address these comments and spin a v2. -ritesh
diff --git a/check b/check index c6ad1d6c0733..6f3a5d47e212 100755 --- a/check +++ b/check @@ -15,6 +15,7 @@ sum_bad=0 bad="" n_notrun=0 notrun="" +tc_status="" interrupt=true diff="diff -u" showme=false @@ -27,6 +28,7 @@ brief_test_summary=false do_report=false DUMP_OUTPUT=false iterations=1 +istop=false # This is a global variable used to pass test failure text to reporting gunk _err_msg="" @@ -68,6 +70,7 @@ check options -T output timestamps -r randomize test order -i <n> iterate the test list <n> times + -istop <n> iterate the test list <n> times, but stops iterating further in case of any test failure -d dump test output to stdout -b brief test summary -R fmt[,fmt] generate report in formats specified. Supported format: [xunit] @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do -n) showme=true ;; -r) randomize=true ;; -i) iterations=$2; shift ;; + -istop) iterations=$2; istop=true; shift ;; -T) timestamp=true ;; -d) DUMP_OUTPUT=true ;; -b) brief_test_summary=true;; @@ -926,6 +930,11 @@ function run_section() for ((iters = 0; iters < $iterations; iters++)) do for section in $HOST_OPTIONS_SECTIONS; do run_section $section + if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then + interrupt=false + status=`expr $sum_bad != 0` + exit + fi done done
Currently with -i <n> option the test can run for many iterations, but in case if we want to stop the iteration in case of a failure, it is much easier to have such an option which could check the failed status and stop the test from further proceeding. This patch adds such an option thereby extending the -i <n> option functionality. Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com> --- check | 9 +++++++++ 1 file changed, 9 insertions(+) -- 2.26.2