diff mbox series

[v2] check: add support for --start-after

Message ID 20240216010803.164750-1-mcgrof@kernel.org (mailing list archive)
State New, archived
Headers show
Series [v2] check: add support for --start-after | expand

Commit Message

Luis Chamberlain Feb. 16, 2024, 1:08 a.m. UTC
Often times one is running a new test baseline we want to continue to
start testing where we left off if the last test was a crash. To do
this the first thing that occurred to me was to use the check.time
file as an expunge file but that doesn't work so well if you crashed
as the file turns out empty.

So instead add super simple argument --start-after which let's you
skip all tests until the test infrastructure has "seen" the test
you want to skip. This does obviously work best if you are not using
a random order, but that is rather implied. If you do use a random
order --start-after still works, the final output will however just
be randomized of course, but it should let you skip a failed known
crash at least. The real value to --start-after though is for when
you use a non-randomized order.

If the target test is not found in your test list we complain and
bail. This is not as obvious when you specify groups, so likewise
we do a special check when you use groups to ensure the test is at
least part of one group.

Demo:

root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/025
Start after test generic/025 not found in any group specified.
Be sure you specify a test present in one of your test run groups if using --start-after.

Your set of groups have these tests:

generic/476 generic/521 generic/522 generic/616 generic/617 generic/642 generic/650

root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/522
SECTION       -- xfs_reflink
RECREATING    -- xfs on /dev/loop16
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64 demo-xfs-reflink 6.5.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1 (2023-11-29)
MKFS_OPTIONS  -- -f -f -m reflink=1,rmapbt=1, -i sparse=1, /dev/loop5
MOUNT_OPTIONS -- /dev/loop5 /media/scratch

generic/616
generic/617
generic/642
generic/650

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---

Changes since v1:

This all addresses Anand Jain's feedback.

 - Skip tests completely which are not going to be run
 - Sanity test to ensure the test is part of a group, if you listed
   groups, and if not provide a useful output giving the list of all
   tests in your group so you can know better which one is a valid test
   to skip
 - Sanity test to ensure the test you specified is valid
 - Moves the trim during file processing now using a routine
   trim_start_after()

 check | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

Comments

Andrey Albershteyn Feb. 16, 2024, 9:03 a.m. UTC | #1
On 2024-02-15 17:08:02, Luis Chamberlain wrote:
> Often times one is running a new test baseline we want to continue to
> start testing where we left off if the last test was a crash. To do
> this the first thing that occurred to me was to use the check.time
> file as an expunge file but that doesn't work so well if you crashed
> as the file turns out empty.
> 
> So instead add super simple argument --start-after which let's you
> skip all tests until the test infrastructure has "seen" the test
> you want to skip. This does obviously work best if you are not using
> a random order, but that is rather implied. If you do use a random
> order --start-after still works, the final output will however just
> be randomized of course, but it should let you skip a failed known
> crash at least. The real value to --start-after though is for when
> you use a non-randomized order.
> 
> If the target test is not found in your test list we complain and
> bail. This is not as obvious when you specify groups, so likewise
> we do a special check when you use groups to ensure the test is at
> least part of one group.
> 
> Demo:
> 
> root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/025
> Start after test generic/025 not found in any group specified.
> Be sure you specify a test present in one of your test run groups if using --start-after.
> 
> Your set of groups have these tests:
> 
> generic/476 generic/521 generic/522 generic/616 generic/617 generic/642 generic/650
> 
> root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/522
> SECTION       -- xfs_reflink
> RECREATING    -- xfs on /dev/loop16
> FSTYP         -- xfs (non-debug)
> PLATFORM      -- Linux/x86_64 demo-xfs-reflink 6.5.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1 (2023-11-29)
> MKFS_OPTIONS  -- -f -f -m reflink=1,rmapbt=1, -i sparse=1, /dev/loop5
> MOUNT_OPTIONS -- /dev/loop5 /media/scratch
> 
> generic/616
> generic/617
> generic/642
> generic/650
> 
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
> 
> Changes since v1:
> 
> This all addresses Anand Jain's feedback.
> 
>  - Skip tests completely which are not going to be run
>  - Sanity test to ensure the test is part of a group, if you listed
>    groups, and if not provide a useful output giving the list of all
>    tests in your group so you can know better which one is a valid test
>    to skip
>  - Sanity test to ensure the test you specified is valid
>  - Moves the trim during file processing now using a routine
>    trim_start_after()
> 
>  check | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 52 insertions(+)
> 
> diff --git a/check b/check
> index 71b9fbd07522..1c76f33192ba 100755
> --- a/check
> +++ b/check
> @@ -18,6 +18,8 @@ showme=false
>  have_test_arg=false
>  randomize=false
>  exact_order=false
> +start_after=false
> +start_after_test=""
>  export here=`pwd`
>  xfile=""
>  subdir_xfile=""
> @@ -80,6 +82,7 @@ check options
>      -b			brief test summary
>      -R fmt[,fmt]	generate report in formats specified. Supported formats: xunit, xunit-quiet
>      --large-fs		optimise scratch device for large filesystems
> +    --start-after	only start testing after the test specified
>      -s section		run only specified section from config file
>      -S section		exclude the specified section from the config file
>      -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
> @@ -120,6 +123,8 @@ examples:
>   check -x stress xfs/*
>   check -X .exclude -g auto
>   check -E ~/.xfstests.exclude
> + check --start-after btrfs/010
> + check -n -g soak --start-after generic/522
>  '
>  	    exit 1
>  }
> @@ -204,6 +209,24 @@ trim_test_list()
>  	rm -f $tmp.grep
>  }
>  
> +# takes the list of tests to run in $tmp.list and skips all tests until
> +# the specified test is found. This will ensure the tests start after the
> +# test specified, it skips the test specified.
> +trim_start_after()
> +{
> +	local skip_test="$1"
> +	local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
> +	local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"

Looks like grep_start_after is not used
Otherwise, LGTM:
Reviewed-by: Andrey Albershteyn <aalbersh@redhat.com>
Luis Chamberlain Feb. 16, 2024, 4:41 p.m. UTC | #2
On Fri, Feb 16, 2024 at 10:03:31AM +0100, Andrey Albershteyn wrote:
> On 2024-02-15 17:08:02, Luis Chamberlain wrote:
> > Often times one is running a new test baseline we want to continue to
> > start testing where we left off if the last test was a crash. To do
> > this the first thing that occurred to me was to use the check.time
> > file as an expunge file but that doesn't work so well if you crashed
> > as the file turns out empty.
> > 
> > So instead add super simple argument --start-after which let's you
> > skip all tests until the test infrastructure has "seen" the test
> > you want to skip. This does obviously work best if you are not using
> > a random order, but that is rather implied. If you do use a random
> > order --start-after still works, the final output will however just
> > be randomized of course, but it should let you skip a failed known
> > crash at least. The real value to --start-after though is for when
> > you use a non-randomized order.
> > 
> > If the target test is not found in your test list we complain and
> > bail. This is not as obvious when you specify groups, so likewise
> > we do a special check when you use groups to ensure the test is at
> > least part of one group.
> > 
> > Demo:
> > 
> > root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/025
> > Start after test generic/025 not found in any group specified.
> > Be sure you specify a test present in one of your test run groups if using --start-after.
> > 
> > Your set of groups have these tests:
> > 
> > generic/476 generic/521 generic/522 generic/616 generic/617 generic/642 generic/650
> > 
> > root@demo-xfs-reflink /var/lib/xfstests # ./check -s xfs_reflink -n -g soak --start-after generic/522
> > SECTION       -- xfs_reflink
> > RECREATING    -- xfs on /dev/loop16
> > FSTYP         -- xfs (non-debug)
> > PLATFORM      -- Linux/x86_64 demo-xfs-reflink 6.5.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1 (2023-11-29)
> > MKFS_OPTIONS  -- -f -f -m reflink=1,rmapbt=1, -i sparse=1, /dev/loop5
> > MOUNT_OPTIONS -- /dev/loop5 /media/scratch
> > 
> > generic/616
> > generic/617
> > generic/642
> > generic/650
> > 
> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> > ---
> > 
> > Changes since v1:
> > 
> > This all addresses Anand Jain's feedback.
> > 
> >  - Skip tests completely which are not going to be run
> >  - Sanity test to ensure the test is part of a group, if you listed
> >    groups, and if not provide a useful output giving the list of all
> >    tests in your group so you can know better which one is a valid test
> >    to skip
> >  - Sanity test to ensure the test you specified is valid
> >  - Moves the trim during file processing now using a routine
> >    trim_start_after()
> > 
> >  check | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 52 insertions(+)
> > 
> > diff --git a/check b/check
> > index 71b9fbd07522..1c76f33192ba 100755
> > --- a/check
> > +++ b/check
> > @@ -18,6 +18,8 @@ showme=false
> >  have_test_arg=false
> >  randomize=false
> >  exact_order=false
> > +start_after=false
> > +start_after_test=""
> >  export here=`pwd`
> >  xfile=""
> >  subdir_xfile=""
> > @@ -80,6 +82,7 @@ check options
> >      -b			brief test summary
> >      -R fmt[,fmt]	generate report in formats specified. Supported formats: xunit, xunit-quiet
> >      --large-fs		optimise scratch device for large filesystems
> > +    --start-after	only start testing after the test specified
> >      -s section		run only specified section from config file
> >      -S section		exclude the specified section from the config file
> >      -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
> > @@ -120,6 +123,8 @@ examples:
> >   check -x stress xfs/*
> >   check -X .exclude -g auto
> >   check -E ~/.xfstests.exclude
> > + check --start-after btrfs/010
> > + check -n -g soak --start-after generic/522
> >  '
> >  	    exit 1
> >  }
> > @@ -204,6 +209,24 @@ trim_test_list()
> >  	rm -f $tmp.grep
> >  }
> >  
> > +# takes the list of tests to run in $tmp.list and skips all tests until
> > +# the specified test is found. This will ensure the tests start after the
> > +# test specified, it skips the test specified.
> > +trim_start_after()
> > +{
> > +	local skip_test="$1"
> > +	local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
> > +	local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"
> 
> Looks like grep_start_after is not used
> Otherwise, LGTM:
> Reviewed-by: Andrey Albershteyn <aalbersh@redhat.com>

Thanks I've made these changes, will send a v3:

diff --git a/check b/check
index 1c76f33192ba..d0bf7ea43870 100755
--- a/check
+++ b/check
@@ -18,7 +18,6 @@ showme=false
 have_test_arg=false
 randomize=false
 exact_order=false
-start_after=false
 start_after_test=""
 export here=`pwd`
 xfile=""
@@ -216,7 +215,6 @@ trim_start_after()
 {
 	local skip_test="$1"
 	local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
-	local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"
 
 	if grep -q $skip_test $tmp.list ; then
 		rm -f $tmp.grep
@@ -266,7 +264,7 @@ _prepare_test_list()
 				exit 1
 			fi
 
-			if [[ $start_after && $start_after_found -ne 1 ]]; then
+			if [[ "$start_after_test" != "" && $start_after_found -ne 1 ]]; then
 				echo $list | grep -q $start_after_test
 				if [[ $? -eq 0 ]]; then
 					start_after_found=1
@@ -278,7 +276,7 @@ _prepare_test_list()
 			done
 			group_all="$group_all $list"
 		done
-		if [[ $start_after && $start_after_found -ne 1 ]]; then
+		if [[ "$start_after_test" != "" && $start_after_found -ne 1 ]]; then
 			group_all=$(echo $group_all | sed -e 's|tests/||g')
 			echo "Start after test $start_after_test not found in any group specified."
 			echo "Be sure you specify a test present in one of your test run groups if using --start-after."
@@ -301,7 +299,7 @@ _prepare_test_list()
 		trim_test_list $list
 	done
 
-	if [[ $start_after ]]; then
+	if [[ "$start_after_test" != "" ]]; then
 		trim_start_after $start_after_test
 	fi
 
@@ -361,7 +359,10 @@ while [ $# -gt 0 ]; do
 		fi
 		;;
 	--start-after)
-		start_after=true
+		if $randomize; then
+			echo "Cannot specify -r and --start-after."
+			exit 1
+		fi
 		start_after_test="$2"
 		shift
 		;;
diff mbox series

Patch

diff --git a/check b/check
index 71b9fbd07522..1c76f33192ba 100755
--- a/check
+++ b/check
@@ -18,6 +18,8 @@  showme=false
 have_test_arg=false
 randomize=false
 exact_order=false
+start_after=false
+start_after_test=""
 export here=`pwd`
 xfile=""
 subdir_xfile=""
@@ -80,6 +82,7 @@  check options
     -b			brief test summary
     -R fmt[,fmt]	generate report in formats specified. Supported formats: xunit, xunit-quiet
     --large-fs		optimise scratch device for large filesystems
+    --start-after	only start testing after the test specified
     -s section		run only specified section from config file
     -S section		exclude the specified section from the config file
     -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
@@ -120,6 +123,8 @@  examples:
  check -x stress xfs/*
  check -X .exclude -g auto
  check -E ~/.xfstests.exclude
+ check --start-after btrfs/010
+ check -n -g soak --start-after generic/522
 '
 	    exit 1
 }
@@ -204,6 +209,24 @@  trim_test_list()
 	rm -f $tmp.grep
 }
 
+# takes the list of tests to run in $tmp.list and skips all tests until
+# the specified test is found. This will ensure the tests start after the
+# test specified, it skips the test specified.
+trim_start_after()
+{
+	local skip_test="$1"
+	local starts_regexp=$(echo $skip_test | sed -e 's|\/|\\/|')
+	local grep_start_after=" | awk 'f;/.*'$starts_regexp'/{f=1}'"
+
+	if grep -q $skip_test $tmp.list ; then
+		rm -f $tmp.grep
+		awk 'f;/.*'$starts_regexp'/{f=1}' $tmp.list > $tmp.tmp
+		mv $tmp.tmp $tmp.list
+	else
+		echo "Test $skip_test not found in test list, be sure to use a valid test if using --start-after"
+		exit 1
+	fi
+}
 
 _wallclock()
 {
@@ -233,6 +256,9 @@  _prepare_test_list()
 		# no test numbers, do everything
 		get_all_tests
 	else
+		local group_all
+		local start_after_found=0
+		list=""
 		for group in $GROUP_LIST; do
 			list=$(get_group_list $group)
 			if [ -z "$list" ]; then
@@ -240,11 +266,28 @@  _prepare_test_list()
 				exit 1
 			fi
 
+			if [[ $start_after && $start_after_found -ne 1 ]]; then
+				echo $list | grep -q $start_after_test
+				if [[ $? -eq 0 ]]; then
+					start_after_found=1
+				fi
+			fi
 			for t in $list; do
 				grep -s "^$t\$" $tmp.list >/dev/null || \
 							echo "$t" >>$tmp.list
 			done
+			group_all="$group_all $list"
 		done
+		if [[ $start_after && $start_after_found -ne 1 ]]; then
+			group_all=$(echo $group_all | sed -e 's|tests/||g')
+			echo "Start after test $start_after_test not found in any group specified."
+			echo "Be sure you specify a test present in one of your test run groups if using --start-after."
+			echo
+			echo "Your set of groups have these tests:"
+			echo
+			echo -e $group_all
+			exit 1
+		fi
 	fi
 
 	# Specified groups to exclude
@@ -258,6 +301,10 @@  _prepare_test_list()
 		trim_test_list $list
 	done
 
+	if [[ $start_after ]]; then
+		trim_start_after $start_after_test
+	fi
+
 	# sort the list of tests into numeric order unless we're running tests
 	# in the exact order specified
 	if ! $exact_order; then
@@ -313,6 +360,11 @@  while [ $# -gt 0 ]; do
 				<(sed "s/#.*$//" $xfile)
 		fi
 		;;
+	--start-after)
+		start_after=true
+		start_after_test="$2"
+		shift
+		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
 	-S)	EXCLUDE_SECTION="$EXCLUDE_SECTION $2"; shift ;;
 	-l)	diff="diff" ;;