| Message ID | 20230414193845.2494120-1-mcgrof@kernel.org (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | selftests: allow runners to override the timeout | expand |
On 4/15/23 12:38 AM, Luis Chamberlain wrote: > The default timeout for selftests tests is 45 seconds. Although > we already have 13 settings for tests of about 96 sefltests which > use a timeout greater than this, we want to try to avoid encouraging > more tests to forcing a higher test timeout as selftests strives to > run all tests quickly. Selftests also uses the timeout as a non-fatal > error. Only tests runners which have control over a system would know > if to treat a timeout as fatal or not. > > To help with all this: > > o Enhance documentation to avoid future increases of insane timeouts > o Add the option to allow overriding the default timeout with test > runners with a command line option > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> > --- > Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++ > tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- > tools/testing/selftests/run_kselftest.sh | 5 +++++ > 3 files changed, 37 insertions(+), 1 deletion(-) > > diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst > index 12b575b76b20..dd214af7b7ff 100644 > --- a/Documentation/dev-tools/kselftest.rst > +++ b/Documentation/dev-tools/kselftest.rst > @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: > > For other features see the script usage output, seen with the `-h` option. > > +Timeout for selftests > +===================== > + > +Selftests are designed to be quick and so a default timeout is used of 45 > +seconds for each test. Tests can override the default timeout by adding > +a settings file in their directory and set a timeout variable there to the > +configured a desired upper timeout for the test. Only a few tests override > +the timeout with a value higher than 45 seconds, selftests strives to keep > +it that way. Timeouts in selftests are not considered fatal because the > +system under which a test runs may change and this can also modify the > +expected time it takes to run a test. If you have control over the systems > +which will run the tests you can configure a test runner on those systems to > +use a greater or lower timeout on the command line as with the `-o` or > +the `--override-timeout` argument. For example to use 165 seconds instead > +one would use: > + > + $ ./run_kselftest.sh --override-timeout 165 > + > +You can look at the TAP output to see if you ran into the timeout. Test > +runners which know a test must run under a specific time can then optionally > +treat these timeouts then as fatal. > + > Packaging selftests > =================== > > diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh > index 294619ade49f..1c952d1401d4 100644 > --- a/tools/testing/selftests/kselftest/runner.sh > +++ b/tools/testing/selftests/kselftest/runner.sh > @@ -8,7 +8,8 @@ export logfile=/dev/stdout > export per_test_logging= > > # Defaults for "settings" file fields: > -# "timeout" how many seconds to let each test run before failing. > +# "timeout" how many seconds to let each test run before running > +# over our soft timeout limit. > export kselftest_default_timeout=45 > > # There isn't a shell-agnostic way to find the path of a sourced file, > @@ -90,6 +91,14 @@ run_one() > done < "$settings" > fi > > + # Command line timeout overrides the settings file > + if [ -n "$kselftest_override_timeout" ]; then > + kselftest_timeout="$kselftest_override_timeout" > + echo "# overriding timeout to $kselftest_timeout" >> "$logfile" > + else > + echo "# timeout set to $kselftest_timeout" >> "$logfile" > + fi > + > TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" > echo "# $TEST_HDR_MSG" > if [ ! -e "$TEST" ]; then > diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh > index 97165a83df63..9a981b36bd7f 100755 > --- a/tools/testing/selftests/run_kselftest.sh > +++ b/tools/testing/selftests/run_kselftest.sh > @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] > -l | --list List the available collection:test entries > -d | --dry-run Don't actually run any tests > -h | --help Show this usage info > + -o | --override-timeout Number of seconds after which we timeout > EOF > exit $1 > } > @@ -33,6 +34,7 @@ EOF > COLLECTIONS="" > TESTS="" > dryrun="" > +kselftest_override_timeout="" > while true; do > case "$1" in > -s | --summary) > @@ -51,6 +53,9 @@ while true; do > -d | --dry-run) > dryrun="echo" > shift ;; > + -o | --override-timeout) > + kselftest_override_timeout="$2" > + shift 2 ;; > -h | --help) > usage 0 ;; > "")
On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum <usama.anjum@collabora.com> wrote: > > On 4/15/23 12:38 AM, Luis Chamberlain wrote: > > The default timeout for selftests tests is 45 seconds. Although > > we already have 13 settings for tests of about 96 sefltests which > > use a timeout greater than this, we want to try to avoid encouraging > > more tests to forcing a higher test timeout as selftests strives to > > run all tests quickly. Selftests also uses the timeout as a non-fatal > > error. Only tests runners which have control over a system would know > > if to treat a timeout as fatal or not. > > > > To help with all this: > > > > o Enhance documentation to avoid future increases of insane timeouts > > o Add the option to allow overriding the default timeout with test > > runners with a command line option > > > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> > Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> Shuah, just a friendly poke! This is needed to allow me to enable full automation for kdevops for selftests. Luis
On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote: > On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum > <usama.anjum@collabora.com> wrote: > > > > On 4/15/23 12:38 AM, Luis Chamberlain wrote: > > > The default timeout for selftests tests is 45 seconds. Although > > > we already have 13 settings for tests of about 96 sefltests which > > > use a timeout greater than this, we want to try to avoid encouraging > > > more tests to forcing a higher test timeout as selftests strives to > > > run all tests quickly. Selftests also uses the timeout as a non-fatal > > > error. Only tests runners which have control over a system would know > > > if to treat a timeout as fatal or not. > > > > > > To help with all this: > > > > > > o Enhance documentation to avoid future increases of insane timeouts > > > o Add the option to allow overriding the default timeout with test > > > runners with a command line option > > > > > > Suggested-by: Shuah Khan <skhan@linuxfoundation.org> > > > Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> > > Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> > > Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> > > Shuah, just a friendly poke! This is needed to allow me to enable full > automation for kdevops for selftests. Shuah, friendly re-poke. Luis
On 5/24/23 16:58, Luis Chamberlain wrote: > On Thu, May 11, 2023 at 08:26:42AM -0700, Luis Chamberlain wrote: >> On Fri, Apr 28, 2023 at 1:34 AM Muhammad Usama Anjum >> <usama.anjum@collabora.com> wrote: >>> >>> On 4/15/23 12:38 AM, Luis Chamberlain wrote: >>>> The default timeout for selftests tests is 45 seconds. Although >>>> we already have 13 settings for tests of about 96 sefltests which >>>> use a timeout greater than this, we want to try to avoid encouraging >>>> more tests to forcing a higher test timeout as selftests strives to >>>> run all tests quickly. Selftests also uses the timeout as a non-fatal >>>> error. Only tests runners which have control over a system would know >>>> if to treat a timeout as fatal or not. >>>> >>>> To help with all this: >>>> >>>> o Enhance documentation to avoid future increases of insane timeouts >>>> o Add the option to allow overriding the default timeout with test >>>> runners with a command line option >>>> >>>> Suggested-by: Shuah Khan <skhan@linuxfoundation.org> >>>> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> >>> Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com> >>> Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com> >> >> Shuah, just a friendly poke! This is needed to allow me to enable full >> automation for kdevops for selftests. > > Shuah, friendly re-poke. > > Luis Thanks. I will apply this for next. thanks, -- Shuah
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 12b575b76b20..dd214af7b7ff 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times:: For other features see the script usage output, seen with the `-h` option. +Timeout for selftests +===================== + +Selftests are designed to be quick and so a default timeout is used of 45 +seconds for each test. Tests can override the default timeout by adding +a settings file in their directory and set a timeout variable there to the +configured a desired upper timeout for the test. Only a few tests override +the timeout with a value higher than 45 seconds, selftests strives to keep +it that way. Timeouts in selftests are not considered fatal because the +system under which a test runs may change and this can also modify the +expected time it takes to run a test. If you have control over the systems +which will run the tests you can configure a test runner on those systems to +use a greater or lower timeout on the command line as with the `-o` or +the `--override-timeout` argument. For example to use 165 seconds instead +one would use: + + $ ./run_kselftest.sh --override-timeout 165 + +You can look at the TAP output to see if you ran into the timeout. Test +runners which know a test must run under a specific time can then optionally +treat these timeouts then as fatal. + Packaging selftests =================== diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index 294619ade49f..1c952d1401d4 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -8,7 +8,8 @@ export logfile=/dev/stdout export per_test_logging= # Defaults for "settings" file fields: -# "timeout" how many seconds to let each test run before failing. +# "timeout" how many seconds to let each test run before running +# over our soft timeout limit. export kselftest_default_timeout=45 # There isn't a shell-agnostic way to find the path of a sourced file, @@ -90,6 +91,14 @@ run_one() done < "$settings" fi + # Command line timeout overrides the settings file + if [ -n "$kselftest_override_timeout" ]; then + kselftest_timeout="$kselftest_override_timeout" + echo "# overriding timeout to $kselftest_timeout" >> "$logfile" + else + echo "# timeout set to $kselftest_timeout" >> "$logfile" + fi + TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" echo "# $TEST_HDR_MSG" if [ ! -e "$TEST" ]; then diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh index 97165a83df63..9a981b36bd7f 100755 --- a/tools/testing/selftests/run_kselftest.sh +++ b/tools/testing/selftests/run_kselftest.sh @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS] -l | --list List the available collection:test entries -d | --dry-run Don't actually run any tests -h | --help Show this usage info + -o | --override-timeout Number of seconds after which we timeout EOF exit $1 } @@ -33,6 +34,7 @@ EOF COLLECTIONS="" TESTS="" dryrun="" +kselftest_override_timeout="" while true; do case "$1" in -s | --summary) @@ -51,6 +53,9 @@ while true; do -d | --dry-run) dryrun="echo" shift ;; + -o | --override-timeout) + kselftest_override_timeout="$2" + shift 2 ;; -h | --help) usage 0 ;; "")
The default timeout for selftests tests is 45 seconds. Although we already have 13 settings for tests of about 96 sefltests which use a timeout greater than this, we want to try to avoid encouraging more tests to forcing a higher test timeout as selftests strives to run all tests quickly. Selftests also uses the timeout as a non-fatal error. Only tests runners which have control over a system would know if to treat a timeout as fatal or not. To help with all this: o Enhance documentation to avoid future increases of insane timeouts o Add the option to allow overriding the default timeout with test runners with a command line option Suggested-by: Shuah Khan <skhan@linuxfoundation.org> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> --- Documentation/dev-tools/kselftest.rst | 22 +++++++++++++++++++++ tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++- tools/testing/selftests/run_kselftest.sh | 5 +++++ 3 files changed, 37 insertions(+), 1 deletion(-)