diff mbox series

[kdevops] fstests: provide kconfig guidance for SOAK_DURATION

Message ID ZbLcXWI9bCdc9PZO@bombadil.infradead.org (mailing list archive)
State New, archived
Headers show
Series [kdevops] fstests: provide kconfig guidance for SOAK_DURATION | expand

Commit Message

Luis Chamberlain Jan. 25, 2024, 10:10 p.m. UTC
The kdevops test runner has supported a custom SOAK_DURATION for
fstests, however we were not providing any guidance. This means folks
likely disable this. Throw a bone and provide some basic guidance and
use 2.5 hours as the default value. There are about 46 tests today
which use soak duration, this means if you are testing serially it
increase total test time by about 5 days than the previously known
total test time.

Note that if you are using kernel-ci and using a max loop goal of 100
that means 500 days extra, so about 1.3 years extra total test time.
If enabling soak duration you may want to then re-evaluate your loop
target goal for kernel-ci for kdevops.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---

Chandan, Amir, lemme know what you think of a default 2.5 hours default
if soak duration is enabled. The only thing is the math indicates that
if you are going to enable kernel-ci we won't finish this year.

To be clear, we've picked up testing with soak duration seriously for
our LBS testing. It is why we've been able to find pretty hard to
reproduce issues even on the page cache for the baseline [0], ie, without
LBS. While folks have seemed to have found value in adopting 2.5 hours
and of the results we have found, it obviously means a scaling issue
to consider to decide when we're done with testing our baseline.

At first I wrote this patch just to provide basic guidance for kdevops,
but after doing a bit of the math on how it also extends total test
time, *with* our kernel-ci effort, it reveals clearly we should probably
reconsider lowering the kernel-ci threshold a bit if adopting soak
duration.

CC'ing a bit wider audience so to get a bit better idea of what folks
might consider a sensible value for your own testing too. From what
we've been observing, SOAK_DURATION allows us to catch bugs faster than
just increasing the kernel-ci count, however, using both let's us catch
even more bugs too.

To help *reduce* the amount of time to test we've deployed many kdevops
XFS clusters to help test the baseline. This is why our count time on
kernel-ci no is about 50-60 with a soak duration of about 2.5 hours.

Also please not that the reported bugs so far are the ones with crashes,
there are other failures too, but we just haven't had the time to disect
and report failures which are non-fatal (crashes) as crashes have been
our priority.

[0] https://github.com/linux-kdevops/kdevops/blob/master/docs/xfs-bugs.md

 playbooks/roles/fstests/defaults/main.yml |  3 +
 workflows/fstests/Kconfig                 | 89 ++++++++++++++++++++---
 workflows/fstests/Makefile.sparsefiles    |  4 +
 3 files changed, 87 insertions(+), 9 deletions(-)

Comments

Darrick J. Wong Jan. 25, 2024, 10:29 p.m. UTC | #1
On Thu, Jan 25, 2024 at 02:10:37PM -0800, Luis Chamberlain wrote:
> The kdevops test runner has supported a custom SOAK_DURATION for
> fstests, however we were not providing any guidance. This means folks
> likely disable this. Throw a bone and provide some basic guidance and
> use 2.5 hours as the default value. There are about 46 tests today
> which use soak duration, this means if you are testing serially it
> increase total test time by about 5 days than the previously known
> total test time.
> 
> Note that if you are using kernel-ci and using a max loop goal of 100
> that means 500 days extra, so about 1.3 years extra total test time.
> If enabling soak duration you may want to then re-evaluate your loop
> target goal for kernel-ci for kdevops.

Yikes, I wouldn't combine multiple runs with large SOAK_DURATION. ;)

You all might consider kicking all the soak tests over to a separate VM
or VMs so that the long soak test do not hold up the rest of the run.

> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
> 
> Chandan, Amir, lemme know what you think of a default 2.5 hours default
> if soak duration is enabled. The only thing is the math indicates that
> if you are going to enable kernel-ci we won't finish this year.
> 
> To be clear, we've picked up testing with soak duration seriously for
> our LBS testing. It is why we've been able to find pretty hard to
> reproduce issues even on the page cache for the baseline [0], ie, without
> LBS. While folks have seemed to have found value in adopting 2.5 hours
> and of the results we have found, it obviously means a scaling issue
> to consider to decide when we're done with testing our baseline.
> 
> At first I wrote this patch just to provide basic guidance for kdevops,
> but after doing a bit of the math on how it also extends total test
> time, *with* our kernel-ci effort, it reveals clearly we should probably
> reconsider lowering the kernel-ci threshold a bit if adopting soak
> duration.
> 
> CC'ing a bit wider audience so to get a bit better idea of what folks
> might consider a sensible value for your own testing too. From what
> we've been observing, SOAK_DURATION allows us to catch bugs faster than
> just increasing the kernel-ci count, however, using both let's us catch
> even more bugs too.
> 
> To help *reduce* the amount of time to test we've deployed many kdevops
> XFS clusters to help test the baseline. This is why our count time on
> kernel-ci no is about 50-60 with a soak duration of about 2.5 hours.
> 
> Also please not that the reported bugs so far are the ones with crashes,
> there are other failures too, but we just haven't had the time to disect
> and report failures which are non-fatal (crashes) as crashes have been
> our priority.
> 
> [0] https://github.com/linux-kdevops/kdevops/blob/master/docs/xfs-bugs.md
> 
>  playbooks/roles/fstests/defaults/main.yml |  3 +
>  workflows/fstests/Kconfig                 | 89 ++++++++++++++++++++---
>  workflows/fstests/Makefile.sparsefiles    |  4 +
>  3 files changed, 87 insertions(+), 9 deletions(-)
> 
> diff --git a/playbooks/roles/fstests/defaults/main.yml b/playbooks/roles/fstests/defaults/main.yml
> index 2f70f9549cde..4a1f5dec5827 100644
> --- a/playbooks/roles/fstests/defaults/main.yml
> +++ b/playbooks/roles/fstests/defaults/main.yml
> @@ -30,6 +30,9 @@ fstests_test_logdev_mkfs_opts: "/dev/null"
>  fstests_test_dev_zns: "/dev/null"
>  fstests_zns_enabled: False
>  
> +fstests_soak_duration_enable: False
> +fstests_soak_duration: 0
> +
>  fstests_uses_no_devices: False
>  fstests_generate_simple_config_enable: False
>  fstests_generate_nvme_live_config_enable: False
> diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
> index 985a7847b6c7..bbd8927b3cd3 100644
> --- a/workflows/fstests/Kconfig
> +++ b/workflows/fstests/Kconfig
> @@ -760,15 +760,23 @@ config FSTESTS_RUN_LARGE_DISK_TESTS
>  	  to run. The "large disk" requirement is test dependent, but
>  	  typically, it means a disk with capacity of at several 10G.
>  
> -config FSTESTS_SOAK_DURATION
> -	int "Custom Soak duration to be used"
> -	default 0
> +config FSTESTS_ENABLE_SOAK_DURATION
> +	bool "Enable custom soak duration time"
>  	help
> -	  Custom Soak duration to be used during test execution. If you set this
> -	  to a non-zero value then fstests will increase the amount of time it
> -	  takes to run certain tests which are time based and support using
> -	  SOAK_DURATION. A moderate high value setting for this is 9900 which is
> -	  2.5 hours.
> +	  Enable soak duration to be used during test execution. If you are not
> +	  interested in extending your testing then leave this disabled.
> +
> +	  Using a custom soak duration to a non-zero value then fstests will
> +	  increase the amount of time it takes to run certain tests which are
> +	  time based and support using SOAK_DURATION. A moderate high value
> +	  setting for this is 9900 which is 2.5 hours.

"A moderately high setting for this is "2.5h" for 2.5 hours."

FWIW, the ./check parser translates floating point numbers with suffixes
to integer seconds; see soak_duration.awk.

The part I don't know is if kdevops merely passes through the value as a
string; or actually treats this as an integer.  If the latter, then
please ignore my comment.

> +
> +	  Note that we have 46 tests today which will be able to use soak
> +	  duration if set. This means your test time will increase by the
> +	  soak duration * these number of tests. When soak duration is
> +	  enabled the test specific watchdog fstests_watchdog.py will be
> +	  aware of tests which require soak duration and consider before
> +	  reporting a possible hang.
>  
>  	  As of 2023-10-31 that consists of the following tests which use either
>  	  fsstress or fsx or fio. Tests either use SOAK_DURATION directly or they
> @@ -786,7 +794,7 @@ config FSTESTS_SOAK_DURATION
>  	  - generic/648 - fsstress + disk failures on loopback
>  	  - generic/650 - fsstress - multithreaded write + CPU hotplug
>  
> -	  The tests below use _scratch_xfs_stress_scrub() to stress
> +	  All the tests below use _scratch_xfs_stress_scrub() to stress
>  	  test an with fsstress with scrub or an alternate xfs_db operation.
>  
>  	  - xfs/285
> @@ -825,4 +833,67 @@ config FSTESTS_SOAK_DURATION
>  	  - xfs/729
>  	  - xfs/800
>  
> +if FSTESTS_ENABLE_SOAK_DURATION
> +
> +choice
> +	prompt "Soak duration value to use"
> +	default FSTESTS_SOAK_DURATION_HIGH
> +
> +config FSTESTS_SOAK_DURATION_CUSTOM
> +	bool "Custom"
> +	help
> +	  You want to specify the value yourself.
> +
> +config FSTESTS_SOAK_DURATION_PATHALOGICAL

"PATHOLOGICAL", and yes that high a setting is pathological. ;)

(Unless you're allocating one soak-fstest per VM in which case "1w"
might be appropriate.)

> +	bool "High (48 hours)"
> +	help
> +	  Use 48 hours for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 92 days, or a bit over 3 months if run
> +	  serially.
> +
> +config FSTESTS_SOAK_DURATION_HIGH
> +	bool "High (2.5 hours)"
> +	help
> +	  Use 2.5 hours for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 5 days if run serially.
> +
> +config FSTESTS_SOAK_DURATION_MID
> +	bool "Mid (1 hour)"
> +	help
> +	  Use 1 hour for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 2 days if run serially.

I wonder, is there any way to scan the number of soak test to generate
these figures automatically at configure time?  I'd guess no, since
kdevops kconfig comes before pulling and compiling, right?

--D

> +
> +config FSTESTS_SOAK_DURATION_LOW
> +	bool "Low (30 minutes)
> +	help
> +	  Use 30 minutes for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 1 day if run serially.
> +
> +endchoice
> +
> +config FSTESTS_SOAK_DURATION_CUSTOM_VAL
> +	int "Custom soak duration value (seconds)"
> +	default 0
> +	depends on FSTESTS_SOAK_DURATION_CUSTOM
> +	help
> +	  Enter your custom soak duration value in seconds.
> +
> +endif # FSTESTS_ENABLE_SOAK_DURATION
> +
> +config FSTESTS_SOAK_DURATION
> +	default 0 if !FSTESTS_ENABLE_SOAK_DURATION
> +	default FSTESTS_SOAK_DURATION_CUSTOM_VAL if FSTESTS_SOAK_DURATION_CUSTOM
> +	default 1800 if FSTESTS_SOAK_DURATION_LOW
> +	default 3600 if FSTESTS_SOAK_DURATION_MID
> +	default 9900 if FSTESTS_SOAK_DURATION_HIGH
> +	default 172800 if FSTESTS_SOAK_DURATION_PATHALOGICAL
> +
>  endif # KDEVOPS_WORKFLOW_ENABLE_FSTESTS
> diff --git a/workflows/fstests/Makefile.sparsefiles b/workflows/fstests/Makefile.sparsefiles
> index c5ca20a9c462..7dd129c4f9cc 100644
> --- a/workflows/fstests/Makefile.sparsefiles
> +++ b/workflows/fstests/Makefile.sparsefiles
> @@ -44,6 +44,10 @@ FSTESTS_ARGS += run_large_disk_tests='$(FSTESTS_RUN_LARGE_DISK_TESTS)'
>  FSTESTS_ARGS += run_auto_group_tests='$(FSTESTS_RUN_AUTO_GROUP_TESTS)'
>  FSTESTS_ARGS += run_custom_group_tests='$(FSTESTS_RUN_CUSTOM_GROUP_TESTS)'
>  FSTESTS_ARGS += exclude_test_groups='$(CONFIG_FSTESTS_EXCLUDE_TEST_GROUPS)'
> +
> +ifeq (y,$(CONFIG_FSTESTS_ENABLE_SOAK_DURATION))
> +FSTESTS_ARGS += fstests_soak_duration_enable='True'
> +endif
>  FSTESTS_ARGS += fstests_soak_duration='$(CONFIG_FSTESTS_SOAK_DURATION)'
>  
>  ifeq (y,$(CONFIG_FSTESTS_ENABLE_RUN_CUSTOM_TESTS))
> -- 
> 2.42.0
> 
>
Matthew Wilcox Jan. 25, 2024, 11:24 p.m. UTC | #2
On Thu, Jan 25, 2024 at 02:10:37PM -0800, Luis Chamberlain wrote:
> The kdevops test runner has supported a custom SOAK_DURATION for
> fstests, however we were not providing any guidance. This means folks
> likely disable this. Throw a bone and provide some basic guidance and
> use 2.5 hours as the default value. There are about 46 tests today
> which use soak duration, this means if you are testing serially it
> increase total test time by about 5 days than the previously known
> total test time.

Do you have any stats on how much more effective it is to soak for 1
hour vs 5 hours vs 24 hours?  Presumably there's some kind of
exponential dropoff, like 20 bugs found at 1 hour, an extra five found
by four hours and three more at 24 hours?
Luis Chamberlain Jan. 29, 2024, 8:49 p.m. UTC | #3
On Thu, Jan 25, 2024 at 02:29:56PM -0800, Darrick J. Wong wrote:
> On Thu, Jan 25, 2024 at 02:10:37PM -0800, Luis Chamberlain wrote:
> > The kdevops test runner has supported a custom SOAK_DURATION for
> > fstests, however we were not providing any guidance. This means folks
> > likely disable this. Throw a bone and provide some basic guidance and
> > use 2.5 hours as the default value. There are about 46 tests today
> > which use soak duration, this means if you are testing serially it
> > increase total test time by about 5 days than the previously known
> > total test time.
> > 
> > Note that if you are using kernel-ci and using a max loop goal of 100
> > that means 500 days extra, so about 1.3 years extra total test time.
> > If enabling soak duration you may want to then re-evaluate your loop
> > target goal for kernel-ci for kdevops.
> 
> Yikes, I wouldn't combine multiple runs with large SOAK_DURATION. ;)

To be clear, the don't run in parallel, it runs serially. The paralellism
happens by having each test section run on a different guest.

> You all might consider kicking all the soak tests over to a separate VM
> or VMs so that the long soak test do not hold up the rest of the run.

Yes indeed, these would be nice optimizations. At hallway talks at Linux
Plumbers it was clear that given sparsefile tests *are* proving
successful in finding issues, we should in theory be able run tests
using k8, and so the resource problems is hanlded by your max system
capacity. Solarflare folks has some PoC k8 [0] tests going with fstests
and it showed how test time reduces considerably, they added that PoC
because they *don't* have enough resources to run many VMs, but
coincidently that solution should help us scale more efficiently.

[0] https://lkml.kernel.org/r/ZThCAv60FDcxoCfI@CMGLRV3

> > diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
> > index 985a7847b6c7..bbd8927b3cd3 100644
> > --- a/workflows/fstests/Kconfig
> > +++ b/workflows/fstests/Kconfig
> > @@ -760,15 +760,23 @@ config FSTESTS_RUN_LARGE_DISK_TESTS
> >  	  to run. The "large disk" requirement is test dependent, but
> >  	  typically, it means a disk with capacity of at several 10G.
> >  
> > -config FSTESTS_SOAK_DURATION
> > -	int "Custom Soak duration to be used"
> > -	default 0
> > +config FSTESTS_ENABLE_SOAK_DURATION
> > +	bool "Enable custom soak duration time"
> >  	help
> > -	  Custom Soak duration to be used during test execution. If you set this
> > -	  to a non-zero value then fstests will increase the amount of time it
> > -	  takes to run certain tests which are time based and support using
> > -	  SOAK_DURATION. A moderate high value setting for this is 9900 which is
> > -	  2.5 hours.
> > +	  Enable soak duration to be used during test execution. If you are not
> > +	  interested in extending your testing then leave this disabled.
> > +
> > +	  Using a custom soak duration to a non-zero value then fstests will
> > +	  increase the amount of time it takes to run certain tests which are
> > +	  time based and support using SOAK_DURATION. A moderate high value
> > +	  setting for this is 9900 which is 2.5 hours.
> 
> "A moderately high setting for this is "2.5h" for 2.5 hours."
> 
> FWIW, the ./check parser translates floating point numbers with suffixes
> to integer seconds; see soak_duration.awk.

Oh neat!

> The part I don't know is if kdevops merely passes through the value as a
> string; or actually treats this as an integer.  If the latter, then
> please ignore my comment.

No we use an int for now. The above magic let's us add a few boools
which represent a sensible range or a custom manual seconds setting.

> > +if FSTESTS_ENABLE_SOAK_DURATION
> > +
> > +choice
> > +	prompt "Soak duration value to use"
> > +	default FSTESTS_SOAK_DURATION_HIGH
> > +
> > +config FSTESTS_SOAK_DURATION_CUSTOM
> > +	bool "Custom"
> > +	help
> > +	  You want to specify the value yourself.
> > +
> > +config FSTESTS_SOAK_DURATION_PATHALOGICAL
> 
> "PATHOLOGICAL", and yes that high a setting is pathological. ;)
> 
> (Unless you're allocating one soak-fstest per VM in which case "1w"
> might be appropriate.)

I think we can certainly get there, but we need first the robustness to
trust each target node / guest / etc. I do suspect the k8 approach might
get us there faster with less resources.

> > +
> > +config FSTESTS_SOAK_DURATION_MID
> > +	bool "Mid (1 hour)"
> > +	help
> > +	  Use 1 hour for soak duration.
> > +
> > +	  Using this with 46 tests known to use soak duration means your test
> > +	  time will increase by about 2 days if run serially.
> 
> I wonder, is there any way to scan the number of soak test to generate
> these figures automatically at configure time? 

You read my mind here, indeed that's desirable but we have no way to do
that today. Also, to get my current set list I had to a series of git
greps, but at least the Kconfig and the kdevops fstests watchdog pyhon
script already has an array with the tests which do use soak duration.
Indeed a much better approach is to have a static annotation we add to
all tests which use it so that we can scrape these later and also use
special resources for them as needed.

> I'd guess no, since kdevops kconfig comes before pulling and compiling, right?

kdevops makes heavy use of scripts on Kconfig for some hard-to-get
values. It also supports 'make dynconfig' which let's us dynamically
generate Kconfig files based on your setup, which otherwise would
not be possible. We do this to support PCI-E hotplug onto guests
for example, so you just have to use the menuconfig interface to setup
PCI hotplug onto guests. Such target could be extended if we want later
to support more special information, such as scraping the fstests git
tree we're going to use for actual tests, etc. We just have no the need
for that yet. At this point I think its overkill but, if we do want to
get there, indeed, its possible.

  Luis
Luis Chamberlain Jan. 29, 2024, 8:57 p.m. UTC | #4
On Thu, Jan 25, 2024 at 11:24:58PM +0000, Matthew Wilcox wrote:
> On Thu, Jan 25, 2024 at 02:10:37PM -0800, Luis Chamberlain wrote:
> > The kdevops test runner has supported a custom SOAK_DURATION for
> > fstests, however we were not providing any guidance. This means folks
> > likely disable this. Throw a bone and provide some basic guidance and
> > use 2.5 hours as the default value. There are about 46 tests today
> > which use soak duration, this means if you are testing serially it
> > increase total test time by about 5 days than the previously known
> > total test time.
> 
> Do you have any stats on how much more effective it is to soak for 1
> hour vs 5 hours vs 24 hours?

Heh no, we just went for the high value of 2.5 hours for our LBS
testing. Note that the soaking is *per* test and hence the summation
of *all* 46 tests need to be considered. Unless of course we run 46
guests all running each respective soak test separately.

> Presumably there's some kind of
> exponential dropoff, like 20 bugs found at 1 hour, an extra five found
> by four hours and three more at 24 hours?

I suspect you might be right. We *may* be able to experiment more with
this once resource scaling can be done automatically for us, ie with
something like k8.

  Luis
diff mbox series

Patch

diff --git a/playbooks/roles/fstests/defaults/main.yml b/playbooks/roles/fstests/defaults/main.yml
index 2f70f9549cde..4a1f5dec5827 100644
--- a/playbooks/roles/fstests/defaults/main.yml
+++ b/playbooks/roles/fstests/defaults/main.yml
@@ -30,6 +30,9 @@  fstests_test_logdev_mkfs_opts: "/dev/null"
 fstests_test_dev_zns: "/dev/null"
 fstests_zns_enabled: False
 
+fstests_soak_duration_enable: False
+fstests_soak_duration: 0
+
 fstests_uses_no_devices: False
 fstests_generate_simple_config_enable: False
 fstests_generate_nvme_live_config_enable: False
diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
index 985a7847b6c7..bbd8927b3cd3 100644
--- a/workflows/fstests/Kconfig
+++ b/workflows/fstests/Kconfig
@@ -760,15 +760,23 @@  config FSTESTS_RUN_LARGE_DISK_TESTS
 	  to run. The "large disk" requirement is test dependent, but
 	  typically, it means a disk with capacity of at several 10G.
 
-config FSTESTS_SOAK_DURATION
-	int "Custom Soak duration to be used"
-	default 0
+config FSTESTS_ENABLE_SOAK_DURATION
+	bool "Enable custom soak duration time"
 	help
-	  Custom Soak duration to be used during test execution. If you set this
-	  to a non-zero value then fstests will increase the amount of time it
-	  takes to run certain tests which are time based and support using
-	  SOAK_DURATION. A moderate high value setting for this is 9900 which is
-	  2.5 hours.
+	  Enable soak duration to be used during test execution. If you are not
+	  interested in extending your testing then leave this disabled.
+
+	  Using a custom soak duration to a non-zero value then fstests will
+	  increase the amount of time it takes to run certain tests which are
+	  time based and support using SOAK_DURATION. A moderate high value
+	  setting for this is 9900 which is 2.5 hours.
+
+	  Note that we have 46 tests today which will be able to use soak
+	  duration if set. This means your test time will increase by the
+	  soak duration * these number of tests. When soak duration is
+	  enabled the test specific watchdog fstests_watchdog.py will be
+	  aware of tests which require soak duration and consider before
+	  reporting a possible hang.
 
 	  As of 2023-10-31 that consists of the following tests which use either
 	  fsstress or fsx or fio. Tests either use SOAK_DURATION directly or they
@@ -786,7 +794,7 @@  config FSTESTS_SOAK_DURATION
 	  - generic/648 - fsstress + disk failures on loopback
 	  - generic/650 - fsstress - multithreaded write + CPU hotplug
 
-	  The tests below use _scratch_xfs_stress_scrub() to stress
+	  All the tests below use _scratch_xfs_stress_scrub() to stress
 	  test an with fsstress with scrub or an alternate xfs_db operation.
 
 	  - xfs/285
@@ -825,4 +833,67 @@  config FSTESTS_SOAK_DURATION
 	  - xfs/729
 	  - xfs/800
 
+if FSTESTS_ENABLE_SOAK_DURATION
+
+choice
+	prompt "Soak duration value to use"
+	default FSTESTS_SOAK_DURATION_HIGH
+
+config FSTESTS_SOAK_DURATION_CUSTOM
+	bool "Custom"
+	help
+	  You want to specify the value yourself.
+
+config FSTESTS_SOAK_DURATION_PATHALOGICAL
+	bool "High (48 hours)"
+	help
+	  Use 48 hours for soak duration.
+
+	  Using this with 46 tests known to use soak duration means your test
+	  time will increase by about 92 days, or a bit over 3 months if run
+	  serially.
+
+config FSTESTS_SOAK_DURATION_HIGH
+	bool "High (2.5 hours)"
+	help
+	  Use 2.5 hours for soak duration.
+
+	  Using this with 46 tests known to use soak duration means your test
+	  time will increase by about 5 days if run serially.
+
+config FSTESTS_SOAK_DURATION_MID
+	bool "Mid (1 hour)"
+	help
+	  Use 1 hour for soak duration.
+
+	  Using this with 46 tests known to use soak duration means your test
+	  time will increase by about 2 days if run serially.
+
+config FSTESTS_SOAK_DURATION_LOW
+	bool "Low (30 minutes)
+	help
+	  Use 30 minutes for soak duration.
+
+	  Using this with 46 tests known to use soak duration means your test
+	  time will increase by about 1 day if run serially.
+
+endchoice
+
+config FSTESTS_SOAK_DURATION_CUSTOM_VAL
+	int "Custom soak duration value (seconds)"
+	default 0
+	depends on FSTESTS_SOAK_DURATION_CUSTOM
+	help
+	  Enter your custom soak duration value in seconds.
+
+endif # FSTESTS_ENABLE_SOAK_DURATION
+
+config FSTESTS_SOAK_DURATION
+	default 0 if !FSTESTS_ENABLE_SOAK_DURATION
+	default FSTESTS_SOAK_DURATION_CUSTOM_VAL if FSTESTS_SOAK_DURATION_CUSTOM
+	default 1800 if FSTESTS_SOAK_DURATION_LOW
+	default 3600 if FSTESTS_SOAK_DURATION_MID
+	default 9900 if FSTESTS_SOAK_DURATION_HIGH
+	default 172800 if FSTESTS_SOAK_DURATION_PATHALOGICAL
+
 endif # KDEVOPS_WORKFLOW_ENABLE_FSTESTS
diff --git a/workflows/fstests/Makefile.sparsefiles b/workflows/fstests/Makefile.sparsefiles
index c5ca20a9c462..7dd129c4f9cc 100644
--- a/workflows/fstests/Makefile.sparsefiles
+++ b/workflows/fstests/Makefile.sparsefiles
@@ -44,6 +44,10 @@  FSTESTS_ARGS += run_large_disk_tests='$(FSTESTS_RUN_LARGE_DISK_TESTS)'
 FSTESTS_ARGS += run_auto_group_tests='$(FSTESTS_RUN_AUTO_GROUP_TESTS)'
 FSTESTS_ARGS += run_custom_group_tests='$(FSTESTS_RUN_CUSTOM_GROUP_TESTS)'
 FSTESTS_ARGS += exclude_test_groups='$(CONFIG_FSTESTS_EXCLUDE_TEST_GROUPS)'
+
+ifeq (y,$(CONFIG_FSTESTS_ENABLE_SOAK_DURATION))
+FSTESTS_ARGS += fstests_soak_duration_enable='True'
+endif
 FSTESTS_ARGS += fstests_soak_duration='$(CONFIG_FSTESTS_SOAK_DURATION)'
 
 ifeq (y,$(CONFIG_FSTESTS_ENABLE_RUN_CUSTOM_TESTS))