xfstests flakiness in generic/095 triggered by fio 3.37

Commit Message

Theodore Ts'o April 15, 2024, 6:22 p.m. UTC

I was updating various components in my xfstests test appliance, and
it appears that fio 3.37 is causing generic/095 to become flaky.
Previously, I had been using fio 3.35 and a quick test shows that fio
3.36 seems to be OK.  So the change seems to be somethinig that was
introduced between fio 3.36 and 3.37.

The test failures seem to be caused by the following unexpected
message in the output file:

fio: first I/O failed. If /xt-vdc/file1 is a zoned block device, consider --zonemode=zbd

The following comment in tests/generic/095 suggests to my suspicious
mind that the above warning message isn't getting suppressed when
--ignore_error=,EIO is passed to fio:

  # There's a known EIO failure to report collisions between directio and buffered
  # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error
  # at here.

.... but I could be wrong about that.

For now, I've just switched to using fio 3.36, since there's nothing
in my testing that really _requires_ 3.37.  It might be that we should
just work around this by filtering out the warning message in
xfstests.  Or maybe we should make a change in fio?

						- Ted



Note: the changes between 4bb62098 and f9077fe2 in xfstests-bld do not
affect the test appliance.  This commits reflect some minor fixes to
xfstests-bld's regression test suite, and adding flex and
libsqlite3-dev to the build chroot, since that's needed by newer
versions of util-linux....

Comments

Jens Axboe April 15, 2024, 6:26 p.m. UTC | #1

On 4/15/24 12:22 PM, Theodore Ts'o wrote:
> I was updating various components in my xfstests test appliance, and
> it appears that fio 3.37 is causing generic/095 to become flaky.
> Previously, I had been using fio 3.35 and a quick test shows that fio
> 3.36 seems to be OK.  So the change seems to be somethinig that was
> introduced between fio 3.36 and 3.37.
> 
> The test failures seem to be caused by the following unexpected
> message in the output file:
> 
> fio: first I/O failed. If /xt-vdc/file1 is a zoned block device, consider --zonemode=zbd
> 
> The following comment in tests/generic/095 suggests to my suspicious
> mind that the above warning message isn't getting suppressed when
> --ignore_error=,EIO is passed to fio:
> 
>   # There's a known EIO failure to report collisions between directio and buffered
>   # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error
>   # at here.
> 
> .... but I could be wrong about that.
> 
> For now, I've just switched to using fio 3.36, since there's nothing
> in my testing that really _requires_ 3.37.  It might be that we should
> just work around this by filtering out the warning message in
> xfstests.  Or maybe we should make a change in fio?

I'll just revert that change, not sure how it slipped by that it also
change it from informational to an error message.

Patch

--- /tmp/b	2024-04-15 14:01:06.048592838 -0400
+++ /tmp/a	2024-04-15 14:01:02.180602551 -0400
@@ -1,18 +1,20 @@ 
-TESTRUNID: tytso-20240415133921
+TESTRUNID: tytso-20240415131932
 KERNEL:    kernel 6.9.0-rc3-xfstests #328 SMP PREEMPT_DYNAMIC Mon Apr 15 13:04:29 EDT 2024 x86_64
 CMDLINE:   -c ext4/4k,xfs/4k -C 20 generic/095
 CPUS:      2
 MEM:       7680
 
-ext4/4k: 20 tests, 86 seconds
-xfs/4k: 20 tests, 75 seconds
-Totals: 40 tests, 0 skipped, 0 failures, 0 errors, 161s
+ext4/4k: 20 tests, 8 failures, 86 seconds
+  Flaky: generic/095: 40% (8/20)
+xfs/4k: 20 tests, 7 failures, 71 seconds
+  Flaky: generic/095: 35% (7/20)
+Totals: 40 tests, 0 skipped, 15 failures, 0 errors, 157s
 
-FSTESTIMG: gce-xfstests/xfstests-amd64-202404151325
+FSTESTIMG: gce-xfstests/xfstests-amd64-202404122347
 FSTESTPRJ: gce-xfstests
 FSTESTVER: blktests 56d7e53 (Fri, 12 Apr 2024 16:03:20 +0900)
 FSTESTVER: e2fsprogs v1.47.0-144-g441741fc (Tue, 8 Aug 2023 16:08:52 -0400)
-FSTESTVER: fio  fio-3.36 (Fri, 20 Oct 2023 04:30:43 -0600)
+FSTESTVER: fio  fio-3.37 (Tue, 26 Mar 2024 15:13:51 -0600)
 FSTESTVER: fsverity v1.6 (Wed, 20 Mar 2024 21:21:46 -0700)
 FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500)
 FSTESTVER: libaio   libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200)
@@ -21,10 +23,10 @@ 
 FSTESTVER: quota  v4.05-56-g00534e7 (Mon, 29 Jan 2024 11:59:57 +0100)
 FSTESTVER: util-linux v2.40 (Wed, 27 Mar 2024 12:46:59 +0100)
 FSTESTVER: xfsprogs v6.6.0 (Mon, 5 Feb 2024 13:43:13 +0100)
-FSTESTVER: xfstests-bld f9077fe2 (Sat, 13 Apr 2024 16:34:23 -0400)
+FSTESTVER: xfstests-bld 4bb62098 12 Apr 2024 20:03:16 -0400)
 FSTESTVER: xfstests v2024.03.31-10-g554b613ae (Fri, 12 Apr 2024 22:06:49 -0400)
 FSTESTVER: zz_build-distro bookworm
 FSTESTCFG: ext4/4k xfs/4k
 FSTESTSET: generic/095
 FSTESTOPT: count 20 fail_loop_count 0 aex
-GCE ID:    7786599483943475363
+GCE ID:    6536244786506030920