[2/2] tests/xfs: test for post umount readahead completion panic
diff mbox

Message ID 1467290974-13203-3-git-send-email-bfoster@redhat.com
State New
Headers show

Commit Message

Brian Foster June 30, 2016, 12:49 p.m. UTC
XFS has a bug where directory readahead completions can occur after
unmount. This can lead to a crash or panic because metadata read
verification attempts to access core XFS data structures (e.g., the log)
after they have been freed and certain pointers have been reset.

Add a test that triggers directory readahead, delays the readahead I/O
and immediately unmounts the filesystem. This test is part of the
dangerous group as it will cause kernels affected by the bug to crash.

Signed-off-by: Brian Foster <bfoster@redhat.com>
---
 tests/xfs/999     | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/999.out |  2 ++
 tests/xfs/group   |  1 +
 3 files changed, 93 insertions(+)
 create mode 100755 tests/xfs/999
 create mode 100644 tests/xfs/999.out

Comments

Eryu Guan June 30, 2016, 1:20 p.m. UTC | #1
On Thu, Jun 30, 2016 at 08:49:34AM -0400, Brian Foster wrote:
> XFS has a bug where directory readahead completions can occur after
> unmount. This can lead to a crash or panic because metadata read
> verification attempts to access core XFS data structures (e.g., the log)
> after they have been freed and certain pointers have been reset.
> 
> Add a test that triggers directory readahead, delays the readahead I/O
> and immediately unmounts the filesystem. This test is part of the
> dangerous group as it will cause kernels affected by the bug to crash.
> 
> Signed-off-by: Brian Foster <bfoster@redhat.com>

It crashed my 4.7-rc5 kernel quickly.

Reviewed-by: Eryu Guan <eguan@redhat.com>
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Dave Chinner July 1, 2016, 1:40 a.m. UTC | #2
On Thu, Jun 30, 2016 at 08:49:34AM -0400, Brian Foster wrote:
> XFS has a bug where directory readahead completions can occur after
> unmount. This can lead to a crash or panic because metadata read
> verification attempts to access core XFS data structures (e.g., the log)
> after they have been freed and certain pointers have been reset.
> 
> Add a test that triggers directory readahead, delays the readahead I/O
> and immediately unmounts the filesystem. This test is part of the
> dangerous group as it will cause kernels affected by the bug to crash.
.....
> +
> +# create a directory large enough for extent format
> +mkdir $SCRATCH_MNT/dir
> +for i in $(seq 0 999); do
> +	touch $SCRATCH_MNT/dir/$i
> +done

minor quibble - what's an "extent format" directory? I think you
mean a directory inode whose data fork is in extent or btree format,
not inline. i.e. not a short-form directory, but rather a block,
leaf or node format directory.

Yeah, I'm being pedantic, but we should use the correct terminology
so when someone reads it in 5 years time....

Cheers,

Dave.
Brian Foster July 1, 2016, 1:03 p.m. UTC | #3
On Fri, Jul 01, 2016 at 11:40:02AM +1000, Dave Chinner wrote:
> On Thu, Jun 30, 2016 at 08:49:34AM -0400, Brian Foster wrote:
> > XFS has a bug where directory readahead completions can occur after
> > unmount. This can lead to a crash or panic because metadata read
> > verification attempts to access core XFS data structures (e.g., the log)
> > after they have been freed and certain pointers have been reset.
> > 
> > Add a test that triggers directory readahead, delays the readahead I/O
> > and immediately unmounts the filesystem. This test is part of the
> > dangerous group as it will cause kernels affected by the bug to crash.
> .....
> > +
> > +# create a directory large enough for extent format
> > +mkdir $SCRATCH_MNT/dir
> > +for i in $(seq 0 999); do
> > +	touch $SCRATCH_MNT/dir/$i
> > +done
> 
> minor quibble - what's an "extent format" directory? I think you
> mean a directory inode whose data fork is in extent or btree format,
> not inline. i.e. not a short-form directory, but rather a block,
> leaf or node format directory.
> 

Indeed. So would you prefer the comment refers to the data fork or
directory format? This really cares more about whether the directory has
an extent count than the format per se, because that's what triggers the
readahead.

Perhaps something like: "insert entries to grow the directory to at
least one extent, which is what triggers readahead on dir open" ?

Eryu,

I'm assuming you can fix this up since you fixed up the previous patch.
Let me know if you want me to send another version.

Brian

> Yeah, I'm being pedantic, but we should use the correct terminology
> so when someone reads it in 5 years time....
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eryu Guan July 1, 2016, 1:37 p.m. UTC | #4
On Fri, Jul 01, 2016 at 09:03:09AM -0400, Brian Foster wrote:
> On Fri, Jul 01, 2016 at 11:40:02AM +1000, Dave Chinner wrote:
> > On Thu, Jun 30, 2016 at 08:49:34AM -0400, Brian Foster wrote:
> > > XFS has a bug where directory readahead completions can occur after
> > > unmount. This can lead to a crash or panic because metadata read
> > > verification attempts to access core XFS data structures (e.g., the log)
> > > after they have been freed and certain pointers have been reset.
> > > 
> > > Add a test that triggers directory readahead, delays the readahead I/O
> > > and immediately unmounts the filesystem. This test is part of the
> > > dangerous group as it will cause kernels affected by the bug to crash.
> > .....
> > > +
> > > +# create a directory large enough for extent format
> > > +mkdir $SCRATCH_MNT/dir
> > > +for i in $(seq 0 999); do
> > > +	touch $SCRATCH_MNT/dir/$i
> > > +done
> > 
> > minor quibble - what's an "extent format" directory? I think you
> > mean a directory inode whose data fork is in extent or btree format,
> > not inline. i.e. not a short-form directory, but rather a block,
> > leaf or node format directory.
> > 
> 
> Indeed. So would you prefer the comment refers to the data fork or
> directory format? This really cares more about whether the directory has
> an extent count than the format per se, because that's what triggers the
> readahead.
> 
> Perhaps something like: "insert entries to grow the directory to at
> least one extent, which is what triggers readahead on dir open" ?
> 
> Eryu,
> 
> I'm assuming you can fix this up since you fixed up the previous patch.
> Let me know if you want me to send another version.

Sure, I can fix it. And I fixed the mount option issue Dave pointed out
in the dm-delay patch.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch
diff mbox

diff --git a/tests/xfs/999 b/tests/xfs/999
new file mode 100755
index 0000000..041d719
--- /dev/null
+++ b/tests/xfs/999
@@ -0,0 +1,90 @@ 
+#! /bin/bash
+# FS QA Test No. 999
+#
+# Test to reproduce an XFS unmount crash due to races with directory readahead.
+# XFS had a bug in which unmount would proceed with a readahead I/O in flight.
+# If the unmount deconstructed the log by the time I/O completion occurs,
+# certain metadata read verifier checks could access invalid memory and cause a
+# panic.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	_scratch_unmount > /dev/null 2>&1
+	_cleanup_delay > /dev/null 2>&1
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/dmdelay
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+
+_require_scratch
+_require_dm_target delay
+
+rm -f $seqres.full
+
+echo "Silence is golden."
+
+_scratch_mkfs_xfs >> $seqres.full 2>&1
+
+_init_delay
+_mount_delay
+
+# create a directory large enough for extent format
+mkdir $SCRATCH_MNT/dir
+for i in $(seq 0 999); do
+	touch $SCRATCH_MNT/dir/$i
+done
+
+# remount to clear the buffer cache
+_unmount_delay
+_mount_delay
+
+# introduce a read I/O delay
+_load_delay_table $DELAY_READ
+
+# Map the directory and immediately unmount. This should invoke an asynchronous
+# readahead on the first block of the directory. The readahead is delayed by
+# dm-delay. If the unmount doesn't properly wait for the readahead completion,
+# the read verifier can run after core data structures have been freed and lead
+# to a crash/panic.
+$XFS_IO_PROG -c "bmap -v" $SCRATCH_MNT/dir >> $seqres.full 2>&1
+_unmount_delay
+
+_cleanup_delay
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/999.out b/tests/xfs/999.out
new file mode 100644
index 0000000..d254382
--- /dev/null
+++ b/tests/xfs/999.out
@@ -0,0 +1,2 @@ 
+QA output created by 999
+Silence is golden.
diff --git a/tests/xfs/group b/tests/xfs/group
index 5a35a76..6752cc0 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -290,3 +290,4 @@ 
 308 auto quick clone
 309 auto clone
 310 auto clone rmap
+999 auto dangerous quick