diff mbox

[v2,1/4] generic/377: Add copy to new file test

Message ID 20160909062136.GY27776@eguan.usersys.redhat.com (mailing list archive)
State New, archived
Headers show

Commit Message

Eryu Guan Sept. 9, 2016, 6:21 a.m. UTC 
On Wed, Sep 07, 2016 at 03:56:16PM -0400, Anna Schumaker wrote:
> This test copies data from various points in a source file to a new
> file.  This is useful for testing the basics of copy_file_range().
> 
> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> ---
>  tests/generic/377     | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/377.out |  21 +++++++++++
>  tests/generic/group   |   1 +
>  3 files changed, 124 insertions(+)
>  create mode 100644 tests/generic/377

File mode of test case should be 755, i.e. it has x permission. So "git
diff" won't complain the mode change after running these tests.

>  create mode 100644 tests/generic/377.out
> 
> diff --git a/tests/generic/377 b/tests/generic/377
> new file mode 100644
> index 0000000..352b9bb
> --- /dev/null
> +++ b/tests/generic/377
> @@ -0,0 +1,102 @@
> +#!/bin/bash
> +# FS QA Test No. 377
> +#
> +# Tests vfs_copy_file_range():
> +#   - Copy a file
> +#   - Copy beginning of original to new file
> +#   - Copy middle of original to a new file
> +#   - Copy end of original to new file
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2016 Netapp, Inc. All rights reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1    # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +}
> +
> +# get standard environment
> +. common/rc
> +. common/filter
> +
> +# real QA test starts here
> +_supported_os Linux

Need "_supported_fs generic" in all these tests.

> +
> +_require_xfs_io_command "copy_range"

Do we need to test the support status on kernel side? e.g. what happens
if filesystems have no "copy_file_range" implemented? Seems
copy_file_range falls back to splice in this case, but I'm not sure. If
so I think it's OK to have no kernel side detection.

> +_require_test
> +
> +testdir=$TEST_DIR/test-$seq
> +rm -rf $testdir
> +mkdir $testdir
> +
> +_checksum_files()

We usually name local functions without the leading "_".

> +{
> +	for f in $*; do
> +		md5sum $testdir/$f | _filter_test_dir
> +	done
> +}
> +
> +rm -f $seqres.full
> +
> +echo "Create the original file and then copy"
> +$XFS_IO_PROG -f -c 'pwrite -S 0x61 0    1000' $testdir/file >> $seqres.full 2>&1
> +$XFS_IO_PROG -f -c 'pwrite -S 0x62 1000 1000' $testdir/file >> $seqres.full 2>&1
> +$XFS_IO_PROG -f -c 'pwrite -S 0x63 2000 1000' $testdir/file >> $seqres.full 2>&1
> +$XFS_IO_PROG -f -c 'pwrite -S 0x64 3000 1000' $testdir/file >> $seqres.full 2>&1
> +$XFS_IO_PROG -f -c 'pwrite -S 0x65 4000 1000' $testdir/file >> $seqres.full 2>&1
> +$XFS_IO_PROG -f -c "copy_range $testdir/file" "$testdir/copy"
> +cmp $testdir/file $testdir/copy
> +echo "Original md5sums:"
> +_checksum_files file copy

I saw test failures when testing on XFS and btrfs, 4.8-rc3 kernel, and
similar failures in other tests. Are these failures expected?

[root@dhcp-66-86-11 xfstests]# diff -u tests/generic/377.out /root/workspace/xfstests/results//xfs_4k_crc/generic/377.out.bad

> +
> +echo "Copy beginning of original file"
> +$XFS_IO_PROG -f -c "copy_range -l 1000 $testdir/file" "$testdir/beginning"
> +cmp -n 1000 $testdir/file $testdir/beginning
> +echo "md5sums after copying beginning:"
> +_checksum_files file beginning
> +
> +echo "Copy middle of original file"
> +$XFS_IO_PROG -f -c "copy_range -s 1000 -l 3000 $testdir/file" "$testdir/middle"
> +cmp -n 3000 $testdir/file $testdir/middle 1000
> +echo "md5sums after copying middle:"
> +_checksum_files file middle
> +
> +echo "Copy end of original file"
> +$XFS_IO_PROG -f -c "copy_range -s 4000 -l 1000 $testdir/file" "$testdir/end"
> +cmp -n 1000 $testdir/file $testdir/end 4000
> +echo "md5sums after copying end:"
> +_checksum_files file end
> +
> +echo "Copy beyond end of original file"
> +$XFS_IO_PROG -f -c "copy_range -s 4000 -l 2000 $testdir/file" "$testdir/beyond"

I noticed that xfs_io hung on copying beyond EOF when testing with ext4
and NFSv4.1, xfs_io was in "R" state for 10 minutes until I killed it.
Is it kernel bug or test is not suitable on such filesystems?

Thanks,
Eryu

> +cmp -n 1000 $testdir/file $testdir/end 4000
> +echo "md5sums after copying beyond:"
> +_checksum_files file beyond
> +
> +#success, all done
> +status=0
> +exit
> diff --git a/tests/generic/377.out b/tests/generic/377.out
> new file mode 100644
> index 0000000..8ad8b53
> --- /dev/null
> +++ b/tests/generic/377.out
> @@ -0,0 +1,21 @@
> +QA output created by 377
> +Create the original file and then copy
> +Original md5sums:
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/copy
> +Copy beginning of original file
> +md5sums after copying beginning:
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> +cabe45dcc9ae5b66ba86600cca6b8ba8  TEST_DIR/test-377/beginning
> +Copy middle of original file
> +md5sums after copying middle:
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> +4197de9da5badfc302715486b82bcdf1  TEST_DIR/test-377/middle
> +Copy end of original file
> +md5sums after copying end:
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> +e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/end
> +Copy beyond end of original file
> +md5sums after copying beyond:
> +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> +e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/beyond
> diff --git a/tests/generic/group b/tests/generic/group
> index bad71bc..dec0d64 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -379,3 +379,4 @@
>  374 auto quick clone dedupe
>  375 auto quick acl
>  376 auto quick metadata
> +377 auto quick copy
> -- 
> 2.9.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Darrick J. Wong Sept. 9, 2016, 6:29 a.m. UTC | #1 
On Fri, Sep 09, 2016 at 02:21:36PM +0800, Eryu Guan wrote:
> On Wed, Sep 07, 2016 at 03:56:16PM -0400, Anna Schumaker wrote:
> > This test copies data from various points in a source file to a new
> > file.  This is useful for testing the basics of copy_file_range().
> > 
> > Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
> > ---
> >  tests/generic/377     | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/377.out |  21 +++++++++++
> >  tests/generic/group   |   1 +
> >  3 files changed, 124 insertions(+)
> >  create mode 100644 tests/generic/377
> 
> File mode of test case should be 755, i.e. it has x permission. So "git
> diff" won't complain the mode change after running these tests.
> 
> >  create mode 100644 tests/generic/377.out
> > 
> > diff --git a/tests/generic/377 b/tests/generic/377
> > new file mode 100644
> > index 0000000..352b9bb
> > --- /dev/null
> > +++ b/tests/generic/377
> > @@ -0,0 +1,102 @@
> > +#!/bin/bash
> > +# FS QA Test No. 377
> > +#
> > +# Tests vfs_copy_file_range():
> > +#   - Copy a file
> > +#   - Copy beginning of original to new file
> > +#   - Copy middle of original to a new file
> > +#   - Copy end of original to new file
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2016 Netapp, Inc. All rights reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1    # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.*
> > +}
> > +
> > +# get standard environment
> > +. common/rc
> > +. common/filter
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> 
> Need "_supported_fs generic" in all these tests.
> 
> > +
> > +_require_xfs_io_command "copy_range"
> 
> Do we need to test the support status on kernel side? e.g. what happens
> if filesystems have no "copy_file_range" implemented? Seems
> copy_file_range falls back to splice in this case, but I'm not sure. If
> so I think it's OK to have no kernel side detection.

But... it's a totally new syscall, so _require_io_command should actually try
calling it so that we can _notrun on old kernels.

> > +_require_test
> > +
> > +testdir=$TEST_DIR/test-$seq
> > +rm -rf $testdir
> > +mkdir $testdir
> > +
> > +_checksum_files()
> 
> We usually name local functions without the leading "_".
> 
> > +{
> > +	for f in $*; do
> > +		md5sum $testdir/$f | _filter_test_dir
> > +	done
> > +}

Or perhaps just "md5sum file copy | _filter_test_dir" directly?

--D

> > +
> > +rm -f $seqres.full
> > +
> > +echo "Create the original file and then copy"
> > +$XFS_IO_PROG -f -c 'pwrite -S 0x61 0    1000' $testdir/file >> $seqres.full 2>&1
> > +$XFS_IO_PROG -f -c 'pwrite -S 0x62 1000 1000' $testdir/file >> $seqres.full 2>&1
> > +$XFS_IO_PROG -f -c 'pwrite -S 0x63 2000 1000' $testdir/file >> $seqres.full 2>&1
> > +$XFS_IO_PROG -f -c 'pwrite -S 0x64 3000 1000' $testdir/file >> $seqres.full 2>&1
> > +$XFS_IO_PROG -f -c 'pwrite -S 0x65 4000 1000' $testdir/file >> $seqres.full 2>&1
> > +$XFS_IO_PROG -f -c "copy_range $testdir/file" "$testdir/copy"
> > +cmp $testdir/file $testdir/copy
> > +echo "Original md5sums:"
> > +_checksum_files file copy
> 
> I saw test failures when testing on XFS and btrfs, 4.8-rc3 kernel, and
> similar failures in other tests. Are these failures expected?
> 
> [root@dhcp-66-86-11 xfstests]# diff -u tests/generic/377.out /root/workspace/xfstests/results//xfs_4k_crc/generic/377.out.bad
> --- tests/generic/377.out       2016-09-09 12:17:40.790000000 +0800
> +++ /root/workspace/xfstests/results//xfs_4k_crc/generic/377.out.bad    2016-09-09 14:14:48.483000000 +0800
> @@ -4,18 +4,22 @@
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/copy
>  Copy beginning of original file
> +cmp: EOF on /mnt/testarea/test/test-377/beginning
>  md5sums after copying beginning:
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> -cabe45dcc9ae5b66ba86600cca6b8ba8  TEST_DIR/test-377/beginning
> +d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/beginning
>  Copy middle of original file
> +cmp: EOF on /mnt/testarea/test/test-377/middle
>  md5sums after copying middle:
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> -4197de9da5badfc302715486b82bcdf1  TEST_DIR/test-377/middle
> +d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/middle
>  Copy end of original file
> +cmp: EOF on /mnt/testarea/test/test-377/end
>  md5sums after copying end:
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> -e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/end
> +d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/end
>  Copy beyond end of original file
> +cmp: EOF on /mnt/testarea/test/test-377/end
>  md5sums after copying beyond:
>  e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> -e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/beyond
> +d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/beyond
> 
> > +
> > +echo "Copy beginning of original file"
> > +$XFS_IO_PROG -f -c "copy_range -l 1000 $testdir/file" "$testdir/beginning"
> > +cmp -n 1000 $testdir/file $testdir/beginning
> > +echo "md5sums after copying beginning:"
> > +_checksum_files file beginning
> > +
> > +echo "Copy middle of original file"
> > +$XFS_IO_PROG -f -c "copy_range -s 1000 -l 3000 $testdir/file" "$testdir/middle"
> > +cmp -n 3000 $testdir/file $testdir/middle 1000
> > +echo "md5sums after copying middle:"
> > +_checksum_files file middle
> > +
> > +echo "Copy end of original file"
> > +$XFS_IO_PROG -f -c "copy_range -s 4000 -l 1000 $testdir/file" "$testdir/end"
> > +cmp -n 1000 $testdir/file $testdir/end 4000
> > +echo "md5sums after copying end:"
> > +_checksum_files file end
> > +
> > +echo "Copy beyond end of original file"
> > +$XFS_IO_PROG -f -c "copy_range -s 4000 -l 2000 $testdir/file" "$testdir/beyond"
> 
> I noticed that xfs_io hung on copying beyond EOF when testing with ext4
> and NFSv4.1, xfs_io was in "R" state for 10 minutes until I killed it.
> Is it kernel bug or test is not suitable on such filesystems?
> 
> Thanks,
> Eryu
> 
> > +cmp -n 1000 $testdir/file $testdir/end 4000
> > +echo "md5sums after copying beyond:"
> > +_checksum_files file beyond
> > +
> > +#success, all done
> > +status=0
> > +exit
> > diff --git a/tests/generic/377.out b/tests/generic/377.out
> > new file mode 100644
> > index 0000000..8ad8b53
> > --- /dev/null
> > +++ b/tests/generic/377.out
> > @@ -0,0 +1,21 @@
> > +QA output created by 377
> > +Create the original file and then copy
> > +Original md5sums:
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/copy
> > +Copy beginning of original file
> > +md5sums after copying beginning:
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> > +cabe45dcc9ae5b66ba86600cca6b8ba8  TEST_DIR/test-377/beginning
> > +Copy middle of original file
> > +md5sums after copying middle:
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> > +4197de9da5badfc302715486b82bcdf1  TEST_DIR/test-377/middle
> > +Copy end of original file
> > +md5sums after copying end:
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> > +e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/end
> > +Copy beyond end of original file
> > +md5sums after copying beyond:
> > +e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
> > +e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/beyond
> > diff --git a/tests/generic/group b/tests/generic/group
> > index bad71bc..dec0d64 100644
> > --- a/tests/generic/group
> > +++ b/tests/generic/group
> > @@ -379,3 +379,4 @@
> >  374 auto quick clone dedupe
> >  375 auto quick acl
> >  376 auto quick metadata
> > +377 auto quick copy
> > -- 
> > 2.9.3
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe fstests" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eryu Guan Sept. 9, 2016, 7:01 a.m. UTC | #2 
On Thu, Sep 08, 2016 at 11:29:25PM -0700, Darrick J. Wong wrote:
> > > +
> > > +_require_xfs_io_command "copy_range"
> > 
> > Do we need to test the support status on kernel side? e.g. what happens
> > if filesystems have no "copy_file_range" implemented? Seems
> > copy_file_range falls back to splice in this case, but I'm not sure. If
> > so I think it's OK to have no kernel side detection.
> 
> But... it's a totally new syscall, so _require_io_command should actually try
> calling it so that we can _notrun on old kernels.

The only reason that I think it's OK to check xfs_io support only is
that the "copy_range" subcommand won't be even compiled in xfs_io if
kernel has no copy_file_range syscall support.

But yeah, I agree that calling it and see how kernel handles it would be
the best option, like how we handle falloc, fpunch in
_require_xfs_io_command.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Dave Chinner Sept. 11, 2016, 11:10 p.m. UTC | #3 
On Fri, Sep 09, 2016 at 03:01:46PM +0800, Eryu Guan wrote:
> On Thu, Sep 08, 2016 at 11:29:25PM -0700, Darrick J. Wong wrote:
> > > > +
> > > > +_require_xfs_io_command "copy_range"
> > > 
> > > Do we need to test the support status on kernel side? e.g. what happens
> > > if filesystems have no "copy_file_range" implemented? Seems
> > > copy_file_range falls back to splice in this case, but I'm not sure. If
> > > so I think it's OK to have no kernel side detection.
> > 
> > But... it's a totally new syscall, so _require_io_command should actually try
> > calling it so that we can _notrun on old kernels.
> 
> The only reason that I think it's OK to check xfs_io support only is
> that the "copy_range" subcommand won't be even compiled in xfs_io if
> kernel has no copy_file_range syscall support.

It's not uncommon to have an xfs_io that will support a new syscall
or syscall flags by directly coding the support when it's
not found by the configure script. e.g. io/prealloc.c support for
all the different fallocate flags, regardless of whether the
underlying kernel or userspace headers support them.

> But yeah, I agree that calling it and see how kernel handles it would be
> the best option, like how we handle falloc, fpunch in
> _require_xfs_io_command.

Which we do for precisely the reason above. :P

Cheers,

Dave.
diff mbox

Patch

--- tests/generic/377.out       2016-09-09 12:17:40.790000000 +0800
+++ /root/workspace/xfstests/results//xfs_4k_crc/generic/377.out.bad    2016-09-09 14:14:48.483000000 +0800
@@ -4,18 +4,22 @@ 
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/copy
 Copy beginning of original file
+cmp: EOF on /mnt/testarea/test/test-377/beginning
 md5sums after copying beginning:
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
-cabe45dcc9ae5b66ba86600cca6b8ba8  TEST_DIR/test-377/beginning
+d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/beginning
 Copy middle of original file
+cmp: EOF on /mnt/testarea/test/test-377/middle
 md5sums after copying middle:
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
-4197de9da5badfc302715486b82bcdf1  TEST_DIR/test-377/middle
+d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/middle
 Copy end of original file
+cmp: EOF on /mnt/testarea/test/test-377/end
 md5sums after copying end:
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
-e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/end
+d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/end
 Copy beyond end of original file
+cmp: EOF on /mnt/testarea/test/test-377/end
 md5sums after copying beyond:
 e11fbace556cba26bf0076e74cab90a3  TEST_DIR/test-377/file
-e68d4a150c4e42f4f9ea3ffe4c9cf4ed  TEST_DIR/test-377/beyond
+d41d8cd98f00b204e9800998ecf8427e  TEST_DIR/test-377/beyond