From patchwork Tue Nov 5 14:56:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228031 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B849114E5 for ; Tue, 5 Nov 2019 14:56:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8D4C521D6C for ; Tue, 5 Nov 2019 14:56:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965793; bh=W5uVvSmMjCeg/kPsx/uOm38SeQSZeJOHRYzDWiGJKUE=; h=From:To:Cc:Subject:Date:List-ID:From; b=lMKBIp4/F7nKEMD4St2XI5xOHP7mUAocDCJPLGOmyRwLQcxDI4l9wc6b2NCqHz7mN rzlxl3nbmD3ajZv5GOUPbBv7d896oZy5Tps8IZvZkGbtSR7NpolEsnqHxd0ZZqNIfI 5gVLqZKxxiT090uXB687PEWiYm0msRbEr1jmaDVA= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389547AbfKEO4d (ORCPT ); Tue, 5 Nov 2019 09:56:33 -0500 Received: from mail.kernel.org ([198.145.29.99]:58578 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388635AbfKEO4d (ORCPT ); Tue, 5 Nov 2019 09:56:33 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AADB1217F5; Tue, 5 Nov 2019 14:56:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965792; bh=W5uVvSmMjCeg/kPsx/uOm38SeQSZeJOHRYzDWiGJKUE=; h=From:To:Cc:Subject:Date:From; b=iPFFZuPfhxj/APLOQUMeU2N4Srzy99Ake3mtCmEYsro8/1/vS6bNP7j/IQyb+q9xY ciwGv+01UJGbldCbgBh/Gg9rvFJfY2b6P9BBdURy+duqR1jG/TTH5fFKAB6ProTwo6 tvizmu2R7FA5/l8FwGlYlV6/6bbbiMs9/y/gyo2I= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 1/3] common: open files in ro mode for extent and hole count helpers Date: Tue, 5 Nov 2019 14:56:22 +0000 Message-Id: <20191105145622.11181-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Filipe Manana The helper functions _count_extents() and _count_holes() open their input file in RW mode to call fiemap, however opening it in RO mode is enough. By opening them in RW mode it also makes it not possible to use them against files residing in btrfs readonly snapshots for example. So just open the files in RO mode in these functions. Signed-off-by: Filipe Manana --- common/rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/common/rc b/common/rc index ef0c2509..238ffef9 100644 --- a/common/rc +++ b/common/rc @@ -3199,12 +3199,12 @@ _require_metadata_journaling() _count_extents() { - $XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l } _count_holes() { - $XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep hole | wc -l + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep hole | wc -l } _count_attr_extents() From patchwork Tue Nov 5 14:56:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228033 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC6AD14E5 for ; Tue, 5 Nov 2019 14:56:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CBCC221A4A for ; Tue, 5 Nov 2019 14:56:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965807; bh=iUobjqWyiOHpQFuEuNChvGI6bw32368NHyot+RqS4gE=; h=From:To:Cc:Subject:Date:List-ID:From; b=Wqw0+vAlAG5Po+gE0DrsJO50pEGXxrPR71tgotT7X/nHLdhWDUbLnh3bgfWbGIIik bFLdvMzMYpP0rGGfO7TEXd40nGm1XMG4IQtFTaU8qTGD1C94OZo9/rov3yRhJBKsA8 Cz/G9IrY/OCXr0+MsPUPSiPhISxyE32Mg+TrqQqE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389601AbfKEO4q (ORCPT ); Tue, 5 Nov 2019 09:56:46 -0500 Received: from mail.kernel.org ([198.145.29.99]:58648 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727889AbfKEO4q (ORCPT ); Tue, 5 Nov 2019 09:56:46 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F3EAD217F5; Tue, 5 Nov 2019 14:56:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965805; bh=iUobjqWyiOHpQFuEuNChvGI6bw32368NHyot+RqS4gE=; h=From:To:Cc:Subject:Date:From; b=BuqTCV1+UmdoY5Xt2Db1tDhx1/FAkYb3t+4ACXlm12sEvfQNa5Dhx79U+zQ39c3Rw 8Xfi560QftGAk74/FJkaNL33N+75VHHmGcdKeKYO40BRWNPJ/GGrCxCmaIyROLaO3U 8vO8xlz6+o7F26gEPAgsVSrlsIImPRDVUmXHlBRs= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 2/3] common: add helper to count number of exclusive extents in a file Date: Tue, 5 Nov 2019 14:56:40 +0000 Message-Id: <20191105145640.11231-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Filipe Manana Add a new helper that is similar to _count_extents() except that extents that are shared several times by the file with itself (reflinked at different file offsets) are accounted 1 time only, instead of N times. This is motivated by a subsequent test for btrfs that will use this new helper, to verify that send streams are issuing reflink operations. Signed-off-by: Filipe Manana --- common/rc | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/common/rc b/common/rc index 238ffef9..3c412178 100644 --- a/common/rc +++ b/common/rc @@ -3202,6 +3202,15 @@ _count_extents() $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l } +# Similar to _count_extents() but if any extent is shared multiples times in +# the file (reflinked to different file offsets), it is accounted as 1 extent +# instead of N extents. +_count_exclusive_extents() +{ + $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep -v hole | \ + cut -d ' ' -f 3 | sort | uniq | wc -l +} + _count_holes() { $XFS_IO_PROG -r -c "fiemap" $1 | tail -n +2 | grep hole | wc -l From patchwork Tue Nov 5 14:57:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 11228039 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCA2A1986 for ; Tue, 5 Nov 2019 14:57:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BB63821D7C for ; Tue, 5 Nov 2019 14:57:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965826; bh=XUeUBwRSU90NHuQYOTLWZCHE0RTyqgxJTHCTijCAmaI=; h=From:To:Cc:Subject:Date:List-ID:From; b=sDwBJyVMDzOH8+HZVubBvBzCx9fFuyHo6NmDWRE0sstovK/zXHwJFvk/DN1XHvxIP Tn+a1JkBV7v7JEZJGJtuIEzYb4nMZpbCg+ve+Ww7JdNC3C+QOhnm9tSATO6n4K95K/ WESZIShznP1h0SX3pNWE0OkZd1WfR+VnsK/HI3Zo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389622AbfKEO5G (ORCPT ); Tue, 5 Nov 2019 09:57:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:58744 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389183AbfKEO5G (ORCPT ); Tue, 5 Nov 2019 09:57:06 -0500 Received: from localhost.localdomain (bl8-197-74.dsl.telepac.pt [85.241.197.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EB690217F5; Tue, 5 Nov 2019 14:57:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572965824; bh=XUeUBwRSU90NHuQYOTLWZCHE0RTyqgxJTHCTijCAmaI=; h=From:To:Cc:Subject:Date:From; b=DeTCHwh5N5HO0JFPHo/rn0Dgm0B1PJOE4IWSHkT4wdkEJSTS7eLhAP15l3eXTs4ym 26dJgiPxe5R6KtGl1d9nnz4iWGyw5lUy3YHh6KaIcgoNcX7wX2IG8jzpdBLSsR8rjh aO5fZK/9OeixRPzbZRZlmZZuyprjGlotiMKMaYmM= From: fdmanana@kernel.org To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Filipe Manana Subject: [PATCH 3/3] btrfs: test that send can issue clone operations within the same file Date: Tue, 5 Nov 2019 14:57:00 +0000 Message-Id: <20191105145700.11281-1-fdmanana@kernel.org> X-Mailer: git-send-email 2.11.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Filipe Manana Verify that both full and incremental send operations can issue clone operations when a file has extents that are shared with itself (at different offsets of course). This currently fails on btrfs but is addressed by a patch for the kernel titled: "Btrfs: send, allow clone operations within the same file" Signed-off-by: Filipe Manana --- tests/btrfs/200 | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/200.out | 17 ++++++++ tests/btrfs/group | 1 + 3 files changed, 140 insertions(+) create mode 100755 tests/btrfs/200 create mode 100644 tests/btrfs/200.out diff --git a/tests/btrfs/200 b/tests/btrfs/200 new file mode 100755 index 00000000..8661f446 --- /dev/null +++ b/tests/btrfs/200 @@ -0,0 +1,122 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test No. 200 +# +# Check that send operations (full and incremental) are able to issue clone +# operations for extents that are shared between the same file. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + rm -fr $send_files_dir +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/reflink +. ./common/punch + +# real QA test starts here +_supported_fs btrfs +_supported_os Linux +_require_fssum +_require_test +_require_scratch_reflink +_require_xfs_io_command "fiemap" + +send_files_dir=$TEST_DIR/btrfs-test-$seq + +rm -f $seqres.full +rm -fr $send_files_dir +mkdir $send_files_dir + +_scratch_mkfs >>$seqres.full 2>&1 +_scratch_mount + +# Create our first test file, which has an extent that is shared only with +# itself and no other files. We want to verify a full send operation will +# clone the extent. +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ + | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ + | _filter_xfs_io + +# Create out second test file which initially, for the first send operation, +# only has a single extent that is not shared. +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ + | _filter_xfs_io + +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base 2>&1 \ + | _filter_scratch + +$BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ + | _filter_scratch + +# Now clone the existing extent in file bar to itself at a different offset. +# We want to verify the incremental send operation below will issue a clone +# operation instead of a write operation. +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ + | _filter_xfs_io + +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr 2>&1 \ + | _filter_scratch + +$BTRFS_UTIL_PROG send -p $SCRATCH_MNT/base -f $send_files_dir/2.snap \ + $SCRATCH_MNT/incr 2>&1 | _filter_scratch + +# Compute digests of the snapshot trees so that later we can compare against +# digests of the trees in the new filesystem, to see if they match (no data or +# metadata corruption happened). +$FSSUM_PROG -A -f -w $send_files_dir/base.fssum $SCRATCH_MNT/base +$FSSUM_PROG -A -f -w $send_files_dir/incr.fssum \ + -x $SCRATCH_MNT/incr/base $SCRATCH_MNT/incr + +# Now recreate the filesystem by receiving both send streams and verify we get +# the same file contents that the original filesystem had and that files foo +# and bar have shared extents. +_scratch_unmount +_scratch_mkfs >>$seqres.full 2>&1 +_scratch_mount + +$BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT +$BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT + +# Compute digests of the snapshot trees in the new filesystem and compare them +# to the ones in the original filesystem, they must match. +$FSSUM_PROG -r $send_files_dir/base.fssum $SCRATCH_MNT/base +$FSSUM_PROG -r $send_files_dir/incr.fssum $SCRATCH_MNT/incr + +num_extents=$(_count_extents $SCRATCH_MNT/base/foo) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/base/foo) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File foo does not have 2 shared extents in the base snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/base/foo +fi + +num_extents=$(_count_extents $SCRATCH_MNT/incr/foo) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/incr/foo) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File foo does not have 2 shared extents in the incr snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/incr/foo +fi + +num_extents=$(_count_extents $SCRATCH_MNT/incr/bar) +num_exclusive_extents=$(_count_exclusive_extents $SCRATCH_MNT/incr/bar) +if [ $num_extents -ne 2 ] || [ $num_exclusive_extents -ne 1 ]; then + echo "File bar does not have 2 shared extents in the incr snapshot" + $XFS_IO_PROG -r -c "fiemap" $SCRATCH_MNT/incr/bar +fi + +status=0 +exit diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out new file mode 100644 index 00000000..3eec567e --- /dev/null +++ b/tests/btrfs/200.out @@ -0,0 +1,17 @@ +QA output created by 200 +wrote 65536/65536 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +linked 65536/65536 bytes at offset 65536 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 65536/65536 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/base' +At subvol SCRATCH_MNT/base +linked 65536/65536 bytes at offset 65536 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/incr' +At subvol SCRATCH_MNT/incr +At subvol base +At snapshot incr +OK +OK diff --git a/tests/btrfs/group b/tests/btrfs/group index c7ab129e..d56dcafa 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -202,3 +202,4 @@ 197 auto quick volume 198 auto quick volume 199 auto quick trim +200 auto quick send clone