From patchwork Fri Jan 28 00:26:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 12727641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0843DC433F5 for ; Fri, 28 Jan 2022 00:28:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344927AbiA1A2D (ORCPT ); Thu, 27 Jan 2022 19:28:03 -0500 Received: from smtp-out2.suse.de ([195.135.220.29]:54122 "EHLO smtp-out2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344607AbiA1A1V (ORCPT ); Thu, 27 Jan 2022 19:27:21 -0500 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 091501F391; Fri, 28 Jan 2022 00:27:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1643329640; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=/41dAJoPSqnWF9niJ62j++xsUj7wWhJgLVNmJY2sFn8=; b=kiONqW8nKszDHeNn6YHkxcF0TnaqAQoJULtWlKBjGPMFdFaIMuqKS5SIJIwzzbhznsCf92 IkLWBapFMGGsZ8tXly2MdiyPM8Ayii5KKB34BVCR6DxbIcLRflkkh4YWbJj3HuNI4i+ooZ sBhHjmG3S8Yd6/WvnqO5cSmuaq/ycvA= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3037C139F7; Fri, 28 Jan 2022 00:27:18 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 6UQeO2Y482FQQAAAMHmgww (envelope-from ); Fri, 28 Jan 2022 00:27:18 +0000 From: Qu Wenruo To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org Subject: [PATCH v3 1/3] btrfs: test defrag with regular and preallocated extents Date: Fri, 28 Jan 2022 08:26:59 +0800 Message-Id: <20220128002701.11971-1-wqu@suse.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org Recent v5.16 has some regression around btrfs autodefrag mount option, and the extra scrutiny around defrag code exposes some questionable behavior from the old code. One behavior is to defrag extents along with the next preallocated extent. This behavior will cause extra IO and convert all the preallocated extent to regular zero filled extents, rendering the preallocated extent useless. The kernel fix is titled: btrfs: defrag: don't try to merge regular extents with preallocated extents Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- Changelog: v2: - Add _require_btrfs_support_sectorsize() helper And use it to make sure the platform supports 4k sectorsize - Use $AWK_PROG to replace awk v3: - Move _get_file_extent_sector() into common/rc --- common/btrfs | 16 ++++++++++ common/rc | 18 +++++++++++ tests/btrfs/255 | 75 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/255.out | 2 ++ 4 files changed, 111 insertions(+) create mode 100755 tests/btrfs/255 create mode 100644 tests/btrfs/255.out diff --git a/common/btrfs b/common/btrfs index 4afe81eb..5de926dd 100644 --- a/common/btrfs +++ b/common/btrfs @@ -480,3 +480,19 @@ _btrfs_no_v1_cache_opt() fi echo -n "-onospace_cache" } + +# Require certain sectorsize support +_require_btrfs_support_sectorsize() +{ + local sectorsize=$1 + + # PAGE_SIZE as sectorsize is always supported + if [ $sectorsize -eq $(get_page_size) ]; then + return + fi + + test -f /sys/fs/btrfs/features/supported_sectorsizes || \ + _notrun "no subpage support found" + grep -wq $sectorsize /sys/fs/btrfs/features/supported_sectorsizes || \ + _notrun "sectorsize $sectorsize is not supported" +} diff --git a/common/rc b/common/rc index b3289de9..8fbb32f8 100644 --- a/common/rc +++ b/common/rc @@ -3767,6 +3767,24 @@ _count_attr_extents() $XFS_IO_PROG -c "fiemap -a" $1 | tail -n +2 | grep -v hole | wc -l } +# Get the sector number of the extent at @offset of @file +_get_file_extent_sector() +{ + local file=$1 + local offset=$2 + local result + + result=$($XFS_IO_PROG -c "fiemap $offset" "$file" | \ + _filter_xfs_io_fiemap | head -n1 | $AWK_PROG '{print $3}') + + # xfs_io fiemap will output nothing if there is only hole, so here + # to replace the empty string with "hole" instead + if [ -z "$result" ]; then + result="hole" + fi + echo "$result" +} + # arg 1 is dev to remove and is output of the below eg. # ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev _devmgt_remove() diff --git a/tests/btrfs/255 b/tests/btrfs/255 new file mode 100755 index 00000000..fb80359c --- /dev/null +++ b/tests/btrfs/255 @@ -0,0 +1,75 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 255 +# +# Make sure btrfs doesn't defrag preallocated extents, nor lone extents +# before preallocated extents. +# + +. ./common/preamble +_begin_fstest auto quick defrag + +# Override the default cleanup function. +# _cleanup() +# { +# cd / +# rm -r -f $tmp.* +# } + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch + +# Needs 4K sectorsize +_require_btrfs_support_sectorsize 4096 + +_scratch_mkfs -s 4k >> $seqres.full 2>&1 + +# Need datacow to make the defragged extents to have different bytenr +_scratch_mount -o datacow + +# Create a file with the following layout: +# 0 4K 8K 16K +# |<- R ->|<-- Preallocated ->| +# R is regular extents. +# +# In this case it makes no sense to defrag any extent. +$XFS_IO_PROG -f -c "pwrite 0 4k" -c sync -c "falloc 4k 12k" \ + "$SCRATCH_MNT/foobar" >> $seqres.full + +echo "=== Initial file extent layout ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full + +# Save the bytenr of both extents +old_regular=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 0) +old_prealloc=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 4096) + +# Now defrag and write the defragged range back to disk +$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/foobar" >> $seqres.full +sync + +echo "=== File extent layout after defrag ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full + +new_regular=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 0) +new_prealloc=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 4096) + +if [ "$old_regular" -ne "$new_regular" ]; then + echo "the single lone sector get defragged" +fi +if [ "$old_prealloc" -ne "$new_prealloc" ]; then + echo "the preallocated extent get defragged" +fi + +echo "Silence is golden" + +# success, all done +status=0 +exit diff --git a/tests/btrfs/255.out b/tests/btrfs/255.out new file mode 100644 index 00000000..7eefb828 --- /dev/null +++ b/tests/btrfs/255.out @@ -0,0 +1,2 @@ +QA output created by 255 +Silence is golden From patchwork Fri Jan 28 00:27:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 12727642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EB61C433FE for ; Fri, 28 Jan 2022 00:28:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344946AbiA1A2G (ORCPT ); Thu, 27 Jan 2022 19:28:06 -0500 Received: from smtp-out2.suse.de ([195.135.220.29]:54128 "EHLO smtp-out2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344985AbiA1A1W (ORCPT ); Thu, 27 Jan 2022 19:27:22 -0500 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4344D1F385; Fri, 28 Jan 2022 00:27:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1643329641; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=10Ij3d7KYP62gCYiX+SIsQd2/6yMD/1U85EwuNMzEds=; b=VEDr1glWq6afTqVVhUf1eQH52COyL3pHEuQu8DTCC3T8wNmjp8HuRDy1xaEzNWBJH/gczQ ok5mRuSURBFaehFmRcXdLF4Q9dJMU7DZqyDbCH8f6upetDtE9rfNj8dBFxnmmDvkHxUOB4 lv5cv1B0WfPIB/hEamVKQqXL9YhYU7k= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 67A0C139F7; Fri, 28 Jan 2022 00:27:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id mBw0DWg482FQQAAAMHmgww (envelope-from ); Fri, 28 Jan 2022 00:27:20 +0000 From: Qu Wenruo To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org Subject: [PATCH v2 2/3] btrfs: test autodefrag with regular and hole extents Date: Fri, 28 Jan 2022 08:27:00 +0800 Message-Id: <20220128002701.11971-2-wqu@suse.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220128002701.11971-1-wqu@suse.com> References: <20220128002701.11971-1-wqu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org In v5.11~v5.15 kernels, there is a regression in autodefrag that if a cluster (up to 256K in size) has even a single hole, the whole cluster will be rejected. This will greatly reduce the efficiency of autodefrag. The behavior is fixed in v5.16 by a full rework, although the rework itself has other problems, it at least solves the problem. Here we add a test case to reproduce the case, where we have a 128K cluster, the first half is fragmented extents which can be defragged. The second half is hole. Make sure autodefrag can defrag the 64K part. Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- Changelog: v2: - Use the previously define _get_file_extent_sector() helper This also removed some out-of-sync error messages - Trigger autodefrag using commit=1 mount option No need for special purpose patch any more. - Use xfs_io -s to skip several sync calls - Shorten the subject of the commit --- tests/btrfs/256 | 80 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/256.out | 2 ++ 2 files changed, 82 insertions(+) create mode 100755 tests/btrfs/256 create mode 100644 tests/btrfs/256.out diff --git a/tests/btrfs/256 b/tests/btrfs/256 new file mode 100755 index 00000000..def83a15 --- /dev/null +++ b/tests/btrfs/256 @@ -0,0 +1,80 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 256 +# +# Make sure btrfs auto defrag can properly defrag clusters which has hole +# in the middle +# +. ./common/preamble +_begin_fstest auto defrag quick + +. ./common/btrfs +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs generic +_require_scratch + +# Needs 4K sectorsize, as larger sectorsize can change the file layout. +_require_btrfs_support_sectorsize 4096 + +_scratch_mkfs >> $seqres.full + +# Need datacow to show which range is defragged, and we're testing +# autodefrag +_scratch_mount -o datacow,autodefrag + +# Create a layout where we have fragmented extents at [0, 64k) (sync write in +# reserve order), then a hole at [64k, 128k) +$XFS_IO_PROG -f -s -c "pwrite 48k 16k" -c "pwrite 32k 16k" \ + -c "pwrite 16k 16k" -c "pwrite 0 16k" \ + $SCRATCH_MNT/foobar >> $seqres.full +truncate -s 128k $SCRATCH_MNT/foobar + +old_csum=$(_md5_checksum $SCRATCH_MNT/foobar) +echo "=== File extent layout before autodefrag ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full +echo "old md5=$old_csum" >> $seqres.full + +old_regular=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 0) +old_hole=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 64k) + +# Now trigger autodefrag, autodefrag is triggered in the cleaner thread, +# which will be woken up by commit thread +_scratch_remount commit=1 +sleep 3 +sync + +new_csum=$(_md5_checksum $SCRATCH_MNT/foobar) +new_regular=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 0) +new_hole=$(_get_file_extent_sector "$SCRATCH_MNT/foobar" 64k) + +echo "=== File extent layout after autodefrag ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full +echo "new md5=$new_csum" >> $seqres.full + +# In v5.11~v5.15 kernels, regular extents won't get defragged, and would trigger +# the following output +if [ $new_regular == $old_regular ]; then + echo "regular extents didn't get defragged" +fi + +# In v5.10 and earlier kernel, autodefrag may choose to defrag holes, +# which should be avoided. +if [ "$new_hole" != "$old_hole" ]; then + echo "hole extents got defragged" +fi + +# Defrag should not change file content +if [ "$new_csum" != "$old_csum" ]; then + echo "file content changed" +fi + +echo "Silence is golden" +# success, all done +status=0 +exit diff --git a/tests/btrfs/256.out b/tests/btrfs/256.out new file mode 100644 index 00000000..7ee8e2e5 --- /dev/null +++ b/tests/btrfs/256.out @@ -0,0 +1,2 @@ +QA output created by 256 +Silence is golden From patchwork Fri Jan 28 00:27:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 12727643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 316CCC433EF for ; Fri, 28 Jan 2022 00:28:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344609AbiA1A2K (ORCPT ); Thu, 27 Jan 2022 19:28:10 -0500 Received: from smtp-out1.suse.de ([195.135.220.28]:47264 "EHLO smtp-out1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344995AbiA1A1Y (ORCPT ); Thu, 27 Jan 2022 19:27:24 -0500 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7B783212CA; Fri, 28 Jan 2022 00:27:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1643329642; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+/MsAhRjM9KOl/KjjYKjfeiOO3x3t2jI0t9NmYtqz94=; b=Fur0Qd9PzdzeIfExhVP2nZ9l0EEVxkrlHZvlKB4zXayVZHcUV3PckO4eaE3kA/ntvIaT/i wWY7btxjmv6Oe/DFVUhuhoH1CrWvEELsLgJduq4YlPXR0vyq2ebBim5bR+gOCOGd5gvyJd 7SRrjIoVRNim8YNN8EVY+OG4OJjNBqM= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A0CF4139F7; Fri, 28 Jan 2022 00:27:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id yBgJG2k482FQQAAAMHmgww (envelope-from ); Fri, 28 Jan 2022 00:27:21 +0000 From: Qu Wenruo To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org Subject: [PATCH 3/3] btrfs: test defrag with compressed extents Date: Fri, 28 Jan 2022 08:27:01 +0800 Message-Id: <20220128002701.11971-3-wqu@suse.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220128002701.11971-1-wqu@suse.com> References: <20220128002701.11971-1-wqu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org There is a long existing bug in btrfs defrag code that it will always try to defrag compressed extents, even they are already at max capacity. This will not reduce the number of extents, but only waste IO/CPU. The kernel fix is titled: btrfs: defrag: don't defrag extents which is already at its max capacity Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- Changelog: v2: - Use fiemap output to compare the difference Now no need to use _get_file_extent_sector() helper at all. - Remove unnecessary mount options - Enlarge the write size to 16M To be future proof - Shorten the subject --- tests/btrfs/257 | 57 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/257.out | 2 ++ 2 files changed, 59 insertions(+) create mode 100755 tests/btrfs/257 create mode 100644 tests/btrfs/257.out diff --git a/tests/btrfs/257 b/tests/btrfs/257 new file mode 100755 index 00000000..bacd0c23 --- /dev/null +++ b/tests/btrfs/257 @@ -0,0 +1,57 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 257 +# +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already +# at their max capacity. +# +. ./common/preamble +_begin_fstest auto quick defrag + +# Import common functions. +. ./common/filter +. ./common/btrfs + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_require_scratch + +_scratch_mkfs >> $seqres.full + +_scratch_mount -o compress + +# Btrfs uses 128K as max extent size for compressed extents, this would result +# several compressed extents all at their max size +$XFS_IO_PROG -f -c "pwrite -S 0xee 0 16m" -c sync \ + $SCRATCH_MNT/foobar >> $seqres.full + +old_csum=$(_md5_checksum $SCRATCH_MNT/foobar) + +echo "=== File extent layout before defrag ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" > $tmp.before + +$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/foobar" >> $seqres.full +sync + +new_csum=$(_md5_checksum $SCRATCH_MNT/foobar) + +echo "=== File extent layout before defrag ===" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" > $tmp.after + +if [ $new_csum != $old_csum ]; then + echo "file content changed" +fi + +diff -q $tmp.before $tmp.after || echo "compressed extents get defragged" + +echo "Silence is golden" + +# success, all done +status=0 +exit diff --git a/tests/btrfs/257.out b/tests/btrfs/257.out new file mode 100644 index 00000000..cc3693f3 --- /dev/null +++ b/tests/btrfs/257.out @@ -0,0 +1,2 @@ +QA output created by 257 +Silence is golden