From patchwork Tue Jan 29 12:32:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 10786127 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4CC176C2 for ; Tue, 29 Jan 2019 12:32:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E2D82C04A for ; Tue, 29 Jan 2019 12:32:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C36A2C06C; Tue, 29 Jan 2019 12:32:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B635F2C04A for ; Tue, 29 Jan 2019 12:32:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727770AbfA2Mc3 (ORCPT ); Tue, 29 Jan 2019 07:32:29 -0500 Received: from mx2.suse.de ([195.135.220.15]:45496 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727215AbfA2Mc3 (ORCPT ); Tue, 29 Jan 2019 07:32:29 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 6BB1DB0FD; Tue, 29 Jan 2019 12:32:27 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: fstests@vger.kernel.org, Filipe Manana Subject: [PATCH v2 1/3] btrfs: Test if btrfs hits EDQUOT without trying to reclaim some space Date: Tue, 29 Jan 2019 20:32:20 +0800 Message-Id: <20190129123222.8698-1-wqu@suse.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit a514d63882c3 ("btrfs: qgroup: Commit transaction in advance to reduce early EDQUOT") is no longer forcing transaction commit to reclaim space, and only commits transaction asynchronously in advance to address it. However the criteria used in async transaction commit is not comprehensive, thus it doesn't reclaim space automatically. This test case will check the behavior by: 1) Falloc a large padding file This file will take 90% of the qgroup limit 2) Sync the fs To reflect the qgroup changes 3) Delete the file Qgroup won't reclaim the space until transaction committed. 4) Try to write a file If kernel not fixed, qgroup will not automatically commit transaction to reclaim the freed space and hit EDQUOT. This bug is going to be fixed by a patch for kernel titled: "btrfs: qgroup: Make qgroup async transaction commit more aggressive". Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- changelog: v2: - Grammar and typo fixes. - Remove unnecessary filter for falloc. - Append '-f' option for rm. --- tests/btrfs/180 | 62 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/180.out | 3 +++ tests/btrfs/group | 1 + 3 files changed, 66 insertions(+) create mode 100755 tests/btrfs/180 create mode 100644 tests/btrfs/180.out diff --git a/tests/btrfs/180 b/tests/btrfs/180 new file mode 100755 index 000000000000..2b4cfcc92e25 --- /dev/null +++ b/tests/btrfs/180 @@ -0,0 +1,62 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 180 +# +# Test if btrfs hits EDQUOT without reclaiming already freed extents when quota +# is enabled. +# +# This bug is going to be fxied by a patch for kernel titled +# "btrfs: qgroup: Make qgroup async transaction commit more aggressive" +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command falloc + +_scratch_mkfs > /dev/null +_scratch_mount + +$BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null +$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null +$BTRFS_UTIL_PROG qgroup limit -e 1G "$SCRATCH_MNT" + +$XFS_IO_PROG -f -c "falloc 0 900M" "$SCRATCH_MNT/padding" + +# Commit transaction to reflect the quota usage +sync + +rm -f "$SCRATCH_MNT/padding" + +# Without the kernel fix, this will trigger EDQUOT. +_pwrite_byte 0xcd 0 512M "$SCRATCH_MNT/real_file" | _filter_xfs_io + +# success, all done +status=0 +exit diff --git a/tests/btrfs/180.out b/tests/btrfs/180.out new file mode 100644 index 000000000000..17187477fc12 --- /dev/null +++ b/tests/btrfs/180.out @@ -0,0 +1,3 @@ +QA output created by 180 +wrote 536870912/536870912 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) diff --git a/tests/btrfs/group b/tests/btrfs/group index 46dd3c9523c2..03eb62f98332 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -182,3 +182,4 @@ 177 auto quick swap balance 178 auto quick send 179 auto qgroup dangerous +180 auto quick qgroup limit From patchwork Tue Jan 29 12:32:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 10786131 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1763139A for ; Tue, 29 Jan 2019 12:32:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C22382C04F for ; Tue, 29 Jan 2019 12:32:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B66C22C065; Tue, 29 Jan 2019 12:32:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 338B72C04F for ; Tue, 29 Jan 2019 12:32:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727996AbfA2Mce (ORCPT ); Tue, 29 Jan 2019 07:32:34 -0500 Received: from mx2.suse.de ([195.135.220.15]:45518 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727964AbfA2Mce (ORCPT ); Tue, 29 Jan 2019 07:32:34 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id B4FB7B102; Tue, 29 Jan 2019 12:32:32 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: fstests@vger.kernel.org, Filipe Manana Subject: [PATCH v3 2/3] btrfs: Test if btrfs will commit too many transactions for balance Date: Tue, 29 Jan 2019 20:32:21 +0800 Message-Id: <20190129123222.8698-2-wqu@suse.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190129123222.8698-1-wqu@suse.com> References: <20190129123222.8698-1-wqu@suse.com> MIME-Version: 1.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Kernel commit 64403612b73a ("btrfs: rework btrfs_check_space_for_delayed_refs") is introducing a regression for btrfs balance performance. Since that commit will cause btrfs to commit too many transactions for nothing during balance/relocation, it will slow balance dramatically even we only need to relocate several megabytes. This test case will catch the problem by using super block generation as failure criteria. For small chunk relocated, we will commit 6 transactions for each block group, and the test case should only have 2 block groups, it should only commit 12 transactions. This test case will use 120 as the threshold to detect the failure. And in my test environment, with kernel fix btrfs committed 14 transactions. While without the fix btrfs committed 209 transactions. So the test case should be enough to detect the regression, while still keep the runtime small enough for failure. Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- changelog: v2: - Remove the unnecessary workaround for ENOSPC balance Since we know the cause of that false ENOSPC error, a simple sync would be enough to take care of the problem - Remove unnecessary mount option of max_inline The same reason above. v3: - Remove unused variable. - Grammar and type fixes. - Full subcommand name for 'btrfs' command. - Always check output for get_super_gen(). --- tests/btrfs/181 | 101 ++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/181.out | 2 + tests/btrfs/group | 1 + 3 files changed, 104 insertions(+) create mode 100755 tests/btrfs/181 create mode 100644 tests/btrfs/181.out diff --git a/tests/btrfs/181 b/tests/btrfs/181 new file mode 100755 index 000000000000..323216e4fa0a --- /dev/null +++ b/tests/btrfs/181 @@ -0,0 +1,101 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 181 +# +# Test if btrfs will commit too many transactions for nothing and cause +# performance regression during balance. +# +# This bug is going to be fixed by a patch for kernel title +# "btrfs: don't end the transaction for delayed refs in throttle" +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_btrfs_command inspect-internal dump-super + +_scratch_mkfs > /dev/null + +_scratch_mount + +nr_files=1024 + +get_super_gen() +{ + local ret=$($BTRFS_UTIL_PROG inspect dump-super "$SCRATCH_DEV" |\ + grep ^generation | awk '{print $2}') + if [ -z $ret ]; then + _fail "failed to get super block generation" + fi + echo "$ret" +} + +$BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/subvol" > /dev/null + +# Create some small files to take up enough metadata reserved space +for ((i = 0; i < $nr_files; i++)) do + _pwrite_byte 0xcd 0 1K "$SCRATCH_MNT/subvol/file_$i" > /dev/null +done + +# Commit the fs so we can get a stable super generation +sync + +before_gen=$(get_super_gen) + +$BTRFS_UTIL_PROG balance start -m "$SCRATCH_MNT" > /dev/null + +after_gen=$(get_super_gen) + +# Since the fs is pretty small, we should have only 1 small metadata chunk and +# one tiny system chunk. +# Relocating such small chunks only needs 6 commits for each, thus 12 commits for +# 2 chunks. +# Here we use 10x the theoretic value as threshold. +theoretic_gen=$(( 6 * 2 )) +threshold_gen=$(( 10 * $theoretic_gen )) +if [ $(( $after_gen - $before_gen )) -gt 120 ]; then + echo "balance committed too many transactions" + echo "super generation before balance: ${before_gen}" + echo "super generation after balance: ${after_gen}" + echo "super generation difference: $((after_gen - before_gen))" + echo "theoretic generation difference: ${theoretic_gen}" + echo "threshold generation difference: ${threshold_gen}" +fi + +echo "super generation before balance: ${before_gen}" >> $seqres.full +echo "super generation after balance: ${after_gen}" >> $seqres.full +echo "super generation difference: $((after_gen - before_gen))" >> $seqres.full +echo "theoretic generation difference: ${theoretic_gen}" >> $seqres.full +echo "threshold generation difference: ${threshold_gen}" >> $seqres.full + +# success, all done +echo "Silence is golden" + +status=0 +exit diff --git a/tests/btrfs/181.out b/tests/btrfs/181.out new file mode 100644 index 000000000000..a5a93be663eb --- /dev/null +++ b/tests/btrfs/181.out @@ -0,0 +1,2 @@ +QA output created by 181 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group index 03eb62f98332..0db485cbe834 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -183,3 +183,4 @@ 178 auto quick send 179 auto qgroup dangerous 180 auto quick qgroup limit +181 auto quick balance From patchwork Tue Jan 29 12:32:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 10786137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 775F017E9 for ; Tue, 29 Jan 2019 12:32:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 692F52C06A for ; Tue, 29 Jan 2019 12:32:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67A9A2C060; Tue, 29 Jan 2019 12:32:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4120D2C060 for ; Tue, 29 Jan 2019 12:32:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728296AbfA2Mcm (ORCPT ); Tue, 29 Jan 2019 07:32:42 -0500 Received: from mx2.suse.de ([195.135.220.15]:45568 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727215AbfA2Mcl (ORCPT ); Tue, 29 Jan 2019 07:32:41 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id B175CB0FD; Tue, 29 Jan 2019 12:32:40 +0000 (UTC) From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: fstests@vger.kernel.org, Filipe Manana Subject: [PATCH v2 3/3] btrfs: Test if btrfs will report false ENOSPC error balancing small metadata chunk Date: Tue, 29 Jan 2019 20:32:22 +0800 Message-Id: <20190129123222.8698-3-wqu@suse.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190129123222.8698-1-wqu@suse.com> References: <20190129123222.8698-1-wqu@suse.com> MIME-Version: 1.0 Sender: fstests-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is a test case for a long existing bug, caused by over-estimated metadata space_info::bytes_may_use. There is one proposed patch for btrfs-progs to fix it, titled: "btrfs-progs: balance: Sync the fs before balancing metadata chunks" The test case itself is almost the same as btrfs/181, which uses small files to bump the reserved space to trigger the false alert. Signed-off-by: Qu Wenruo Reviewed-by: Filipe Manana --- changelog: v2: - Grammar and type fixes. - Use full subcommand name for 'btrfs' command. --- tests/btrfs/182 | 60 +++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/182.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 63 insertions(+) create mode 100755 tests/btrfs/182 create mode 100644 tests/btrfs/182.out diff --git a/tests/btrfs/182 b/tests/btrfs/182 new file mode 100755 index 000000000000..2b347011aaa8 --- /dev/null +++ b/tests/btrfs/182 @@ -0,0 +1,60 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 182 +# +# Test if balance will report false ENOSPC error +# +# This is a long existing bug, caused by over-estimated metadata +# space_info::bytes_may_use. +# There is one proposed patch for btrfs-progs to fix it, titled: +# "btrfs-progs: balance: Sync the fs before balancing metadata chunks" +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch + +nr_files=1024 + +_scratch_mkfs > /dev/null +_scratch_mount + +$BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/subvol" > /dev/null + +# Create some small files to take up enough metadata reserved space +for ((i = 0; i < $nr_files; i++)) do + _pwrite_byte 0xcd 0 1K "$SCRATCH_MNT/subvol/file_$i" > /dev/null +done + +$BTRFS_UTIL_PROG balance start -m "$SCRATCH_MNT" > /dev/null + +# success, all done +echo "Silence is golden" +status=0 +exit diff --git a/tests/btrfs/182.out b/tests/btrfs/182.out new file mode 100644 index 000000000000..53d28e50c68b --- /dev/null +++ b/tests/btrfs/182.out @@ -0,0 +1,2 @@ +QA output created by 182 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group index 0db485cbe834..e32c84e4b253 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -184,3 +184,4 @@ 179 auto qgroup dangerous 180 auto quick qgroup limit 181 auto quick balance +182 auto quick balance