From patchwork Fri May 20 01:23:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 12856180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98A58C433F5 for ; Fri, 20 May 2022 01:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243737AbiETBXm (ORCPT ); Thu, 19 May 2022 21:23:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232799AbiETBXl (ORCPT ); Thu, 19 May 2022 21:23:41 -0400 Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 80F8727CC7 for ; Thu, 19 May 2022 18:23:40 -0700 (PDT) Received: from dread.disaster.area (pa49-181-2-147.pa.nsw.optusnet.com.au [49.181.2.147]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id 9163A10E6838 for ; Fri, 20 May 2022 11:23:39 +1000 (AEST) Received: from discord.disaster.area ([192.168.253.110]) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1nrrMf-00E40U-Rm for fstests@vger.kernel.org; Fri, 20 May 2022 11:23:37 +1000 Received: from dave by discord.disaster.area with local (Exim 4.95) (envelope-from ) id 1nrrMf-006TK7-Qi for fstests@vger.kernel.org; Fri, 20 May 2022 11:23:37 +1000 From: Dave Chinner To: fstests@vger.kernel.org Subject: [PATCH 1/2] fstests: tests should not source common/rc directly Date: Fri, 20 May 2022 11:23:35 +1000 Message-Id: <20220520012336.1542637-2-david@fromorbit.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220520012336.1542637-1-david@fromorbit.com> References: <20220520012336.1542637-1-david@fromorbit.com> MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.4 cv=VuxAv86n c=1 sm=1 tr=0 ts=6286ed9b a=ivVLWpVy4j68lT4lJFbQgw==:117 a=ivVLWpVy4j68lT4lJFbQgw==:17 a=oZkIemNP1mAA:10 a=20KFwNOVAAAA:8 a=gbopmE6BPGZhA-chkOMA:9 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Dave Chinner _begin_fstest() does this as every test needs it, so remove it from all the tests that source it a second time. Signed-off-by: Dave Chinner Reviewed-by: Darrick J. Wong Acked-by: Christian Brauner (Microsoft) --- tests/btrfs/243 | 1 - tests/btrfs/245 | 1 - tests/ext4/053 | 1 - tests/generic/644 | 1 - tests/generic/645 | 1 - tests/generic/647 | 1 - tests/generic/677 | 1 - tests/generic/678 | 1 - tests/generic/679 | 1 - tests/generic/690 | 1 - 10 files changed, 10 deletions(-) diff --git a/tests/btrfs/243 b/tests/btrfs/243 index 750b4323..09a63bcd 100755 --- a/tests/btrfs/243 +++ b/tests/btrfs/243 @@ -20,7 +20,6 @@ _cleanup() rm -r -f $tmp.* } -. ./common/rc . ./common/filter . ./common/dmflakey diff --git a/tests/btrfs/245 b/tests/btrfs/245 index dadc8492..426aab1c 100755 --- a/tests/btrfs/245 +++ b/tests/btrfs/245 @@ -11,7 +11,6 @@ _begin_fstest auto quick idmapped subvol # get standard environment, filters and checks -. ./common/rc . ./common/filter # real QA test starts here diff --git a/tests/ext4/053 b/tests/ext4/053 index bf6e3f6b..187a2515 100755 --- a/tests/ext4/053 +++ b/tests/ext4/053 @@ -28,7 +28,6 @@ _cleanup() } # get standard environment, filters and checks -. ./common/rc . ./common/filter . ./common/quota diff --git a/tests/generic/644 b/tests/generic/644 index edf9b03e..c48338d8 100755 --- a/tests/generic/644 +++ b/tests/generic/644 @@ -10,7 +10,6 @@ _begin_fstest auto quick cap idmapped mount # get standard environment, filters and checks -. ./common/rc . ./common/filter # real QA test starts here diff --git a/tests/generic/645 b/tests/generic/645 index 74e0f589..556d2450 100755 --- a/tests/generic/645 +++ b/tests/generic/645 @@ -10,7 +10,6 @@ _begin_fstest auto quick idmapped mount # get standard environment, filters and checks -. ./common/rc . ./common/filter # real QA test starts here diff --git a/tests/generic/647 b/tests/generic/647 index fa6edadc..8484fa8d 100755 --- a/tests/generic/647 +++ b/tests/generic/647 @@ -18,7 +18,6 @@ _cleanup() } # get standard environment, filters and checks -. ./common/rc . ./common/filter # real QA test starts here diff --git a/tests/generic/677 b/tests/generic/677 index 1d4eaa53..39af90a9 100755 --- a/tests/generic/677 +++ b/tests/generic/677 @@ -18,7 +18,6 @@ _cleanup() rm -r -f $tmp.* } -. ./common/rc . ./common/filter . ./common/dmflakey . ./common/punch diff --git a/tests/generic/678 b/tests/generic/678 index 1c3cf6e3..3c3c05b7 100755 --- a/tests/generic/678 +++ b/tests/generic/678 @@ -22,7 +22,6 @@ _cleanup() } # get standard environment, filters and checks -. ./common/rc . ./common/filter # real QA test starts here diff --git a/tests/generic/679 b/tests/generic/679 index c32d42b9..a0094e48 100755 --- a/tests/generic/679 +++ b/tests/generic/679 @@ -11,7 +11,6 @@ . ./common/preamble _begin_fstest auto quick prealloc -. ./common/rc . ./common/filter . ./common/punch diff --git a/tests/generic/690 b/tests/generic/690 index f03295a5..cef8d6e8 100755 --- a/tests/generic/690 +++ b/tests/generic/690 @@ -24,7 +24,6 @@ _cleanup() rm -r -f $tmp.* } -. ./common/rc . ./common/filter . ./common/dmflakey From patchwork Fri May 20 01:23:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 12856182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFEAFC433F5 for ; Fri, 20 May 2022 01:23:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245324AbiETBXo (ORCPT ); Thu, 19 May 2022 21:23:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245387AbiETBXm (ORCPT ); Thu, 19 May 2022 21:23:42 -0400 Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6424F29831 for ; Thu, 19 May 2022 18:23:41 -0700 (PDT) Received: from dread.disaster.area (pa49-181-2-147.pa.nsw.optusnet.com.au [49.181.2.147]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 8FC195345DF for ; Fri, 20 May 2022 11:23:39 +1000 (AEST) Received: from discord.disaster.area ([192.168.253.110]) by dread.disaster.area with esmtp (Exim 4.92.3) (envelope-from ) id 1nrrMf-00E40V-T0 for fstests@vger.kernel.org; Fri, 20 May 2022 11:23:37 +1000 Received: from dave by discord.disaster.area with local (Exim 4.95) (envelope-from ) id 1nrrMf-006TKB-Rt for fstests@vger.kernel.org; Fri, 20 May 2022 11:23:37 +1000 From: Dave Chinner To: fstests@vger.kernel.org Subject: [PATCH 2/2] common/rc: fix test init performance regression on XFS Date: Fri, 20 May 2022 11:23:36 +1000 Message-Id: <20220520012336.1542637-3-david@fromorbit.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220520012336.1542637-1-david@fromorbit.com> References: <20220520012336.1542637-1-david@fromorbit.com> MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.4 cv=VuxAv86n c=1 sm=1 tr=0 ts=6286ed9b a=ivVLWpVy4j68lT4lJFbQgw==:117 a=ivVLWpVy4j68lT4lJFbQgw==:17 a=oZkIemNP1mAA:10 a=20KFwNOVAAAA:8 a=u6XUrllArdfvgz3kJkIA:9 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Dave Chinner I've been having problems over the past month with tests randomly running much slower than they should. The overall effect has been that running an auto group iterate has blown out from about 3 hours to almost 5 hours. The problem has been that the root disks have been thrashing. Thousands of iops and the backing store getting IO bound so all test vms with root drive images hosted on that backing store then also go slow. I finally got to the bottom of it - it looked for all the world like a qemu image format write amplification problem, because I couldn't capture anything unusual from inside the guest. I converted all the root images to raw format on fast SSD storage, and the problem didn't go away - while some tests were faster, other tests were still randomly slow. Finally I tracked it down with blktrace/blkparse. All the IO was coming from processes with the command line like: /sbin/mkfs.xfs -d file name=/tmp/791409.img size=512m A quick grep then pointed me at common/rc::init_rc() where there is a test for XFS specific on-disk format behaviour that uses that exact command line. init_rc() is run from _begin_fstest() so it is executed on every test start. We run mkfs to create a 512MB filesystem in $tmp on every test start. That's ..... not smart. And we do it so we can change the setup of $XFS_COPY_PROG, which is only used in a handful of tests. That's also .... not smart. And then the penny dropped: we recently increased the default log size of small filesystems, so this is now zeroing a 64MB log instead of a 5-10MB log. And, of course it's on the root drive because of the $tmp image prefix, which in this case is ext3, and it's fragmenting the crap out of the writes and so it increases runtime of the mkfs.xfs execution from nothing to 5-10s. Then when one of these mkfs invocations co-incides with journal flushing driven writeback, everything just stops until the writeback stops thrashing. And so all tests are 5-10s slows, and random tests take anywhere between 20-100s longer to start up.... Fix it by moving the setup of XFS_COPY_PROG to _require_xfs_copy() so only the tests that use xfs_copy run this code. Numbers don't lie: generic/001 11s ... 6s generic/002 9s ... 1s generic/003 17s ... 10s generic/004 12s ... 1s generic/005 9s ... 1s generic/006 9s ... 2s generic/007 11s ... 3s generic/008 5s ... 2s generic/009 10s ... 2s generic/010 5s ... 0s generic/011 9s ... 1s generic/012 7s ... 2s generic/013 10s ... 5s generic/014 11s ... 1s generic/015 7s ... 1s generic/016 7s ... 2s ..... Signed-off-by: Dave Chinner Reviewed-by: Darrick J. Wong Reviewed-by: Zorro Lang --- common/rc | 13 ------------- common/xfs | 12 ++++++++++++ 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/common/rc b/common/rc index f5ead044..67b4cc77 100644 --- a/common/rc +++ b/common/rc @@ -4522,19 +4522,6 @@ init_rc() # it is supported. $XFS_IO_PROG -i -c quit 2>/dev/null && \ export XFS_IO_PROG="$XFS_IO_PROG -i" - - # xfs_copy on v5 filesystems do not require the "-d" option if xfs_db - # can change the UUID on v5 filesystems - if [ "$FSTYP" == "xfs" ]; then - touch /tmp/$$.img - $MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=512m >/dev/null 2>&1 - # xfs_db will return 0 even if it can't generate a new uuid, so - # check the output to make sure if it can change UUID of V5 xfs - $XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \ - | grep -q "invalid UUID\|supported on V5 fs" \ - && export XFS_COPY_PROG="$XFS_COPY_PROG -d" - rm -f /tmp/$$.img - fi } # get real device path name by following link diff --git a/common/xfs b/common/xfs index ac1d021e..2123a4ab 100644 --- a/common/xfs +++ b/common/xfs @@ -1160,6 +1160,18 @@ _require_xfs_copy() [ -n "$XFS_COPY_PROG" ] || _notrun "xfs_copy binary not yet installed" [ "$USE_EXTERNAL" = yes ] && \ _notrun "Cannot xfs_copy with external devices" + + # xfs_copy on v5 filesystems do not require the "-d" option if xfs_db + # can change the UUID on v5 filesystems + touch /tmp/$$.img + $MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=64m >/dev/null 2>&1 + + # xfs_db will return 0 even if it can't generate a new uuid, so + # check the output to make sure if it can change UUID of V5 xfs + $XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \ + | grep -q "invalid UUID\|supported on V5 fs" \ + && export XFS_COPY_PROG="$XFS_COPY_PROG -d" + rm -f /tmp/$$.img } __xfs_cowgc_interval_knob1="/proc/sys/fs/xfs/speculative_cow_prealloc_lifetime"