From patchwork Fri Nov 22 16:50:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883401 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 026257082F; Fri, 22 Nov 2024 16:50:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294244; cv=none; b=hojR9yCf2NmaHtI18dBoNchhRRMqtFqJVACnz3b5aeHv0PF5P4pBbfJl7vLXKvJelc53xPEcngFIEHwkzz/xGqAbTgfcOkE4N7joWvFCS29+RRk9Bfs5myS/7aZPeKiUOBMf9n2q+KfQvAgif5XFZzerQIFda2yTnjbb+NNtZrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294244; c=relaxed/simple; bh=118F3cLeQLECfTPe2n1rCyAfJUReJzNfz7TOatchghk=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kmQyQ+3a4SHj3biQ994Zapo2Z9JQxzv4CfBn2QKiCNX6zc2c6u1W2fDRefLMe+yLyVRJSxwVxJu4YBV6Gr2Sw3+10fL4SnTYg2QSoLBoJPFXNdDK+T8pDFQVeuwSa/WMTswGVlVwpVMEdVGD5/lO3ZQ674lrZqTJKuXPLwyxkNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UHlfgO7H; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UHlfgO7H" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A5BFC4CED0; Fri, 22 Nov 2024 16:50:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294243; bh=118F3cLeQLECfTPe2n1rCyAfJUReJzNfz7TOatchghk=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=UHlfgO7HOVDGo7P4spHb0h7hLQtNmSxbpkt4l8J+ciSA0L+Zu4qbVozhR/g+x4aIj OdMJNJTErAzWNlmuwGLBtNJrasKu6wNl7x7v1rdyPyCTy+EVpfhurVmXmgNXiWMHds YkQHNtjlQ38WAKzIV8W7p/hy0kgWwm6yJqdZkxI1ok7/NTrx5Mf1MlxSbXXHjJNahx kSf/xINsyM3F+einNTrDQ9MBknAucr9dyugfyc9dtc8KLyLv82rbUWkXZccHeWyny3 ZO1RI18Hq/xMYTuPiwicc7f+bmb58RQoGJXvbocNrgL+RGOUsjyvr5of7VHqHIjzlq MAWVxNYl20UYw== Date: Fri, 22 Nov 2024 08:50:43 -0800 Subject: [PATCH 01/17] generic/757: fix various bugs in this test From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: hch@lst.de, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420029.358248.9101250955454761474.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Fix this test so the check doesn't fail on XFS, and restrict runtime to 100 loops because otherwise this test takes many hours. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/757 | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/generic/757 b/tests/generic/757 index 0ff5a8ac00182b..37cf49e6bc7fd9 100755 --- a/tests/generic/757 +++ b/tests/generic/757 @@ -63,9 +63,14 @@ prev=$(_log_writes_mark_to_entry_number mkfs) cur=$(_log_writes_find_next_fua $prev) [ -z "$cur" ] && _fail "failed to locate next FUA write" -while [ ! -z "$cur" ]; do +while _soak_loop_running $((100 * TIME_FACTOR)); do _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full + # xfs_repair won't run if the log is dirty + if [ $FSTYP = "xfs" ]; then + _scratch_mount + _scratch_unmount + fi _check_scratch_fs prev=$cur From patchwork Fri Nov 22 16:50:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883402 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E0445D477; Fri, 22 Nov 2024 16:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294259; cv=none; b=lsjkVPwXUfbEiG6G0K4QlE54bAl2qWEMTGChmelPxY+DBidI4B5sXMmVImEknhvzdxvLT2b/NWgNRBec2u40KEIYEHcKDzXesSk7Tsq/fnty6VfbJd71pHTFT+VZ5qgtqdhvQjHgzZ7Eqsd+3DqE3bTImUkMYuGAq0cp5tPyhwI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294259; c=relaxed/simple; bh=vDejHN868eLxHTxDPPabBAQeKxYhNyPu8rgk3Fp0FjE=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XtyW6VgTd5QMXjInhNZmEs2Fo4hu4Qv7bP9RInCOxl1j2aNLUxjUJGJ/tVkgWfQGc5dbD1J08CjZHDNGRNBLW9JEMIR6E8Ij/D61kFuBfhrl6+Mo5+rWyO6B+9279FpvBbtE/zjm01jJf39W1amWpe+9N8o2qQnHOzKZq38voq4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o7v58s/z; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o7v58s/z" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 114A2C4CECE; Fri, 22 Nov 2024 16:50:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294259; bh=vDejHN868eLxHTxDPPabBAQeKxYhNyPu8rgk3Fp0FjE=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=o7v58s/zRMnKItOoC8P5nLx6nbBTc86ycixv5Cj9vwy0LlOMAclBhb8lJ8VeaIHz6 T1GONfGrcvWpxlQBd1eyKCONcwXgrC7pETo1UAWrC8YMJ/ESp7zmYSd0/vvaz6bPiG 8tNW6D39T6Oj7vbPScvTuDWLpO6bEs90HGxYBiMHz8qElGNyo+JIN4CFptGO7unHXg +W6MROEWgYO+QOphElCq34TAtg2haWTnVqr7YKhGccX9fbTL0Ufs8ZvNedshXgk4ys YNG5ly4C4bQjp0S+ABsd9ZTMG1TtxuiWq9CULVrMlyWXG78aXtek3ETXxoK6IqITi6 HNAaOMv1rmc+g== Date: Fri, 22 Nov 2024 08:50:58 -0800 Subject: [PATCH 02/17] generic/757: convert to thinp From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420045.358248.13209894180925094165.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Convert this test to use dm-thinp so that discards always zero the data. This prevents weird replay problems if the scratch device doesn't guarantee that read after discard returns zeroes. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/757 | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/tests/generic/757 b/tests/generic/757 index 37cf49e6bc7fd9..6c13c6af41c57c 100755 --- a/tests/generic/757 +++ b/tests/generic/757 @@ -8,12 +8,13 @@ # This can be seen on subpage FSes on Linux 6.4. # . ./common/preamble -_begin_fstest auto quick metadata log recoveryloop aio +_begin_fstest auto quick metadata log recoveryloop aio thin _cleanup() { cd / _log_writes_cleanup &> /dev/null + _dmthin_cleanup rm -f $tmp.* } @@ -23,11 +24,14 @@ _cleanup() fio_config=$tmp.fio +. ./common/dmthin . ./common/dmlogwrites -_require_scratch +# Use thin device as replay device, which requires $SCRATCH_DEV +_require_scratch_nocheck _require_aiodio _require_log_writes +_require_dm_target thin-pool cat >$fio_config <> $seqres.full -_log_writes_init $SCRATCH_DEV +# Use a thin device to provide deterministic discard behavior. Discards are used +# by the log replay tool for fast zeroing to prevent out-of-order replay issues. +_test_unmount +sectors=$(blockdev --getsz $SCRATCH_DEV) +sectors=$((sectors * 90 / 100)) +_dmthin_init $sectors $sectors +_log_writes_init $DMTHIN_VOL_DEV _log_writes_mkfs >> $seqres.full 2>&1 _log_writes_mark mkfs @@ -64,14 +74,13 @@ cur=$(_log_writes_find_next_fua $prev) [ -z "$cur" ] && _fail "failed to locate next FUA write" while _soak_loop_running $((100 * TIME_FACTOR)); do - _log_writes_replay_log_range $cur $SCRATCH_DEV >> $seqres.full + _log_writes_replay_log_range $cur $DMTHIN_VOL_DEV >> $seqres.full # xfs_repair won't run if the log is dirty if [ $FSTYP = "xfs" ]; then - _scratch_mount - _scratch_unmount + _dmthin_mount fi - _check_scratch_fs + _dmthin_check_fs prev=$cur cur=$(_log_writes_find_next_fua $(($cur + 1))) From patchwork Fri Nov 22 16:51:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883403 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1ED3222339; Fri, 22 Nov 2024 16:51:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294275; cv=none; b=XcpliSjY8nRymePrmo9GXj30LvVpqtvzyPwOfIr/m/qcHn1ktZWrygGSb7HxSu3C1ITXc02j01vP5NxOXbe4LMRVi23d6jjW8VsvKb1kTe94vd1OH03/gJV53lgbXNF9th1NUYmDRjSx7V6O2a9zpT/lWujD+s4bvfKmr5SAkKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294275; c=relaxed/simple; bh=SrLOsJZykfMgnBXOHMK19RBxhkmcaAb8bJQxpR8wtwI=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nNVOe2YmnKLzZhlCKtBZXC0yEMaj6S1gxoBbV8TvDtx7Am1dnl1J2mHYCjQWUXGSVrYsCehtthKgCocHDasAKlPgQc7GqhGQxmOmv0IsfKINKk+CPbGVDOZUmMfzFq30H4mxe48VudF+kbZtQIGdlyiw8gR0TmNRA+khuifwvvc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XcFVIlQA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XcFVIlQA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A73DAC4CECE; Fri, 22 Nov 2024 16:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294274; bh=SrLOsJZykfMgnBXOHMK19RBxhkmcaAb8bJQxpR8wtwI=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=XcFVIlQAgl1uHevfhD1jg6Fuj8TH6Y0dld9uem5DrgyPE9j/XohOHe9/17u5e3RBW EX/b8QldjzilaN6/Bef3Yv91IRGq9X23j49T0sBVNM3sE+/RsRyaGw8dM396PMJgz/ 4oSiWKp/DzhShgdQdR7JywzbHek6NXcWkwtd+gq53JxcvkrsatM/S2EX+AsRSBtIUf W3BV2R+EJCD/5CDUK6SSQGhk5qd0koaPY3fj1f3XAINVCXHtpdGXb78ddB7TKm3JCj fsSCYlVf8P8IsCT5kRlcYKTfIZ2B3tJq3MqHC+kfsivlio6vIw+Zb/70EK+0NLCACq QFTwFdKk3OmNw== Date: Fri, 22 Nov 2024 08:51:14 -0800 Subject: [PATCH 03/17] logwrites: warn if we don't think read after discard returns zeroes From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420060.358248.11054238752146807489.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong The logwrites replay program expects that it can issue a DISCARD against the block device passed to _log_writes_init and that will cause all subsequent reads to return zeroes. This is required for correct log recovery on filesystems such as XFS that skip recovering buffers if newer ones are found on disk. Unfortunately, there's no way to discover if a device's discard implementation actually guarantees zeroes. There used to be a sysfs knob keyed to an allowlist, but it is now hardwired to return 0. So either we need a magic device that does discard-and-zero, or we need to do the zeroing ourselves. The logwrites program does its own zeroing if there is no discard support, and some tests do their own zeroing. The only devices we know to work reliably are the software defined ones that are provided by the kernel itself -- which means dm-thinp. Warn if we have a device that supports discard that isn't thinp and the test fails. Signed-off-by: "Darrick J. Wong" --- common/dmlogwrites | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/common/dmlogwrites b/common/dmlogwrites index c1c85de9dd43ac..24a8a25ace277f 100644 --- a/common/dmlogwrites +++ b/common/dmlogwrites @@ -59,6 +59,35 @@ _require_log_writes_dax_mountopt() fi } +_log_writes_check_bdev() +{ + local sysfs="/sys/block/$(_short_dev $1)" + + # Some filesystems (e.g. XFS) optimize log recovery by assuming that + # they can elide replay of metadata blocks if the block has a higher + # log serial number than the transaction being recovered. This is a + # problem if the filesystem log contents can go back in time, which is + # what the logwrites replay program does. + # + # The logwrites replay program begins by erasing the block device's + # contents. This can be done very quickly with DISCARD provided the + # device guarantees that all reads after a DISCARD return zeroes, or + # very slowly by writing zeroes to the device. Fast is preferable, but + # there's no longer any way to detect that DISCARD actually unmaps + # zeroes, so warn the user about this requirement if the test happens + # to fail. + + # No discard support means the logwrites will do its own zeroing + test "$(cat "$sysfs/queue/discard_max_bytes")" -eq 0 && return + + # dm-thinp guarantees that reads after discards return zeroes + dmsetup status "$blkdev" 2>/dev/null | grep -q '^0.* thin ' && return + + echo "HINT: $blkdev doesn't guarantee that reads after DISCARD will return zeroes" >> $seqres.hints + echo " This is required for correct journal replay on some filesystems (e.g. xfs)" >> $seqres.hints + echo >> $seqres.hints +} + # Set up a dm-log-writes device # # blkdev: the specified target device @@ -84,6 +113,8 @@ _log_writes_init() LOGWRITES_NAME=logwrites-test LOGWRITES_DMDEV=/dev/mapper/$LOGWRITES_NAME LOGWRITES_TABLE="0 $BLK_DEV_SIZE log-writes $blkdev $LOGWRITES_DEV" + + _log_writes_check_bdev "$blkdev" _dmsetup_create $LOGWRITES_NAME --table "$LOGWRITES_TABLE" || \ _fail "failed to create log-writes device" } From patchwork Fri Nov 22 16:51:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883404 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E55A460890; Fri, 22 Nov 2024 16:51:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294291; cv=none; b=TClNqRmRrj4x/NnHug/kDBIwwG90nMFMIbinDAA1YAOoK9FYc5mcNc1qRsb1DloV3KvPlStnhNDSaD44hSeQcG5C2PjFxmVj9Saqr5r0N0wEkASiLiy65uLHO4VQP0SSlVT6EGMaufnzgtH4/Nl2mrV1JFTENXsSY86MQVXHZkE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294291; c=relaxed/simple; bh=1b3WRsNnivAxyi55OjZu3eEOrtb0Id4ruAwv8YsLp24=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FfmB99pV0iL3zN669isremIfIm47F7DUqt2bBUUGmjMWDFGjNKxspfwQLEo6Qp6kDKXuLvEbtQuAQ4gVGGcxMZ/Bv5WsKPUKFtA1IWEbz9pY84Sxm7Egb+sO5VRY2N+5vFNWtVRxIeEmSpagHLlgQc11qvI+BL5RNNjv0kSaFlY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Dy0sxI3Q; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Dy0sxI3Q" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 569D7C4CECE; Fri, 22 Nov 2024 16:51:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294290; bh=1b3WRsNnivAxyi55OjZu3eEOrtb0Id4ruAwv8YsLp24=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=Dy0sxI3Qz5QQjTmpOXDFSA2z6rdSma6VGEQ/gAEdxTWDZy+3HRX0sjgwDlna9W7W2 mgrKYvDMrvhzgGuBmObjOdDYypCcGHNMEaPayTKytzk5BN/7m8vi7KHDfRSzDfuuq7 O75KtkImEQlYW8ZTHhs4/bYtBBeoNdl6yLnWMdFlTMajMQSUKx9bbULN8P9QvXnwbw 5Yf1ERt7A/a4Ce3wl2J87Zp24dsId7ffRD+uGXfII20RpPOCJNTrQPHRtp704LC1b/ U0Sg84wfoVb/M/b1XHz5oa6YcrnRqhfdWSEhpsdB5xVOswuHtQWjMA6NBDWjioM1mE 27gmG8rtgal4A== Date: Fri, 22 Nov 2024 08:51:29 -0800 Subject: [PATCH 04/17] logwrites: use BLKZEROOUT if it's available From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420076.358248.10925789878371514364.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Use the BLKZEROOUT ioctl instead of writing zeroed buffers if the kernel supports it. Signed-off-by: "Darrick J. Wong" --- src/log-writes/log-writes.c | 10 ++++++++++ src/log-writes/log-writes.h | 1 + 2 files changed, 11 insertions(+) diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c index aa53473974d9e8..8f94ae5629e085 100644 --- a/src/log-writes/log-writes.c +++ b/src/log-writes/log-writes.c @@ -42,6 +42,7 @@ static int discard_range(struct log *log, u64 start, u64 len) static int zero_range(struct log *log, u64 start, u64 len) { + u64 range[2] = { start, len }; u64 bufsize = len; ssize_t ret; char *buf = NULL; @@ -54,6 +55,15 @@ static int zero_range(struct log *log, u64 start, u64 len) return 0; } + if (!(log->flags & LOG_ZEROOUT_NOT_SUPP)) { + if (ioctl(log->replayfd, BLKZEROOUT, &range) < 0) { + if (log_writes_verbose) + printf( + "replay device doesn't support zeroout, switching to writing zeros\n"); + log->flags |= LOG_ZEROOUT_NOT_SUPP; + } + } + while (!buf) { buf = malloc(bufsize); if (!buf) diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h index b9f571ac3b2384..f659931634e64a 100644 --- a/src/log-writes/log-writes.h +++ b/src/log-writes/log-writes.h @@ -63,6 +63,7 @@ struct log_write_entry { #define LOG_IGNORE_DISCARD (1 << 0) #define LOG_DISCARD_NOT_SUPP (1 << 1) +#define LOG_ZEROOUT_NOT_SUPP (1 << 2) struct log { int logfd; From patchwork Fri Nov 22 16:51:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883405 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C5F722339; Fri, 22 Nov 2024 16:51:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294306; cv=none; b=OabfP+CudKuvayGYBJr8Yw+EHcN2Sphj+9kZQIttoMcXHCQefL8u3DQWdM9loOwz9+xrDHF1SUVStnl16g/LQ9oyswx1NXUQd4xy8k863Wk/mhh2jlrXf4sR5IRiXw28CWZEkQuKXmG4JiLr48FGC27a3yh9GVx3aFEnaATI/vc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294306; c=relaxed/simple; bh=XwDXAj8ZTLJd/AYQ9y/0nH0aQPW2AvFBH+YaBLeB52E=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PWKb8VuxVdzOgAbwgxcUyfdEL7LcAm2+VUDpjD12W3dMqM3IAymlmcuqPA2EsCF1R5DkQumsKQJoP8DD5GewnkisTp0KkK6ukdK03CZ1LOkQJwh2z7Ub51kk6SOC8dZP26mjAXU3URX1Pqr23niVSxQpb33UPM/eDUAbtgr8CUc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i8TuXT11; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i8TuXT11" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF5ECC4CECE; Fri, 22 Nov 2024 16:51:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294306; bh=XwDXAj8ZTLJd/AYQ9y/0nH0aQPW2AvFBH+YaBLeB52E=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=i8TuXT11TVeYY8RAxJXufEH1P1LYOYkruI0SGYKjh7b1j0+ZykZSEOTwST2Xdycti 287KsQU9Xb7XpntZXQUllDRl2enHhjV16Wk+14Tiv+jKt3bkOWkwxnCQ9h01J7r6kN TIb2Ab9CYWHn3xRUGHQO1Sm2vW24guPVEEZ8SjQHnAB4i4ZC7pX6SrA+UerRS5UL7G qL+K52fbVgMi+sEPeskK34FjS0AVJ48lUMSyiMJUut8RCSQyv17CDTzJ1ys/C7nKv9 kbyHuYa0TERmh/ZebqK9YMSh1Bq5HuYRh8j4Mk2jdn8Ac/wV+P6wDFUwJxK0Ri4ieU PHkC/cscQdEuA== Date: Fri, 22 Nov 2024 08:51:45 -0800 Subject: [PATCH 05/17] logwrites: only use BLKDISCARD if we know discard zeroes data From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420088.358248.16258645435851782707.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Building off the checks established in the previous patch, only enable the use of BLKDISCARD if we know that the logwrites device guarantees that reads after a discard return zeroes. Signed-off-by: "Darrick J. Wong" --- common/dmlogwrites | 10 ++++++++-- src/log-writes/replay-log.c | 8 ++++++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/common/dmlogwrites b/common/dmlogwrites index 24a8a25ace277f..7a8a9078cb8b65 100644 --- a/common/dmlogwrites +++ b/common/dmlogwrites @@ -81,7 +81,10 @@ _log_writes_check_bdev() test "$(cat "$sysfs/queue/discard_max_bytes")" -eq 0 && return # dm-thinp guarantees that reads after discards return zeroes - dmsetup status "$blkdev" 2>/dev/null | grep -q '^0.* thin ' && return + if dmsetup status "$blkdev" 2>/dev/null | grep -q '^0.* thin '; then + LOGWRITES_REPLAY_ARGS+=(--discard-zeroes-data) + return + fi echo "HINT: $blkdev doesn't guarantee that reads after DISCARD will return zeroes" >> $seqres.hints echo " This is required for correct journal replay on some filesystems (e.g. xfs)" >> $seqres.hints @@ -110,6 +113,7 @@ _log_writes_init() BLK_DEV_SIZE=$((length / blksz)) fi + LOGWRITES_REPLAY_ARGS=() LOGWRITES_NAME=logwrites-test LOGWRITES_DMDEV=/dev/mapper/$LOGWRITES_NAME LOGWRITES_TABLE="0 $BLK_DEV_SIZE log-writes $blkdev $LOGWRITES_DEV" @@ -161,7 +165,8 @@ _log_writes_replay_log() [ $? -ne 0 ] && _fail "mark '$_mark' does not exist" $here/src/log-writes/replay-log --log $LOGWRITES_DEV --replay $_blkdev \ - --end-mark $_mark >> $seqres.full 2>&1 + --end-mark $_mark "${LOGWRITES_REPLAY_ARGS[@]}" \ + >> $seqres.full 2>&1 [ $? -ne 0 ] && _fail "replay failed" } @@ -231,6 +236,7 @@ _log_writes_replay_log_range() echo "=== replay to $end ===" >> $seqres.full $here/src/log-writes/replay-log -vv --log $LOGWRITES_DEV \ --replay $blkdev --limit $(($end + 1)) \ + "${LOGWRITES_REPLAY_ARGS[@]}" \ >> $seqres.full 2>&1 [ $? -ne 0 ] && _fail "replay failed" } diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c index 968c82ab64a9ad..e07401f63af573 100644 --- a/src/log-writes/replay-log.c +++ b/src/log-writes/replay-log.c @@ -18,6 +18,7 @@ enum option_indexes { FIND, NUM_ENTRIES, NO_DISCARD, + DISCARD_ZEROES_DATA, FSCK, CHECK, START_MARK, @@ -37,6 +38,7 @@ static struct option long_options[] = { {"find", no_argument, NULL, 0}, {"num-entries", no_argument, NULL, 0}, {"no-discard", no_argument, NULL, 0}, + {"discard-zeroes-data", no_argument, NULL, 0}, {"fsck", required_argument, NULL, 0}, {"check", required_argument, NULL, 0}, {"start-mark", required_argument, NULL, 0}, @@ -155,6 +157,7 @@ int main(int argc, char **argv) int ret; int print_num_entries = 0; int discard = 1; + int use_kernel_discard = 0; enum log_replay_check_mode check_mode = 0; while ((c = getopt_long(argc, argv, "v", long_options, @@ -242,6 +245,9 @@ int main(int argc, char **argv) case NO_DISCARD: discard = 0; break; + case DISCARD_ZEROES_DATA: + use_kernel_discard = 1; + break; case FSCK: fsck_command = strdup(optarg); if (!fsck_command) { @@ -299,6 +305,8 @@ int main(int argc, char **argv) if (!discard) log->flags |= LOG_IGNORE_DISCARD; + if (!use_kernel_discard) + log->flags |= LOG_DISCARD_NOT_SUPP; log->start_sector = start_sector; log->end_sector = end_sector; From patchwork Fri Nov 22 16:52:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883406 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 092B922339; Fri, 22 Nov 2024 16:52:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294322; cv=none; b=KICuRXeef4Yr02gi7cZA2AvPusTF+kAICxROr3qsea0jn/PK0zYliSmFzi2vHv9Hy3W0e6nT2dVOiTrYrz2gU0WZ0i9U11LM/9PdsFOgbCh+tXcEktfZGbtjg393R0YmnmZ4WC6omEk5sjUdSFexTEWg8SjNHMtMpDUJ7LzDRvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294322; c=relaxed/simple; bh=KD0t3PuSiDqGGJor5+114IwU8rWTQoK7E4io5F5i2Y4=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YHrASZMQ+gSmOB04KpjzH4GPffCzS1hcY1XfLUdjThrdCMMKROuRjQT10JlvF2nlnUrdWx27k0RZxEsHuP5Xqo84H85c5W6wnLogg6f1Q2MoBnYcVG3fUdCmfOFsuKuVhy6SfpIi4TsDwAlJmLoB6xIHCAQ+VDoLXvv+3D9+/xk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AfptBZVi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AfptBZVi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C6F4C4CECE; Fri, 22 Nov 2024 16:52:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294321; bh=KD0t3PuSiDqGGJor5+114IwU8rWTQoK7E4io5F5i2Y4=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=AfptBZViF17z2uaTQim2C8JKYsHQUFjyakFA8IbbewyCjcdo30mU7xRginKoOFxhk hMQu2L7B0+gtFF8m3IZRaS9wvwytyDzNg6PoHlE9YPiUWthz7dVTzmcBXCEbdK22c0 DzVzwWYZP54rRH9Q7wmVrEFPdDjzn2X6MS0QR7C6AHihiIIKAfvWPUfO9OwcQbxefE LRY7L9+FBzwd7d23Bc6DT0zgYZSTrFlP6Uu+gBktlsMCL2FIDIbfkn3a6tBHK0O8W9 1insqMq9MHZV42t9FswzSjmqguOujOM5ExclKYwuC8OCW8eaqC2tsPNndEb+dvETzD kPwHvF29Qhssw== Date: Fri, 22 Nov 2024 08:52:01 -0800 Subject: [PATCH 06/17] xfs/113: fix failure to corrupt the entire directory From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420102.358248.14679957569173990136.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong This test tries to corrupt the data blocks of a directory, but it doesn't take into account the fact that __populate_check_xfs_dir can remove enough entries to cause sparse holes in the directory. If that happens, this "file data block is unmapped" logic will cause the corruption loop to exit early. Then we can add to the directory, which causes the test to fail. Instead, create a list of mappable dir block offsets, and run 100 corruptions at a time to reduce the amount of time we spend initializing xfs_db. This fixes the regressions that I see with 32k/64k block sizes. Cc: # v2022.05.01 Fixes: c8e6dbc8812653 ("xfs: test directory metadata corruption checking and repair") Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/xfs/113 | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/tests/xfs/113 b/tests/xfs/113 index 094ab71f2aefec..22ac8c3fd51b80 100755 --- a/tests/xfs/113 +++ b/tests/xfs/113 @@ -52,13 +52,34 @@ _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail echo "+ check dir" __populate_check_xfs_dir "${inode}" btree +dir_data_offsets() { + _scratch_xfs_db -c "inode ${inode}" -c 'bmap' | \ + awk -v leaf_lblk=$leaf_lblk \ + '{ + if ($3 >= leaf_lblk) + exit; + for (i = 0; i < $8; i++) + printf("%d\n", $3 + i); + }' +} + echo "+ corrupt dir" -loff=0 -while true; do - _scratch_xfs_db -x -c "inode ${inode}" -c "dblock ${loff}" -c "stack" | grep -q 'file data block is unmapped' && break - _scratch_xfs_db -x -c "inode ${inode}" -c "dblock ${loff}" -c "stack" -c "blocktrash -x 32 -y $((blksz * 8)) -z ${FUZZ_ARGS}" >> $seqres.full - loff="$((loff + 1))" -done +subcommands=() +while read loff; do + # run 100 commands at a time + if [ "${#subcommands[@]}" -lt 600 ]; then + subcommands+=(-c "inode ${inode}") + subcommands+=(-c "dblock ${loff}") + subcommands+=(-c "blocktrash -x 32 -y $((blksz * 8)) -z ${FUZZ_ARGS}") + continue + fi + + _scratch_xfs_db -x "${subcommands[@]}" >> $seqres.full + subcommands=() +done < <(dir_data_offsets) +if [ "${#subcommands[@]}" -gt 0 ]; then + _scratch_xfs_db -x "${subcommands[@]}" >> $seqres.full +fi echo "+ mount image && modify dir" if _try_scratch_mount >> $seqres.full 2>&1; then From patchwork Fri Nov 22 16:52:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883407 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C238A22339; Fri, 22 Nov 2024 16:52:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294337; cv=none; b=FIOYttvapJwDlj3UCQZ+BLFCWIQudZ17PIkIqfC9yb9IP6tdSEelWSHiuSBRGozRVnj+35MKVx75TsV+hwKv5d8NGXPHBd/pJ6kW5WxhKow/uf9nmMnlAu7LGS3tS5jYCsSS99hQncI0jSbEMDoDDZtjOVGt/KC85toub9mbpOk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294337; c=relaxed/simple; bh=cSj5vLFu+VCywZD0eb6RbpiB0kdG47gWWWDsOEAE7yw=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rVaSKOjT3VPSuQYYMyPTSLqA1aWgmQK4qgBQLW3nvtMAZhJR6undvLdA8yoB4IbowVQFHevluYzQNjLUVPgOLsMg4P4zZs+C9WGZiNLGKkjo3pdlL8mm2/Eqq7ccuPgNmEogEJmwJycB6JQtD0Vm4aRvWlMyYUuHJ7zLznU1DqU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qE2MYcly; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qE2MYcly" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4362BC4CECE; Fri, 22 Nov 2024 16:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294337; bh=cSj5vLFu+VCywZD0eb6RbpiB0kdG47gWWWDsOEAE7yw=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=qE2MYclydg2KBzz6FWrA8KEr00zqEoTIm//eY2PM9lnGHdnQ802+NReFKQ1hFJkMT qTN8DNXkWG8VpYxO8uFsPdoPS5li8nAMKWsTC4EJJrZ+t+jnA59Yx1XnZrDkiEkK78 ZymRt20mtEZ//wXHmklPkejuoRycLNCXDCokl4Yrfc60eGiJ4tGEMerYcOLDgN6+Wm wkymdxJonBt4JTFen1acMUi5XurrlQ/HtRl/obRTkmHQNBNe6qpVOf45+gaL5HuqcP tK3pg9e6LTF9xh9rE9Tiw2tOlvAPq1wsnHKCPVqm7DChqMZ7ssbUewwFzzfFYCIurL 8/P1tEbXQ7vng== Date: Fri, 22 Nov 2024 08:52:16 -0800 Subject: [PATCH 07/17] xfs/508: fix test for 64k blocksize From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420117.358248.8488570912527103936.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong It turns out that icreate transactions will try to reserve quite a bit of space on a 64k fsblock filesystem -- enough to handle the worst case parent directory expansion, a new inode chunk, and these days a parent pointer as well. This can work out to quite a bit of space: fsblock reservation 1k 172K 4k 368K 16k 1136K 64k 3650K Unfortunately, this test sets its block quota limits at 1-2MB, so we can't even create a child file. Bump the limits up by 10x so that this test will pass even if there's more metadata size creep in the future. Fixes: f769a923f576df ("xfs: project quota ineritance flag test") Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/xfs/508 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/xfs/508 b/tests/xfs/508 index ee1a0371db7d6d..1bd13e98c9f641 100755 --- a/tests/xfs/508 +++ b/tests/xfs/508 @@ -44,7 +44,7 @@ do_quota_nospc() local exp=$2 echo "Write $file, expect $exp:" | _filter_scratch - $XFS_IO_PROG -t -f -c "pwrite 0 5m" $file 2>&1 >/dev/null | \ + $XFS_IO_PROG -t -f -c "pwrite 0 50m" $file 2>&1 >/dev/null | \ _filter_xfs_io_error rm -f $file } @@ -56,7 +56,7 @@ _require_prjquota $SCRATCH_DEV mkdir $SCRATCH_MNT/dir $QUOTA_CMD -x -c 'project -s test' $SCRATCH_MNT >>$seqres.full 2>&1 -$QUOTA_CMD -x -c 'limit -p bsoft=1m bhard=2m test' $SCRATCH_MNT +$QUOTA_CMD -x -c 'limit -p bsoft=10m bhard=20m test' $SCRATCH_MNT # test the Project inheritance bit is a directory only flag, and it's set on # directory by default. Expect no complain about "project inheritance flag is From patchwork Fri Nov 22 16:52:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883408 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 728F422339; Fri, 22 Nov 2024 16:52:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294353; cv=none; b=JYUgSKOeLLUMjh5lrrFHV7ig+P9NVw42ihK3H+Ecsf2jaspdww8HhousTVIMf0yFgoPy+A4Lz+W7Ph+rorOl5WCpCpEL9iG6Adb1r4oRQsJRekhrH3UkQODiNcjrAyPgKMoZZ0oigwA5ZveGTeNfTnS/B3N2STi5TVWuFEBZ+Q0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294353; c=relaxed/simple; bh=VLe+iO2CGjCnXmc18PilZKB7CW203XR0IjhQyPHSbvU=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bOkDEOZo+UVe65W5wVSOz884lNFF81MNYEe2DsT5tNTapP20ln33eAhUVKSgOXosi/SshsLr7F2krgMGLoBCfreEnxwAOgXvjEy5XOPKo1nsbS3w4WbJDA7+Zu3CfqdZWmvJCfuTj3/rT9Bu2JAtxoShPW83TZy7dvXfz9G/1Oo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=b+TmqGHz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="b+TmqGHz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E59FAC4CECE; Fri, 22 Nov 2024 16:52:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294353; bh=VLe+iO2CGjCnXmc18PilZKB7CW203XR0IjhQyPHSbvU=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=b+TmqGHzRSvj5mDhWa9z5h0HKwiCpNXJgtVTfEq9YjoF+dcorFaxoDIyDC+RrqxUN w7OAD6VzWcf+uJ2pXf76GJIszwBk/H9iMDhWMVlfwo376XFspE06/GNM5ctDGz+8iG fBms4seDqlpPlZtuwCcMoEZbXVcq+UUNEZxM9Cmn/zmuPMszjv2w/6x19ABovP6WFl EaruX3MW4tOj1y03w5zpbHposADj5XhOhvFaJJCHqABWIuiN6OAp6oi/T8kdntWfp1 Cp4SegJPixvWtXo34vBZg4MeQC/JkR/YB85EXpmKYJd5w/zpEUySc4l2NdFNxUMOFZ YcAd3mPMRObGg== Date: Fri, 22 Nov 2024 08:52:32 -0800 Subject: [PATCH 08/17] common/rc: capture dmesg when oom kills happen From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420132.358248.17693136056902875765.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Capture the dmesg output if the OOM killer is invoked during fstests. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- common/rc | 1 + 1 file changed, 1 insertion(+) diff --git a/common/rc b/common/rc index 2ee46e5101e168..70a0f1d1c6acd9 100644 --- a/common/rc +++ b/common/rc @@ -4538,6 +4538,7 @@ _check_dmesg() -e "INFO: possible circular locking dependency detected" \ -e "general protection fault:" \ -e "BUG .* remaining" \ + -e "oom-kill" \ -e "UBSAN:" \ $seqres.dmesg if [ $? -eq 0 ]; then From patchwork Fri Nov 22 16:52:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883409 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06F5522339; Fri, 22 Nov 2024 16:52:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294369; cv=none; b=Qa8uk73jkU8IXpRmsEbE1VnFQpdYLw3KNaR+OYgbCWk38bOQe/uBXgjSetyKoUpfCQC/MFxk2JQEfglNC7/+ehWNlo3616i2+NyyyTkRc4FvcGy2ZOjJVdbwv79CVMtghkqLFYzR/x4KeaorhlCk/Nz/qkXceviiQr6BAsxTk/s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294369; c=relaxed/simple; bh=jtDg7rV60EHbv2fundEmPAeiRtCpunRCaDZWcIOnfxo=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=WXRhy9H4zuc7wBrJH8m8Iy3Go6EYutJarEY2Gf8rToFx2e2PwXT1n/YGC2tItWu29voBFNu6IDUQY5kwsCAJXM6AfNcdS/ClEHcZWT24sjIztR4WTq3vjLmomA3KV6z7+3ISRqNSZ1MTE8lAzffdlpZELgA5kjWgCdP5Cdw/g68= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nBTyq7ai; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nBTyq7ai" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81F51C4CECE; Fri, 22 Nov 2024 16:52:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294368; bh=jtDg7rV60EHbv2fundEmPAeiRtCpunRCaDZWcIOnfxo=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=nBTyq7aiHEeVFYR42FdzZq1yhWg1m1FfRdP340nG8hXmV4VVGdvZSJQlTxmLDPkBd zSL9a8XR6VzwJb6xSGt5L4/jCs4/fusLvEw1tBwDRTMDxWzN9CCq8njrhAeANjSB9+ 0MOJybkhnCN0KssjVLdGmFh+G8PidO5ygm/csAhE9gFKMUiAvdH2d5rcbwEPgSYuTQ sd67THhY6L6mcmSxEIXOd1CGhoahJ5cYUHcwJ2Kn7ntbF/9zgCa7qaH1Vy9pY+IUFU C/CpifjUHenS+vYNeS3SXH1UtGLqvkbjxZgn8ELGd5B8ZgalCW0KLqvmMaxleyOaZ/ q+fejYjxwQDXg== Date: Fri, 22 Nov 2024 08:52:48 -0800 Subject: [PATCH 09/17] generic/562: handle ENOSPC while cloning gracefully From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420148.358248.4652252209849197144.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong This test creates a couple of patterned files on a tiny filesystem, fragments the free space, clones one patterned file to the other, and checks that the entire file was cloned. However, this test doesn't work on a 64k fsblock filesystem because we've used up all the free space reservation for the rmapbt, and that causes the FICLONE to error out with ENOSPC partway through. Hence we need to detect the ENOSPC and _notrun the test. That said, it turns out that XFS has been silently dropping error codes if we managed to make some progress cloning extents. That's ok if the operation has REMAP_FILE_CAN_SHORTEN like copy_file_range does, but FICLONE/FICLONERANGE do not permit partial results, so the dropped error codes is actually an error. Therefore, this testcase now becomes a regression test for the patch to fix that. Signed-off-by: "Darrick J. Wong" --- tests/generic/562 | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/generic/562 b/tests/generic/562 index 91360c4154a6a2..36bd02911c96b8 100755 --- a/tests/generic/562 +++ b/tests/generic/562 @@ -15,6 +15,9 @@ _begin_fstest auto clone punch . ./common/filter . ./common/reflink +test "$FSTYP" = "xfs" && \ + _fixed_by_kernel_commit XXXXXXXXXX "xfs: don't drop errno values when we fail to ficlone the entire range" + _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch" @@ -48,8 +51,11 @@ while true; do done # Now clone file bar into file foo. This is supposed to succeed and not fail -# with ENOSPC for example. -_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full +# with ENOSPC for example. However, XFS will sometimes run out of space. +_reflink $SCRATCH_MNT/bar $SCRATCH_MNT/foo >>$seqres.full 2> $tmp.err +cat $tmp.err +test "$FSTYP" = "xfs" && grep -q 'No space left on device' $tmp.err && \ + _notrun "ran out of space while cloning" # Unmount and mount the filesystem again to verify the operation was durably # persisted. From patchwork Fri Nov 22 16:53:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883410 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 975D722339; Fri, 22 Nov 2024 16:53:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294384; cv=none; b=Qbdo3cZmHyod2U4+Swglff5+FMRgIkj0Wx0SsGMcikTOaDdDKQW5CWRUCQy/Z2sSzQLt9eEIaiNQYZUFeyhVcphpJC0Jd1AVCl36tLMo139aVQcB8rRcLnYR8FB9ywbZASTzzec4L410VWB8WDFLY3OKfz+jn0Ix57bRKYgbGbc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294384; c=relaxed/simple; bh=gExprrLIh6QkDIw05PgeX5U6Q/m9Rxofjk2IItTszR8=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nwjJ6wi17yV9UqtiP4yo9ALrMCaUOLeFO5NuPMjZ0xmMHFLX3GCg+tXYqRo9M1E1m6m0KxUmdrLSj+GMJuOmW/psqGYczC/FfHHHiT6PH0+r7q8z8HzsXoW+DUCjpeJ4jsEfz0IzFVDb3oamfUDGzBNIoByi7chnlYcw+2Wuibo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DSm87hm9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DSm87hm9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 230B4C4CECE; Fri, 22 Nov 2024 16:53:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294384; bh=gExprrLIh6QkDIw05PgeX5U6Q/m9Rxofjk2IItTszR8=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=DSm87hm9vvEJHLFbaHSVJu7dgVbNdxGdl4zudq4cZtOjzrWg35ZaOEI88/99+MZHA JRNWj/7xbxgkIEANtVI0GSON6LtSyTEIksGs2dUMElQqCt1ewDWORPRGLg4iZtNm4z SKp7a2r2/d/jdR+QDYzUqKglgZVO0yBIliJGq+KeOUUK4vHJwUWRwabae7ukuy7Mft uvfIfUGSrL2jhlj4lsuzvg2B4EvZc1B1VR3tH78EcHUHxXIdsOZDyebFaL4vdw9r3V jsBET+ECLpn3w/U8XJx18d3OcgMIUl7HK7kaizmaWL5QDsOMALXrF28g0dtqjdJBp8 rGL2f8f00OdCw== Date: Fri, 22 Nov 2024 08:53:03 -0800 Subject: [PATCH 10/17] xfs/163: skip test if we can't shrink due to enospc issues From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: hch@lst.de, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420163.358248.12881809744914882048.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong If this test fails due to insufficient space, skip this test. This can happen if a realtime volume is enabled on the filesystem and we cannot shrink due to the rtbitmap. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/xfs/163 | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/tests/xfs/163 b/tests/xfs/163 index 2bd94060222f96..75c3113dc2fd03 100755 --- a/tests/xfs/163 +++ b/tests/xfs/163 @@ -17,13 +17,20 @@ _begin_fstest auto quick growfs shrinkfs test_shrink() { - $XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1 + $XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT &> $tmp.growfs ret=$? _scratch_unmount _check_scratch_fs _scratch_mount + # If we couldn't shrink the filesystem due to lack of space, we're + # done with this test. + [ $1 -ne $dblocks ] && \ + grep -q 'No space left on device' $tmp.growfs && \ + _notrun "Could not shrink due to lack of space" + cat $tmp.growfs >> $seqres.full + $XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null . $tmp.growfs [ $ret -eq 0 -a $1 -eq $dblocks ] From patchwork Fri Nov 22 16:53:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883411 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 239107082F; Fri, 22 Nov 2024 16:53:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294400; cv=none; b=CrZd+LjZiHMAkPSBfUfC2H20lmDc+w82vmgz58imWdFFdL2T4kfbxtbOnBG6OF1SsAsBL4KlE60B4MwnQjtVb3XY5vWQvVwnQIfplgLHDplkpB9GXRLJkw57Tk/ag8psqX2g+UBfUw5e6RmXyDE9IhHWDb9Eyq8kETmTrG84F2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294400; c=relaxed/simple; bh=BJP7oRpeUsyY8KQ7I8nt5/4GCsG2VhbI6uP5nDt1imE=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=N8aTrnrPSKCcrHHtoiy+yxALsOGjH94KPYtdkznxBjTcx7t7MewUY0bdD/Aww77UGuzp78fS8FEp/qhNfdQNnvYPJNCuMl4maTs7Yc4KxJcTV3MhcHHYPvueKzu7DX3qH0d6oYBVdIEG9nEczMmPY34zD4GPf3czCNrmWnyNlpE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jbHo0Y9+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jbHo0Y9+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B78D3C4CECE; Fri, 22 Nov 2024 16:53:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294399; bh=BJP7oRpeUsyY8KQ7I8nt5/4GCsG2VhbI6uP5nDt1imE=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=jbHo0Y9+A6VKjj9vFJ1U9PliycB6j7mX62K+V+SZWac22+OtQ//NvMhNTe+Dy6yUQ zFHim74mB+uRRQN6aK2JPD5aL/PyXKj+F7zfrHglpLiSpA81hxOvgC93M59tCu3Yk4 tt5nUg3p4ULNppXWwTFuwI8IiJDOhFMfYt10qP17Q7xdKL9XDzrSEe0J2lFgZL5iWn aRXqv4WkkSZM1X/h/+oV2ThtXvXX3DntyqYCYOWjnXxnf01Oz1RHuR1cfyh6UGlvKa RkmzKn/9m78H7dtAEbXV2jY/wV4Ba7KGrLYGjyKXIJyHhrCu2e+rGo3m482OaayZz+ C2bmBT39rU2Dg== Date: Fri, 22 Nov 2024 08:53:19 -0800 Subject: [PATCH 11/17] xfs/009: allow logically contiguous preallocations From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: hch@lst.de, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420178.358248.14522170594906217361.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong The new rtgroups feature implements a simplistic rotor to pick the rtgroup for an initial allocation to a file. This causes test failures if the preallocations are spread across two rtgroups, which happens if there are more subtests than rtgroups. One way to fix this would be to reset the rotor then each subtest starts allocating from rtgroup 0, but the only way to do that is to cycle the scratch mount, which is a bit gross. Instead, report logically contiguous mappings as a single mapping even if the physical space is not contiguous. Unfortunately, there's not enough context in the comments to know if the test actually was checking for physical contiguity? Or if this is just an exerciser of the old preallocation calls, and it's fine as long as the file ranges are mapped (or unmapped) as desired. Messing with some awk is a lot cheaper than umount/mount cycling. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/xfs/009 | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/tests/xfs/009 b/tests/xfs/009 index dde505f079f4f8..bb42ce32490df5 100755 --- a/tests/xfs/009 +++ b/tests/xfs/009 @@ -49,13 +49,26 @@ _filesize() _block_filter() { $AWK_PROG -v bsize="$bsize" ' + BEGIN { + br_pos = 0 + br_len = 0 + } + function dump_blockrange() { + if (br_len == 0) + return + printf(" [%d,%d]: BLOCKRANGE\n", br_pos, br_len) + br_pos = 0 + br_len = 0 + } /blocksize/ { + dump_blockrange() printf(" blocksize BSIZE\n") next } /CMD/ { + dump_blockrange() split($3, off, "=") offset = strtonum(off[2]) if (offset != -1) @@ -72,6 +85,7 @@ _block_filter() } /MAP/ { + dump_blockrange() split($2, off, "=") offset = strtonum(off[2]) if (offset != -1) @@ -90,6 +104,7 @@ _block_filter() } /TRUNCATE/ { + dump_blockrange() split($2, off, "=") offset = strtonum(off[2]) / bsize @@ -99,16 +114,28 @@ _block_filter() } /\[[0-9]+,[0-9]+\]:/ { - printf(" %s BLOCKRANGE\n", $1) + rangestr = gensub(/\[([0-9]+),([0-9]+)\]:/, "\\1,\\2", "g", $1); + split(rangestr, off, ",") + if (br_pos + br_len == off[1]) { + br_len += off[2]; + } else { + dump_blockrange() + br_pos = off[1]; + br_len = off[2]; + } next } { + dump_blockrange() print next } + END { + dump_blockrange() + } ' } From patchwork Fri Nov 22 16:53:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883412 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 829E422339; Fri, 22 Nov 2024 16:53:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294415; cv=none; b=ZUpiYGi7VYwkGq/p1MgDy922nwhcl2qlbVIENuTNcAivreelvrsxun+eOhSFRVOelukeUJSBcIRXMXNpwQM/YIZA0/7r3ztq4eRMuveD7SfSkttiUK4fLZrDLkeoh36eYB+tVNYzbJyZD5DSOKwKStDbVONn2BgzCAlH153jxS8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294415; c=relaxed/simple; bh=y8PpZm+HcG5jNGFNt/C/yNnVzaez8vzAR40bqTAe2d0=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=l55Pai6RuGDhRfavqfCX8F9BT3hLou4AxDPFyiQutJLoRR3PtnU6ygL6ZHtBOzBouxJFjz/TR9B7u6z0tW9FkguxCo2heKGTqE3Hl7PKTr+BKulgue7zwRCn6ZxOeH+V9JCVKAnRCeFEBTI0sU1SHfNGkbm5vkFjK1yZweWRxvg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pv6uSL1l; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pv6uSL1l" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59902C4CECE; Fri, 22 Nov 2024 16:53:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294415; bh=y8PpZm+HcG5jNGFNt/C/yNnVzaez8vzAR40bqTAe2d0=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=pv6uSL1lEVy6PG1XbRD3sjrjr/5FlRs2XbdrVtTZCXNzW40LR3JNPFhC3R5QRnWMn p/9KLEyY9Y4fc0GRLRd5/yMVtNSufB3huGO84TfF3SRa2VjRAZKzV+7w1kIfF5he1v hOCxdn6WcIqVvLra+IjxUXkGUtFCqJX2DfIwC/JscVSiQohPdD/kl+yrSxvfDjAISh vQZeeMk9nccQ6jnPE9pZFZ1Vv6jfK7wm9kD/wVqG8mDOeaehboyfOVwh7x55KyNEPI kxmR/5wJ0HSdd7hc5IJ7+muqB51hjkPQgVXr6LwqNbBroUYfXl0xHM9HwiHVK8IEQH OaadDKOH4JonA== Date: Fri, 22 Nov 2024 08:53:34 -0800 Subject: [PATCH 12/17] generic/251: use sentinel files to kill the fstrim loop From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: hch@lst.de, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420194.358248.7959632387252983213.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Apparently the subshell kill doesn't always take, and then the test runs for hours and hours because nothing stops it. Instead, use a sentinel file to detect when fstrim_loop should stop execing background fstrims. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/251 | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/generic/251 b/tests/generic/251 index b432fb11937911..d59e91c3e0a33a 100755 --- a/tests/generic/251 +++ b/tests/generic/251 @@ -125,12 +125,15 @@ fstrim_loop() wait $fpid fi while [ $start -lt $fsize ] ; do + test -s $tmp.fstrim_loop || break $FSTRIM_PROG -m ${minlen}k -o ${start}k -l ${step}k $SCRATCH_MNT & fpid=$! wait $fpid start=$(( $start + $step )) done + test -s $tmp.fstrim_loop || break done + rm -f $tmp.fstrim_loop } function check_sums() { @@ -188,6 +191,7 @@ find -P . -xdev -type f -print0 | xargs -0 md5sum | sort -o $tmp/content.sums echo -n "Running the test: " pids="" +echo run > $tmp.fstrim_loop fstrim_loop & fstrim_pid=$! p=1 @@ -199,8 +203,10 @@ done echo "done." wait $pids -kill $fstrim_pid -wait $fstrim_pid +truncate -s 0 $tmp.fstrim_loop +while test -e $tmp.fstrim_loop; do + sleep 1 +done status=0 From patchwork Fri Nov 22 16:53:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883413 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27E0B22339; Fri, 22 Nov 2024 16:53:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294431; cv=none; b=LKDBjtKn+mLWVU886aLb0+qxvrXkwnFKejpjD77A8TZgmxYoQOxix9GTsz96WwO8ZDuGwFdw7dc4M1gfuQzcB4ZMN6qZTZBVYW6ey5WUcMDez7nyBu0MyzIkpUyZuIx3Esh+xxwGVrHh8mVLeH6JJ7CzdcxnuFFYJ0RPBka4hmU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294431; c=relaxed/simple; bh=FqXto+XvEKJb19Mg6eHjG09gKuWcl9gRB9HoBxKkqIQ=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OD/6Uca01TmURQv50fz2ifFAkOCqND2yVorzDp3YQ+5NhSFxOi/XKMa3Xz6HuASw/94NDyUCeX14EpaWcViz4Fv33GAyLrRs0nsJd5wjVyJEaepi9MoEy+l++SIrBV1Mu0YQCgG3ntqCRhizyUyHa3Kn/Rp3vVEuy0K/67RBncI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OlQ7JRQi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OlQ7JRQi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02B5DC4CECE; Fri, 22 Nov 2024 16:53:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294431; bh=FqXto+XvEKJb19Mg6eHjG09gKuWcl9gRB9HoBxKkqIQ=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=OlQ7JRQieuj8nz+x+UIjZ0HGKLF4HcAYqC/+didKK/siPz/iqMRqsWcr7QrDC3mCf p5XHtxlKfAoVTxOfUohQh4w5+plrWKNRDXIulde78kedPTMzdvv43OosAbUoHH8zRn MdCFPAQ92jnW3yUAQD04xI9fbT7pgYjpi2GvBYeV8Ys2nn0imN2r9w+9ro5HwOkv8k lk4LvDxssjyXFR8acaiukbpIISZ93qHTwBH4gtBnlekpSYxgVSzWqMp793QhA/wwMV LJHnOLlnw99yffREzkQB/Vshyy+nLhBv+/EFNSICUQ1vHMsaOH8Jkbgmdp7YIPWAHJ PleEXG9h/yzvA== Date: Fri, 22 Nov 2024 08:53:50 -0800 Subject: [PATCH 13/17] generic/251: constrain runtime via time/load/soak factors From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420209.358248.6636819948700619006.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong On my test fleet, this test can run for well in excess of 20 minutes: 613 generic/251 616 generic/251 624 generic/251 630 generic/251 634 generic/251 652 generic/251 675 generic/251 749 generic/251 777 generic/251 808 generic/251 832 generic/251 946 generic/251 1082 generic/251 1221 generic/251 1241 generic/251 1254 generic/251 1305 generic/251 1366 generic/251 1646 generic/251 1936 generic/251 1952 generic/251 2358 generic/251 4359 generic/251 5325 generic/251 34046 generic/251 because it hardcodes 20 threads and 10 copies. It's not great to have a test that results in a significant fraction of the total test runtime. Fix the looping and load on this test to use LOAD and TIME_FACTOR to scale up its operations, along with the usual SOAK_DURATION override. That brings the default runtime down to less than a minute. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/251 | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/tests/generic/251 b/tests/generic/251 index d59e91c3e0a33a..b4ddda10cef403 100755 --- a/tests/generic/251 +++ b/tests/generic/251 @@ -15,7 +15,6 @@ _begin_fstest ioctl trim auto tmp=`mktemp -d` trap "_cleanup; exit \$status" 0 1 3 trap "_destroy; exit \$status" 2 15 -chpid=0 mypid=$$ # Import common functions. @@ -151,29 +150,28 @@ function check_sums() { function run_process() { local p=$1 - repeat=10 + if [ -n "$SOAK_DURATION" ]; then + local duration="$SOAK_DURATION" + else + local duration="$((30 * TIME_FACTOR))" + fi + local stopat="$(( $(date +%s) + duration))" - sleep $((5*$p))s & - export chpid=$! && wait $chpid &> /dev/null - chpid=0 - - while [ $repeat -gt 0 ]; do + sleep $((5*$p))s + while [ "$(date +%s)" -lt "$stopat" ]; do # Remove old directories. rm -rf $SCRATCH_MNT/$p - export chpid=$! && wait $chpid &> /dev/null # Copy content -> partition. mkdir $SCRATCH_MNT/$p cp -axT $content/ $SCRATCH_MNT/$p/ - export chpid=$! && wait $chpid &> /dev/null check_sums - repeat=$(( $repeat - 1 )) done } -nproc=20 +nproc=$((4 * LOAD_FACTOR)) # Copy $here to the scratch fs and make coipes of the replica. The fstests # output (and hence $seqres.full) could be in $here, so we need to snapshot @@ -194,11 +192,9 @@ pids="" echo run > $tmp.fstrim_loop fstrim_loop & fstrim_pid=$! -p=1 -while [ $p -le $nproc ]; do +for ((p = 1; p < nproc; p++)); do run_process $p & pids="$pids $!" - p=$(($p+1)) done echo "done." From patchwork Fri Nov 22 16:54:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883419 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F30195D477; Fri, 22 Nov 2024 16:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294448; cv=none; b=jxJfn5xpbzid5jkxb3qh0HlHUKK+UjyHpVcmyDr5qLBFShmj+3He+Du62XBd1oE499QsBABlH+9/B88YcTBzLXBnQvNfjsIMT34j+4VoOx6YXBGuBYYHYziUYE0gWN61sB2u4zHPSE9ZwJwwozOI1O9meClsRsTK7w8RNOyzsK4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294448; c=relaxed/simple; bh=UQN4iBjjS9X0X0vwLd34ITFdSrcN8+d5IPfqQNonASI=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qPMnk9ygnVJ08wQHYkgADhx/h6mq80x6xtbVcPzpmp8rWdIB7HP97SqLaUlfAYe9OQ2dfRFE8vA6Kaf9mU7kKnw5rmX4Vz03jbPeahp824GcwE9R/ilJgf7yto8FH1vof3aGC9sTMV/UMH1IVSb9EMw78Sigwz4fpDwFdK3bE2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XWBU7reF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XWBU7reF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97AC7C4CECE; Fri, 22 Nov 2024 16:54:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294446; bh=UQN4iBjjS9X0X0vwLd34ITFdSrcN8+d5IPfqQNonASI=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=XWBU7reF/UxQpVBuzpxIYpL7vw5x+ytQjH5rBu2iHUNH35A/V4qDaT8+JjdwaGS1P cT+dFMuCx7yAYhiRznXTDqASjzPwBxwFe72/U3wV92eQaXhSt4/8v3v2EP2gFZIIrU zeSnGUv6JeffqVkBrXjoTEyQuK15Dq/eMgB3oIq5a7M12PSQl8UIYxQnsBdvERdIQx JldT7hXAvCAESDUoNUGesTSOeHkUsG3g+nIZR8TsHNCMmm7T2AstqhuRIOPUyNi1uT 4rEaW8sE3l6gHx3s8jACvhd3BtA0zRFX1Zhnp71ak7AHuWuaS7F5i9PSAt3d5HWO3+ 1wEMIKqU4JHTQ== Date: Fri, 22 Nov 2024 08:54:06 -0800 Subject: [PATCH 14/17] generic/251: don't copy the fsstress source code From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420224.358248.12570640675092195188.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Run fsstress for a short time to generate test data to replicate on the scratch device so that we don't blow out the test runtimes on unintentionally copying .git directories or large corefiles from the developer's systems, etc. Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/251 | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/tests/generic/251 b/tests/generic/251 index b4ddda10cef403..ec486c277c6828 100755 --- a/tests/generic/251 +++ b/tests/generic/251 @@ -173,13 +173,11 @@ function run_process() { nproc=$((4 * LOAD_FACTOR)) -# Copy $here to the scratch fs and make coipes of the replica. The fstests -# output (and hence $seqres.full) could be in $here, so we need to snapshot -# $here before computing file checksums. +# Use fsstress to create a directory tree with some variability content=$SCRATCH_MNT/orig mkdir -p $content -cp -axT $here/ $content/ - +FSSTRESS_ARGS=$(_scale_fsstress_args -p 4 -d $content -n 1000 $FSSTRESS_AVOID) +$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full mkdir -p $tmp ( From patchwork Fri Nov 22 16:54:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883420 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C595022339; Fri, 22 Nov 2024 16:54:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294462; cv=none; b=ZB6nbhcP+kPSrppJBZLFOGWBdHqOwSvW+heohv4V+EkJfrz2NJ78TNTkhU5KIelhI0m2AU9nVtS5XC8UBtZWpJziwaTH6/yvoxMWi50Z+ADYZc68ITyjV/igZJyOErVfWEkhA3oWEnja/VR2TOvhWgGyi6Tj0E95gXdwpPLTcPA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294462; c=relaxed/simple; bh=aZ92cBGikadawS733SmHE9j0KKxvkfJZgfCH1/QFwe8=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=r8bZLrtppGAjOq/gl1MErp9XS8Sjc/BOUtKEqgbSnunRmHQLnYTozkW076iDpZaGZOOor6hoiiVa0ccp6Q6MvvLw7QW3GvTWPe28vZDpgtB3pqgS83k/BrXApyZxiCw/fcHmrfYU1wOQwhbM9oWerAUuR9CZUu6do2jiCIW/vp8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=b6QCWvW5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="b6QCWvW5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3EEA7C4CECE; Fri, 22 Nov 2024 16:54:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294462; bh=aZ92cBGikadawS733SmHE9j0KKxvkfJZgfCH1/QFwe8=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=b6QCWvW5DI1ML2nvqXQVV/s7In+4Sj9wqcg/sEj73+eoQXXvHSOQDl+t9vU+8tvXB iBrqj0MWGZBFnwewFkv73e8V9NAZn/iOKtLF9shlvrdy6GsIzf0u4HPehg1jCIpzIm eOcWt5SP3BuDwlwy6twce7jKzVyojvwbsifTy9x4VzTtrz4TwIY3pUBt9Lk7ZFh4ue MRRyvY6+OqEFGOAvXjj4bCipITpL109Pt0kQm7kdT+HtzeA9QvRU2t7vMgwMsV2S4b m51I1uTzGJuGEU6cxvrDBBrkO7w6MA1u/a730xCvDwpnIsKHwd6JfRmtHfh6KgUJ1m XB247kmlb/Oww== Date: Fri, 22 Nov 2024 08:54:21 -0800 Subject: [PATCH 15/17] common/rc: _scratch_mkfs_sized supports extra arguments From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: zlang@kernel.org, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420240.358248.13252517041858687590.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zorro Lang To give more arguments to _scratch_mkfs_sized, we generally do as: MKFS_OPTIONS="-L oldlabel $MKFS_OPTIONS" _scratch_mkfs_sized $fs_size to give "-L oldlabel" to it. But if _scratch_mkfs_sized fails, it will get rid of the whole MKFS_OPTIONS and try to mkfs again. Likes: ** mkfs failed with extra mkfs options added to "-L oldlabel -m rmapbt=1" by test 157 ** ** attempting to mkfs using only test 157 options: -d size=524288000 -b size=4096 ** But that's not the fault of "-L oldlabel". So for keeping the mkfs options ("-L oldlabel") we need, we'd better to let the scratch_mkfs_sized to support extra arguments, rather than using global MKFS_OPTIONS. Signed-off-by: Zorro Lang Reviewed-by: "Darrick J. Wong" [djwong: fix string quoting issues] Signed-off-by: "Darrick J. Wong" --- common/rc | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/common/rc b/common/rc index 70a0f1d1c6acd9..6acacbe4c88eea 100644 --- a/common/rc +++ b/common/rc @@ -1026,11 +1026,13 @@ _small_fs_size_mb() } # Create fs of certain size on scratch device -# _try_scratch_mkfs_sized [optional blocksize] +# _try_scratch_mkfs_sized [optional blocksize] [other options] _try_scratch_mkfs_sized() { local fssize=$1 - local blocksize=$2 + shift + local blocksize=$1 + shift local def_blksz local blocksize_opt local rt_ops @@ -1094,10 +1096,10 @@ _try_scratch_mkfs_sized() # don't override MKFS_OPTIONS that set a block size. echo $MKFS_OPTIONS |grep -E -q "b\s*size=" if [ $? -eq 0 ]; then - _try_scratch_mkfs_xfs -d size=$fssize $rt_ops + _try_scratch_mkfs_xfs -d size=$fssize $rt_ops "$@" else _try_scratch_mkfs_xfs -d size=$fssize $rt_ops \ - -b size=$blocksize + -b size=$blocksize "$@" fi ;; ext2|ext3|ext4) @@ -1108,7 +1110,7 @@ _try_scratch_mkfs_sized() _notrun "Could not make scratch logdev" MKFS_OPTIONS="$MKFS_OPTIONS -J device=$SCRATCH_LOGDEV" fi - ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize $SCRATCH_DEV $blocks + ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize "$@" $SCRATCH_DEV $blocks ;; gfs2) # mkfs.gfs2 doesn't automatically shrink journal files on small @@ -1123,13 +1125,13 @@ _try_scratch_mkfs_sized() (( journal_size >= min_journal_size )) || journal_size=$min_journal_size MKFS_OPTIONS="-J $journal_size $MKFS_OPTIONS" fi - ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS -O -b $blocksize $SCRATCH_DEV $blocks + ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS -O -b $blocksize "$@" $SCRATCH_DEV $blocks ;; ocfs2) - yes | ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize $SCRATCH_DEV $blocks + yes | ${MKFS_PROG} -t $FSTYP -F $MKFS_OPTIONS -b $blocksize "$@" $SCRATCH_DEV $blocks ;; udf) - $MKFS_UDF_PROG $MKFS_OPTIONS -b $blocksize $SCRATCH_DEV $blocks + $MKFS_UDF_PROG $MKFS_OPTIONS -b $blocksize "$@" $SCRATCH_DEV $blocks ;; btrfs) local mixed_opt= @@ -1137,33 +1139,33 @@ _try_scratch_mkfs_sized() # the device is not zoned. Ref: btrfs-progs: btrfs_min_dev_size() (( fssize < $((256 * 1024 * 1024)) )) && ! _scratch_btrfs_is_zoned && mixed_opt='--mixed' - $MKFS_BTRFS_PROG $MKFS_OPTIONS $mixed_opt -b $fssize $SCRATCH_DEV + $MKFS_BTRFS_PROG $MKFS_OPTIONS $mixed_opt -b $fssize "$@" $SCRATCH_DEV ;; jfs) - ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS $SCRATCH_DEV $blocks + ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS "$@" $SCRATCH_DEV $blocks ;; reiserfs) - ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS -b $blocksize $SCRATCH_DEV $blocks + ${MKFS_PROG} -t $FSTYP $MKFS_OPTIONS -b $blocksize "$@" $SCRATCH_DEV $blocks ;; reiser4) # mkfs.resier4 requires size in KB as input for creating filesystem - $MKFS_REISER4_PROG $MKFS_OPTIONS -y -b $blocksize $SCRATCH_DEV \ + $MKFS_REISER4_PROG $MKFS_OPTIONS -y -b $blocksize "$@" $SCRATCH_DEV \ `expr $fssize / 1024` ;; f2fs) # mkfs.f2fs requires # of sectors as an input for the size local sector_size=`blockdev --getss $SCRATCH_DEV` - $MKFS_F2FS_PROG $MKFS_OPTIONS $SCRATCH_DEV `expr $fssize / $sector_size` + $MKFS_F2FS_PROG $MKFS_OPTIONS "$@" $SCRATCH_DEV `expr $fssize / $sector_size` ;; tmpfs) local free_mem=`_free_memory_bytes` if [ "$free_mem" -lt "$fssize" ] ; then _notrun "Not enough memory ($free_mem) for tmpfs with $fssize bytes" fi - export MOUNT_OPTIONS="-o size=$fssize $TMPFS_MOUNT_OPTIONS" + export MOUNT_OPTIONS="-o size=$fssize "$@" $TMPFS_MOUNT_OPTIONS" ;; bcachefs) - $MKFS_BCACHEFS_PROG $MKFS_OPTIONS --fs_size=$fssize $blocksize_opt $SCRATCH_DEV + $MKFS_BCACHEFS_PROG $MKFS_OPTIONS --fs_size=$fssize $blocksize_opt "$@" $SCRATCH_DEV ;; *) _notrun "Filesystem $FSTYP not supported in _scratch_mkfs_sized" @@ -1173,7 +1175,7 @@ _try_scratch_mkfs_sized() _scratch_mkfs_sized() { - _try_scratch_mkfs_sized $* || _notrun "_scratch_mkfs_sized failed with ($*)" + _try_scratch_mkfs_sized "$@" || _notrun "_scratch_mkfs_sized failed with ($*)" } # Emulate an N-data-disk stripe w/ various stripe units From patchwork Fri Nov 22 16:54:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883421 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 665A222339; Fri, 22 Nov 2024 16:54:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294478; cv=none; b=B1/IPF6ZKuaQhh4Ypnt791OQ4Bt7dNvdz2dR1jKafKG5tE57rxjILOjuRyO1fU4282dRhkg1DNb3p4gJuxo660isL/OZe5AuZzRlFuwRq1LrY0sc598ZduNQTA+J77xlUImvGP3DHL25SwGaF2TZvNTx/wvhRwVNVf3+9nhFGi0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294478; c=relaxed/simple; bh=xoZYZcbhaXPVuUUb3eqxkHGXvjFqsgU9jp/QiJ6gyyE=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=se/io7lPm2m8LUJYDxJfPVmsHQOfkHbhjgJwyeNvWTxwJ9t/a1kWCsI7hHmAhQ+dOeJccna5xDc6sytqeURKCXIme76hBi5p8rNG8O7P3l65JsxrZ3vN0+pu/zV8heaR7Dx9l9l21ucgnpY7rdcvp4Kv2Yy0XNukevPFd398Rc4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pp+8XBcY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pp+8XBcY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D64DBC4CECE; Fri, 22 Nov 2024 16:54:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294477; bh=xoZYZcbhaXPVuUUb3eqxkHGXvjFqsgU9jp/QiJ6gyyE=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=pp+8XBcYIHsxZr+a+k9fivDXNxxQqxNCAX6hwFYp99C0WxfXkni/SUsrCywhh2D3v 49XWsDalx8Nv8+a5Wfhtv/hI9Zv60bOjKtN58ROsZxLoU3uUoEy2u/RHb5R2tEWiqp LrOI1RxKILCCanryT/lZtMXRzp48HGP1Lo+S8B0YZy7jPWJ2Z0QzQggvka/ShnXMcA L+LUX07LOw3rYER24F0TWaXxcI5zYOWdjMkIBpW64fvs8Ag3moZ71c1xkxKuqha+Cb TGMTvig/AYMLn4oy68vRGc/G5GWFGoiazjPxp/+VcGaM7Q4Wywc25cO75Rc/7LZEN/ QPHZ2wfMWpp1w== Date: Fri, 22 Nov 2024 08:54:37 -0800 Subject: [PATCH 16/17] xfs/157: do not drop necessary mkfs options From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: zlang@kernel.org, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420255.358248.5980322427420325625.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Zorro Lang To give the test option "-L oldlabel" to _scratch_mkfs_sized, xfs/157 does: MKFS_OPTIONS="-L oldlabel $MKFS_OPTIONS" _scratch_mkfs_sized $fs_size but the _scratch_mkfs_sized trys to keep the $fs_size, when mkfs fails with incompatible $MKFS_OPTIONS options, likes this: ** mkfs failed with extra mkfs options added to "-L oldlabel -m rmapbt=1" by test 157 ** ** attempting to mkfs using only test 157 options: -d size=524288000 -b size=4096 ** but the "-L oldlabel" is necessary, we shouldn't drop it. To avoid that, we give the "-L oldlabel" to _scratch_mkfs_sized through function parameters, not through global MKFS_OPTIONS. Signed-off-by: Zorro Lang Reviewed-by: "Darrick J. Wong" [djwong: fix more string quoting issues] Signed-off-by: "Darrick J. Wong" --- tests/xfs/157 | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/xfs/157 b/tests/xfs/157 index 9b5badbaeb3c76..e102a5a10abe4b 100755 --- a/tests/xfs/157 +++ b/tests/xfs/157 @@ -66,8 +66,7 @@ scenario() { } check_label() { - MKFS_OPTIONS="-L oldlabel $MKFS_OPTIONS" _scratch_mkfs_sized $fs_size \ - >> $seqres.full + _scratch_mkfs_sized "$fs_size" "" -L oldlabel >> $seqres.full 2>&1 _scratch_xfs_db -c label _scratch_xfs_admin -L newlabel "$@" >> $seqres.full _scratch_xfs_db -c label From patchwork Fri Nov 22 16:54:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13883422 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEF1F7082F; Fri, 22 Nov 2024 16:54:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294494; cv=none; b=KXZJS3LBdU/2muV/gm0hFUpnqPolQxzOxYbAk6eWK2NQbiJyv/PSwd/415CDJbVZV/BAJNjIm43z9PwgTNLfc/VOTSGV4EJAkHHm1XSYe0k+a4rXRNThdVPLdMfxla1TDJhWScERLNIeMJd3VLFMQCtOrpXH05k6P14m6pofLpE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732294494; c=relaxed/simple; bh=u/Fh2gjxTg+tmrMRCsGA2+Mcd2OI98kMFiliB2UDlqo=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JlI5vfwjsKji5uvxZvdCMkbmPT2qraeO64ulBbzSu6PrD5yd4cueE+U4SU/eEeTXwT3mNmj3Hlrop/bWajo6YUmdyubuveArGYhTgaKl9xX7wR9RvcHeYKIAfiERbnWFx1hA7RBnBJY8MmO0sfdv7aP1BPqWC5sMuxYP3030an8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VmVUQBLb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VmVUQBLb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6ECDDC4CECE; Fri, 22 Nov 2024 16:54:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1732294493; bh=u/Fh2gjxTg+tmrMRCsGA2+Mcd2OI98kMFiliB2UDlqo=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=VmVUQBLb3DWJIQ4XJTX4g8/9VAnBHcsjEtTb0LNzLqAGeQ7ZAy5d+4pyQ5ejz73/A zDukWogTv2qa2724rmYULJU9XYOKE6PN4M2Zj8iw5AGGA5UiIo18mY/V6kMijFWPzO Y/NFxxlEOfwANRyPnjigu5fcISkuZ2rrZsmyC0FyLnRFNRHwLVEWgJ4s+EC6IXhZZy wbH/h+bAnn2Xk1xK/xyxmIOioyb/EdpYG6nFLPpZylAi2LCdQkBtwiubWH/qArCxcc qx27DrqqFMBM8pY8ZpdtGUSbQx8FUlwDbgt8L8vfxV1XHGRTmYKs4JHnroyNQL/7Iq eNbJTpt1fDvPA== Date: Fri, 22 Nov 2024 08:54:53 -0800 Subject: [PATCH 17/17] generic/366: fix directio requirements checking From: "Darrick J. Wong" To: zlang@redhat.com, djwong@kernel.org Cc: fstests@vger.kernel.org, fstests@vger.kernel.org, linux-xfs@vger.kernel.org Message-ID: <173229420269.358248.6325435085396026110.stgit@frogsfrogsfrogs> In-Reply-To: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> References: <173229419991.358248.8516467437316874374.stgit@frogsfrogsfrogs> Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong On a system with 4k-sector storage devices, this test fails with: --- /tmp/fstests/tests/generic/366.out 2024-11-17 09:04:53.161104479 -0800 +++ /var/tmp/fstests/generic/366.out.bad 2024-11-20 21:02:30.948000000 -0800 @@ -1,2 +1,34 @@ QA output created by 366 +fio: io_u error on file /opt/file1: Invalid argument: read offset=15360, buflen=512 +fio: io_u error on file /opt/file1: Invalid argument: read offset=15360, buflen=512 The cause of this failure is that we cannot do 512byte directios to a device with 4k LBAs. Update the precondition checking to exclude this scenario. Cc: # v2024.11.17 Fixes: 4c1629ae3a3a56 ("generic: new test case to verify if certain fio load will hang the filesystem") Signed-off-by: "Darrick J. Wong" Reviewed-by: Christoph Hellwig --- tests/generic/366 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/generic/366 b/tests/generic/366 index 6e7dd7279218c4..b322bcca72fecc 100755 --- a/tests/generic/366 +++ b/tests/generic/366 @@ -20,7 +20,7 @@ _begin_fstest auto quick rw . ./common/filter _require_scratch -_require_odirect +_require_odirect 512 # see fio job1 config below _require_aio _fixed_by_kernel_commit xxxxxxxxxxxx \