From patchwork Mon Aug 21 06:37:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naohiro Aota X-Patchwork-Id: 13359190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A1CEE49AC for ; Mon, 21 Aug 2023 06:37:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233455AbjHUGhN (ORCPT ); Mon, 21 Aug 2023 02:37:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjHUGhN (ORCPT ); Mon, 21 Aug 2023 02:37:13 -0400 Received: from esa6.hgst.iphmx.com (esa6.hgst.iphmx.com [216.71.154.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A1BCAB; Sun, 20 Aug 2023 23:37:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1692599831; x=1724135831; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rhV3HXwgsDIqHBdHBS2K2HmQpdMw41VPG9k19oglC8M=; b=fhr6hfb5KdMPnt3awZEXRfzLc6Be6CDTBaThBOWdF3XL3/ub7pBWfzS0 gaPVloT0WpHnE0ndmkFMhOADu/Rc1PZ6lKZQmYn57kx2wdMgqGIE/rzjT OaZm0UyhFNICDlRp64kSsJ4vx2VW/sbWJZkK63xb0lG8UBPYGkXqXg2iG Et02LgLYfX97f8uDItq2ltCAfLVukpbmsJKmJvdsLDsZTUyIQzF5DPHiY OWeuuGnabWuveAb2W1kNOrsEllWPUymdQcn8bUaeORaZDvpk5SvNTnQbU bK55Jl7To1Cf92T+vLhe1Fg2yePzwVGNnZA43FkbwMoZFLKCKT160tTq8 w==; X-IronPort-AV: E=Sophos;i="6.01,189,1684771200"; d="scan'208";a="241991951" Received: from uls-op-cesaip02.wdc.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 21 Aug 2023 14:37:10 +0800 IronPort-SDR: kKH9Tz2kdpgPxPBrgjY+0fwRBqSWLxHS6gZqQKBRey8XRax1jgu4QyNZhrv7aAgwBcLqONP55J PdbgyRmwTQr8kWt7ts2NUfa/dOXQrh1gnR+0WbmM2PLQWCaNt7SsyrHvB6OJaVCMuXMXC4Spfo 509NrxetN6WGMWdOXm7V2dPeBNgfEY0evrR4cPyrPgsQ7rNKzYIn7UGicuGSzBqhhCcbDV8hqW kB16D0tsfOS+ALqjWiHhHdNFC3Gp+U0GSLVPEYPjgxtFfY0HYKFYnyjk9OnfTIjSWwR2XWZLyC 66A= Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep02.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 20 Aug 2023 22:44:42 -0700 IronPort-SDR: 3UgSJsiqSUVUIIi54+djIIlXXN5uyQZypysH+PrUV/ye7dQXFxRQZO8HyMkCNGWr4l4uXqxTUx 21T+LrOOznluzRtISlADbDoR6mIgwNw4hUkUkOrvJWiwYqRn4mpAmOj2fiA45fK5ctpURGIPd6 InQoyQkXyz3Lp34xe+sXukdHdAeel1zOXpv/8SdwGD/3R8+jDmn8gBXgzWA/moIgC2KY+276rn LYm5YYniMGSXSWb231iIFf6YE/HDIlvg6sElSRV9VIFI+RkOa5vciqZTeVwu3r5+CzdCzpcXTY WYA= WDCIronportException: Internal Received: from unknown (HELO naota-xeon.wdc.com) ([10.225.163.96]) by uls-op-cesaip02.wdc.com with ESMTP; 20 Aug 2023 23:37:11 -0700 From: Naohiro Aota To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Naohiro Aota Subject: [PATCH v2 1/3] common/rc: introduce _random_file() helper Date: Mon, 21 Aug 2023 15:37:02 +0900 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Currently, we use "ls ... | sort -R | head -n1" (or tail) to choose a random file in a directory.It sorts the files with "ls", sort it randomly and pick the first line, which wastes the "ls" sort. Also, using "sort -R | head -n1" is inefficient. For example, in a directory with 1000000 files, it takes more than 15 seconds to pick a file. $ time bash -c "ls -U | sort -R | head -n 1 >/dev/null" bash -c "ls -U | sort -R | head -n 1 >/dev/null" 15.38s user 0.14s system 99% cpu 15.536 total $ time bash -c "ls -U | shuf -n 1 >/dev/null" bash -c "ls -U | shuf -n 1 >/dev/null" 0.30s user 0.12s system 138% cpu 0.306 total So, we should just use "ls -U" and "shuf -n 1" to choose a random file. Introduce _random_file() helper to do it properly. Signed-off-by: Naohiro Aota --- common/rc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/common/rc b/common/rc index 5c4429ed0425..2bdda30f497a 100644 --- a/common/rc +++ b/common/rc @@ -5224,6 +5224,13 @@ _soak_loop_running() { return 0 } +# Return a random file in a directory. A directory is *not* followed +# recursively. +_random_file() { + local basedir=$1 + ls -U "${basedir}" | shuf -n 1 +} + init_rc ################################################################################ From patchwork Mon Aug 21 06:37:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naohiro Aota X-Patchwork-Id: 13359193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DD83EE49B1 for ; Mon, 21 Aug 2023 06:37:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233469AbjHUGhO (ORCPT ); Mon, 21 Aug 2023 02:37:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233456AbjHUGhN (ORCPT ); Mon, 21 Aug 2023 02:37:13 -0400 Received: from esa6.hgst.iphmx.com (esa6.hgst.iphmx.com [216.71.154.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 176C3AC; Sun, 20 Aug 2023 23:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1692599831; x=1724135831; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DIRbr2XPDPI1DdxdkY2jF6TmEaYD7n7iiQAKN2jCW1U=; b=V4+IH7A4I/Hsq4zyvgvz6okxzEfAezP6ut0RH7/1y7T+dkeunyS8Wpuq tOAzGiJsZ0w1aRqwyHK68UZDVN17J5vDmaWygIFIecnmMcquujT4sYipW +/TgYzgbPNeZQOlxjXxB3IBRRcMdGhe2z7I/pAgt6bX4M+9OmErLfdy6T Wt9jTfoTZyIqeIAQRTNttmFsj8DajuXTRTY9DngihU5VOVFHFh4bwrd5U 8kbcfg2tPrrzdXwgwYlxdLDOlp4llHgY7oq+JZT5GZbfibIR4gWvswwks 3S4VBsIM/s6r/xZri7uAiwTEYswLxh60UADOJFnjktXsrN/iQEyEIs+s8 A==; X-IronPort-AV: E=Sophos;i="6.01,189,1684771200"; d="scan'208";a="241991952" Received: from uls-op-cesaip02.wdc.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 21 Aug 2023 14:37:11 +0800 IronPort-SDR: lat7GHsd8FzLd7NbIqd3CzD57RaAns45q4N4IQDDdPUuWa6nVxkb6BfWOOmrXslvkEjpR5l7Nn 2tvlVJgR/fxAr29qzr0rYhawdeL+JaDDnmGrhiM8gyzBBae9/xYI7xy/PFsc6fTxE0dJvzX4RM r6GZgzc2VDvPXknGO/YsaKDp0dqCIG3Fc3PbGz4byv8U2qE1SwhYMBQKBg4YEDKsq0b00QP4Xz SWhBQNDTbq8r7spIHWpxHSFwI+k7ov6F3sw/loc44DvcsEwhcVPckogT6JZ/4co6V5ozj6JWoa 6VM= Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep02.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 20 Aug 2023 22:44:42 -0700 IronPort-SDR: rS9+34tp6t84Nz4FcCTp0AY8Uml3dJFKSnGuDnlicX1T7N1YSxsV4UWFbgKT1pQ8fsLkfbNkri m+bRu0gT0p2ntcxB2jcBvCgDL8bdlyBA08WLp5E7bnGb5NEJGvc/0hHxaTU2SINK/9Zm+YIY2S kWq0yO8zEjqcQcae9WKWO+5lGk558z2SBsmZEHw1XMYsb4Y1Ri3az6wTUIJJMCVPO6FDDkHLjg 5RzsPXBGTKWUz2AlvDa4cfwOgBnDVlgC7p/vyKzhjvkCi0X6NVuofG2FlBhRdA/9gDQZPjEG7b MW0= WDCIronportException: Internal Received: from unknown (HELO naota-xeon.wdc.com) ([10.225.163.96]) by uls-op-cesaip02.wdc.com with ESMTP; 20 Aug 2023 23:37:12 -0700 From: Naohiro Aota To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Naohiro Aota Subject: [PATCH v2 2/3] fstests/btrfs: use _random_file() helper Date: Mon, 21 Aug 2023 15:37:03 +0900 Message-ID: <9c0f4efcabbd3c4cfcd726910f7295d98115d66c.1692599767.git.naohiro.aota@wdc.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Use _random_file() helper to choose a random file in a directory. Signed-off-by: Naohiro Aota --- tests/btrfs/179 | 4 ++-- tests/btrfs/192 | 14 ++++---------- 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/tests/btrfs/179 b/tests/btrfs/179 index 2f17c9f9fb4a..33a854d70401 100755 --- a/tests/btrfs/179 +++ b/tests/btrfs/179 @@ -45,7 +45,7 @@ fill_workload() # Randomly remove some files for every 5 loop if [ $(( $i % 5 )) -eq 0 ]; then - victim=$(ls "$SCRATCH_MNT/src" | sort -R | head -n1) + victim=$(_random_file "$SCRATCH_MNT/src") rm "$SCRATCH_MNT/src/$victim" fi i=$((i + 1)) @@ -69,7 +69,7 @@ delete_workload() trap "wait; exit" SIGTERM while true; do sleep $((sleep_time * 2)) - victim=$(ls "$SCRATCH_MNT/snapshots" | sort -R | head -n1) + victim=$(_random_file "$SCRATCH_MNT/snapshots") if [ -z "$victim" ]; then # No snapshots available, sleep and retry later. continue diff --git a/tests/btrfs/192 b/tests/btrfs/192 index bcf14ebb8e3b..7324c9e39833 100755 --- a/tests/btrfs/192 +++ b/tests/btrfs/192 @@ -69,12 +69,6 @@ $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/src > /dev/null mkdir -p $SCRATCH_MNT/snapshots mkdir -p $SCRATCH_MNT/src/padding -random_file() -{ - local basedir=$1 - echo "$basedir/$(ls $basedir | sort -R | tail -1)" -} - snapshot_workload() { trap "wait; exit" SIGTERM @@ -85,9 +79,9 @@ snapshot_workload() $SCRATCH_MNT/src $SCRATCH_MNT/snapshots/$i \ > /dev/null # Do something small to make snapshots different - rm -f "$(random_file $SCRATCH_MNT/src/padding)" - rm -f "$(random_file $SCRATCH_MNT/src/padding)" - touch "$(random_file $SCRATCH_MNT/src/padding)" + rm -f "$(_random_file $SCRATCH_MNT/src/padding)" + rm -f "$(_random_file $SCRATCH_MNT/src/padding)" + touch "$(_random_file $SCRATCH_MNT/src/padding)" touch "$SCRATCH_MNT/src/padding/random_$RANDOM" i=$(($i + 1)) @@ -102,7 +96,7 @@ delete_workload() while true; do sleep 2 $BTRFS_UTIL_PROG subvolume delete \ - "$(random_file $SCRATCH_MNT/snapshots)" \ + "$(_random_file $SCRATCH_MNT/snapshots)" \ > /dev/null 2>&1 done } From patchwork Mon Aug 21 06:37:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naohiro Aota X-Patchwork-Id: 13359192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 393F3EE49AE for ; Mon, 21 Aug 2023 06:37:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233466AbjHUGhO (ORCPT ); Mon, 21 Aug 2023 02:37:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjHUGhN (ORCPT ); Mon, 21 Aug 2023 02:37:13 -0400 Received: from esa6.hgst.iphmx.com (esa6.hgst.iphmx.com [216.71.154.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B4A4A9; Sun, 20 Aug 2023 23:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1692599832; x=1724135832; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G9ZyhRGZh2z81PLUydMfxH4iIes0hBOJauITXx85a2s=; b=XQ6wl/MGQYIKCXuWyZCwgHf7eEHvj0+mc+tfD58PY/jKrh4vBoMcfPeG xw/O7tMv10dCPC0UHK/mt3VIeRtaHe6zOIvdVVfZyFV5mZj0S4v1xaYfJ 41+rXFln2Md86YloWK2T1mlotFasz++7SSjigxE516FPc1A+N6OVWnkjE OC10twT4xNHfpOtQUYFljIUrbkZLrMZqocKf6TOY0zINbRHw+toWzRA14 XH87ymLBd5RHWOnk3x1Uu4ObLFtzqoJ/uP8td0GAGqLTbOl0Qu+crlBsU pvoi8AxKYHYAPjddtS+7ZjhQT04RH5e0IxNyLloGu705ibGK8NvYrCMdL g==; X-IronPort-AV: E=Sophos;i="6.01,189,1684771200"; d="scan'208";a="241991953" Received: from uls-op-cesaip02.wdc.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 21 Aug 2023 14:37:11 +0800 IronPort-SDR: 2APoeC2xbV7fi38+cOZ9HohAWFqPmRaI2nKNHtX09/QidExbpbVpO1q2tjbr+67phs8bnsSD7v v808R33Jv64oyhWeFfIh0e2IXSdxP4vuFKC3soUyNvCsOR93h0FaiEIcGuTT5xJE0W+r6HRTvf kEAWyGRotIbqimLwP8j/C0GVHE0zLCXad8e+Dyi3c925IlaHJTFNAltzoUq3QmcGQLjzl+MXGX Fjtt4Ndclxtb7NCbKe3Y7e3/YU82ic+94b6rypp2LiM58Oj7I3IjJZApocIrDSCdCdlGv4GqbJ 8ek= Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep02.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 20 Aug 2023 22:44:43 -0700 IronPort-SDR: 8B9YpI4ticd2Yi8e1aIL8ZXiwuwLSk5nm46pmREJIkZCc4QzasQGAG8jAbw0eY311qMysLo3Hf ihZjKp5sjZmZ+hr6WcbmlDjeI/xHFXVDbA3zlFFKccWHuXZOOEu2Glmpfe3U4cDA0iYgTGHwdC IZ14Bk7z3PSHJTVpHsXKJi4AAPF8QQeRlbsQl63IU6+XPrs2muRlGfMWB4YvbS1p9gMtWzsIr8 3Q4tW3OmFgjKZEpwFfQ5LrNoj3LU1X5TPMMBXKcLX39h8gLsJT65MiBhCCTvBYJf9jvLqEimBb okw= WDCIronportException: Internal Received: from unknown (HELO naota-xeon.wdc.com) ([10.225.163.96]) by uls-op-cesaip02.wdc.com with ESMTP; 20 Aug 2023 23:37:12 -0700 From: Naohiro Aota To: fstests@vger.kernel.org Cc: linux-btrfs@vger.kernel.org, Naohiro Aota Subject: [PATCH v2 3/3] btrfs/004: use shuf to shuffle the file lines Date: Mon, 21 Aug 2023 15:37:04 +0900 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org The "sort -R" is slower than "shuf" even with the full output because "sort -R" actually sort them to group the identical keys. $ time bash -c "seq 1000000 | shuf >/dev/null" bash -c "seq 1000000 | shuf >/dev/null" 0.18s user 0.03s system 104% cpu 0.196 total $ time bash -c "seq 1000000 | sort -R >/dev/null" bash -c "seq 1000000 | sort -R >/dev/null" 19.61s user 0.03s system 99% cpu 19.739 total Since the "find"'s outputs never be identical, we can just use "shuf" to optimize the selection. Signed-off-by: Naohiro Aota --- tests/btrfs/004 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/btrfs/004 b/tests/btrfs/004 index ea40dbf62880..78df6a3af6b1 100755 --- a/tests/btrfs/004 +++ b/tests/btrfs/004 @@ -201,7 +201,7 @@ workout() cnt=0 errcnt=0 dir="$SCRATCH_MNT/$snap_name/" - for file in `find $dir -name f\* -size +0 | sort -R`; do + for file in `find $dir -name f\* -size +0 | shuf`; do extents=`_check_file_extents $file` ret=$? if [ $ret -ne 0 ]; then