From patchwork Thu Apr 3 08:58:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nirjhar Roy (IBM)" X-Patchwork-Id: 14037313 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3C772459C2; Thu, 3 Apr 2025 08:59:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670749; cv=none; b=gx6PvdappNmvrPABh/iLblJ5wi7VfcXk1tfalZrrP/CNUJPE4oSvALU3QsmIIqjcAD7cAeQbdyOKl/6KxITF84B2eGXadhqRZGlzA5Ip2NdJxQ2tX0dFj+veeulWQj1xNdCAzfTVGbIq6Rfb/GYplqkJqetyxHaYbMDOKlMFEdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670749; c=relaxed/simple; bh=quOSX+dukN8bp5qyLFWL/jkZsrmomvQkWn1v4f4g+cU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WEteFuBRqPZrJLYg04mV4wEVAp7C9AJAH3VKgqlaahUcbSvaKk/KrB9ZUy05xxN/jbqlWALwn6lNK5s+mhSXGtttSIwqNA3cQgIhQcingEOCbdFv6wEvbdpEnbXlRVRt1QHSavl/aciD05SYH6/ygflxglfsKYtzorDEm88fUPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f5Vp7EzH; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f5Vp7EzH" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-22409077c06so9949345ad.1; Thu, 03 Apr 2025 01:59:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743670746; x=1744275546; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N+oisU0LaLd3zUsHySudAGJLRXcJ0NxAaiTaaIJXLlI=; b=f5Vp7EzHEHZuVkwWi1K9PEk7KS789KPNwKolW5eq407O/94IbFmMwdYS0hzXixDZRe fUbtGP4bsCnO8Pv08i9HUbp8p9CObtH5QmvL3Sux0lPoX9PfuP3dZm8vcRgrPQtzlAcU 8YRiNHWPzecPKKInw3KGkDBE1kwRe2nj+aMZTqAuHdYQz+LRNcj2ioeYYDH1g0MwFGFR ERkLf8h7WcQOWoEJuNQwVHqGTYoQPem4zI/fZSA/FbzRs3ZtBQKS9qW4+5YXBFH+0VpV RC0eyxud60ZVrVSYnv4lq+Bjqhmme2Y4QG/OnPHAx6qhoZrq185bs1OJuuZKT7XzZY41 HlMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743670746; x=1744275546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N+oisU0LaLd3zUsHySudAGJLRXcJ0NxAaiTaaIJXLlI=; b=DH2T40NsgXZx5eRFcLEhILVR7tVKbu1XlkDMSPKCffucelTCEachVocaC2xDDiUHJu huOlZ8UBhLSX++ZYWgnNa9KMdBDQWEuDP5VMYGx+3mQaOXVsuCsRDU5pI/uAptZY/6zr n2m/mvhQNKz0Uw0MMTCdS3l38+j/dznHs9db41hrBbuSbqVPp9dd0YOXR17sr2CZ+GGb fXksyzwxOBSdy8DWofO8D5j24nBcertq04vC6kjBYWYEts4sWS/k81OR7wrx8v3v4tOs Zb3/e9Qt65KJAXsE+u4NA+Sep0Uk1j2sjcP20enWCIMcJWh4SxMI3kaHxdcYLa7zVlpl ICZw== X-Forwarded-Encrypted: i=1; AJvYcCVT4lRBJqMwl1YNPijOQG914xA1JaI9xVzDOVwslXquuJp79jhvMCoTkDKimvzevRESy8DQRM/Vrlk=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4T5vwL7PtgrCTWlzBRtyrOPYJ/id/sjq0ryBdadff0HKkPFk3 QlsM+KHn4/2YqxDpbnfzNq+biBK0cpFaumowEFe+QKMZeDZJenLpe0l3hw== X-Gm-Gg: ASbGnctgaXaapOxVSAFPWDGDejQMqMyclurcvnewbGaO8sLwpVuEYEwXJKn2K0kCPIP 3fRjvHjwpxhn7ZGBANnToKLQcsfp4i4lKJNu0lJzpRFp+IukhKZbNMtcscSxBPosnmFBZkHbWB2 ZogMT/+tS/Of24I5fwpvi69xNDMUtAFW7t73PNbcb9CSmfm5N37zEZGWyHe24YvskO0p0i6lkOr ylIRaGx4x2Xtv6yrc2nKLy9cs/dK9zGRO1nZ1IazdZoh/ZVVEqlGPHzcOBYlIObzGKTIiU6YO1q JMFv5VBei2DhjO3yoDmgoSqYp3Q5XHQsyCJa95Ab9CvrUUvsEv1AFEFD8zg= X-Google-Smtp-Source: AGHT+IHOUVV2Xe5N2hUS8UaAQ41GettknD29OhlQoehrC9qAx5fHmhc6a5hq821xyo+/A2Cf6zYCKw== X-Received: by 2002:a17:902:e805:b0:224:192a:9154 with SMTP id d9443c01a7336-2292f97a6aemr296988875ad.26.1743670746126; Thu, 03 Apr 2025 01:59:06 -0700 (PDT) Received: from citest-1.. ([49.205.34.162]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-229785c01cdsm9535715ad.99.2025.04.03.01.59.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 01:59:05 -0700 (PDT) From: "Nirjhar Roy (IBM)" To: fstests@vger.kernel.org Cc: linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, ritesh.list@gmail.com, ojaswin@linux.ibm.com, djwong@kernel.org, zlang@kernel.org, david@fromorbit.com, nirjhar.roy.lists@gmail.com Subject: [PATCH v2 1/3] tests/selftest: Add a new pseudo flaky test. Date: Thu, 3 Apr 2025 08:58:18 +0000 Message-Id: <176a3c6b995fde1390e152bc430689a39796dc76.1743670253.git.nirjhar.roy.lists@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This test is to simulate the behavior of a flaky test. This will be required when we will make some modifications to the pass/fail metric calculation of the test infrastructure where we will need a test with non-zero pass and non-zero failure rate. Reviewed-by: Ritesh Harjani (IBM) Reviewed-by: "Darrick J. Wong" Signed-off-by: Nirjhar Roy (IBM) --- tests/selftest/007 | 21 +++++++++++++++++++++ tests/selftest/007.out | 2 ++ 2 files changed, 23 insertions(+) create mode 100755 tests/selftest/007 create mode 100644 tests/selftest/007.out diff --git a/tests/selftest/007 b/tests/selftest/007 new file mode 100755 index 00000000..f100ec5f --- /dev/null +++ b/tests/selftest/007 @@ -0,0 +1,21 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 IBM Corporation. All Rights Reserved. +# Author: Nirjhar Roy (IBM) +# +# FS QA Test 007 +# +# This test is to simulate the behavior of a flakey test. +# + +. ./common/preamble +_begin_fstest selftest + +if (($RANDOM % 2)); then + echo "Silence is golden" +else + echo "Silence is flakey" +fi + +status=0 +exit diff --git a/tests/selftest/007.out b/tests/selftest/007.out new file mode 100644 index 00000000..fd3590e6 --- /dev/null +++ b/tests/selftest/007.out @@ -0,0 +1,2 @@ +QA output created by 007 +Silence is golden From patchwork Thu Apr 3 08:58:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nirjhar Roy (IBM)" X-Patchwork-Id: 14037314 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D32F24DFE2; Thu, 3 Apr 2025 08:59:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670772; cv=none; b=JX9ePa+IbdiARiuZZpVqlhI+bIvU8wvY7tDwBuaTc6jnYW+SGz+Rx2QGyK2CtiwWeTR7wESzEK7lCfCG6bvo/5wg/0q1Xxwa7fG+7TXOAxCk8Ybh2inPT+zAcHjK3J+ereblt2PWXVXu4jxxkRmbpo2IcZklh2AsbDqqSK76oes= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670772; c=relaxed/simple; bh=YPsD6BmQF9RNGFUOzhC8nd5c/Q5+Ap+meRgORq8s5bw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q+BrGQS86VZcr9+uRJtc4fLDCOpqPf04MWuNh9djoDbTwPAHFUA9n8H2EKE6VJON3N24Ah6lBBwNloM01EJPliY3J+WYCyqk9xorZ4li0+8k+/g4PVo3Qbt9t38fsIRsjVtqsqkKU0OaNj/H6+lzQqDo1vrdsQG9gEMZuZjlmBo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TEYoQdBK; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TEYoQdBK" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-227914acd20so19284895ad.1; Thu, 03 Apr 2025 01:59:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743670760; x=1744275560; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4PXwhjc8Cv5eEL1uUpHB3F+rJFG0NxsfWWqy/FqzBHk=; b=TEYoQdBKfdw8nzk4k1rXj2Aw0gV9jBfAZoiAiV0drXv/3kYW68Ho5sE93ZxLzFSw5K mvoy8LqInkaKMvp42QChA6MCm35RvWKc/ZIQWsrBl5RXiBi9UqT+RYw3PCgVSoFkYLmr hcAMZsEnwShzqcnQmNz9sk5CtkURSC4ENDywvFL9WXdcXY6NrffD2rcJLH9ookoS+aAk pc2vzutVTEZWLXQHs0G77BvKvCqjDXOaGLOfaD4Xt9p9zPVvG6Q//6J4O7i2O4H4bc9A 69cgZWtkMAQNJYDhigU/FA6fksxd6lZGXxFTsj3EHnvNjN+rC2kdoZLt8JcspmV9fid0 9rog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743670760; x=1744275560; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4PXwhjc8Cv5eEL1uUpHB3F+rJFG0NxsfWWqy/FqzBHk=; b=XxyGqojM4D3rox3f+wUzEIKtq+CkYVR2qZ87n4DO9sbzu+Px90jF0wz5tupGHaMyAJ MwxtS9Zig6pYRN2X7a/hEHIYP2ABDiyz0PwnfV6D9lu4C5vKtQybtQlKW4LevvkHk3Gu Gpv5fG+qGmFjLgGGYAe/+wjqP+7YqXNZh0NNpJ3GlwI3r+BQAlAKFXhW0ki89wR3DMQ4 MFQGbHNzSfvn4vcm39ka6nNbaK+E7ZQfeK9lQbCoxnOYuFXZdAEY/pLLWlWp0NifuN6e WB+s2ANQcOrS9StBu4yLbG6YCgO7xfz7lBYjG29qtgFPXpQyrkPuV4+mVFu8LBFe64qS RPuQ== X-Forwarded-Encrypted: i=1; AJvYcCUY4SzkyQE2RmXBIHCqarvKFY3xUipYD9f1t8GjvQ50GxeTN6Nrdt5FZLK3qkpVcuUGobAwixue7fM=@vger.kernel.org X-Gm-Message-State: AOJu0YwNQ/050g92OWm3eNtwwD/TGBTZnma3lf0YK8/ozbYNWHlQ/tdN Up0jDuHesUTof+Pt+K2ZLiWDnnut4PYJ+A6TzOQyisrfXJhHnZsV/qUl6A== X-Gm-Gg: ASbGncvOp5KHeeuQzyZiUzHpc6yoxPPoRZ2B94ivOleuGLTO4+F59wBegz33uXJRFVi K84Mu751JNaCGisSqdfjYJFhs++PLqzL6+FNe6PZW94B985tJAZ0loiek9pSrJ6X8N7fFIRwlld GQ7U+FvV5sV+rq/13qki+DbigpGLYXXVungLhM8nJtTQg6bDhnp167lwvaijnEWVDxmpll661eD C1awLpQIUEvkcXv/58Aj/xV9mV2y7IMwETENVe8mVLg7Hck8M4fk8y8RSHyBGLAhEfXsO0Npe/6 NBuCq/zhVl3uyVLTCCy7kde51zszy//D2T8AE5ME7R2Myz3O X-Google-Smtp-Source: AGHT+IFsCHRVkS0W9RQ4ljdzof/v2lVMfyPbA0C1bnWDvE4blEZwx0AmNiv4mJmNOdWKiJpfF+NZrw== X-Received: by 2002:a17:903:1b0c:b0:215:a303:24e9 with SMTP id d9443c01a7336-229765ba844mr35776985ad.3.1743670760402; Thu, 03 Apr 2025 01:59:20 -0700 (PDT) Received: from citest-1.. ([49.205.34.162]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-229785c01cdsm9535715ad.99.2025.04.03.01.59.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 01:59:20 -0700 (PDT) From: "Nirjhar Roy (IBM)" To: fstests@vger.kernel.org Cc: linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, ritesh.list@gmail.com, ojaswin@linux.ibm.com, djwong@kernel.org, zlang@kernel.org, david@fromorbit.com, nirjhar.roy.lists@gmail.com Subject: [PATCH v2 2/3] check: Add -q option to support unconditional looping. Date: Thu, 3 Apr 2025 08:58:19 +0000 Message-Id: <762d80d522724f975df087c1e92cdd202fd18cae.1743670253.git.nirjhar.roy.lists@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch adds -q option through which one can run a given test times unconditionally. It also prints pass/fail metrics at the end. The advantage of this over -L and -i/-I is that: a. -L will not re-run a flakey test if the test passes for the first time. b. -I/-i sets up devices during each iteration and hence slower. Note -q will override -L . Also rename _stash_fail_loop_files() to _stash_loop_files() because this function will now be used even when the test doesn't fail (i.e. when ran with -q ). Suggested-by: Ritesh Harjani (IBM) Signed-off-by: Nirjhar Roy (IBM) Co-developed-by: Ritesh Harjani (IBM) Signed-off-by: Ritesh Harjani (IBM) --- check | 46 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/check b/check index 32890470..f6007750 100755 --- a/check +++ b/check @@ -11,6 +11,7 @@ needsum=true try=() sum_bad=0 bad=() +is_bad_test=false notrun=() interrupt=true diff="diff -u" @@ -27,6 +28,7 @@ DUMP_OUTPUT=false iterations=1 istop=false loop_on_fail=0 +loop_unconditional=0 exclude_tests=() # This is a global variable used to pass test failure text to reporting gunk @@ -83,6 +85,7 @@ check options -s section run only specified section from config file -S section exclude the specified section from the config file -L loop tests times following a failure, measuring aggregate pass/fail metrics + -q loop tests times irrespective of a pass or a failure, measuring aggregate pass/fail metrics testlist options -g group[,group...] include tests from these groups @@ -341,7 +344,9 @@ while [ $# -gt 0 ]; do -L) [[ $2 =~ ^[0-9]+$ ]] || usage loop_on_fail=$2; shift ;; - + -q) [[ $2 =~ ^[0-9]+$ ]] || usage + loop_unconditional=$(($2 - 1)); shift + ;; -*) usage ;; *) # not an argument, we've got tests now. have_test_arg=true ;; @@ -357,6 +362,11 @@ while [ $# -gt 0 ]; do shift done +# -q overrides -L +if [ "$loop_unconditional" -gt 0 ]; then + loop_on_fail=0 +fi + # we need common/rc, that also sources common/config. We need to source it # after processing args, overlay needs FSTYP set before sourcing common/config if ! . ./common/rc; then @@ -609,7 +619,7 @@ _expunge_test() } # retain files which would be overwritten in subsequent reruns of the same test -_stash_fail_loop_files() { +_stash_loop_files() { local seq_prefix="${REPORT_DIR}/${1}" local cp_suffix="$2" @@ -633,10 +643,18 @@ _stash_test_status() { fi if ((${#loop_status[*]} > 0)); then - # continuing or completing rerun-on-failure loop - _stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}" + # continuing or completing rerun loop + _stash_loop_files "$test_seq" ".rerun${#loop_status[*]}" loop_status+=("$test_status") - if ((${#loop_status[*]} > loop_on_fail)); then + + # only stash @bad result for initial failure in loop + if [[ "$test_status" == "fail" ]] && ! $is_bad_test; then + bad+=("$test_seq") + is_bad_test=true + fi + + if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \ + ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then printf "%s aggregate results across %d runs: " \ "$test_seq" "${#loop_status[*]}" awk "BEGIN { @@ -650,23 +668,30 @@ _stash_test_status() { }' echo loop_status=() + is_bad_test=false fi - return # only stash @bad result for initial failure in loop + return fi case "$test_status" in fail) - if ((loop_on_fail > 0)); then - # initial failure, start rerun-on-failure loop - _stash_fail_loop_files "$test_seq" ".rerun0" + # re-run if either of the loop argument is set + if ((loop_on_fail > 0)) || ((loop_unconditional > 0)); then + _stash_loop_files "$test_seq" ".rerun0" loop_status+=("$test_status") fi bad+=("$test_seq") + is_bad_test=true ;; list|notrun) notrun+=("$test_seq") ;; pass|expunge) + # re-run if loop_unconditional argument is set + if ((loop_unconditional > 0)); then + _stash_loop_files "$test_seq" ".rerun0" + loop_status+=("$test_status") + fi ;; *) echo "Unexpected test $test_seq status: $test_status" @@ -852,7 +877,8 @@ function run_section() seqres="$check" _check_test_fs - loop_status=() # track rerun-on-failure state + is_bad_test=false + loop_status=() # track loop rerun state local tc_status ix local -a _list=( $list ) for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do From patchwork Thu Apr 3 08:58:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nirjhar Roy (IBM)" X-Patchwork-Id: 14037315 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25ED524339C; Thu, 3 Apr 2025 08:59:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670783; cv=none; b=bYCK1ZES0x7zjlLcbm16y0q41Ej2rbwctGewzcEf1Fv0ASTgk+MSJCI0cadq2Z/c5KgWdSPWWTgSKEkEDD3MLWLuHQHGr73QQsQhBF9djEtwCfL42GUgiMLVFPqcNoQ9ZcZw707wKbJsguzkvJXdxKYErDIIX7N0ZwUXUWiikgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743670783; c=relaxed/simple; bh=cXpMG4evM+/eBh0rh9cOx22wN60fLnLdyOex1885nT0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I5YPZ7Bh99aobuic3uXKxwFpNriFo/FNYgfyVa/Z6qC+c+JpxPonQB9wM5LPDWRKyGB/m0NcXPCihS24kSP026cgtC0XisftM4IeDPbWsSNcUl2zS7fNpT4Qft8qBDmDW70ZNdo2ccqTk28uJQNlj6CshVUspyLWVWf9eWndQqU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cnG1vZR3; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cnG1vZR3" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-22409077c06so9954525ad.1; Thu, 03 Apr 2025 01:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743670781; x=1744275581; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NR4dPmRB+WlVaBFBqvFj4uKQeU+r3bP802+oB47JMSg=; b=cnG1vZR3aJ7KW3zxwm+5273mHxVvzGMXV2bIAH6D+zrU4XJfbsEF0gyFRRqhmf8U7b gg7x/2LmIaS8KKOrDWZig2RtNXjic0Vwo2cGuQRr5Z+/E8RB2sdQOD7OOvC83tE3Nm3A HSfrpnJWbfJOVT+YnumgmqNbJPqdWbiz0jrOn0zrS8oDVyJNDMscfEeKeOB9/52O73Ux 1/n/3kyxOcibY/qb2ueF28snDPykp4SmcXZIeQCruYiq35VqQHXrpfkifKXrnq+sSMH7 KY9hVv/L32OSW85p+8SEejwvdcgewpehm+04iZF3HCtdEsPWlCrLQq9ecR/KRZvCpGo/ rFGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743670781; x=1744275581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NR4dPmRB+WlVaBFBqvFj4uKQeU+r3bP802+oB47JMSg=; b=HRiumoi/DmXu+196xZ6Pht5DYyaDlqRHD5eoN8nwuWjnWg+XGOE/DjMWl7gijwXnv4 Y9C4pMsyDcRrBOilFPduI/WGTMTMlMnZ3bJUvwyNFH6w8WfOWA1RWv4+Er5ISIvWdRPv 9NBtoXvyby3VsAgma47Uuia/kQM0nP99nNw2Qr+j4w9dqDLDbd6tCo9bGdB2cBAvB7wD 3KeZ7l4I5c54V21MrOSQV7xi1RIgoc9oDuBi9L3jdzkWZ9tCLWtlkVTbqIfmFBIhAnup Jw+yUZFhVvpHWo1w/3LPB/fYj4D7POHzIXVsMmnmJVI17Q7ysgBAeOlW0W82Rw5So2Ne xRMg== X-Forwarded-Encrypted: i=1; AJvYcCWP++72HSKrDuVWX/uQQ00YDySH4wqHY8kxKPKJq7PjAxMJyaUeZj3dlyLo5kHQWlqfwGwvkDez798=@vger.kernel.org X-Gm-Message-State: AOJu0YxVxphUyTVRp3lbPbL/ROpbvIU3yrARhznAgJUAhigN/EPhOFze 0UZ/TNyohG1rt+/Ni0PocWYiSpFXcXZ+/rcfhKuE2wQ7cVCRgOlgDx4xOw== X-Gm-Gg: ASbGncsf3t0ee/PMvpnnf4KXKIFWbs+Wj1KS3O0VLDCeq3eArkVzw6NJ4fpDtNPTYuz 6TbOZcsZPCmj8G1ZaaTR9QpuEGVEQB/kkG3H7C8E5/NVYOCYn16X/4y4J+AulLv/kUMySUf2jES BUix+9zFwTj52duLha+/zwgBO6ZK6nwLEpkOeI+7ApqhpKyrcDalB+e1oFu/khAs4jlDAl15iZE t8dYPfjwI8GtWKz85Z2+LbDfEoI+QAYMTDsyuIcDMpmHNh+FEM9bCU99C72/9mM9n6fz7c2qj8A VUz5BxNSP+3XzBJveHdet8kGQry92mM72DGDbLuDQOekMLuk1Gu/s8ySoCU= X-Google-Smtp-Source: AGHT+IHC0+dVJMNf5ScPuEOnU+uWmFtn7jonuLrkEs1Gl0dp4lJ4z5Yberxu3G9N6t2PTIb1OjGQTA== X-Received: by 2002:a17:902:cf10:b0:21b:b3c9:38ff with SMTP id d9443c01a7336-2292f9f54d6mr304729445ad.37.1743670780789; Thu, 03 Apr 2025 01:59:40 -0700 (PDT) Received: from citest-1.. ([49.205.34.162]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-229785c01cdsm9535715ad.99.2025.04.03.01.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 01:59:40 -0700 (PDT) From: "Nirjhar Roy (IBM)" To: fstests@vger.kernel.org Cc: linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, ritesh.list@gmail.com, ojaswin@linux.ibm.com, djwong@kernel.org, zlang@kernel.org, david@fromorbit.com, nirjhar.roy.lists@gmail.com Subject: [PATCH v2 3/3] check: Improve pass/fail metrics and section config output Date: Thu, 3 Apr 2025 08:58:20 +0000 Message-Id: <72dd490d0d00ce2e35c168d21983c015ae471354.1743670253.git.nirjhar.roy.lists@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: fstests@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch improves the output by: - Adding pass/fail aggregate results in the stdout for each section's summary. Also adds the location of the xunit xml files in the section's summary. - Adds section config details into the result's global log file. e.g. $ ./check -q 2 -R xunit-quiet selftest/001 SECTION -- s1 ========================= selftest/001 aggregate results across 2 runs: pass=2 (100.0%) Ran: selftest/001 Passed all 1 tests Xunit report: /home/ubuntu/xfstests/results//s1/result.xml SECTION -- s2 ========================= selftest/001 aggregate results across 2 runs: pass=2 (100.0%) Ran: selftest/001 Passed all 1 tests Xunit report: /home/ubuntu/xfstests/results//s2/result.xml Output in results/check.log file Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty Wed Jan 8 11:23:57 UTC 2025 SECTION -- s1 ========================= FSTYP -- ext4 PLATFORM -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024 MKFS_OPTIONS -- -F /dev/loop1 MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch selftest/001 aggregate results across 2 runs: pass=2 (100.0%) Ran: selftest/001 Passed all 1 tests Xunit report: /home/ubuntu/xfstests/results//s1/result.xml Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty Wed Jan 8 11:24:02 UTC 2025 SECTION -- s2 ========================= FSTYP -- ext4 PLATFORM -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024 MKFS_OPTIONS -- -F /dev/loop1 MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch selftest/001 aggregate results across 2 runs: pass=2 (100.0%) Ran: selftest/001 Passed all 1 tests Xunit report: /home/ubuntu/xfstests/results//s2/result.xml Suggested-by: Ritesh Harjani Co-developed-by: Ritesh Harjani (IBM) Signed-off-by: Ritesh Harjani (IBM) Signed-off-by: Nirjhar Roy (IBM) --- check | 67 ++++++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 21 deletions(-) diff --git a/check b/check index f6007750..bb91cf9d 100755 --- a/check +++ b/check @@ -468,6 +468,25 @@ if [ -n "$REPORT_GCOV" ]; then _gcov_check_report_gcov fi +_print_list() +{ + local -n list=$1 + local item + for item in "${list[@]}"; do + echo "$item" + done +} + +_display_test_configuration() +{ + echo "FSTYP -- `_full_fstyp_details`" + echo "PLATFORM -- `_full_platform_details`" + if [ ! -z "$SCRATCH_DEV" ]; then + echo "MKFS_OPTIONS -- `_scratch_mkfs_options`" + echo "MOUNT_OPTIONS -- `_scratch_mount_options`" + fi +} + _wrapup() { seq="check.$$" @@ -505,11 +524,19 @@ _wrapup() echo "SECTION -- $section" >>$tmp.summary echo "=========================" >>$tmp.summary + + _global_log "SECTION -- $section" + _global_log "=========================" >>$tmp.summary + _global_log "$(_display_test_configuration)" if ((${#try[*]} > 0)); then + local test_aggr_stats=$(_print_list loop_test_stats_per_section) if [ $brief_test_summary == "false" ]; then + [[ -n "$test_aggr_stats" ]] && echo "$test_aggr_stats" >> \ + $tmp.summary echo "Ran: ${try[*]}" echo "Ran: ${try[*]}" >>$tmp.summary fi + _global_log "$test_aggr_stats" _global_log "Ran: ${try[*]}" fi @@ -539,12 +566,15 @@ _wrapup() _global_log "Passed all ${#try[*]} tests" echo "Passed all ${#try[*]} tests" >>$tmp.summary fi - echo "" >>$tmp.summary if $do_report; then _make_section_report "$section" "${#try[*]}" \ "${#bad[*]}" "${#notrun[*]}" \ "$((sect_stop - sect_start))" + local out_fn="$REPORT_DIR/result.xml" + echo "Xunit report: $out_fn" >> $tmp.summary + _global_log "Xunit report: $out_fn" fi + echo "" >>$tmp.summary # Generate code coverage report if [ -n "$REPORT_GCOV" ]; then @@ -655,18 +685,19 @@ _stash_test_status() { if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \ ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then - printf "%s aggregate results across %d runs: " \ - "$test_seq" "${#loop_status[*]}" - awk "BEGIN { - n=split(\"${loop_status[*]}\", arr);"' - for (i = 1; i <= n; i++) - stats[arr[i]]++; - for (x in stats) - printf("%s=%d (%.1f%%)", - (i-- > n ? x : ", " x), - stats[x], 100 * stats[x] / n); - }' - echo + local test_stats=$(printf "%s aggregate results across %d runs: " \ + "$test_seq" "${#loop_status[*]}") + test_stats+=$(awk "BEGIN { + n=split(\"${loop_status[*]}\", arr);"' + for (i = 1; i <= n; i++) + stats[arr[i]]++; + for (x in stats) + printf("%s=%d (%.1f%%)", + (i-- > n ? x : ", " x), + stats[x], 100 * stats[x] / n); + }') + echo "$test_stats" + loop_test_stats_per_section+=("$test_stats") loop_status=() is_bad_test=false fi @@ -834,14 +865,7 @@ function run_section() rm -f $check.full [ -f $check.time ] || touch $check.time - - # print out our test configuration - echo "FSTYP -- `_full_fstyp_details`" - echo "PLATFORM -- `_full_platform_details`" - if [ ! -z "$SCRATCH_DEV" ]; then - echo "MKFS_OPTIONS -- `_scratch_mkfs_options`" - echo "MOUNT_OPTIONS -- `_scratch_mount_options`" - fi + _display_test_configuration echo test -n "$REPORT_GCOV" && _gcov_reset needwrap=true @@ -879,6 +903,7 @@ function run_section() is_bad_test=false loop_status=() # track loop rerun state + loop_test_stats_per_section=() # store loop test statistics per section local tc_status ix local -a _list=( $list ) for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do