From patchwork Thu Dec 14 16:24:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13493235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3F94C4332F for ; Thu, 14 Dec 2023 16:24:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 304848D00CE; Thu, 14 Dec 2023 11:24:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 28E828D00C7; Thu, 14 Dec 2023 11:24:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12F0D8D00CE; Thu, 14 Dec 2023 11:24:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F20A38D00C7 for ; Thu, 14 Dec 2023 11:24:46 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BDB06C0C82 for ; Thu, 14 Dec 2023 16:24:46 +0000 (UTC) X-FDA: 81565947372.04.8477F44 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id EBAE440015 for ; Thu, 14 Dec 2023 16:24:43 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702571084; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=LlNzN7VVDItKRPWufXvVn8vMWsP7MVj1khCyLzsvbTA=; b=qoYjkSTQzqeB/Dv6H3o0fljuQcNZ3n7qRDOinEAH5wsvtFjFxqO6sGUUbAJuqLvBE9+UJ5 5xXZqT/VJGOVbywZV2rsu3RG+7Ta4GV81tg4fJXChGdoNrf7NOMAQmWml+QGugIb57Uz0e Z5why7/0FgYWHA7os0G89t+Po2XQ0pk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702571084; a=rsa-sha256; cv=none; b=CepD0BeTJOS2tJxjuXFuhH4urFW+wD92nbKeU7Ytnx14VNZnBKtucSj+XIgDBhI81FTnpQ 11Fv9PVJPJQhdp9HMhRjX5v24L9LvLKDFr1/O+be/dDs39xPfB1vkdNOx67g8i362rg6s8 Ss/WNqiRk4185biJ+LiD1QIsFtDMS78= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9B87AC15; Thu, 14 Dec 2023 08:25:28 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 051153F5A1; Thu, 14 Dec 2023 08:24:41 -0800 (PST) From: Ryan Roberts To: Andrew Morton , Shuah Khan , Peter Xu , Mark Brown , Aishwarya TCV Cc: Ryan Roberts , linux-mm@kvack.org, linux-kselftest@vger.kernel.org Subject: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format Date: Thu, 14 Dec 2023 16:24:34 +0000 Message-Id: <20231214162434.3580009-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: EBAE440015 X-Rspam-User: X-Stat-Signature: d845aqdwpjrx4ue83mwkqn4w3icstwet X-Rspamd-Server: rspam01 X-HE-Tag: 1702571083-10205 X-HE-Meta: U2FsdGVkX1+h+Zkg6JT7IZYT4qyySF1iZsqpADV9ltm/b7KPOUYHIBoskVNqBpkgIxjm76qER51CvfmBptoBxbCNelIbSkuxryJnb6FKdxvysEbchvkD88ebiia9iaO+qt4OTBa3cjzqxNwYqia8/EUienzJualh8lGnv8oBL9F06eg2FKB/aVs2M+G/B2rgpiR2newF4Oocgm9ctKHdfBBk0H+rPKeoDEeaONye1ClKoD6LAMbI4jM/mIWpxyqR9urtNuwaID1m0C6/X0I8D3r9/CEImM2f6JENqs9kCcaDJeCl2KE9Ci6TbtN+iliHia+weOi3sZ0qgPWBJfBAXpIAT/m2pD9+SpUfY922L8MNtYzoipYIlB0AJjxHCY/ltmrg3fIDu9v8KIQxSHO/yQpHn5K31b1Uc9Ikoz+ozqi78uaEEVd4EahJWrSqFsGby82lM0xUCZE03nXXWoiillP574hIQJdiCyjUYZTPSDnzqCZmgSKGkh3fgq55GI4+wInNtH9vRSfkSEJnIFMUwxgfF9PAwTdveMoivQAi+tLZHPbNVF/hBA61dGW8L7PCmp0r/Yr3866YBpUbLXY8D2pkaF1kpo7o32+KQEwtdtIDJWsl9SQsRv+V61nGGLNXL96pQ+fS6OSBptjxwQE0DLTRtZEe80sUm4cJFDrBlcsD5+FpSiIMEvAjJhkv1Ih2/8zs6IItJcLGWFJWKPcGWDw0rOVWdPMzuM9oQPdKjEhi5hfDxOiUsburDyr5lxiVEE5bGQKEbxEXTehl3gzxO0qR8d9lXbhRWVcO43LMgwGkKY4BrvAfJmjfYSBRqGL8jhviukryVclvr/ylJsL/NNAWu2eAbaMNjolgHXtbkrTHuCVxm20ZGgpduNXEf7dbNG9W9FzEeyntBgiXh+mz+xFGiiJqcpf5TkGEJbwLDPKgD7DuDdR5wNKBk5SR6J7paKk8DJMVzo0YjMHP/Hg U2H+HzkK 5if63xHAAYCtU8KaKi9WYNqXhd8GquE/rYcurTqaWVLjzRD4lYmgAS26/veIfOX0j/DAG/36kyIg1IBqBnYgEELgu6+15JtNGm58UPANjoBQAC4/fCMGCd8s8pZYoxbn20765K+988pe7eWM3SlVdru36G1UJ1OdPc3rB5xI1L9D7tVXNrvBS2fVNbu3n7aZR6eF3rdD3hO128S3o8sPdG4OQO+hyjrYWnJmM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When running tests on a CI system (e.g. LAVA) it is useful to output test results in TAP format so that the CI can parse the fine-grained results to show regressions. Many of the mm selftest binaries already output using the TAP format. And the kselftests runner (run_kselftest.sh) also uses the format. CI systems such as LAVA can already handle nested TAP reports. However, with the mm selftests we have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh -> individual test binaries) and the middle level did not previously support TAP, which breaks the parser. Let's fix that by teaching run_vmtests.sh to output using the TAP format. Ideally this would be opt-in via a command line argument to avoid the possibility of breaking anyone's existing scripts that might scrape the output. However, it is not possible to pass arguments to tests invoked via run_kselftest.sh. So I've implemented an opt-out option (-n), which will revert to the existing output format. Future changes to this file should be aware of 2 new conventions: - output that is part of the TAP reporting is piped through tap_output - general output is piped through tap_prefix Signed-off-by: Ryan Roberts Reviewed-by: Mark Brown Tested-by: John Hubbard --- tools/testing/selftests/mm/run_vmtests.sh | 51 +++++++++++++++++------ 1 file changed, 39 insertions(+), 12 deletions(-) -- 2.25.1 diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 87f513f5cf91..246d53a5d7f2 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -5,6 +5,7 @@ # Kselftest framework requirement - SKIP code is 4. ksft_skip=4 +count_total=0 count_pass=0 count_fail=0 count_skip=0 @@ -17,6 +18,7 @@ usage: ${BASH_SOURCE[0]:-$0} [ options ] -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message + -n: disable TAP output The default behavior is to run required tests only. If -a is specified, will run all tests. @@ -77,12 +79,14 @@ EOF } RUN_ALL=false +TAP_PREFIX="# " -while getopts "aht:" OPT; do +while getopts "aht:n" OPT; do case ${OPT} in "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; + "n") TAP_PREFIX= ;; esac done shift $((OPTIND -1)) @@ -184,30 +188,52 @@ fi VADDR64=0 echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1 +tap_prefix() { + sed -e "s/^/${TAP_PREFIX}/" +} + +tap_output() { + if [[ ! -z "$TAP_PREFIX" ]]; then + read str + echo $str + fi +} + +pretty_name() { + echo "$*" | sed -e 's/^\(bash \)\?\.\///' +} + # Usage: run_test [test binary] [arbitrary test arguments...] run_test() { if test_selected ${CATEGORY}; then + local test=$(pretty_name "$*") local title="running $*" local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -) - printf "%s\n%s\n%s\n" "$sep" "$title" "$sep" + printf "%s\n%s\n%s\n" "$sep" "$title" "$sep" | tap_prefix - "$@" - local ret=$? + ("$@" 2>&1) | tap_prefix + local ret=${PIPESTATUS[0]} + count_total=$(( count_total + 1 )) if [ $ret -eq 0 ]; then count_pass=$(( count_pass + 1 )) - echo "[PASS]" + echo "[PASS]" | tap_prefix + echo "ok ${count_total} ${test}" | tap_output elif [ $ret -eq $ksft_skip ]; then count_skip=$(( count_skip + 1 )) - echo "[SKIP]" + echo "[SKIP]" | tap_prefix + echo "ok ${count_total} ${test} # SKIP" | tap_output exitcode=$ksft_skip else count_fail=$(( count_fail + 1 )) - echo "[FAIL]" + echo "[FAIL]" | tap_prefix + echo "not ok ${count_total} ${test} # exit=$ret" | tap_output exitcode=1 fi fi # test_selected } +echo "TAP version 13" | tap_output + CATEGORY="hugetlb" run_test ./hugepage-mmap shmmax=$(cat /proc/sys/kernel/shmmax) @@ -231,9 +257,9 @@ CATEGORY="hugetlb" run_test ./hugetlb_fault_after_madv echo "$nr_hugepages_tmp" > /proc/sys/vm/nr_hugepages if test_selected "hugetlb"; then - echo "NOTE: These hugetlb tests provide minimal coverage. Use" - echo " https://github.com/libhugetlbfs/libhugetlbfs.git for" - echo " hugetlb regression testing." + echo "NOTE: These hugetlb tests provide minimal coverage. Use" | tap_prefix + echo " https://github.com/libhugetlbfs/libhugetlbfs.git for" | tap_prefix + echo " hugetlb regression testing." | tap_prefix fi CATEGORY="mmap" run_test ./map_fixed_noreplace @@ -312,7 +338,7 @@ CATEGORY="hmm" run_test bash ./test_hmm.sh smoke # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests CATEGORY="madv_populate" run_test ./madv_populate -echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope +(echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 2>&1) | tap_prefix CATEGORY="memfd_secret" run_test ./memfd_secret # KSM KSM_MERGE_TIME_HUGE_PAGES test with size of 100 @@ -369,6 +395,7 @@ CATEGORY="mkdirty" run_test ./mkdirty CATEGORY="mdwe" run_test ./mdwe_test -echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" +echo "SUMMARY: PASS=${count_pass} SKIP=${count_skip} FAIL=${count_fail}" | tap_prefix +echo "1..${count_total}" | tap_output exit $exitcode