From patchwork Tue Jul 17 21:02:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Bronnikov X-Patchwork-Id: 10530659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7333D600F4 for ; Tue, 17 Jul 2018 21:02:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63086290D0 for ; Tue, 17 Jul 2018 21:02:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 54FC7295D3; Tue, 17 Jul 2018 21:02:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 10EEF290D0 for ; Tue, 17 Jul 2018 21:02:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730055AbeGQVhA (ORCPT ); Tue, 17 Jul 2018 17:37:00 -0400 Received: from ns1.bronevichok.ru ([159.100.254.207]:12353 "EHLO ns1.bronevichok.ru" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729714AbeGQVhA (ORCPT ); Tue, 17 Jul 2018 17:37:00 -0400 Received: from ns1.bronevichok.ru (localhost [127.0.0.1]) by ns1.bronevichok.ru (OpenSMTPD) with ESMTP id 6f6afaa8; Tue, 17 Jul 2018 23:02:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bronevichok.ru; h=date :from:to:subject:message-id:mime-version:content-type; s= selector1; bh=nr5QXamxD5o/fFl0Gej0bNwU4bM=; b=J0VFplvIislzdByAXk Ny9R9qkQHrTqHE96mWEl4FjIksXC3mxoUFYdcCaH71nKGI0UXm//uOG1OuvTvIax W7Zq6IQKjDYncKyurXbbTzPXaPzBfVsUGgQhpFf8A4TEJHq/Eh4wPKfSv9h2bYJi Wxtbyopkh3Adi48xPa/Oarjbh5aXf6Kxp1hna/IE8Goq/q6QcmY4GWeNHcUdla1D 6khl3BFarvRDS0y+yTLUAdx218kAU2ua014VsSb8JQlcDYpQF6auFuQV7iLBQcWt kkl9NLnHO2WFks7t4/O3IXG7YSFTbonHevNhwLGvyWlpToSRZFFyZnWgZ1EGyAc8 1rfg== DomainKey-Signature: a=rsa-sha1; c=nofws; d=bronevichok.ru; h=date:from :to:subject:message-id:mime-version:content-type; q=dns; s= selector1; b=GaJqkzeEfIzezRwuQHB5PXObfYfLmQCCyQUGesjXXX0XyWVkuzM 9lmso/47uEgPso+sBIRqDHK+7U/HLXWnGFL/O+YcaN+Bit+rwnqrgzC/fSIogtQp Zq6FNzo63NfVCC69bA0exaD1FD6SsVlu85oLKa6J5mzH4tt6spfmCZZAz1Hx15s0 wxxTsgCLgeyNFqdR/upjPd0tS58Ye9o1aptOjFBm0Y2TzeWQDxnXedjc14di2tbb u3wxg/wbixzt2iKluVYesTHTWhCQmsuVCx+3XaabduTlKncTORp9HubIc7rO6QRh xHhY6cC5qPMWVjK0yHkUUW2Oj8UeK5LRzfw== Received: by mx1.bronevichok.ru (OpenSMTPD) with ESMTPSA id f794c79b (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Tue, 17 Jul 2018 23:02:26 +0200 (CEST) Date: Wed, 18 Jul 2018 00:02:22 +0300 From: Sergey Bronnikov To: kvm@vger.kernel.org Subject: [kvm-unit-tests PATCH][v3] scripts: add option to output in TAP format Message-ID: <20180717210222.GA10311@pony.bronevichok.ru> MIME-Version: 1.0 Content-Disposition: inline X-Operating-System: OpenBSD X-Url: https://bronevichok.ru/ User-Agent: Mutt/1.9.4 (2018-02-28) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Option -t adds ability to output in standard machine-readable format - TAP (TestAnythingProtocol). Example: $ ./run_tests.sh -t -j5 TAP version 13 ok smptest ok ioapic-split ok ioapic ok smptest3 ok vmexit_cpuid ok vmexit_vmcall ok vmexit_mov_from_cr8 ok vmexit_mov_to_cr8 ok vmexit_inl_pmtimer ok vmexit_ipi ok vmexit_ipi_halt ... Signed-off-by: Sergey Bronnikov Reviewed-by: Andrew Jones --- run_tests.sh | 21 +++++++++++++++++++-- scripts/runtime.bash | 48 +++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/run_tests.sh b/run_tests.sh index aa2e65f..102c806 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -1,6 +1,7 @@ #!/usr/bin/env bash verbose="no" +tap_output="no" run_all_tests="no" # don't run nodefault tests if [ ! -f config.mak ]; then @@ -14,7 +15,7 @@ function usage() { cat < $unittest_log_dir/SUMMARY +if [[ $tap_output == "yes" ]]; then + test_number=0 + echo "TAP version 13" +fi trap "wait; exit 130" SIGINT for_each_unittest $config run_task # wait until all tasks finish wait + +if [[ $tap_output == "yes" ]]; then + echo "1..$test_number" +fi diff --git a/scripts/runtime.bash b/scripts/runtime.bash index a31ae91..aac8383 100644 --- a/scripts/runtime.bash +++ b/scripts/runtime.bash @@ -50,6 +50,36 @@ skip_nodefault() done } +function print_result() +{ + # output test results in a TAP format + # https://testanything.org/tap-version-13-specification.html + + local status="$1" + local testname="$2" + local summary="$3" + local reason="$4" + + if [[ $tap_output == "no" ]]; then + if [ -z "$reason" ]; then + echo "`$status` $testname $summary" + else + echo "`$status` $testname ($reason)" + fi + return + fi + + if [[ $status == "FAIL" ]]; then + echo "not ok $testname $reason" + elif [[ $status == "PASS" ]]; then + echo "ok $testname" + elif [[ $status == "SKIP" ]]; then + echo "ok $testname # SKIP $reason" + else + echo "not ok # TODO unknown test status" + fi +} + function run() { local testname="$1" @@ -72,12 +102,12 @@ function run() if [ -z "$only_group" ] && grep -qw "nodefault" <<<$groups && skip_nodefault; then - echo -e "`SKIP` $testname (test marked as manual run only)" + print_result "SKIP" $testname "" "test marked as manual run only" return; fi if [ -n "$arch" ] && [ "$arch" != "$ARCH" ]; then - echo "`SKIP` $1 ($arch only)" + print_result "SKIP" $testname "" "$arch only" return 2 fi @@ -88,13 +118,13 @@ function run() path=${check_param%%=*} value=${check_param#*=} if [ "$path" ] && [ "$(cat $path)" != "$value" ]; then - echo "`SKIP` $1 ($path not equal to $value)" + print_result "SKIP" $testname "" "$path not equal to $value" return 2 fi done last_line=$(premature_failure > >(tail -1)) && { - echo "`SKIP` $1 ($last_line)" + print_result "SKIP" $testname "" "$last_line" return 77 } @@ -115,15 +145,15 @@ function run() [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $kernel) if [ $ret -eq 0 ]; then - echo "`PASS` $1 $summary" + print_result "PASS" $testname "$summary" elif [ $ret -eq 77 ]; then - echo "`SKIP` $1 $summary" + print_result "SKIP" $testname "$summary" elif [ $ret -eq 124 ]; then - echo "`FAIL` $1 (timeout; duration=$timeout)" + print_result "FAIL" $testname "" "timeout; duration=$timeout" elif [ $ret -gt 127 ]; then - echo "`FAIL` $1 (terminated on SIG$(kill -l $(($ret - 128))))" + print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))" else - echo "`FAIL` $1 $summary" + print_result "FAIL" $testname "$summary" fi return $ret