From patchwork Tue Jul 25 11:57:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony PERARD X-Patchwork-Id: 9861919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C351B602B1 for ; Tue, 25 Jul 2017 12:19:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9E9F27E63 for ; Tue, 25 Jul 2017 12:19:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ADDC427F2B; Tue, 25 Jul 2017 12:19:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E338427E63 for ; Tue, 25 Jul 2017 12:18:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dZylM-0004pJ-TN; Tue, 25 Jul 2017 12:16:32 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dZylL-0004mr-QM for xen-devel@lists.xenproject.org; Tue, 25 Jul 2017 12:16:31 +0000 Received: from [85.158.143.35] by server-10.bemta-6.messagelabs.com id 74/94-03582-E9637795; Tue, 25 Jul 2017 12:16:30 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprNIsWRWlGSWpSXmKPExsXitHSDve48s/J IgxN7ZS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1ow1r9rYC7aZVazetpulgXGrUhcjJ4eEgL9E z4tuNhCbTcBAYsX0q4xdjBwcIgIqErf3GoCEmQUiJF7cWAVWIixQJPFsai8jiM0ioCpxa91vF pByXgF7iS/7IiEmyktM7J0GVsIJFP65/zo7iC0kYCexeecGKFtN4sbCZSwgNq+AoMTJmU9YIF ZJSBx88YJ5AiPvLCSpWUhSCxiZVjFqFKcWlaUW6Rpa6CUVZaZnlOQmZuboGhqY6eWmFhcnpqf mJCYV6yXn525iBAYOAxDsYLy5MeAQoyQHk5Io7zfd8kghvqT8lMqMxOKM+KLSnNTiQ4wyHBxK ErzPTYFygkWp6akVaZk5wBCGSUtw8CiJ8J4zAUrzFhck5hZnpkOkTjFacmxYvf4LE8eUd7uA5 KsJ/78xCbHk5eelSonzzgOZJwDSkFGaBzcOFmeXGGWlhHkZgQ4U4ilILcrNLEGVf8UozsGoJM zrAzKFJzOvBG7rK6CDmIAOmjOjFOSgkkSElFQDI7vhrsJP6lGClcuWHKrcrjblJEeVrPTGFRl LNN9qHn0ZPKcjKCD4usCjld/6z647nT2B+ftP7k8l4jq6kUnFPN+jqu0f+sj0x34wMv3spy9z liO82VJmpWV5mUdV0GHB/1PXv06stZru/WT96vKrN7Yx9Bi07/gVqlSx0FjGrlSHc/VTa4vXS izFGYmGWsxFxYkAE7wQy64CAAA= X-Env-Sender: prvs=372f2a2d8=anthony.perard@citrix.com X-Msg-Ref: server-7.tower-21.messagelabs.com!1500984981!74878560!4 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 27864 invoked from network); 25 Jul 2017 12:16:30 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-7.tower-21.messagelabs.com with RC4-SHA encrypted SMTP; 25 Jul 2017 12:16:30 -0000 X-IronPort-AV: E=Sophos;i="5.40,411,1496102400"; d="scan'208";a="441196223" From: Anthony PERARD To: Date: Tue, 25 Jul 2017 12:57:54 +0100 Message-ID: <20170725115759.21895-20-anthony.perard@citrix.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170725115759.21895-1-anthony.perard@citrix.com> References: <20170725115759.21895-1-anthony.perard@citrix.com> MIME-Version: 1.0 Cc: Anthony PERARD , Ian Jackson Subject: [Xen-devel] [OSSTEST PATCH v13 19/24] TestSupport: Implement target_subunit_cmd a subunit stream parser into substeps X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP target_subunit_cmd can be used like target_cmd, but the command would needs to output a subunit v1 stream, which will be parsed and turned into osstest substeps. The command can be `| subunit-2to1` in order to turn a subunit v2 stream into v1. Currently, time is not taken into account, and all substeps will have bogus timestamp as the output of the command is parsed after it has runned. This is a description of the subunit v1 protocol, taken from python-subunit README, or https://pypi.python.org/pypi/python-subunit test|testing|test:|testing: test LABEL success|success:|successful|successful: test LABEL success|success:|successful|successful: test LABEL DETAILS failure: test LABEL failure: test LABEL DETAILS error: test LABEL error: test LABEL DETAILS skip[:] test LABEL skip[:] test LABEL DETAILS xfail[:] test LABEL xfail[:] test LABEL DETAILS uxsuccess[:] test LABEL uxsuccess[:] test LABEL DETAILS progress: [+|-]X progress: push progress: pop tags: [-]TAG ... time: YYYY-MM-DD HH:MM:SSZ LABEL: UTF8* NAME: UTF8* DETAILS ::= BRACKETED | MULTIPART BRACKETED ::= '[' CR UTF8-lines ']' CR MULTIPART ::= '[ multipart' CR PART* ']' CR PART ::= PART_TYPE CR NAME CR PART_BYTES CR PART_TYPE ::= Content-Type: type/sub-type(;parameter=value,parameter=value) PART_BYTES ::= (DIGITS CR LF BYTE{DIGITS})* '0' CR LF Signed-off-by: Anthony PERARD --- Notes: Changes in v13: - also parse multipart output - add every possible test result - use target_cmd_stashed Osstest/TestSupport.pm | 117 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm index 135289b..ba55967 100644 --- a/Osstest/TestSupport.pm +++ b/Osstest/TestSupport.pm @@ -55,7 +55,7 @@ BEGIN { target_cmd_root_status target_cmd_output_root_status target_cmd_root target_cmd target_cmd_build - target_cmd_stashed + target_cmd_stashed target_subunit_cmd target_cmd_output_root target_cmd_output target_cmd_inputfh_root sshuho target_getfile target_getfile_root @@ -775,6 +775,121 @@ sub target_cmd_stashed ($$$;$$) { return "$stash/$$leafref"; } +sub subunit_result_to_osstest_result ($) { + my ($ret) = @_; + return "pass" if $ret eq "success" or $ret eq "successful"; + return "fail" if $ret eq "failure"; + return "skip" if $ret eq "skip"; + return "fail" if $ret eq "error"; + # expected failure + return "pass" if $ret eq "xfail"; + # unexpected success + return "fail" if $ret eq "uxsuccess"; + die "subunit_result_to_osstest_result unexpected result $ret"; +} +sub subunit_sanitize_testname ($) { + my ($testname) = @_; + $testname =~ s'[^_.()\[\]/~0-9a-zA-Z-]'_'g; + return $testname; +} + +# Like target_cmd, but parse the command output as a subunit v1 stream and make +# a substep out of each subunit test. +sub target_subunit_cmd ($$;$$) { + my ($tho,$tcmd,$timeout,$extrasshopts) = @_; + my $filename = "subunit-output"; + my $path = target_cmd_stashed($tho, \$filename, $tcmd, $timeout, + $extrasshopts); + + open my $stdout, "$path" or die "$path: $!"; + + my $logfilename = undef; + my $fh = undef; + + while (<$stdout>) { + if (/^time: (\d+)-(\d+)-(\d+) (\d+):(\d+):(\d+)(\.\d+)?Z$/) { + # This is the timestamp for the next events + } elsif (/^test(?:ing)?:? (.+)\n/) { + # Start of a new test. + $logfilename = subunit_sanitize_testname($1) . '.log'; + $fh = open_unique_stashfile(\$logfilename); + substep_start(subunit_sanitize_testname($1), $logfilename); + } elsif (/^(success(?:ful)?|failure|skip|error|xfail|uxsuccess): + \ (.+?)(\ \[(\ multipart)?)?$/x) { + # Result of a test, with its output. + my $result = $1; + my $testname = $2; + my $have_details = $3; + my $is_multipart = $4; + + if ($have_details) { + if ($is_multipart) { + # Test output + while (<$stdout>) { + # part content-type + # from https://tools.ietf.org/html/rfc6838#section-4.2 + my $restricted_name = qr'[a-zA-Z0-9][a-zA-Z0-9!#$&^_.+-]*'; + if (m{ ^Content-Type:\s+ + $restricted_name/$restricted_name # type/sub-type + # parameters + (?:\s*;\s*$restricted_name=[^,]+ + (?:,\s*$restricted_name=[^,]+)*) + \s*$ + }xi) { + print $fh $_ or die $!; + + # part name + my $line = <$stdout>; + print $fh $line or die $!; + + # Read chunks of a part + while (<$stdout>) { + if (/^([0-9A-F]+)\r$/i) { + my $chunk_size = hex($1); + my $chunk; + + last if $chunk_size == 0; + read $stdout, $chunk, $chunk_size; + print $fh $chunk or die $!; + } else { + # Unexpected output + chomp; + logm("*** $_"); + } + } + } elsif (/^\]$/) { + last; + } else { + # Unexpected output + chomp; + logm("*** $_"); + } + } + } else { + # Simple non-multipart test output. + while (<$stdout>) { + last if (/^\]$/); + print $fh $_ or die $!; + } + } + } + close $fh or die $!; + substep_finish(subunit_sanitize_testname($testname), + subunit_result_to_osstest_result($result)); + } elsif (/^tags: .+/) { + # unused + } elsif (/^progress: (?:[+-]?\d+|push|pop)$/) { + # unused + } else { + # Unexpected output + chomp; + logm("*** $_"); + } + } + + close $stdout or die $!; +} + sub poll_loop ($$$&) { my ($maxwait, $interval, $what, $code) = @_; # $code should return undef when all is well