[2/8] report: derive an xml schema for the xunit report

Commit Message

Darrick J. Wong Dec. 20, 2022, 12:01 a.m. UTC

From: Darrick J. Wong <[e-mail]>

The "xunit" report format emits an XML document that more or less
follows the junit xml schema.  However, there are two major exceptions:

1. fstests does not emit an @errors attribute on the testsuite element
because we don't have the concept of unanticipated errors such as
"unchecked throwables".

2. The system-out/system-err elements sound like they belong under the
testcase element, though the schema itself imprecisely says "while the
test was executed".  The schema puts them under the top-level testsuite
element, but we put them under the testcase element.

Define an xml schema for the xunit report format, and update the xml
headers to link to the schema file.  This enables consumers of the
reports to check mechanically that the incoming document follows the
format.

Signed-off-by: Darrick J. Wong <[e-mail]>
---
 common/report |   15 +++-
 doc/xunit.xsd |  226 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 238 insertions(+), 3 deletions(-)
 create mode 100644 doc/xunit.xsd

Comments

Qu Wenruo Dec. 20, 2022, 2:18 a.m. UTC | #1

On 2022/12/20 08:01, Darrick J. Wong wrote:
> From: Darrick J. Wong <[e-mail]>
> 
> The "xunit" report format emits an XML document that more or less
> follows the junit xml schema.  However, there are two major exceptions:
> 
> 1. fstests does not emit an @errors attribute on the testsuite element
> because we don't have the concept of unanticipated errors such as
> "unchecked throwables".
> 
> 2. The system-out/system-err elements sound like they belong under the
> testcase element, though the schema itself imprecisely says "while the
> test was executed".  The schema puts them under the top-level testsuite
> element, but we put them under the testcase element.
> 
> Define an xml schema for the xunit report format, and update the xml
> headers to link to the schema file.  This enables consumers of the
> reports to check mechanically that the incoming document follows the
> format.

One thing is, does the official XMLs use tabs as indents?

We got some lines definitely too long for human to read.
Any way to make them a little better?

But overall, it really defines a good standard for us to follow.
This is definitely a good start.

> 
> Signed-off-by: Darrick J. Wong <[e-mail]>
> ---
[...]
> +						<xs:choice minOccurs="0" maxOccurs="2">

For this, I prefer maxOccurs to be at least 3.

We have 3 different possible outputs:

- $seqnum.out.bad
- $seqnum.full
- $seqnum.dmesg

[...]

> +								</xs:annotation>
> +								<xs:simpleType>
> +									<xs:restriction base="pre-string">
> +										<xs:whiteSpace value="preserve"/>
> +									</xs:restriction>
> +								</xs:simpleType>
> +							</xs:element>
> +							<xs:element name="system-err" minOccurs="0" maxOccurs="1">
> +								<xs:annotation>
> +									<xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>

We don't use stderr, but $seqnum.full and $seqnum.dmesg.

Or can we just rename the "system-out" and "system-err" to something 
fstests specific? E.g.

- system-output
- system-full
- system-dmesg

Or the system-err/out thing is mostly to keep the compatibility?
If so, I'd prefer some properties to make it explicit which output 
represents which fstests specific output.

> +								</xs:annotation>
> +								<xs:simpleType>
> +									<xs:restriction base="pre-string">
> +										<xs:whiteSpace value="preserve"/>
> +									</xs:restriction>
> +								</xs:simpleType>
> +							</xs:element>
> +						</xs:choice>
> +					</xs:sequence>
> +					<xs:attribute name="name" type="xs:token" use="required">
> +						<xs:annotation>
> +							<xs:documentation xml:lang="en">Name of the test method</xs:documentation>

Can we update the description to something more fstests specific, better 
with an example?
Like "test case number, e.g. generic/001".

This can apply to most description copied from the JUnit doc.

[...]
> +		<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
> +			<xs:annotation>
> +				<xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
> +			</xs:annotation>

This means the start time, thus all our existing timestamp is not 
following the spec already.

Thanks,
Qu

Darrick J. Wong Feb. 14, 2023, 6:54 p.m. UTC | #2

On Tue, Dec 20, 2022 at 10:18:18AM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/12/20 08:01, Darrick J. Wong wrote:
> > From: Darrick J. Wong <[e-mail]>
> > 
> > The "xunit" report format emits an XML document that more or less
> > follows the junit xml schema.  However, there are two major exceptions:
> > 
> > 1. fstests does not emit an @errors attribute on the testsuite element
> > because we don't have the concept of unanticipated errors such as
> > "unchecked throwables".
> > 
> > 2. The system-out/system-err elements sound like they belong under the
> > testcase element, though the schema itself imprecisely says "while the
> > test was executed".  The schema puts them under the top-level testsuite
> > element, but we put them under the testcase element.
> > 
> > Define an xml schema for the xunit report format, and update the xml
> > headers to link to the schema file.  This enables consumers of the
> > reports to check mechanically that the incoming document follows the
> > format.
> 
> One thing is, does the official XMLs use tabs as indents?

XML doesn't care one way or another:
https://www.w3.org/TR/xml/#sec-white-space

> We got some lines definitely too long for human to read.
> Any way to make them a little better?

Eh, I guess I could change them to two spaces, given the indentyness of
the schema.

> But overall, it really defines a good standard for us to follow.
> This is definitely a good start.

<nod>

> > 
> > Signed-off-by: Darrick J. Wong <[e-mail]>
> > ---
> [...]
> > +						<xs:choice minOccurs="0" maxOccurs="2">
> 
> For this, I prefer maxOccurs to be at least 3.
> 
> We have 3 different possible outputs:
> 
> - $seqnum.out.bad
> - $seqnum.full
> - $seqnum.dmesg
> 
> [...]
> 
> > +								</xs:annotation>
> > +								<xs:simpleType>
> > +									<xs:restriction base="pre-string">
> > +										<xs:whiteSpace value="preserve"/>
> > +									</xs:restriction>
> > +								</xs:simpleType>
> > +							</xs:element>
> > +							<xs:element name="system-err" minOccurs="0" maxOccurs="1">
> > +								<xs:annotation>
> > +									<xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
> 
> We don't use stderr, but $seqnum.full and $seqnum.dmesg.
> 
> Or can we just rename the "system-out" and "system-err" to something fstests
> specific? E.g.
> 
> - system-output
> - system-full
> - system-dmesg
> 
> Or the system-err/out thing is mostly to keep the compatibility?
> If so, I'd prefer some properties to make it explicit which output
> represents which fstests specific output.

I'll change those, since that's one of the major divergences from the
upstream junit xml schema.  junit says system-out should capture the
stdout of the whole testsuite, not an individual testcase.

> 
> > +								</xs:annotation>
> > +								<xs:simpleType>
> > +									<xs:restriction base="pre-string">
> > +										<xs:whiteSpace value="preserve"/>
> > +									</xs:restriction>
> > +								</xs:simpleType>
> > +							</xs:element>
> > +						</xs:choice>
> > +					</xs:sequence>
> > +					<xs:attribute name="name" type="xs:token" use="required">
> > +						<xs:annotation>
> > +							<xs:documentation xml:lang="en">Name of the test method</xs:documentation>
> 
> Can we update the description to something more fstests specific, better
> with an example?
> Like "test case number, e.g. generic/001".
> 
> This can apply to most description copied from the JUnit doc.

Ok.

> [...]
> > +		<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
> > +			<xs:annotation>
> > +				<xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
> > +			</xs:annotation>
> 
> This means the start time, thus all our existing timestamp is not following
> the spec already.

I wrote my comments about this part in the thread about patch 1, so
let's leave the discussion there.

--D

> Thanks,
> Qu

Patch

diff --git a/common/report b/common/report
index 64f9c86648..1d84650270 100644
--- a/common/report
+++ b/common/report
@@ -48,9 +48,18 @@  _xunit_make_section_report()
 	if [ -z "$date_time" ]; then
 		date_time=$(date +"%F %T")
 	fi
-	local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
-	local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
-	echo "<testsuite name=\"xfstests\" $stats  $hw_info >" >> $REPORT_DIR/result.xml
+
+	local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+	cat >> $REPORT_DIR/result.xml << ENDL
+<testsuite
+ xmlns="$fstests_ns"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="$fstests_ns $fstests_ns/tree/doc/xunit.xsd"
+
+ name="xfstests"
+ failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
+ hostname="$HOST" timestamp="${date_time/ /T}">
+ENDL
 
 	# Properties
 	echo -e "\t<properties>" >> $REPORT_DIR/result.xml
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
new file mode 100644
index 0000000000..0aef8a9839
--- /dev/null
+++ b/doc/xunit.xsd
@@ -0,0 +1,226 @@ 
+<?xml version="1.0" encoding="UTF-8"?>
+<xs:schema
+ xmlns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+
+ targetNamespace="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ elementFormDefault="qualified"
+ attributeFormDefault="unqualified">
+	<xs:annotation>
+		<xs:documentation xml:lang="en">fstests xunit test result schema, derived from https://github.com/windyroad/JUnit-Schema</xs:documentation>
+	</xs:annotation>
+	<xs:element name="testsuite" type="testsuite"/>
+	<xs:simpleType name="ISO8601_DATETIME_PATTERN">
+		<xs:restriction base="xs:dateTime">
+			<xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
+		</xs:restriction>
+	</xs:simpleType>
+	<xs:element name="testsuites">
+		<xs:annotation>
+			<xs:documentation xml:lang="en">Contains an aggregation of testsuite results</xs:documentation>
+		</xs:annotation>
+		<xs:complexType>
+			<xs:sequence>
+				<xs:element name="testsuite" minOccurs="0" maxOccurs="unbounded">
+					<xs:complexType>
+						<xs:complexContent>
+							<xs:extension base="testsuite">
+								<xs:attribute name="package" type="xs:token" use="required">
+									<xs:annotation>
+										<xs:documentation xml:lang="en">Derived from testsuite/@name in the non-aggregated documents</xs:documentation>
+									</xs:annotation>
+								</xs:attribute>
+								<xs:attribute name="id" type="xs:int" use="required">
+									<xs:annotation>
+										<xs:documentation xml:lang="en">Starts at '0' for the first testsuite and is incremented by 1 for each following testsuite</xs:documentation>
+									</xs:annotation>
+								</xs:attribute>
+							</xs:extension>
+						</xs:complexContent>
+					</xs:complexType>
+				</xs:element>
+			</xs:sequence>
+		</xs:complexType>
+	</xs:element>
+	<xs:complexType name="testsuite">
+		<xs:annotation>
+			<xs:documentation xml:lang="en">Contains the results of executing a testsuite</xs:documentation>
+		</xs:annotation>
+		<xs:sequence>
+			<xs:element name="properties">
+				<xs:annotation>
+					<xs:documentation xml:lang="en">Properties (e.g., environment settings) set during test execution</xs:documentation>
+				</xs:annotation>
+				<xs:complexType>
+					<xs:sequence>
+						<xs:element name="property" minOccurs="0" maxOccurs="unbounded">
+							<xs:complexType>
+								<xs:attribute name="name" use="required">
+									<xs:simpleType>
+										<xs:restriction base="xs:token">
+											<xs:minLength value="1"/>
+										</xs:restriction>
+									</xs:simpleType>
+								</xs:attribute>
+								<xs:attribute name="value" type="xs:string" use="required"/>
+							</xs:complexType>
+						</xs:element>
+					</xs:sequence>
+				</xs:complexType>
+			</xs:element>
+			<xs:element name="testcase" minOccurs="0" maxOccurs="unbounded">
+				<xs:complexType>
+					<xs:sequence>
+						<xs:choice minOccurs="0" maxOccurs="1">
+							<xs:element name="skipped" minOccurs="0" maxOccurs="1">
+								<xs:annotation>
+									<xs:documentation xml:lang="en">Indicates that the test was skipped.</xs:documentation>
+								</xs:annotation>
+								<xs:complexType>
+									<xs:simpleContent>
+										<xs:extension base="pre-string">
+											<xs:attribute name="message" type="xs:string">
+												<xs:annotation>
+													<xs:documentation xml:lang="en">The message specifying why the test case was skipped</xs:documentation>
+												</xs:annotation>
+											</xs:attribute>
+										</xs:extension>
+									</xs:simpleContent>
+								</xs:complexType>
+							</xs:element>
+							<xs:element name="error" minOccurs="0" maxOccurs="1">
+								<xs:annotation>
+									<xs:documentation xml:lang="en">Indicates that the test errored.  An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test. Contains as a text node relevant data for the error, e.g., a stack trace</xs:documentation>
+								</xs:annotation>
+								<xs:complexType>
+									<xs:simpleContent>
+										<xs:extension base="pre-string">
+											<xs:attribute name="message" type="xs:string">
+												<xs:annotation>
+													<xs:documentation xml:lang="en">The error message. e.g., if a java exception is thrown, the return value of getMessage()</xs:documentation>
+												</xs:annotation>
+											</xs:attribute>
+											<xs:attribute name="type" type="xs:string" use="required">
+												<xs:annotation>
+													<xs:documentation xml:lang="en">The type of error that occured. e.g., if a java execption is thrown the full class name of the exception.</xs:documentation>
+												</xs:annotation>
+											</xs:attribute>
+										</xs:extension>
+									</xs:simpleContent>
+								</xs:complexType>
+							</xs:element>
+							<xs:element name="failure">
+								<xs:annotation>
+									<xs:documentation xml:lang="en">Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals. Contains as a text node relevant data for the failure, e.g., a stack trace</xs:documentation>
+								</xs:annotation>
+								<xs:complexType>
+									<xs:simpleContent>
+										<xs:extension base="pre-string">
+											<xs:attribute name="message" type="xs:string">
+												<xs:annotation>
+													<xs:documentation xml:lang="en">The message specified in the assert</xs:documentation>
+												</xs:annotation>
+											</xs:attribute>
+											<xs:attribute name="type" type="xs:string" use="required">
+												<xs:annotation>
+													<xs:documentation xml:lang="en">The type of the assert.</xs:documentation>
+												</xs:annotation>
+											</xs:attribute>
+										</xs:extension>
+									</xs:simpleContent>
+								</xs:complexType>
+							</xs:element>
+						</xs:choice>
+						<xs:choice minOccurs="0" maxOccurs="2">
+							<xs:element name="system-out" minOccurs="0" maxOccurs="1">
+								<xs:annotation>
+									<xs:documentation xml:lang="en">Data that was written to standard out while the test was executed</xs:documentation>
+								</xs:annotation>
+								<xs:simpleType>
+									<xs:restriction base="pre-string">
+										<xs:whiteSpace value="preserve"/>
+									</xs:restriction>
+								</xs:simpleType>
+							</xs:element>
+							<xs:element name="system-err" minOccurs="0" maxOccurs="1">
+								<xs:annotation>
+									<xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
+								</xs:annotation>
+								<xs:simpleType>
+									<xs:restriction base="pre-string">
+										<xs:whiteSpace value="preserve"/>
+									</xs:restriction>
+								</xs:simpleType>
+							</xs:element>
+						</xs:choice>
+					</xs:sequence>
+					<xs:attribute name="name" type="xs:token" use="required">
+						<xs:annotation>
+							<xs:documentation xml:lang="en">Name of the test method</xs:documentation>
+						</xs:annotation>
+					</xs:attribute>
+					<xs:attribute name="classname" type="xs:token" use="required">
+						<xs:annotation>
+							<xs:documentation xml:lang="en">Full class name for the class the test method is in.</xs:documentation>
+						</xs:annotation>
+					</xs:attribute>
+					<xs:attribute name="time" type="xs:decimal" use="required">
+						<xs:annotation>
+							<xs:documentation xml:lang="en">Time taken (in seconds) to execute the test</xs:documentation>
+						</xs:annotation>
+					</xs:attribute>
+				</xs:complexType>
+			</xs:element>
+		</xs:sequence>
+		<xs:attribute name="name" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents</xs:documentation>
+			</xs:annotation>
+			<xs:simpleType>
+				<xs:restriction base="xs:token">
+					<xs:minLength value="1"/>
+				</xs:restriction>
+			</xs:simpleType>
+		</xs:attribute>
+		<xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
+			</xs:annotation>
+		</xs:attribute>
+		<xs:attribute name="hostname" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined.</xs:documentation>
+			</xs:annotation>
+			<xs:simpleType>
+				<xs:restriction base="xs:token">
+					<xs:minLength value="1"/>
+				</xs:restriction>
+			</xs:simpleType>
+		</xs:attribute>
+		<xs:attribute name="tests" type="xs:int" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">The total number of tests in the suite</xs:documentation>
+			</xs:annotation>
+		</xs:attribute>
+		<xs:attribute name="failures" type="xs:int" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals</xs:documentation>
+			</xs:annotation>
+		</xs:attribute>
+		<xs:attribute name="skipped" type="xs:int" use="optional">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">The total number of ignored or skipped tests in the suite.</xs:documentation>
+			</xs:annotation>
+		</xs:attribute>
+		<xs:attribute name="time" type="xs:decimal" use="required">
+			<xs:annotation>
+				<xs:documentation xml:lang="en">Time taken (in seconds) to execute the tests in the suite</xs:documentation>
+			</xs:annotation>
+		</xs:attribute>
+	</xs:complexType>
+	<xs:simpleType name="pre-string">
+		<xs:restriction base="xs:string">
+			<xs:whiteSpace value="preserve"/>
+		</xs:restriction>
+	</xs:simpleType>
+</xs:schema>