[kvm-unit-test,5/5] travis.yml: Expect that at least one test succeeds
diff mbox series

Message ID 20191113112649.14322-6-thuth@redhat.com
State New
Headers show
Series
  • Improvements for the Travis CI
Related show

Commit Message

Thomas Huth Nov. 13, 2019, 11:26 a.m. UTC
While working on the travis.yml file, I've run into cases where
all tests are reported as "SKIP" by the run_test.sh script (e.g.
when QEMU could not be started). This should not result in a
successful test run, so mark it as failed if not at least one
test passed.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

Comments

Alex Bennée Nov. 13, 2019, 3:06 p.m. UTC | #1
Thomas Huth <thuth@redhat.com> writes:

> While working on the travis.yml file, I've run into cases where
> all tests are reported as "SKIP" by the run_test.sh script (e.g.
> when QEMU could not be started). This should not result in a
> successful test run, so mark it as failed if not at least one
> test passed.

But doesn't this mean you could have everything fail except one pass and
still report success?

>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .travis.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index 9ceb04d..aacf7d2 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -115,3 +115,4 @@ script:
>    - make -j3
>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>    - if grep -q FAIL results.txt ; then exit 1 ; fi
> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi


--
Alex Bennée
Thomas Huth Nov. 13, 2019, 3:17 p.m. UTC | #2
On 13/11/2019 16.06, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> While working on the travis.yml file, I've run into cases where
>> all tests are reported as "SKIP" by the run_test.sh script (e.g.
>> when QEMU could not be started). This should not result in a
>> successful test run, so mark it as failed if not at least one
>> test passed.
> 
> But doesn't this mean you could have everything fail except one pass and
> still report success?

The FAILs are already handled one line earlier...

>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .travis.yml | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 9ceb04d..aacf7d2 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -115,3 +115,4 @@ script:
>>    - make -j3
>>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>>    - if grep -q FAIL results.txt ; then exit 1 ; fi

... here -----------^

>> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi

Maybe it would also be nicer to provide proper exit values in the
run_tests.sh script, but the logic there gives me a bad headache...
grep'ing for FAIL and PASS in the yml script is way more easy.

 Thomas
Alex Bennée Nov. 13, 2019, 4:03 p.m. UTC | #3
Thomas Huth <thuth@redhat.com> writes:

> On 13/11/2019 16.06, Alex Bennée wrote:
>>
>> Thomas Huth <thuth@redhat.com> writes:
>>
>>> While working on the travis.yml file, I've run into cases where
>>> all tests are reported as "SKIP" by the run_test.sh script (e.g.
>>> when QEMU could not be started). This should not result in a
>>> successful test run, so mark it as failed if not at least one
>>> test passed.
>>
>> But doesn't this mean you could have everything fail except one pass and
>> still report success?
>
> The FAILs are already handled one line earlier...

Oops I should have noticed that:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>
>>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>  .travis.yml | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 9ceb04d..aacf7d2 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -115,3 +115,4 @@ script:
>>>    - make -j3
>>>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>>>    - if grep -q FAIL results.txt ; then exit 1 ; fi
>
> ... here -----------^
>
>>> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi
>
> Maybe it would also be nicer to provide proper exit values in the
> run_tests.sh script, but the logic there gives me a bad headache...
> grep'ing for FAIL and PASS in the yml script is way more easy.
>
>  Thomas


--
Alex Bennée

Patch
diff mbox series

diff --git a/.travis.yml b/.travis.yml
index 9ceb04d..aacf7d2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -115,3 +115,4 @@  script:
   - make -j3
   - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
   - if grep -q FAIL results.txt ; then exit 1 ; fi
+  - if ! grep -q PASS results.txt ; then exit 1 ; fi