diff mbox series

[v2,02/20] Acceptance tests: show avocado test execution by default

Message ID 20190202005610.24048-3-crosa@redhat.com (mailing list archive)
State New, archived
Headers show
Series Acceptance Tests: target architecture support | expand

Commit Message

Cleber Rosa Feb. 2, 2019, 12:55 a.m. UTC
The current version of the "check-acceptance" target will only show
one line for execution of all tests.  That's probably OK if the tests
to be run are quick enough and they're always the same.

But, there's already one test alone that takes on average ~5 seconds
to run, we intend to adapt the list of tests to match the user's build
environment (among other choices).

Because of that, let's present the default Avocado UI by default.
Users can always choose a different output by setting the AVOCADO_SHOW
variable.

Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml            | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Comments

Cornelia Huck Feb. 6, 2019, 2:36 p.m. UTC | #1
On Fri,  1 Feb 2019 19:55:52 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> The current version of the "check-acceptance" target will only show
> one line for execution of all tests.  That's probably OK if the tests
> to be run are quick enough and they're always the same.
> 
> But, there's already one test alone that takes on average ~5 seconds
> to run, we intend to adapt the list of tests to match the user's build
> environment (among other choices).

Btw: What are our expectations regarding execution time for tests?
Especially if we continue adding tests, and architecture-specific tests
are bound to be slower if run on a foreign architecture via tcg.

Would a make check-acceptance-quick command make sense? ("I only want
to verify quickly that I didn't break too much, so run the quicker
tests only, probably only for my host architecture")

> 
> Because of that, let's present the default Avocado UI by default.
> Users can always choose a different output by setting the AVOCADO_SHOW
> variable.
> 
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .travis.yml            | 2 +-
>  tests/Makefile.include | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
Cleber Rosa Feb. 6, 2019, 5:02 p.m. UTC | #2
On 2/6/19 9:36 AM, Cornelia Huck wrote:
> On Fri,  1 Feb 2019 19:55:52 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
>> The current version of the "check-acceptance" target will only show
>> one line for execution of all tests.  That's probably OK if the tests
>> to be run are quick enough and they're always the same.
>>
>> But, there's already one test alone that takes on average ~5 seconds
>> to run, we intend to adapt the list of tests to match the user's build
>> environment (among other choices).
> 
> Btw: What are our expectations regarding execution time for tests?
> Especially if we continue adding tests, and architecture-specific tests
> are bound to be slower if run on a foreign architecture via tcg.
> 
> Would a make check-acceptance-quick command make sense? ("I only want
> to verify quickly that I didn't break too much, so run the quicker
> tests only, probably only for my host architecture")
> 

Yes, it definitely makes sense.  Now, let me know if the following also
makes sense to you:

1) Because these tests focus on functional testing, the default
target/shortcut ("make check-acceptance") should run the complete set of
test cases (including the slow ones).

2) Requirements vary greatly from user to user, to while adding a
"check-acceptance-quick" is fine, you just mentioned one extra test
execution variation ("for my host architecture").  For those, the idea
is that:

 a) "make check-acceptance[-quick]" will adapt to the build environment
(if you only built s390x targets, that's all it's going to use)

 b) "avocado" command line interface *should* be easy enough to fulfill
other requirements, and not necessarily require a "make" target.  For
instance, if you're only interested in your host arch and one specific
machine type, a command line such as the following should do the trick:

  $ make check-venv
  $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t
machine:WHAT_I_CARE_ABOUT tests/acceptance/

How does that sound?

Regards,
- Cleber.

>>
>> Because of that, let's present the default Avocado UI by default.
>> Users can always choose a different output by setting the AVOCADO_SHOW
>> variable.
>>
>> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  .travis.yml            | 2 +-
>>  tests/Makefile.include | 2 +-
>>  2 files changed, 2 insertions(+), 2 deletions(-)
Cornelia Huck Feb. 6, 2019, 5:20 p.m. UTC | #3
On Wed, 6 Feb 2019 12:02:55 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> On 2/6/19 9:36 AM, Cornelia Huck wrote:
> > On Fri,  1 Feb 2019 19:55:52 -0500
> > Cleber Rosa <crosa@redhat.com> wrote:
> >   
> >> The current version of the "check-acceptance" target will only show
> >> one line for execution of all tests.  That's probably OK if the tests
> >> to be run are quick enough and they're always the same.
> >>
> >> But, there's already one test alone that takes on average ~5 seconds
> >> to run, we intend to adapt the list of tests to match the user's build
> >> environment (among other choices).  
> > 
> > Btw: What are our expectations regarding execution time for tests?
> > Especially if we continue adding tests, and architecture-specific tests
> > are bound to be slower if run on a foreign architecture via tcg.
> > 
> > Would a make check-acceptance-quick command make sense? ("I only want
> > to verify quickly that I didn't break too much, so run the quicker
> > tests only, probably only for my host architecture")
> >   
> 
> Yes, it definitely makes sense.  Now, let me know if the following also
> makes sense to you:
> 
> 1) Because these tests focus on functional testing, the default
> target/shortcut ("make check-acceptance") should run the complete set of
> test cases (including the slow ones).

Nod.

> 
> 2) Requirements vary greatly from user to user, to while adding a
> "check-acceptance-quick" is fine, you just mentioned one extra test
> execution variation ("for my host architecture").  For those, the idea
> is that:
> 
>  a) "make check-acceptance[-quick]" will adapt to the build environment
> (if you only built s390x targets, that's all it's going to use)

Yes. However, I usually build some extra targets (just to verify that
files that e.g. include headers I modify don't break for other
environments), but don't necessarily want to run all acceptance tests
for them (as I don't expect functional changes for them). That's
probably something I only want to do when I run the full set anyway.

> 
>  b) "avocado" command line interface *should* be easy enough to fulfill
> other requirements, and not necessarily require a "make" target.  For
> instance, if you're only interested in your host arch and one specific
> machine type, a command line such as the following should do the trick:
> 
>   $ make check-venv
>   $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t
> machine:WHAT_I_CARE_ABOUT tests/acceptance/
> 
> How does that sound?

Yes, that looks quite usable (I can easily define aliases locally for
shortcuts) and solves the case I mentioned above.

So, to summarize:
- add 'make check-acceptance-quick' that excludes tests marked as 'slow'
- use the avocado command line interface to further narrow down
  architectures and machines, if wanted
?
Cleber Rosa Feb. 6, 2019, 5:36 p.m. UTC | #4
On 2/6/19 12:20 PM, Cornelia Huck wrote:
> On Wed, 6 Feb 2019 12:02:55 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
>> On 2/6/19 9:36 AM, Cornelia Huck wrote:
>>> On Fri,  1 Feb 2019 19:55:52 -0500
>>> Cleber Rosa <crosa@redhat.com> wrote:
>>>   
>>>> The current version of the "check-acceptance" target will only show
>>>> one line for execution of all tests.  That's probably OK if the tests
>>>> to be run are quick enough and they're always the same.
>>>>
>>>> But, there's already one test alone that takes on average ~5 seconds
>>>> to run, we intend to adapt the list of tests to match the user's build
>>>> environment (among other choices).  
>>>
>>> Btw: What are our expectations regarding execution time for tests?
>>> Especially if we continue adding tests, and architecture-specific tests
>>> are bound to be slower if run on a foreign architecture via tcg.
>>>
>>> Would a make check-acceptance-quick command make sense? ("I only want
>>> to verify quickly that I didn't break too much, so run the quicker
>>> tests only, probably only for my host architecture")
>>>   
>>
>> Yes, it definitely makes sense.  Now, let me know if the following also
>> makes sense to you:
>>
>> 1) Because these tests focus on functional testing, the default
>> target/shortcut ("make check-acceptance") should run the complete set of
>> test cases (including the slow ones).
> 
> Nod.
> 
>>
>> 2) Requirements vary greatly from user to user, to while adding a
>> "check-acceptance-quick" is fine, you just mentioned one extra test
>> execution variation ("for my host architecture").  For those, the idea
>> is that:
>>
>>  a) "make check-acceptance[-quick]" will adapt to the build environment
>> (if you only built s390x targets, that's all it's going to use)
> 
> Yes. However, I usually build some extra targets (just to verify that
> files that e.g. include headers I modify don't break for other
> environments), but don't necessarily want to run all acceptance tests
> for them (as I don't expect functional changes for them). That's
> probably something I only want to do when I run the full set anyway.
> 

Right, that's an interesting point I haven't thought of.

>>
>>  b) "avocado" command line interface *should* be easy enough to fulfill
>> other requirements, and not necessarily require a "make" target.  For
>> instance, if you're only interested in your host arch and one specific
>> machine type, a command line such as the following should do the trick:
>>
>>   $ make check-venv
>>   $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t
>> machine:WHAT_I_CARE_ABOUT tests/acceptance/
>>
>> How does that sound?
> 
> Yes, that looks quite usable (I can easily define aliases locally for
> shortcuts) and solves the case I mentioned above.
> 
> So, to summarize:
> - add 'make check-acceptance-quick' that excludes tests marked as 'slow'

Yes... with one caveat: since the default is to have functional tests
that are not necessarily quick, the other approach has been used.  Tests
that are known to be quick are currently marked as such.

Anyway, this task is being tracked here:

https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target

> - use the avocado command line interface to further narrow down
>   architectures and machines, if wanted
> ?
> 

Yes.  Do you think this deserves a section in the docs?

Thanks!
- Cleber.
Cornelia Huck Feb. 7, 2019, 10:25 a.m. UTC | #5
On Wed, 6 Feb 2019 12:36:23 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> On 2/6/19 12:20 PM, Cornelia Huck wrote:
> > So, to summarize:
> > - add 'make check-acceptance-quick' that excludes tests marked as 'slow'  
> 
> Yes... with one caveat: since the default is to have functional tests
> that are not necessarily quick, the other approach has been used.  Tests
> that are known to be quick are currently marked as such.
> 
> Anyway, this task is being tracked here:
> 
> https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target

Yes, that sounds good as well.

> 
> > - use the avocado command line interface to further narrow down
> >   architectures and machines, if wanted
> > ?
> >   
> 
> Yes.  Do you think this deserves a section in the docs?

I know that I'll find myself searching the docs for that info at some
point in the future, so yes :)
Cleber Rosa Feb. 7, 2019, 6:32 p.m. UTC | #6
On 2/7/19 5:25 AM, Cornelia Huck wrote:
> On Wed, 6 Feb 2019 12:36:23 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
>> On 2/6/19 12:20 PM, Cornelia Huck wrote:
>>> So, to summarize:
>>> - add 'make check-acceptance-quick' that excludes tests marked as 'slow'  
>>
>> Yes... with one caveat: since the default is to have functional tests
>> that are not necessarily quick, the other approach has been used.  Tests
>> that are known to be quick are currently marked as such.
>>
>> Anyway, this task is being tracked here:
>>
>> https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target
> 
> Yes, that sounds good as well.
> 
>>
>>> - use the avocado command line interface to further narrow down
>>>   architectures and machines, if wanted
>>> ?
>>>   
>>
>> Yes.  Do you think this deserves a section in the docs?
> 
> I know that I'll find myself searching the docs for that info at some
> point in the future, so yes :)
> 

Sure thing, tracking it here:

https://trello.com/c/7Fxbtx6F/80-document-how-to-use-the-avocado-command-line-and-tags

Regards,
- Cleber.
Cornelia Huck Feb. 8, 2019, 9:54 a.m. UTC | #7
On Thu, 7 Feb 2019 13:32:54 -0500
Cleber Rosa <crosa@redhat.com> wrote:

> On 2/7/19 5:25 AM, Cornelia Huck wrote:
> > On Wed, 6 Feb 2019 12:36:23 -0500
> > Cleber Rosa <crosa@redhat.com> wrote:
> >   
> >> On 2/6/19 12:20 PM, Cornelia Huck wrote:  
> >>> So, to summarize:
> >>> - add 'make check-acceptance-quick' that excludes tests marked as 'slow'    
> >>
> >> Yes... with one caveat: since the default is to have functional tests
> >> that are not necessarily quick, the other approach has been used.  Tests
> >> that are known to be quick are currently marked as such.
> >>
> >> Anyway, this task is being tracked here:
> >>
> >> https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target  
> > 
> > Yes, that sounds good as well.
> >   
> >>  
> >>> - use the avocado command line interface to further narrow down
> >>>   architectures and machines, if wanted
> >>> ?
> >>>     
> >>
> >> Yes.  Do you think this deserves a section in the docs?  
> > 
> > I know that I'll find myself searching the docs for that info at some
> > point in the future, so yes :)
> >   
> 
> Sure thing, tracking it here:
> 
> https://trello.com/c/7Fxbtx6F/80-document-how-to-use-the-avocado-command-line-and-tags

Great, thanks!
diff mbox series

Patch

diff --git a/.travis.yml b/.travis.yml
index 87d9fa971c..1b43ed4df6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -190,7 +190,7 @@  matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
-        - TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
+        - TEST_CMD="make check-acceptance"
       addons:
         apt:
           packages:
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 75ad9c0dd3..67e222eeff 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1093,7 +1093,7 @@  TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
-AVOCADO_SHOW=none
+AVOCADO_SHOW=app
 
 ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)