diff mbox

[v3,2/3] .travis.yml: run make check for all matrix targets

Message ID 1453991009-32736-3-git-send-email-alex.bennee@linaro.org (mailing list archive)
State New, archived
Headers show

Commit Message

Alex Bennée Jan. 28, 2016, 2:23 p.m. UTC
We only ran make check once before it used to be an unreliable target.
It was only a stop gap measure and we should be able to revert it now.
This also stops us needing a large all-MMU build.

We disable "make check" for a couple of the extra config targets which
are currently broken.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .travis.yml | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

Comments

David Gibson Jan. 30, 2016, 12:35 p.m. UTC | #1
On Thu, Jan 28, 2016 at 02:23:28PM +0000, Alex Bennée wrote:
> We only ran make check once before it used to be an unreliable target.
> It was only a stop gap measure and we should be able to revert it now.
> This also stops us needing a large all-MMU build.
> 
> We disable "make check" for a couple of the extra config targets which
> are currently broken.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

So, in general I like the idea of running make check more widely.

However.. I was wondering - what's the rationale for having separate
matrix builds for each target (or small group) rather than just doing
one build with all the targets?

I can't see any obvious benefit to splitting the build that way, but
it does increase the total build time significantly - and will do so
rather more so with make check added.

> ---
>  .travis.yml | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 4a0c23a..16be23f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -40,7 +40,7 @@ notifications:
>      on_failure: always
>  env:
>    global:
> -    - TEST_CMD=""
> +    - TEST_CMD="make check"
>      - EXTRA_CONFIG=""
>    matrix:
>      # Group major targets together with their linux-user counterparts
> @@ -73,17 +73,14 @@ script:
>  matrix:
>    # We manually include a number of additional build for non-standard bits
>    include:
> -    # Make check target (we only do this once)
> -    - env:
> -        - TARGETS=alpha-softmmu,arm-softmmu,aarch64-softmmu,cris-softmmu,i386-softmmu,x86_64-softmmu,m68k-softmmu,microblaze-softmmu,microblazeel-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,or32-softmmu,ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,s390x-softmmu,sh4-softmmu,sh4eb-softmmu,sparc-softmmu,sparc64-softmmu,unicore32-softmmu,lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
> -          TEST_CMD="make check"
> -      compiler: gcc
>      # Debug related options
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>             EXTRA_CONFIG="--enable-debug"
>        compiler: gcc
> +    # We currently disable "make check"
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>             EXTRA_CONFIG="--enable-debug --enable-tcg-interpreter"
> +           TEST_CMD=""
>        compiler: gcc
>      # Disable a few of the optional features
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
> @@ -104,11 +101,15 @@ matrix:
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>             EXTRA_CONFIG="--enable-trace-backends=simple"
>        compiler: gcc
> +    # We currently disable "make check"
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>             EXTRA_CONFIG="--enable-trace-backends=ftrace"
> +           TEST_CMD=""
>        compiler: gcc
> +    # We currently disable "make check"
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
> -          EXTRA_CONFIG="--enable-trace-backends=ust"
> +           EXTRA_CONFIG="--enable-trace-backends=ust"
> +           TEST_CMD=""
>        compiler: gcc
>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>             EXTRA_CONFIG="--enable-modules"
Alex Bennée Jan. 31, 2016, 8:37 a.m. UTC | #2
David Gibson <david@gibson.dropbear.id.au> writes:

> On Thu, Jan 28, 2016 at 02:23:28PM +0000, Alex Bennée wrote:
>> We only ran make check once before it used to be an unreliable target.
>> It was only a stop gap measure and we should be able to revert it now.
>> This also stops us needing a large all-MMU build.
>>
>> We disable "make check" for a couple of the extra config targets which
>> are currently broken.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> So, in general I like the idea of running make check more widely.
>
> However.. I was wondering - what's the rationale for having separate
> matrix builds for each target (or small group) rather than just doing
> one build with all the targets?

Each individual part of the matrix can be run in parallel with the
others so it makes sense to keep the build component small (as each
softmmu target rebuilds a significant chunk of the build).

Having said that there is a fair amount of repetition as we are
repeating all the generic qtests each time just so we can run the extra
${TARGET}-qtest binaries.

Travis does has an option for using ccache so it might be worth
experimenting with that to see if things are improved.

> I can't see any obvious benefit to splitting the build that way, but
> it does increase the total build time significantly - and will do so
> rather more so with make check added.

Elapsed and total are the ones to look at:

https://travis-ci.org/stsquad/qemu/builds/105401126

vs

https://travis-ci.org/qemu/qemu/builds/105711606

However it looks like Travis are having scaling growing pains because
there "old style" VM approach is running a lot faster than it used to.

>
>> ---
>>  .travis.yml | 15 ++++++++-------
>>  1 file changed, 8 insertions(+), 7 deletions(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 4a0c23a..16be23f 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -40,7 +40,7 @@ notifications:
>>      on_failure: always
>>  env:
>>    global:
>> -    - TEST_CMD=""
>> +    - TEST_CMD="make check"
>>      - EXTRA_CONFIG=""
>>    matrix:
>>      # Group major targets together with their linux-user counterparts
>> @@ -73,17 +73,14 @@ script:
>>  matrix:
>>    # We manually include a number of additional build for non-standard bits
>>    include:
>> -    # Make check target (we only do this once)
>> -    - env:
>> -        - TARGETS=alpha-softmmu,arm-softmmu,aarch64-softmmu,cris-softmmu,i386-softmmu,x86_64-softmmu,m68k-softmmu,microblaze-softmmu,microblazeel-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,or32-softmmu,ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,s390x-softmmu,sh4-softmmu,sh4eb-softmmu,sparc-softmmu,sparc64-softmmu,unicore32-softmmu,lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
>> -          TEST_CMD="make check"
>> -      compiler: gcc
>>      # Debug related options
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>>             EXTRA_CONFIG="--enable-debug"
>>        compiler: gcc
>> +    # We currently disable "make check"
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>>             EXTRA_CONFIG="--enable-debug --enable-tcg-interpreter"
>> +           TEST_CMD=""
>>        compiler: gcc
>>      # Disable a few of the optional features
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>> @@ -104,11 +101,15 @@ matrix:
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>>             EXTRA_CONFIG="--enable-trace-backends=simple"
>>        compiler: gcc
>> +    # We currently disable "make check"
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>>             EXTRA_CONFIG="--enable-trace-backends=ftrace"
>> +           TEST_CMD=""
>>        compiler: gcc
>> +    # We currently disable "make check"
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>> -          EXTRA_CONFIG="--enable-trace-backends=ust"
>> +           EXTRA_CONFIG="--enable-trace-backends=ust"
>> +           TEST_CMD=""
>>        compiler: gcc
>>      - env: TARGETS=i386-softmmu,x86_64-softmmu
>>             EXTRA_CONFIG="--enable-modules"


--
Alex Bennée
David Gibson Jan. 31, 2016, 10:27 a.m. UTC | #3
On Sun, Jan 31, 2016 at 08:37:49AM +0000, Alex Bennée wrote:
> 
> David Gibson <david@gibson.dropbear.id.au> writes:
> 
> > On Thu, Jan 28, 2016 at 02:23:28PM +0000, Alex Bennée wrote:
> >> We only ran make check once before it used to be an unreliable target.
> >> It was only a stop gap measure and we should be able to revert it now.
> >> This also stops us needing a large all-MMU build.
> >>
> >> We disable "make check" for a couple of the extra config targets which
> >> are currently broken.
> >>
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> > So, in general I like the idea of running make check more widely.
> >
> > However.. I was wondering - what's the rationale for having separate
> > matrix builds for each target (or small group) rather than just doing
> > one build with all the targets?
> 
> Each individual part of the matrix can be run in parallel with the
> others so it makes sense to keep the build component small (as each
> softmmu target rebuilds a significant chunk of the build).

It does rebuild a significant chunk, but there's a significant chunk
that isn't rebuilt as well.  When I tried this I found a recombined
build marginally decreases the elapsed time and significantly (maybe
30-40%) reduced the total time.  Given the load the travis system is
under, it seems to me that we should try to keep our total demand on
its resources down when it doens't significantly lower our coverage.

> Having said that there is a fair amount of repetition as we are
> repeating all the generic qtests each time just so we can run the extra
> ${TARGET}-qtest binaries.

That too.

> Travis does has an option for using ccache so it might be worth
> experimenting with that to see if things are improved.

That does sound like something worth looking at.

One thing that does annoy me about travis is that it will do a full
rebuild if you have two branches on exactly the same commit, or if you
revert a branch to an earlier commit which was built previously.

> > I can't see any obvious benefit to splitting the build that way, but
> > it does increase the total build time significantly - and will do so
> > rather more so with make check added.
> 
> Elapsed and total are the ones to look at:
> 
> https://travis-ci.org/stsquad/qemu/builds/105401126
> 
> vs
> 
> https://travis-ci.org/qemu/qemu/builds/105711606
> 
> However it looks like Travis are having scaling growing pains because
> there "old style" VM approach is running a lot faster than it used
> to.

Not terribly surprising TBH.
diff mbox

Patch

diff --git a/.travis.yml b/.travis.yml
index 4a0c23a..16be23f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -40,7 +40,7 @@  notifications:
     on_failure: always
 env:
   global:
-    - TEST_CMD=""
+    - TEST_CMD="make check"
     - EXTRA_CONFIG=""
   matrix:
     # Group major targets together with their linux-user counterparts
@@ -73,17 +73,14 @@  script:
 matrix:
   # We manually include a number of additional build for non-standard bits
   include:
-    # Make check target (we only do this once)
-    - env:
-        - TARGETS=alpha-softmmu,arm-softmmu,aarch64-softmmu,cris-softmmu,i386-softmmu,x86_64-softmmu,m68k-softmmu,microblaze-softmmu,microblazeel-softmmu,mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu,or32-softmmu,ppc-softmmu,ppc64-softmmu,ppcemb-softmmu,s390x-softmmu,sh4-softmmu,sh4eb-softmmu,sparc-softmmu,sparc64-softmmu,unicore32-softmmu,lm32-softmmu,moxie-softmmu,tricore-softmmu,xtensa-softmmu,xtensaeb-softmmu
-          TEST_CMD="make check"
-      compiler: gcc
     # Debug related options
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-debug"
       compiler: gcc
+    # We currently disable "make check"
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-debug --enable-tcg-interpreter"
+           TEST_CMD=""
       compiler: gcc
     # Disable a few of the optional features
     - env: TARGETS=i386-softmmu,x86_64-softmmu
@@ -104,11 +101,15 @@  matrix:
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-trace-backends=simple"
       compiler: gcc
+    # We currently disable "make check"
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-trace-backends=ftrace"
+           TEST_CMD=""
       compiler: gcc
+    # We currently disable "make check"
     - env: TARGETS=i386-softmmu,x86_64-softmmu
-          EXTRA_CONFIG="--enable-trace-backends=ust"
+           EXTRA_CONFIG="--enable-trace-backends=ust"
+           TEST_CMD=""
       compiler: gcc
     - env: TARGETS=i386-softmmu,x86_64-softmmu
            EXTRA_CONFIG="--enable-modules"