tests/acceptance: Add boot tests for some of the QEMU advent calendar images
diff mbox series

Message ID 20200124170325.30072-1-thuth@redhat.com
State New
Headers show
Series
  • tests/acceptance: Add boot tests for some of the QEMU advent calendar images
Related show

Commit Message

Thomas Huth Jan. 24, 2020, 5:03 p.m. UTC
The 2018 edition of the QEMU advent calendar 2018 featured Linux images
for various non-x86 machines. We can use them for a boot tests in our
acceptance test suite.

Let's also make sure that we build the corresponding machines in Travis,
and while we're there, drop the superfluous --python parameter (python3
is now the only supported version anyway).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml                            |  2 +-
 tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 1 deletion(-)

Comments

Liam Merwick Jan. 24, 2020, 7:14 p.m. UTC | #1
On 24/01/2020 17:03, Thomas Huth wrote:
> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
> for various non-x86 machines. We can use them for a boot tests in our
> acceptance test suite.
> 
> Let's also make sure that we build the corresponding machines in Travis,
> and while we're there, drop the superfluous --python parameter (python3
> is now the only supported version anyway).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>


Reviewed-by: Liam Merwick <liam.merwick@oracle.com>


> ---
>   .travis.yml                            |  2 +-
>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>   2 files changed, 97 insertions(+), 1 deletion(-)
>
Wainer dos Santos Moschetta Jan. 24, 2020, 9:28 p.m. UTC | #2
On 1/24/20 3:03 PM, Thomas Huth wrote:
> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
> for various non-x86 machines. We can use them for a boot tests in our
> acceptance test suite.
>
> Let's also make sure that we build the corresponding machines in Travis,
> and while we're there, drop the superfluous --python parameter (python3
> is now the only supported version anyway).

Yes, please, removal of --python was in my wish list.

>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .travis.yml                            |  2 +-
>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>   2 files changed, 97 insertions(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 6c1038a0f1..73ca12c921 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -268,7 +268,7 @@ matrix:
>   
>       # Acceptance (Functional) tests
>       - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
> +        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"


Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:

--target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....


>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e03add2989..f7ac2a3a59 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -584,3 +584,99 @@ class BootLinuxConsole(Test):
>           self.wait_for_console_pattern(console_pattern)
>           console_pattern = 'No filesystem could mount root'
>           self.wait_for_console_pattern(console_pattern)
> +
> +    def do_test_advcal_2018(self, day, tar_hash, kernel_name):
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day' + day + '.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        archive.extract(file_path, self.workdir)
> +        self.vm.set_console()
> +        self.vm.add_args('-kernel',
> +                         self.workdir + '/day' + day + '/' + kernel_name)
> +        self.vm.launch()
> +        self.wait_for_console_pattern('QEMU advent calendar')
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        self.vm.add_args('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb')
> +        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage')
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
> +
> +    def test_or1k_sim(self):
> +        """
> +        :avocado: tags=arch:or1k
> +        :avocado: tags=machine:or1k-sim
> +        """
> +        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
> +        self.do_test_advcal_2018('20', tar_hash, 'vmlinux')
> +
> +    def test_nios2_10m50(self):
> +        """
> +        :avocado: tags=arch:nios2
> +        :avocado: tags=machine:10m50-ghrd
> +        """
> +        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
> +        self.do_test_advcal_2018('14', tar_hash, 'vmlinux.elf')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        self.vm.add_args('-cpu', 'e5500')
> +        self.do_test_advcal_2018('19', tar_hash, 'uImage')
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        self.vm.add_args('-cpu', 'dc233c')
> +        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf')
Thomas Huth Jan. 25, 2020, 4:43 p.m. UTC | #3
On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
> 
> On 1/24/20 3:03 PM, Thomas Huth wrote:
>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>> for various non-x86 machines. We can use them for a boot tests in our
>> acceptance test suite.
>>
>> Let's also make sure that we build the corresponding machines in Travis,
>> and while we're there, drop the superfluous --python parameter (python3
>> is now the only supported version anyway).
> 
> Yes, please, removal of --python was in my wish list.
> 
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .travis.yml                            |  2 +-
>>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>>   2 files changed, 97 insertions(+), 1 deletion(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 6c1038a0f1..73ca12c921 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -268,7 +268,7 @@ matrix:
>>         # Acceptance (Functional) tests
>>       - env:
>> -        - CONFIG="--python=/usr/bin/python3
>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>
>> +        -
>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>
> 
> 
> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:
> 
> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....

Not sure ... while it is a nice way to shorten the line here, it adds a
dependecy to that variable ... and MAIN_SOFTMMU_TARGETS has been changed
a couple of times during the course of time, so we might risk to lose
some testing coverage here in case someone removes a target from
MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
should better use the explicit list here instead.

 Thomas
Philippe Mathieu-Daudé Jan. 25, 2020, 6:38 p.m. UTC | #4
On 1/25/20 5:43 PM, Thomas Huth wrote:
> On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
>>
>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>> for various non-x86 machines. We can use them for a boot tests in our
>>> acceptance test suite.

Excellent idea!

>>>
>>> Let's also make sure that we build the corresponding machines in Travis,
>>> and while we're there, drop the superfluous --python parameter (python3
>>> is now the only supported version anyway).

I'd rather see this change as another commit.

>>
>> Yes, please, removal of --python was in my wish list.
>>
>>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   .travis.yml                            |  2 +-
>>>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>>>   2 files changed, 97 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 6c1038a0f1..73ca12c921 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -268,7 +268,7 @@ matrix:
>>>         # Acceptance (Functional) tests
>>>       - env:
>>> -        - CONFIG="--python=/usr/bin/python3
>>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>>
>>> +        -
>>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>>
>>
>>
>> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:
>>
>> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....
> 
> Not sure ... while it is a nice way to shorten the line here, it adds a
> dependecy to that variable ... and MAIN_SOFTMMU_TARGETS has been changed
> a couple of times during the course of time, so we might risk to lose
> some testing coverage here in case someone removes a target from
> MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
> should better use the explicit list here instead.

Maybe related: "Split enterprise vs. hobbyist acceptance test job"
https://www.mail-archive.com/qemu-devel@nongnu.org/msg644683.html

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Thomas Huth Jan. 27, 2020, 10:38 a.m. UTC | #5
On 25/01/2020 19.38, Philippe Mathieu-Daudé wrote:
> On 1/25/20 5:43 PM, Thomas Huth wrote:
>> On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
>>>
>>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>>> for various non-x86 machines. We can use them for a boot tests in our
>>>> acceptance test suite.
> 
> Excellent idea!
> 
>>>>
>>>> Let's also make sure that we build the corresponding machines in Travis,
>>>> and while we're there, drop the superfluous --python parameter (python3
>>>> is now the only supported version anyway).
> 
> I'd rather see this change as another commit.

I thought so, too, often it is good to separate stuff into separate
commits, but in this case, it's not source code but just the CI yml
file, and the change is really trivial, so it rather sounds like
superfluous churn to me in this case to do it in a separate patch...

>>> Yes, please, removal of --python was in my wish list.
>>>
>>>>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>>   .travis.yml                            |  2 +-
>>>>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>>>>   2 files changed, 97 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/.travis.yml b/.travis.yml
>>>> index 6c1038a0f1..73ca12c921 100644
>>>> --- a/.travis.yml
>>>> +++ b/.travis.yml
>>>> @@ -268,7 +268,7 @@ matrix:
>>>>         # Acceptance (Functional) tests
>>>>       - env:
>>>> -        - CONFIG="--python=/usr/bin/python3
>>>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>>>
>>>> +        -
>>>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>>>
>>>
>>>
>>> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:
>>>
>>> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....
>>
>> Not sure ... while it is a nice way to shorten the line here, it adds a
>> dependency to that variable ... and MAIN_SOFTMMU_TARGETS has been changed
>> a couple of times during the course of time, so we might risk to lose
>> some testing coverage here in case someone removes a target from
>> MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
>> should better use the explicit list here instead.
> 
> Maybe related: "Split enterprise vs. hobbyist acceptance test job"
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg644683.html

I guess naming/splitting is hard in this case ... many targets contain
both, "enterprise" and "hobbyist" machines ... and as long as the Travis
job runs reasonably fast, it's also not really necessary to split it.
The only real ugliness is the very long line - I haven't found a way yet
to correctly wrap this in the YML file to multiple lines .... does
anybody know how to do this in YML?

> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Thanks!

 Thomas
Wainer dos Santos Moschetta Jan. 27, 2020, 1:22 p.m. UTC | #6
On 1/25/20 2:43 PM, Thomas Huth wrote:
> On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>> for various non-x86 machines. We can use them for a boot tests in our
>>> acceptance test suite.
>>>
>>> Let's also make sure that we build the corresponding machines in Travis,
>>> and while we're there, drop the superfluous --python parameter (python3
>>> is now the only supported version anyway).
>> Yes, please, removal of --python was in my wish list.
>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>    .travis.yml                            |  2 +-
>>>    tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>>>    2 files changed, 97 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 6c1038a0f1..73ca12c921 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -268,7 +268,7 @@ matrix:
>>>          # Acceptance (Functional) tests
>>>        - env:
>>> -        - CONFIG="--python=/usr/bin/python3
>>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>>
>>> +        -
>>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>>
>>
>> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:
>>
>> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....
> Not sure ... while it is a nice way to shorten the line here, it adds a
> dependecy to that variable ... and MAIN_SOFTMMU_TARGETS has been changed
> a couple of times during the course of time, so we might risk to lose
> some testing coverage here in case someone removes a target from
> MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
> should better use the explicit list here instead.

Your reasoning makes sense to me. Feel free to ignore my suggestion.

Thanks,

Wainer

>
>   Thomas
>
>
Wainer dos Santos Moschetta Jan. 27, 2020, 2:05 p.m. UTC | #7
On 1/27/20 8:38 AM, Thomas Huth wrote:
> On 25/01/2020 19.38, Philippe Mathieu-Daudé wrote:
>> On 1/25/20 5:43 PM, Thomas Huth wrote:
>>> On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
>>>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>>>> for various non-x86 machines. We can use them for a boot tests in our
>>>>> acceptance test suite.
>> Excellent idea!
>>
>>>>> Let's also make sure that we build the corresponding machines in Travis,
>>>>> and while we're there, drop the superfluous --python parameter (python3
>>>>> is now the only supported version anyway).
>> I'd rather see this change as another commit.
> I thought so, too, often it is good to separate stuff into separate
> commits, but in this case, it's not source code but just the CI yml
> file, and the change is really trivial, so it rather sounds like
> superfluous churn to me in this case to do it in a separate patch...
>
>>>> Yes, please, removal of --python was in my wish list.
>>>>
>>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>>> ---
>>>>>    .travis.yml                            |  2 +-
>>>>>    tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>>>>>    2 files changed, 97 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/.travis.yml b/.travis.yml
>>>>> index 6c1038a0f1..73ca12c921 100644
>>>>> --- a/.travis.yml
>>>>> +++ b/.travis.yml
>>>>> @@ -268,7 +268,7 @@ matrix:
>>>>>          # Acceptance (Functional) tests
>>>>>        - env:
>>>>> -        - CONFIG="--python=/usr/bin/python3
>>>>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>>>>
>>>>> +        -
>>>>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>>>>
>>>>
>>>> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets, like:
>>>>
>>>> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....
>>> Not sure ... while it is a nice way to shorten the line here, it adds a
>>> dependency to that variable ... and MAIN_SOFTMMU_TARGETS has been changed
>>> a couple of times during the course of time, so we might risk to lose
>>> some testing coverage here in case someone removes a target from
>>> MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
>>> should better use the explicit list here instead.
>> Maybe related: "Split enterprise vs. hobbyist acceptance test job"
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg644683.html
> I guess naming/splitting is hard in this case ... many targets contain
> both, "enterprise" and "hobbyist" machines ... and as long as the Travis
> job runs reasonably fast, it's also not really necessary to split it.
> The only real ugliness is the very long line - I haven't found a way yet
> to correctly wrap this in the YML file to multiple lines .... does
> anybody know how to do this in YML?


Try to break the lines with '\' (note: next line should not be 
indented). It worked out on following code:

[wainersm@localhost ~]$ cat /tmp/foo.py
import yaml

yml="""
matrix:
   include:
     - env:
       - CONFIG="-target-list=x86_64-softmmu,\
mips-softmmu,mips64el-softmmu"
"""
print(yaml.safe_load(yml))
[wainersm@localhost ~]$ python3 /tmp/foo.py
{'matrix': {'include': [{'env': 
['CONFIG="-target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu"']}]}}

>
>> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Thanks!
>
>   Thomas
Thomas Huth Jan. 27, 2020, 2:10 p.m. UTC | #8
On 27/01/2020 15.05, Wainer dos Santos Moschetta wrote:
> 
> On 1/27/20 8:38 AM, Thomas Huth wrote:
>> On 25/01/2020 19.38, Philippe Mathieu-Daudé wrote:
>>> On 1/25/20 5:43 PM, Thomas Huth wrote:
>>>> On 24/01/2020 22.28, Wainer dos Santos Moschetta wrote:
>>>>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>>>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux
>>>>>> images
>>>>>> for various non-x86 machines. We can use them for a boot tests in our
>>>>>> acceptance test suite.
>>> Excellent idea!
>>>
>>>>>> Let's also make sure that we build the corresponding machines in
>>>>>> Travis,
>>>>>> and while we're there, drop the superfluous --python parameter
>>>>>> (python3
>>>>>> is now the only supported version anyway).
>>> I'd rather see this change as another commit.
>> I thought so, too, often it is good to separate stuff into separate
>> commits, but in this case, it's not source code but just the CI yml
>> file, and the change is really trivial, so it rather sounds like
>> superfluous churn to me in this case to do it in a separate patch...
>>
>>>>> Yes, please, removal of --python was in my wish list.
>>>>>
>>>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>>>> ---
>>>>>>    .travis.yml                            |  2 +-
>>>>>>    tests/acceptance/boot_linux_console.py | 96
>>>>>> ++++++++++++++++++++++++++
>>>>>>    2 files changed, 97 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/.travis.yml b/.travis.yml
>>>>>> index 6c1038a0f1..73ca12c921 100644
>>>>>> --- a/.travis.yml
>>>>>> +++ b/.travis.yml
>>>>>> @@ -268,7 +268,7 @@ matrix:
>>>>>>          # Acceptance (Functional) tests
>>>>>>        - env:
>>>>>> -        - CONFIG="--python=/usr/bin/python3
>>>>>> --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>>>>>
>>>>>>
>>>>>> +        -
>>>>>> CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>>>>>>
>>>>>>
>>>>>
>>>>> Perhaps use MAIN_SOFTMMU_TARGETS in only append the other targets,
>>>>> like:
>>>>>
>>>>> --target-list=${MAIN_SOFTMMU_TARGETS},alpha-softmmu,sparc-softmmu,....
>>>> Not sure ... while it is a nice way to shorten the line here, it adds a
>>>> dependency to that variable ... and MAIN_SOFTMMU_TARGETS has been
>>>> changed
>>>> a couple of times during the course of time, so we might risk to lose
>>>> some testing coverage here in case someone removes a target from
>>>> MAIN_SOFTMMU_TARGETS but forgets to add it here again...? I think we
>>>> should better use the explicit list here instead.
>>> Maybe related: "Split enterprise vs. hobbyist acceptance test job"
>>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg644683.html
>> I guess naming/splitting is hard in this case ... many targets contain
>> both, "enterprise" and "hobbyist" machines ... and as long as the Travis
>> job runs reasonably fast, it's also not really necessary to split it.
>> The only real ugliness is the very long line - I haven't found a way yet
>> to correctly wrap this in the YML file to multiple lines .... does
>> anybody know how to do this in YML?
> 
> 
> Try to break the lines with '\' (note: next line should not be
> indented). It worked out on following code:
> 
> [wainersm@localhost ~]$ cat /tmp/foo.py
> import yaml
> 
> yml="""
> matrix:
>   include:
>     - env:
>       - CONFIG="-target-list=x86_64-softmmu,\
> mips-softmmu,mips64el-softmmu"
> """

Ok, thanks ... but without indentation, it looks somewhat ugly, too.
... hmm, maybe we should just keep adding acceptance tests until we've
got all targets covered, then we can simply replace --target-list with
--disable-user instead :-)

 Thomas
Wainer dos Santos Moschetta Jan. 27, 2020, 5:31 p.m. UTC | #9
On 1/24/20 3:03 PM, Thomas Huth wrote:
> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
> for various non-x86 machines. We can use them for a boot tests in our
> acceptance test suite.
>
> Let's also make sure that we build the corresponding machines in Travis,
> and while we're there, drop the superfluous --python parameter (python3
> is now the only supported version anyway).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .travis.yml                            |  2 +-
>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>   2 files changed, 97 insertions(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 6c1038a0f1..73ca12c921 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -268,7 +268,7 @@ matrix:
>   
>       # Acceptance (Functional) tests
>       - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
> +        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e03add2989..f7ac2a3a59 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -584,3 +584,99 @@ class BootLinuxConsole(Test):
>           self.wait_for_console_pattern(console_pattern)
>           console_pattern = 'No filesystem could mount root'
>           self.wait_for_console_pattern(console_pattern)
> +
> +    def do_test_advcal_2018(self, day, tar_hash, kernel_name):
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day' + day + '.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        archive.extract(file_path, self.workdir)
> +        self.vm.set_console()
> +        self.vm.add_args('-kernel',
> +                         self.workdir + '/day' + day + '/' + kernel_name)
> +        self.vm.launch()
> +        self.wait_for_console_pattern('QEMU advent calendar')
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        self.vm.add_args('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb')
> +        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage')
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
> +
> +    def test_or1k_sim(self):
> +        """
> +        :avocado: tags=arch:or1k
> +        :avocado: tags=machine:or1k-sim
> +        """
> +        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
> +        self.do_test_advcal_2018('20', tar_hash, 'vmlinux')
> +
> +    def test_nios2_10m50(self):
> +        """
> +        :avocado: tags=arch:nios2
> +        :avocado: tags=machine:10m50-ghrd
> +        """
> +        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
> +        self.do_test_advcal_2018('14', tar_hash, 'vmlinux.elf')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        self.vm.add_args('-cpu', 'e5500')
> +        self.do_test_advcal_2018('19', tar_hash, 'uImage')
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')

Hi Thomas, let me check one thing...

The VM will be launched as:

----

ppc-softmmu/qemu-system-ppc -display none -vga none -chardev 
socket,id=mon,path=/tmp/tmpvdokyvs3/qemu-41146-monitor.sock -mon 
chardev=mon,mode=control -machine g3beige -chardev 
socket,id=console,path=/tmp/tmpvdokyvs3/qemu-41146-console.sock,server,nowait 
-serial chardev:console -M graphics=off -kernel 
/tmp/avocado_g3uccfo5/avocado_job_61gglyz3/02-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_ppc_g3beige/day15/invaders.elf

----

Note that it passes '[..] -machine g3beige [...] -M graphics=off [...]'. 
I suspect you wanted '-machine g3beige,graphics=off'. am I right? or 
QEMU will interpret the -M option as a parameter of the already set 
machine type? or the -M overwrites -machine, and because  g3beige is the 
default type it just works?

Anyway, It happens that avocado_qemu will automatically convert the 
':avocado: tags=machine:g3beige' into QEMU's '-machine g3beige', but  
`self.vm.add_args('-M', 'graphics=off')` doesn't actually append 
'graphics=off` to '-machine' (i.e. `add_args` appends to QEMU's 
parameters list instead). So if you want '-machine g3beige,graphics=off' 
you should overwrite its value with:

self.vm.set_machine('-machine g3beige,graphics=off')

Thanks,

Wainer

> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        self.vm.add_args('-cpu', 'dc233c')
> +        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf')
Thomas Huth Jan. 29, 2020, 2:49 p.m. UTC | #10
On 27/01/2020 18.31, Wainer dos Santos Moschetta wrote:
> 
> On 1/24/20 3:03 PM, Thomas Huth wrote:
>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>> for various non-x86 machines. We can use them for a boot tests in our
>> acceptance test suite.
>>
>> Let's also make sure that we build the corresponding machines in Travis,
>> and while we're there, drop the superfluous --python parameter (python3
>> is now the only supported version anyway).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
[...]
>> +    def test_ppc_g3beige(self):
>> +        """
>> +        :avocado: tags=arch:ppc
>> +        :avocado: tags=machine:g3beige
>> +        """
>> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
>> +        self.vm.add_args('-M', 'graphics=off')
>> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> 
> Hi Thomas, let me check one thing...
> 
> The VM will be launched as:
> 
> ----
> 
> ppc-softmmu/qemu-system-ppc -display none -vga none -chardev
> socket,id=mon,path=/tmp/tmpvdokyvs3/qemu-41146-monitor.sock -mon
> chardev=mon,mode=control -machine g3beige -chardev
> socket,id=console,path=/tmp/tmpvdokyvs3/qemu-41146-console.sock,server,nowait
> -serial chardev:console -M graphics=off -kernel
> /tmp/avocado_g3uccfo5/avocado_job_61gglyz3/02-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_ppc_g3beige/day15/invaders.elf
> 
> 
> ----
> 
> Note that it passes '[..] -machine g3beige [...] -M graphics=off [...]'.
> I suspect you wanted '-machine g3beige,graphics=off'. am I right?

No, the -M graphics=off was intended.

> or
> QEMU will interpret the -M option as a parameter of the already set
> machine type?

That's exactly the case.

> or the -M overwrites -machine, and because  g3beige is the
> default type it just works?

No, the default machine of qemu-system-ppc is mac99.

And just to make sure that the g3beige test is really running with a G3
and not a G4 (i.e. mac99):

$ avocado --show=console run
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc_g3beige
| grep motherboard
console: PowerMac motherboard: PowerMac G3 (Silk)

 Thomas
BALATON Zoltan Jan. 29, 2020, 5:48 p.m. UTC | #11
On Wed, 29 Jan 2020, Thomas Huth wrote:
> On 27/01/2020 18.31, Wainer dos Santos Moschetta wrote:
>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>> for various non-x86 machines. We can use them for a boot tests in our
>>> acceptance test suite.
>>>
>>> Let's also make sure that we build the corresponding machines in Travis,
>>> and while we're there, drop the superfluous --python parameter (python3
>>> is now the only supported version anyway).
>>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
> [...]
>>> +    def test_ppc_g3beige(self):
>>> +        """
>>> +        :avocado: tags=arch:ppc
>>> +        :avocado: tags=machine:g3beige
>>> +        """
>>> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
>>> +        self.vm.add_args('-M', 'graphics=off')
>>> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
>>
>> Hi Thomas, let me check one thing...
>>
>> The VM will be launched as:
>>
>> ----
>>
>> ppc-softmmu/qemu-system-ppc -display none -vga none -chardev
>> socket,id=mon,path=/tmp/tmpvdokyvs3/qemu-41146-monitor.sock -mon
>> chardev=mon,mode=control -machine g3beige -chardev
>> socket,id=console,path=/tmp/tmpvdokyvs3/qemu-41146-console.sock,server,nowait
>> -serial chardev:console -M graphics=off -kernel
>> /tmp/avocado_g3uccfo5/avocado_job_61gglyz3/02-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_ppc_g3beige/day15/invaders.elf
>>
>>
>> ----
>>
>> Note that it passes '[..] -machine g3beige [...] -M graphics=off [...]'.
>> I suspect you wanted '-machine g3beige,graphics=off'. am I right?
>
> No, the -M graphics=off was intended.
>
>> or
>> QEMU will interpret the -M option as a parameter of the already set
>> machine type?
>
> That's exactly the case.
>
>> or the -M overwrites -machine, and because  g3beige is the
>> default type it just works?
>
> No, the default machine of qemu-system-ppc is mac99.

I think you're wrong about the default machine type:

$ qemu-system-ppc -M help
Supported machines are:
40p                  IBM RS/6000 7020 (40p)
bamboo               bamboo
g3beige              Heathrow based PowerMAC (default)

g3beige is marked as default and seems to be created without any -M option 
but having two -M options e.g. -M mac99 -M graphics=off creates mac99 
machine so multiple options seem to work as you described.

Regards,
BALATON Zoltan
Thomas Huth Jan. 30, 2020, 8:16 a.m. UTC | #12
On 29/01/2020 18.48, BALATON Zoltan wrote:
> On Wed, 29 Jan 2020, Thomas Huth wrote:
>> On 27/01/2020 18.31, Wainer dos Santos Moschetta wrote:
>>> On 1/24/20 3:03 PM, Thomas Huth wrote:
>>>> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
>>>> for various non-x86 machines. We can use them for a boot tests in our
>>>> acceptance test suite.
>>>>
>>>> Let's also make sure that we build the corresponding machines in
>>>> Travis,
>>>> and while we're there, drop the superfluous --python parameter (python3
>>>> is now the only supported version anyway).
>>>>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>> [...]
>>>> +    def test_ppc_g3beige(self):
>>>> +        """
>>>> +        :avocado: tags=arch:ppc
>>>> +        :avocado: tags=machine:g3beige
>>>> +        """
>>>> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
>>>> +        self.vm.add_args('-M', 'graphics=off')
>>>> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
>>>
>>> Hi Thomas, let me check one thing...
>>>
>>> The VM will be launched as:
>>>
>>> ----
>>>
>>> ppc-softmmu/qemu-system-ppc -display none -vga none -chardev
>>> socket,id=mon,path=/tmp/tmpvdokyvs3/qemu-41146-monitor.sock -mon
>>> chardev=mon,mode=control -machine g3beige -chardev
>>> socket,id=console,path=/tmp/tmpvdokyvs3/qemu-41146-console.sock,server,nowait
>>>
>>> -serial chardev:console -M graphics=off -kernel
>>> /tmp/avocado_g3uccfo5/avocado_job_61gglyz3/02-tests_acceptance_boot_linux_console.py_BootLinuxConsole.test_ppc_g3beige/day15/invaders.elf
>>>
>>>
>>>
>>> ----
>>>
>>> Note that it passes '[..] -machine g3beige [...] -M graphics=off [...]'.
>>> I suspect you wanted '-machine g3beige,graphics=off'. am I right?
>>
>> No, the -M graphics=off was intended.
>>
>>> or
>>> QEMU will interpret the -M option as a parameter of the already set
>>> machine type?
>>
>> That's exactly the case.
>>
>>> or the -M overwrites -machine, and because  g3beige is the
>>> default type it just works?
>>
>> No, the default machine of qemu-system-ppc is mac99.
> 
> I think you're wrong about the default machine type:
> 
> $ qemu-system-ppc -M help
> Supported machines are:
> 40p                  IBM RS/6000 7020 (40p)
> bamboo               bamboo
> g3beige              Heathrow based PowerMAC (default)
> 
> g3beige is marked as default and seems to be created without any -M
> option

Oops, thanks, I stand corrected.

 Thomas
Wainer dos Santos Moschetta Jan. 30, 2020, 2:12 p.m. UTC | #13
On 1/24/20 3:03 PM, Thomas Huth wrote:
> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
> for various non-x86 machines. We can use them for a boot tests in our
> acceptance test suite.
>
> Let's also make sure that we build the corresponding machines in Travis,
> and while we're there, drop the superfluous --python parameter (python3
> is now the only supported version anyway).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .travis.yml                            |  2 +-
>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>   2 files changed, 97 insertions(+), 1 deletion(-)


Overall result from the discussions I raised on this thread: this patch 
is ready to go as is, IMO.

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Thanks Thomas!

- Wainer


>
> diff --git a/.travis.yml b/.travis.yml
> index 6c1038a0f1..73ca12c921 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -268,7 +268,7 @@ matrix:
>   
>       # Acceptance (Functional) tests
>       - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
> +        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e03add2989..f7ac2a3a59 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -584,3 +584,99 @@ class BootLinuxConsole(Test):
>           self.wait_for_console_pattern(console_pattern)
>           console_pattern = 'No filesystem could mount root'
>           self.wait_for_console_pattern(console_pattern)
> +
> +    def do_test_advcal_2018(self, day, tar_hash, kernel_name):
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day' + day + '.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        archive.extract(file_path, self.workdir)
> +        self.vm.set_console()
> +        self.vm.add_args('-kernel',
> +                         self.workdir + '/day' + day + '/' + kernel_name)
> +        self.vm.launch()
> +        self.wait_for_console_pattern('QEMU advent calendar')
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        self.vm.add_args('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb')
> +        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage')
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
> +
> +    def test_or1k_sim(self):
> +        """
> +        :avocado: tags=arch:or1k
> +        :avocado: tags=machine:or1k-sim
> +        """
> +        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
> +        self.do_test_advcal_2018('20', tar_hash, 'vmlinux')
> +
> +    def test_nios2_10m50(self):
> +        """
> +        :avocado: tags=arch:nios2
> +        :avocado: tags=machine:10m50-ghrd
> +        """
> +        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
> +        self.do_test_advcal_2018('14', tar_hash, 'vmlinux.elf')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        self.vm.add_args('-cpu', 'e5500')
> +        self.do_test_advcal_2018('19', tar_hash, 'uImage')
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        self.vm.add_args('-cpu', 'dc233c')
> +        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf')
Philippe Mathieu-Daudé Jan. 30, 2020, 11:10 p.m. UTC | #14
On 1/24/20 6:03 PM, Thomas Huth wrote:
> The 2018 edition of the QEMU advent calendar 2018 featured Linux images
> for various non-x86 machines. We can use them for a boot tests in our
> acceptance test suite.
> 
> Let's also make sure that we build the corresponding machines in Travis,
> and while we're there, drop the superfluous --python parameter (python3
> is now the only supported version anyway).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .travis.yml                            |  2 +-
>   tests/acceptance/boot_linux_console.py | 96 ++++++++++++++++++++++++++
>   2 files changed, 97 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 6c1038a0f1..73ca12c921 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -268,7 +268,7 @@ matrix:
>   
>       # Acceptance (Functional) tests
>       - env:
> -        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
> +        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index e03add2989..f7ac2a3a59 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -584,3 +584,99 @@ class BootLinuxConsole(Test):
>           self.wait_for_console_pattern(console_pattern)
>           console_pattern = 'No filesystem could mount root'
>           self.wait_for_console_pattern(console_pattern)
> +
> +    def do_test_advcal_2018(self, day, tar_hash, kernel_name):
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day' + day + '.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        archive.extract(file_path, self.workdir)
> +        self.vm.set_console()
> +        self.vm.add_args('-kernel',
> +                         self.workdir + '/day' + day + '/' + kernel_name)
> +        self.vm.launch()
> +        self.wait_for_console_pattern('QEMU advent calendar')
> +
> +    def test_arm_vexpressa9(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:vexpress-a9
> +        """
> +        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
> +        self.vm.add_args('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb')
> +        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage')
> +
> +    def test_m68k_mcf5208evb(self):
> +        """
> +        :avocado: tags=arch:m68k
> +        :avocado: tags=machine:mcf5208evb
> +        """
> +        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> +        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
> +
> +    def test_microblaze_s3adsp1800(self):
> +        """
> +        :avocado: tags=arch:microblaze
> +        :avocado: tags=machine:petalogix-s3adsp1800
> +        """
> +        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
> +        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
> +
> +    def test_or1k_sim(self):
> +        """
> +        :avocado: tags=arch:or1k
> +        :avocado: tags=machine:or1k-sim
> +        """
> +        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
> +        self.do_test_advcal_2018('20', tar_hash, 'vmlinux')
> +
> +    def test_nios2_10m50(self):
> +        """
> +        :avocado: tags=arch:nios2
> +        :avocado: tags=machine:10m50-ghrd
> +        """
> +        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
> +        self.do_test_advcal_2018('14', tar_hash, 'vmlinux.elf')
> +
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        self.vm.add_args('-cpu', 'e5500')
> +        self.do_test_advcal_2018('19', tar_hash, 'uImage')
> +
> +    def test_ppc_g3beige(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:g3beige
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_ppc_mac99(self):
> +        """
> +        :avocado: tags=arch:ppc
> +        :avocado: tags=machine:mac99
> +        """
> +        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
> +        self.vm.add_args('-M', 'graphics=off')
> +        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
> +
> +    def test_sparc_ss20(self):
> +        """
> +        :avocado: tags=arch:sparc
> +        :avocado: tags=machine:SS-20
> +        """
> +        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
> +        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
> +
> +    def test_xtensa_lx60(self):
> +        """
> +        :avocado: tags=arch:xtensa
> +        :avocado: tags=machine:lx60
> +        """
> +        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
> +        self.vm.add_args('-cpu', 'dc233c')
> +        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf')
> 

Thanks, applied to my python-next tree:
https://gitlab.com/philmd/qemu/commits/python-next

Patch
diff mbox series

diff --git a/.travis.yml b/.travis.yml
index 6c1038a0f1..73ca12c921 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -268,7 +268,7 @@  matrix:
 
     # Acceptance (Functional) tests
     - env:
-        - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
+        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index e03add2989..f7ac2a3a59 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -584,3 +584,99 @@  class BootLinuxConsole(Test):
         self.wait_for_console_pattern(console_pattern)
         console_pattern = 'No filesystem could mount root'
         self.wait_for_console_pattern(console_pattern)
+
+    def do_test_advcal_2018(self, day, tar_hash, kernel_name):
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day' + day + '.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        archive.extract(file_path, self.workdir)
+        self.vm.set_console()
+        self.vm.add_args('-kernel',
+                         self.workdir + '/day' + day + '/' + kernel_name)
+        self.vm.launch()
+        self.wait_for_console_pattern('QEMU advent calendar')
+
+    def test_arm_vexpressa9(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:vexpress-a9
+        """
+        tar_hash = '32b7677ce8b6f1471fb0059865f451169934245b'
+        self.vm.add_args('-dtb', self.workdir + '/day16/vexpress-v2p-ca9.dtb')
+        self.do_test_advcal_2018('16', tar_hash, 'winter.zImage')
+
+    def test_m68k_mcf5208evb(self):
+        """
+        :avocado: tags=arch:m68k
+        :avocado: tags=machine:mcf5208evb
+        """
+        tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
+        self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
+
+    def test_microblaze_s3adsp1800(self):
+        """
+        :avocado: tags=arch:microblaze
+        :avocado: tags=machine:petalogix-s3adsp1800
+        """
+        tar_hash = '08bf3e3bfb6b6c7ce1e54ab65d54e189f2caf13f'
+        self.do_test_advcal_2018('17', tar_hash, 'ballerina.bin')
+
+    def test_or1k_sim(self):
+        """
+        :avocado: tags=arch:or1k
+        :avocado: tags=machine:or1k-sim
+        """
+        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
+        self.do_test_advcal_2018('20', tar_hash, 'vmlinux')
+
+    def test_nios2_10m50(self):
+        """
+        :avocado: tags=arch:nios2
+        :avocado: tags=machine:10m50-ghrd
+        """
+        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
+        self.do_test_advcal_2018('14', tar_hash, 'vmlinux.elf')
+
+    def test_ppc64_e500(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:ppce500
+        """
+        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
+        self.vm.add_args('-cpu', 'e5500')
+        self.do_test_advcal_2018('19', tar_hash, 'uImage')
+
+    def test_ppc_g3beige(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:g3beige
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        self.vm.add_args('-M', 'graphics=off')
+        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
+
+    def test_ppc_mac99(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:mac99
+        """
+        tar_hash = 'e0b872a5eb8fdc5bed19bd43ffe863900ebcedfc'
+        self.vm.add_args('-M', 'graphics=off')
+        self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
+
+    def test_sparc_ss20(self):
+        """
+        :avocado: tags=arch:sparc
+        :avocado: tags=machine:SS-20
+        """
+        tar_hash = 'b18550d5d61c7615d989a06edace051017726a9f'
+        self.do_test_advcal_2018('11', tar_hash, 'zImage.elf')
+
+    def test_xtensa_lx60(self):
+        """
+        :avocado: tags=arch:xtensa
+        :avocado: tags=machine:lx60
+        """
+        tar_hash = '49e88d9933742f0164b60839886c9739cb7a0d34'
+        self.vm.add_args('-cpu', 'dc233c')
+        self.do_test_advcal_2018('02', tar_hash, 'santas-sleigh-ride.elf')