| Message ID | 20240716112614.1755692-1-thuth@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Convert avocado tests to normal Python unittests | expand |
On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote: > The Avocado v88 that we use in QEMU is already on a life support > system: It is not supported by upstream anymore, and with the latest > versions of Python, it won't work anymore since it depends on the > "imp" module that has been removed in Python 3.12. > > There have been several attempts to update the test suite in QEMU > to a newer version of Avocado, but so far no attempt has successfully > been merged yet. > > Additionally, the whole "make check" test suite in QEMU is using the > meson test runner nowadays, so running the python-based tests via the > Avocodo test runner looks and feels quite like an oddball, requiring > the users to deal with the knowledge of multiple test runners in > parallel (e.g. the timeout settings work completely differently). > > So instead of trying to update the python-based test suite in QEMU > to a newer version of Avocado, we should maybe try to better integrate > it with the meson test runner instead. Indeed most tests work quite > nicely without the Avocado framework already, as you can see with > this patch series - it does not convert all tests, just a subset so > far, but this already proves that many tests only need small modifi- > cations to work without Avocado. > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > classes (e.g. based on cloud-init images or using SSH) really depend > on the Avocado framework, so we'd need a solution for those if we > want to continue using them. One solution might be to simply use the > required functions from avocado.utils for these tests, and still run > them via the meson test runner instead, but that needs some further > investigation that will be done later. > > > Now if you want to try out these patches: Apply the patches, then > recompile and then run: > > make check-functional > > You can also run single targets e.g. with: > > make check-functional-ppc > > You can also run the tests without any test runner now by > setting the PYTHONPATH environment variable to the "python" folder > of your source tree, and by specifying the build directory via > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the > QEMU binary via QEMU_TEST_QEMU_BINARY. For example: > > export PYTHONPATH=$HOME/qemu/python > export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 > export PYTHONPATH=$HOME/qemu/build > ~/qemu/tests/functional/test_virtio_version.py For the whole series as is Tested-by: Daniel P. Berrangé <berrange@redhat.com> as it does what you claim it does here when I tried it. > The logs of the tests can be found in the build directory under > tests/functional/<testname> - console log and general logs will > be put in separate files there. As an example, one dir name appears to be: __main__.MemAddrCheck.test_phybits_ok_pentium_pae I'd rather prefer it if the dir name matched the test script file name - in this case test_mem_addr_space.py, as I don't want to have to lookup which class names were defined inside each test script. We could drop the "test_" prefix from the method name too IOW, could we make this dir name be: test_mem_addr_space.phybits_ok_pentium_pae With regards, Daniel
On 16/07/2024 18.51, Daniel P. Berrangé wrote: > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote: ... >> So instead of trying to update the python-based test suite in QEMU >> to a newer version of Avocado, we should maybe try to better integrate >> it with the meson test runner instead. Indeed most tests work quite >> nicely without the Avocado framework already, as you can see with >> this patch series - it does not convert all tests, just a subset so >> far, but this already proves that many tests only need small modifi- >> cations to work without Avocado. ... >> Now if you want to try out these patches: Apply the patches, then >> recompile and then run: >> >> make check-functional >> >> You can also run single targets e.g. with: >> >> make check-functional-ppc >> >> You can also run the tests without any test runner now by >> setting the PYTHONPATH environment variable to the "python" folder >> of your source tree, and by specifying the build directory via >> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the >> QEMU binary via QEMU_TEST_QEMU_BINARY. For example: >> >> export PYTHONPATH=$HOME/qemu/python >> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 >> export PYTHONPATH=$HOME/qemu/build >> ~/qemu/tests/functional/test_virtio_version.py > > For the whole series as is > > Tested-by: Daniel P. Berrangé <berrange@redhat.com> > > as it does what you claim it does here when I tried it. Thanks! >> The logs of the tests can be found in the build directory under >> tests/functional/<testname> - console log and general logs will >> be put in separate files there. > > As an example, one dir name appears to be: > > __main__.MemAddrCheck.test_phybits_ok_pentium_pae > > I'd rather prefer it if the dir name matched the test script > file name - in this case test_mem_addr_space.py, as I don't > want to have to lookup which class names were defined inside > each test script. We could drop the "test_" prefix from the > method name too > > IOW, could we make this dir name be: > > test_mem_addr_space.phybits_ok_pentium_pae I can try to change that, indeed ... but the boilerplate code will increase a little bit, I guess, since I cannot simply rely on the unittest.id() function in that case anymore... Thomas
Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com> ha scritto: > The Avocado v88 that we use in QEMU is already on a life support > system: It is not supported by upstream anymore, and with the latest > versions of Python, it won't work anymore since it depends on the > "imp" module that has been removed in Python 3.12. > > There have been several attempts to update the test suite in QEMU > to a newer version of Avocado, but so far no attempt has successfully > been merged yet. > I think we should take another look at that. Avocado 92 should work, though I am not sure if it's also depending on "imp", and if I recall correctly the problem with more recent version was more that it conflicted with distros that didn't have it packaged. That should also not be a problem anymore with the pythondeps.toml mechanism. Additionally, the whole "make check" test suite in QEMU is using the > meson test runner nowadays, so running the python-based tests via the > Avocodo test runner looks and feels quite like an oddball, requiring > the users to deal with the knowledge of multiple test runners in > parallel (e.g. the timeout settings work completely differently). > This is true. Paolo > So instead of trying to update the python-based test suite in QEMU > to a newer version of Avocado, we should maybe try to better integrate > it with the meson test runner instead. Indeed most tests work quite > nicely without the Avocado framework already, as you can see with > this patch series - it does not convert all tests, just a subset so > far, but this already proves that many tests only need small modifi- > cations to work without Avocado. > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > classes (e.g. based on cloud-init images or using SSH) really depend > on the Avocado framework, so we'd need a solution for those if we > want to continue using them. One solution might be to simply use the > required functions from avocado.utils for these tests, and still run > them via the meson test runner instead, but that needs some further > investigation that will be done later. > > > Now if you want to try out these patches: Apply the patches, then > recompile and then run: > > make check-functional > > You can also run single targets e.g. with: > > make check-functional-ppc > > You can also run the tests without any test runner now by > setting the PYTHONPATH environment variable to the "python" folder > of your source tree, and by specifying the build directory via > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the > QEMU binary via QEMU_TEST_QEMU_BINARY. For example: > > export PYTHONPATH=$HOME/qemu/python > export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 > export PYTHONPATH=$HOME/qemu/build > ~/qemu/tests/functional/test_virtio_version.py > > The logs of the tests can be found in the build directory under > tests/functional/<testname> - console log and general logs will > be put in separate files there. > > Still to be done: Update the documentation for this new test framework. > > RFC -> v1: > - Now using pycotap for running the tests instead of "pytest" > - Change the name from "tests/pytest" to "tests/functional" accordingly > - Make it possible to run the tests directly > - Use Python's urllib instead of wget for downloading > - Lots of makefile / meson integration improvements > - Converted more tests > - Update MAINTAINERS file accordingly > - Added a patch to run check-functional in the gitlab-CI > - ... lots of other changes I forgot about ... in fact, I changed so > many things that I also did not dare to pick up the Reviewed-bys > from the RFC > > Thomas Huth (11): > tests/functional: Add base classes for the upcoming pytest-based tests > tests/functional: Convert simple avocado tests into standalone python > tests > tests/functional: Convert avocado tests that just need a small > adjustment > tests/functional: Add python-based tests to the meson build system > tests/functional: Implement fetch_asset() method for downloading > assets > tests/functional: Convert some tests that download files via > fetch_asset() > tests/functional: Add a function for extracting files from an archive > tests/functional: Convert some avocado tests that needed > avocado.utils.archive > tests/functional: Set up logging > tests/functional: Convert the s390x avocado tests into standalone > tests > gitlab-ci: Add "check-functional" to the build tests > > MAINTAINERS | 22 +- > .gitlab-ci.d/buildtest-template.yml | 3 +- > .gitlab-ci.d/buildtest.yml | 60 +-- > pythondeps.toml | 3 +- > tests/Makefile.include | 18 +- > tests/functional/meson.build | 112 +++++ > tests/functional/qemu_test/__init__.py | 384 ++++++++++++++++++ > tests/functional/qemu_test/utils.py | 28 ++ > .../test_arm_canona1100.py} | 21 +- > .../test_arm_n8x0.py} | 25 +- > .../test_avr_mega2560.py} | 11 +- > .../test_cpu_queries.py} | 7 +- > .../test_empty_cpu_model.py} | 7 +- > .../test_info_usernet.py} | 11 +- > .../test_loongarch64_virt.py} | 16 +- > .../test_mem_addr_space.py} | 53 +-- > .../test_mips64el_loongson3v.py} | 26 +- > .../test_netdev_ethtool.py} | 32 +- > .../test_pc_cpu_hotplug_props.py} | 11 +- > .../ppc_405.py => functional/test_ppc_405.py} | 19 +- > .../test_ppc_74xx.py} | 74 ++-- > .../test_ppc_bamboo.py} | 23 +- > .../test_s390x_ccw_virtio.py} | 32 +- > .../test_s390x_topology.py} | 70 ++-- > .../test_sparc64_sun4u.py} | 25 +- > .../version.py => functional/test_version.py} | 13 +- > .../test_virtio_version.py} | 8 +- > tests/meson.build | 1 + > 28 files changed, 795 insertions(+), 320 deletions(-) > create mode 100644 tests/functional/meson.build > create mode 100644 tests/functional/qemu_test/__init__.py > create mode 100644 tests/functional/qemu_test/utils.py > rename tests/{avocado/machine_arm_canona1100.py => > functional/test_arm_canona1100.py} (71%) > mode change 100644 => 100755 > rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} > (71%) > mode change 100644 => 100755 > rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} > (90%) > mode change 100644 => 100755 > rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} > (89%) > mode change 100644 => 100755 > rename tests/{avocado/empty_cpu_model.py => > functional/test_empty_cpu_model.py} (84%) > mode change 100644 => 100755 > rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} > (87%) > mode change 100644 => 100755 > rename tests/{avocado/machine_loongarch.py => > functional/test_loongarch64_virt.py} (89%) > mode change 100644 => 100755 > rename tests/{avocado/mem-addr-space-check.py => > functional/test_mem_addr_space.py} (92%) > mode change 100644 => 100755 > rename tests/{avocado/machine_mips_loongson3v.py => > functional/test_mips64el_loongson3v.py} (55%) > mode change 100644 => 100755 > rename tests/{avocado/netdev-ethtool.py => > functional/test_netdev_ethtool.py} (81%) > mode change 100644 => 100755 > rename tests/{avocado/pc_cpu_hotplug_props.py => > functional/test_pc_cpu_hotplug_props.py} (90%) > mode change 100644 => 100755 > rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%) > mode change 100644 => 100755 > rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) > mode change 100644 => 100755 > rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} > (75%) > mode change 100644 => 100755 > rename tests/{avocado/machine_s390_ccw_virtio.py => > functional/test_s390x_ccw_virtio.py} (95%) > mode change 100644 => 100755 > rename tests/{avocado/s390_topology.py => > functional/test_s390x_topology.py} (90%) > mode change 100644 => 100755 > rename tests/{avocado/machine_sparc64_sun4u.py => > functional/test_sparc64_sun4u.py} (60%) > mode change 100644 => 100755 > rename tests/{avocado/version.py => functional/test_version.py} (78%) > mode change 100644 => 100755 > rename tests/{avocado/virtio_version.py => > functional/test_virtio_version.py} (98%) > mode change 100644 => 100755 > > -- > 2.45.2 > >
On 16/07/2024 19.03, Thomas Huth wrote: > On 16/07/2024 18.51, Daniel P. Berrangé wrote: >> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote: > ... >>> So instead of trying to update the python-based test suite in QEMU >>> to a newer version of Avocado, we should maybe try to better integrate >>> it with the meson test runner instead. Indeed most tests work quite >>> nicely without the Avocado framework already, as you can see with >>> this patch series - it does not convert all tests, just a subset so >>> far, but this already proves that many tests only need small modifi- >>> cations to work without Avocado. > ... >>> Now if you want to try out these patches: Apply the patches, then >>> recompile and then run: >>> >>> make check-functional >>> >>> You can also run single targets e.g. with: >>> >>> make check-functional-ppc >>> >>> You can also run the tests without any test runner now by >>> setting the PYTHONPATH environment variable to the "python" folder >>> of your source tree, and by specifying the build directory via >>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the >>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example: >>> >>> export PYTHONPATH=$HOME/qemu/python >>> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 >>> export PYTHONPATH=$HOME/qemu/build >>> ~/qemu/tests/functional/test_virtio_version.py >> >> For the whole series as is >> >> Tested-by: Daniel P. Berrangé <berrange@redhat.com> >> >> as it does what you claim it does here when I tried it. > > Thanks! > >>> The logs of the tests can be found in the build directory under >>> tests/functional/<testname> - console log and general logs will >>> be put in separate files there. >> >> As an example, one dir name appears to be: >> >> __main__.MemAddrCheck.test_phybits_ok_pentium_pae >> >> I'd rather prefer it if the dir name matched the test script >> file name - in this case test_mem_addr_space.py, as I don't >> want to have to lookup which class names were defined inside >> each test script. We could drop the "test_" prefix from the >> method name too >> >> IOW, could we make this dir name be: >> >> test_mem_addr_space.phybits_ok_pentium_pae > > I can try to change that, indeed ... but the boilerplate code will increase > a little bit, I guess, since I cannot simply rely on the unittest.id() > function in that case anymore... After looking at this for a while, I think it's maybe best to ditch the idea of making the .py files directly runnable and run the tests via a simple pycotap runner instead. Then you get proper module names: $ pyvenv/bin/python3 -m pycotap test_virtio_version TAP version 13 ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs 1..2 Thomas
On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote: > On 16/07/2024 19.03, Thomas Huth wrote: > > On 16/07/2024 18.51, Daniel P. Berrangé wrote: > > > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote: > > ... > > > > So instead of trying to update the python-based test suite in QEMU > > > > to a newer version of Avocado, we should maybe try to better integrate > > > > it with the meson test runner instead. Indeed most tests work quite > > > > nicely without the Avocado framework already, as you can see with > > > > this patch series - it does not convert all tests, just a subset so > > > > far, but this already proves that many tests only need small modifi- > > > > cations to work without Avocado. > > ... > > > > Now if you want to try out these patches: Apply the patches, then > > > > recompile and then run: > > > > > > > > make check-functional > > > > > > > > You can also run single targets e.g. with: > > > > > > > > make check-functional-ppc > > > > > > > > You can also run the tests without any test runner now by > > > > setting the PYTHONPATH environment variable to the "python" folder > > > > of your source tree, and by specifying the build directory via > > > > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the > > > > QEMU binary via QEMU_TEST_QEMU_BINARY. For example: > > > > > > > > export PYTHONPATH=$HOME/qemu/python > > > > export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 > > > > export PYTHONPATH=$HOME/qemu/build > > > > ~/qemu/tests/functional/test_virtio_version.py > > > > > > For the whole series as is > > > > > > Tested-by: Daniel P. Berrangé <berrange@redhat.com> > > > > > > as it does what you claim it does here when I tried it. > > > > Thanks! > > > > > > The logs of the tests can be found in the build directory under > > > > tests/functional/<testname> - console log and general logs will > > > > be put in separate files there. > > > > > > As an example, one dir name appears to be: > > > > > > __main__.MemAddrCheck.test_phybits_ok_pentium_pae > > > > > > I'd rather prefer it if the dir name matched the test script > > > file name - in this case test_mem_addr_space.py, as I don't > > > want to have to lookup which class names were defined inside > > > each test script. We could drop the "test_" prefix from the > > > method name too > > > > > > IOW, could we make this dir name be: > > > > > > test_mem_addr_space.phybits_ok_pentium_pae > > > > I can try to change that, indeed ... but the boilerplate code will > > increase a little bit, I guess, since I cannot simply rely on the > > unittest.id() function in that case anymore... > > After looking at this for a while, I think it's maybe best to ditch the idea > of making the .py files directly runnable and run the tests via a simple > pycotap runner instead. Then you get proper module names: I'd really not want to loose that. To me, eliminating the test harness entirely when debugging is the single biggest improvement of this new approach, especially when I want to 'strace' the test without extraneous processes. > $ pyvenv/bin/python3 -m pycotap test_virtio_version > TAP version 13 > ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs > ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs > 1.. With the following change, you get the same output with direct execution, by making argv look the same as you'd get when running your pycotap example. diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py index cc49fd4c94..3a3e65252d 100644 --- a/tests/functional/qemu_test/__init__.py +++ b/tests/functional/qemu_test/__init__.py @@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash): def main(): tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, test_output_log = pycotap.LogMode.LogToError) - unittest.main(testRunner = tr) + import sys + import os.path + path = os.path.basename(sys.argv[0])[:-3] + unittest.main(module = None, testRunner = tr, argv=["__dummy__", path]) class QemuSystemTest(QemuBaseTest): With regards, Daniel
On 17/07/2024 10.37, Daniel P. Berrangé wrote: > On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote: >> On 16/07/2024 19.03, Thomas Huth wrote: >>> On 16/07/2024 18.51, Daniel P. Berrangé wrote: >>>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote: >>> ... >>>>> So instead of trying to update the python-based test suite in QEMU >>>>> to a newer version of Avocado, we should maybe try to better integrate >>>>> it with the meson test runner instead. Indeed most tests work quite >>>>> nicely without the Avocado framework already, as you can see with >>>>> this patch series - it does not convert all tests, just a subset so >>>>> far, but this already proves that many tests only need small modifi- >>>>> cations to work without Avocado. >>> ... >>>>> Now if you want to try out these patches: Apply the patches, then >>>>> recompile and then run: >>>>> >>>>> make check-functional >>>>> >>>>> You can also run single targets e.g. with: >>>>> >>>>> make check-functional-ppc >>>>> >>>>> You can also run the tests without any test runner now by >>>>> setting the PYTHONPATH environment variable to the "python" folder >>>>> of your source tree, and by specifying the build directory via >>>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the >>>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example: >>>>> >>>>> export PYTHONPATH=$HOME/qemu/python >>>>> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 >>>>> export PYTHONPATH=$HOME/qemu/build >>>>> ~/qemu/tests/functional/test_virtio_version.py >>>> >>>> For the whole series as is >>>> >>>> Tested-by: Daniel P. Berrangé <berrange@redhat.com> >>>> >>>> as it does what you claim it does here when I tried it. >>> >>> Thanks! >>> >>>>> The logs of the tests can be found in the build directory under >>>>> tests/functional/<testname> - console log and general logs will >>>>> be put in separate files there. >>>> >>>> As an example, one dir name appears to be: >>>> >>>> __main__.MemAddrCheck.test_phybits_ok_pentium_pae >>>> >>>> I'd rather prefer it if the dir name matched the test script >>>> file name - in this case test_mem_addr_space.py, as I don't >>>> want to have to lookup which class names were defined inside >>>> each test script. We could drop the "test_" prefix from the >>>> method name too >>>> >>>> IOW, could we make this dir name be: >>>> >>>> test_mem_addr_space.phybits_ok_pentium_pae >>> >>> I can try to change that, indeed ... but the boilerplate code will >>> increase a little bit, I guess, since I cannot simply rely on the >>> unittest.id() function in that case anymore... >> >> After looking at this for a while, I think it's maybe best to ditch the idea >> of making the .py files directly runnable and run the tests via a simple >> pycotap runner instead. Then you get proper module names: > > I'd really not want to loose that. To me, eliminating the test harness > entirely when debugging is the single biggest improvement of this new > approach, especially when I want to 'strace' the test without > extraneous processes. > >> $ pyvenv/bin/python3 -m pycotap test_virtio_version >> TAP version 13 >> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs >> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs >> 1.. > > With the following change, you get the same output with direct > execution, by making argv look the same as you'd get when > running your pycotap example. > > diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py > index cc49fd4c94..3a3e65252d 100644 > --- a/tests/functional/qemu_test/__init__.py > +++ b/tests/functional/qemu_test/__init__.py > @@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash): > def main(): > tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError, > test_output_log = pycotap.LogMode.LogToError) > - unittest.main(testRunner = tr) > + import sys > + import os.path > + path = os.path.basename(sys.argv[0])[:-3] > + unittest.main(module = None, testRunner = tr, argv=["__dummy__", path]) Sweet, thank you very much, looks like this will do the job! Thomas
On 16/07/2024 19.57, Paolo Bonzini wrote: > > > Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com > <mailto:thuth@redhat.com>> ha scritto: > > The Avocado v88 that we use in QEMU is already on a life support > system: It is not supported by upstream anymore, and with the latest > versions of Python, it won't work anymore since it depends on the > "imp" module that has been removed in Python 3.12. > > There have been several attempts to update the test suite in QEMU > to a newer version of Avocado, but so far no attempt has successfully > been merged yet. > > > I think we should take another look at that. Avocado 92 should work, though > I am not sure if it's also depending on "imp" FWIW, the "imp" problem had been fixed here: https://github.com/avocado-framework/avocado/commit/dfc340f1a0b0aeddbf6699e49e94b75ed080137c That had just been released with Avocado version 99, so updating to 92 does not really help. Thomas
On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote: > > The Avocado v88 that we use in QEMU is already on a life support > system: It is not supported by upstream anymore, and with the latest > versions of Python, it won't work anymore since it depends on the > "imp" module that has been removed in Python 3.12. > Hi Thomas, Let me start by saying that this is my attempt to explain the context of the lack of updates on QEMU when it comes to the Avocado update, and point to the actual updates. > There have been several attempts to update the test suite in QEMU > to a newer version of Avocado, but so far no attempt has successfully > been merged yet. > So, we've seen in the past an attempt to update Avocado from 88.1 to a regular release, and the troubles it caused, including a revert. My take was that a LTS version should be used, but during this time, Avocado experienced a rewrite and having it replacing the old implementation in a production level project such as QEMU was tricky. Then 103.0 LTS was released, and there was extensive work to test the QEMU tests before that release was cut. Additionally, there was further work, but unfortunately not posted yet, to make use of 103.0 features in the existing tests[2]. I've tested on GitLab with tests running in parallel, cutting job times in 1/3[2]. A side node is that, because 103.0 is an LTS release, it will receive the needed bug fixes and updates that we deem necessary, including things we find in QEMU tests. In fact, 103.1[3] is in the works. > Additionally, the whole "make check" test suite in QEMU is using the > meson test runner nowadays, so running the python-based tests via the > Avocodo test runner looks and feels quite like an oddball, requiring > the users to deal with the knowledge of multiple test runners in > parallel (e.g. the timeout settings work completely differently). > Now I believe we can be very much in sync here. I've thought for a while that there's no reason for Avocado to cooperate or be compatible with Meson. There's no reason why users can't simply pick how the test gets run. In fact, with the new Avocado architecture, you don't even need to run "avocado" to run an "avocado-instrumented" test. You could pretty much run "avocado-runner-avocado-instrumented" with the right parameters through Meson. > So instead of trying to update the python-based test suite in QEMU > to a newer version of Avocado, we should maybe try to better integrate > it with the meson test runner instead. Indeed most tests work quite > nicely without the Avocado framework already, as you can see with > this patch series - it does not convert all tests, just a subset so > far, but this already proves that many tests only need small modifi- > cations to work without Avocado. > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > classes (e.g. based on cloud-init images or using SSH) really depend > on the Avocado framework, so we'd need a solution for those if we > want to continue using them. One solution might be to simply use the > required functions from avocado.utils for these tests, and still run > them via the meson test runner instead, but that needs some further > investigation that will be done later. > So, I believe this type of higher level testing is something that needs to remain, and even grow. Speaking for Red Hat, I see the movement of QE contributing more Avocado-VT style tests into QEMU itself. This means way more libraries and features that go into a common set of utilities and features (more on that later) than it currently exists in avocado.utils. This brings the autils[4] initiative into the picture. We're about 80% done with the project structure, and after that, it will be a common utility project (such as the cloudinit and ssh) which can be released automatically when the maintainer votes (through GitHub) that a new release is needed. With that, we can have all the existing QEMU tests (and more) running in either meson or Avocado (for users that want Avocado runner only features, such as dependency resolution, container support, etc). Now, if you think any of this makes sense, I'd love to cooperate with you on an update that does both: update Avocado, and allow for the execution of all tests through meson. Thanks and I'm very very sorry for the delay in this update. - Cleber. [1] https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/avocado_test_updates_lts?ref_type=heads [2] https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478 [3] https://github.com/avocado-framework/avocado/pull/5953 [4] https://github.com/avocado-framework/autils
On 25/07/2024 16.21, Cleber Rosa wrote: > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote: ... >> There have been several attempts to update the test suite in QEMU >> to a newer version of Avocado, but so far no attempt has successfully >> been merged yet. > > So, we've seen in the past an attempt to update Avocado from 88.1 to a > regular release, and the troubles it caused, including a revert. My > take was that a LTS version should be used, but during this time, > Avocado experienced a rewrite and having it replacing the old > implementation in a production level project such as QEMU was tricky. > Then 103.0 LTS was released, and there was extensive work to test the > QEMU tests before that release was cut. Additionally, there was > further work, but unfortunately not posted yet, to make use of 103.0 > features in the existing tests[2]. I've tested on GitLab with tests > running in parallel, cutting job times in 1/3[2]. A side node is > that, because 103.0 is an LTS release, it will receive the needed bug > fixes and updates that we deem necessary, including things we find in > QEMU tests. In fact, 103.1[3] is in the works. Hi Cleber, thanks for the explanation, but we really need to replace v88 rather *now* since v88 does not work with the latest versions of Python anymore (there is a work-around on Fedora fortunately, but it's completely broken on Ubuntu 24.04 as far as I know). So even a single-threaded execution with v103 would have been better than waiting forever for your update to land. The problem with v88 being broken has been raised a couple of times already, but it's incredibly hard to get a response from you Avocado folks, so with hardly any help from the Avocado side, and nobody on the QEMU side being really familiar with the Avocado stuff, and with the meson test runner being used by all other subsystems in QEMU already, I think it's best if we continue with this series here. >> Additionally, the whole "make check" test suite in QEMU is using the >> meson test runner nowadays, so running the python-based tests via the >> Avocodo test runner looks and feels quite like an oddball, requiring >> the users to deal with the knowledge of multiple test runners in >> parallel (e.g. the timeout settings work completely differently). >> > > Now I believe we can be very much in sync here. I've thought for a > while that there's no reason for Avocado to cooperate or be compatible > with Meson. There's no reason why users can't simply pick how the > test gets run. In fact, with the new Avocado architecture, you don't > even need to run "avocado" to run an "avocado-instrumented" test. You > could pretty much run "avocado-runner-avocado-instrumented" with the > right parameters through Meson. Ok, good to know, we could maybe use that eventually for the tests that really require the Avocado framework (i.e. the cloud-init based tests). For the others, as Daniel said in an earlier mail, it's much more convenient if you can also run the tests directly instead, without such a layer in between - that makes debugging way easier. >> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn >> classes (e.g. based on cloud-init images or using SSH) really depend >> on the Avocado framework, so we'd need a solution for those if we >> want to continue using them. One solution might be to simply use the >> required functions from avocado.utils for these tests, and still run >> them via the meson test runner instead, but that needs some further >> investigation that will be done later. > > So, I believe this type of higher level testing is something that > needs to remain, and even grow. Speaking for Red Hat, I see the > movement of QE contributing more Avocado-VT style tests into QEMU > itself. I didn't really see such a movement in recent times yet ... Could you point to an example? > This means way more libraries and features that go into a > common set of utilities and features (more on that later) than it > currently exists in avocado.utils. > > This brings the autils[4] initiative into the picture. We're about > 80% done with the project structure, and after that, it will be a > common utility project (such as the cloudinit and ssh) which can be > released automatically when the maintainer votes (through GitHub) that > a new release is needed. autils sounds promising, but I just had a look at the repository, and there does not seem to be that much code available there yet, so I guess it will take still quite a long time 'til that's ready? Thomas
On Fri, Jul 26, 2024 at 6:07 AM Thomas Huth <thuth@redhat.com> wrote: > > On 25/07/2024 16.21, Cleber Rosa wrote: > > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote: > ... > >> There have been several attempts to update the test suite in QEMU > >> to a newer version of Avocado, but so far no attempt has successfully > >> been merged yet. > > > > So, we've seen in the past an attempt to update Avocado from 88.1 to a > > regular release, and the troubles it caused, including a revert. My > > take was that a LTS version should be used, but during this time, > > Avocado experienced a rewrite and having it replacing the old > > implementation in a production level project such as QEMU was tricky. > > Then 103.0 LTS was released, and there was extensive work to test the > > QEMU tests before that release was cut. Additionally, there was > > further work, but unfortunately not posted yet, to make use of 103.0 > > features in the existing tests[2]. I've tested on GitLab with tests > > running in parallel, cutting job times in 1/3[2]. A side node is > > that, because 103.0 is an LTS release, it will receive the needed bug > > fixes and updates that we deem necessary, including things we find in > > QEMU tests. In fact, 103.1[3] is in the works. > > Hi Cleber, > > thanks for the explanation, but we really need to replace v88 rather *now* > since v88 does not work with the latest versions of Python anymore (there is > a work-around on Fedora fortunately, but it's completely broken on Ubuntu > 24.04 as far as I know). So even a single-threaded execution with v103 would > have been better than waiting forever for your update to land. The problem > with v88 being broken has been raised a couple of times already, but it's > incredibly hard to get a response from you Avocado folks, so with hardly any > help from the Avocado side, and nobody on the QEMU side being really > familiar with the Avocado stuff, and with the meson test runner being used > by all other subsystems in QEMU already, I think it's best if we continue > with this series here. > Hi Thomas, I agree with the urgency. I've posted the updates here: https://lists.gnu.org/archive/html/qemu-devel/2024-07/msg06236.html Sorry for the delay, and I hope this gives more time to make the evolution towards a solution that better suits QEMU. Regards, - Cleber. > >> Additionally, the whole "make check" test suite in QEMU is using the > >> meson test runner nowadays, so running the python-based tests via the > >> Avocodo test runner looks and feels quite like an oddball, requiring > >> the users to deal with the knowledge of multiple test runners in > >> parallel (e.g. the timeout settings work completely differently). > >> > > > > Now I believe we can be very much in sync here. I've thought for a > > while that there's no reason for Avocado to cooperate or be compatible > > with Meson. There's no reason why users can't simply pick how the > > test gets run. In fact, with the new Avocado architecture, you don't > > even need to run "avocado" to run an "avocado-instrumented" test. You > > could pretty much run "avocado-runner-avocado-instrumented" with the > > right parameters through Meson. > > Ok, good to know, we could maybe use that eventually for the tests that > really require the Avocado framework (i.e. the cloud-init based tests). > > For the others, as Daniel said in an earlier mail, it's much more convenient > if you can also run the tests directly instead, without such a layer in > between - that makes debugging way easier. > > >> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > >> classes (e.g. based on cloud-init images or using SSH) really depend > >> on the Avocado framework, so we'd need a solution for those if we > >> want to continue using them. One solution might be to simply use the > >> required functions from avocado.utils for these tests, and still run > >> them via the meson test runner instead, but that needs some further > >> investigation that will be done later. > > > > So, I believe this type of higher level testing is something that > > needs to remain, and even grow. Speaking for Red Hat, I see the > > movement of QE contributing more Avocado-VT style tests into QEMU > > itself. > > I didn't really see such a movement in recent times yet ... Could you point > to an example? > Indeed, there is no public evidence of that, yet. > > This means way more libraries and features that go into a > > common set of utilities and features (more on that later) than it > > currently exists in avocado.utils. > > > > This brings the autils[4] initiative into the picture. We're about > > 80% done with the project structure, and after that, it will be a > > common utility project (such as the cloudinit and ssh) which can be > > released automatically when the maintainer votes (through GitHub) that > > a new release is needed. > > autils sounds promising, but I just had a look at the repository, and there > does not seem to be that much code available there yet, so I guess it will > take still quite a long time 'til that's ready? > Like I said previously, we were working on the project infrastructure. We haven't ported much code over there yet. There are very few things missing, and all of them have been assigned to be worked on. It's been tracked here: https://github.com/avocado-framework/autils/issues/1 My estimation is that 2-4 weeks for the project infrastructure to be complete. Regards, - Cleber. > Thomas >
On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote: > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote: > > There have been several attempts to update the test suite in QEMU > > to a newer version of Avocado, but so far no attempt has successfully > > been merged yet. > > > > So, we've seen in the past an attempt to update Avocado from 88.1 to a > regular release, and the troubles it caused, including a revert. My > take was that a LTS version should be used, but during this time, > Avocado experienced a rewrite and having it replacing the old > implementation in a production level project such as QEMU was tricky. IMHO the problems with updating avocado in QEMU are a sign that we're suffering from excess complexity, as it shouldn't be this difficult to update the test harness. That we need to stick on an LTS release to protect us from further instability further suggests that we would benefit from a much simpler solution here. > > Additionally, the whole "make check" test suite in QEMU is using the > > meson test runner nowadays, so running the python-based tests via the > > Avocodo test runner looks and feels quite like an oddball, requiring > > the users to deal with the knowledge of multiple test runners in > > parallel (e.g. the timeout settings work completely differently). > > > > Now I believe we can be very much in sync here. I've thought for a > while that there's no reason for Avocado to cooperate or be compatible > with Meson. There's no reason why users can't simply pick how the > test gets run. In fact, with the new Avocado architecture, you don't > even need to run "avocado" to run an "avocado-instrumented" test. You > could pretty much run "avocado-runner-avocado-instrumented" with the > right parameters through Meson. Looking back to when we introduced avocado, we still had our original home grown build system, and multiple different ways to run integrate testing betwen our unit tests / qtests and block iotests. Adding avocado to the mix introduced a third way to run tests. All round it was rather a mess of inconsistent approaches to testing, but that was a fact of life with our old home grown build/test process. Today we have adopted a much more standardized build system via meson. While we do still have the block iotests harness, we now expose every test to meson individually to reduce what the intermediate harness needs to be involved in doing. Thomas' series here is bringing the same simplified integration to our functional tests, such that each functional test is known to meson individually. Overall we'll now have a consistent approach to integrating tests into meson, without intermediate harnesses adding their own logic. I think this will make it easier for QEMU maintainers to understand how the tests are working, and make it easier to maintain them and troubleshoot failures. So overall I'm not seeing a compelling reason for us to offer users a choice between meson & avocado how they run tests. Such choice is a maint burden over the long term, as we need to validate that both options keep working. Focusing on a single option reduces the complexity and maint work and gives us more flexiblity to evolve the code over time. > > So instead of trying to update the python-based test suite in QEMU > > to a newer version of Avocado, we should maybe try to better integrate > > it with the meson test runner instead. Indeed most tests work quite > > nicely without the Avocado framework already, as you can see with > > this patch series - it does not convert all tests, just a subset so > > far, but this already proves that many tests only need small modifi- > > cations to work without Avocado. > > > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > > classes (e.g. based on cloud-init images or using SSH) really depend > > on the Avocado framework, so we'd need a solution for those if we > > want to continue using them. One solution might be to simply use the > > required functions from avocado.utils for these tests, and still run > > them via the meson test runner instead, but that needs some further > > investigation that will be done later. > > > > So, I believe this type of higher level testing is something that > needs to remain, and even grow. Speaking for Red Hat, I see the > movement of QE contributing more Avocado-VT style tests into QEMU > itself. This means way more libraries and features that go into a > common set of utilities and features (more on that later) than it > currently exists in avocado.utils. The series here is showing how simple it is to adapt our existing tests to the new execution approach, so shouldn't have a significant impact on ability of people to contribute futher tests in future. With regards, Daniel
On Mon, Jul 29, 2024 at 8:44 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote: > > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote: > > > There have been several attempts to update the test suite in QEMU > > > to a newer version of Avocado, but so far no attempt has successfully > > > been merged yet. > > > > > > > So, we've seen in the past an attempt to update Avocado from 88.1 to a > > regular release, and the troubles it caused, including a revert. My > > take was that a LTS version should be used, but during this time, > > Avocado experienced a rewrite and having it replacing the old > > implementation in a production level project such as QEMU was tricky. > > IMHO the problems with updating avocado in QEMU are a sign that we're > suffering from excess complexity, as it shouldn't be this difficult to > update the test harness. That we need to stick on an LTS release to > protect us from further instability further suggests that we would > benefit from a much simpler solution here. > Hi Daniel, Some other projects using Avocado have picked different, non-LTS versions, to use as their pinned versions. The consequences of using rolling releases versus LTS-like releases is not something specific to one software package IMO. The suggestion to use LTS was to minimize the impact on users writing and running tests, while allowing for only needed fixes or very important features (say supporting a new Python version) to be addressed. > > > Additionally, the whole "make check" test suite in QEMU is using the > > > meson test runner nowadays, so running the python-based tests via the > > > Avocodo test runner looks and feels quite like an oddball, requiring > > > the users to deal with the knowledge of multiple test runners in > > > parallel (e.g. the timeout settings work completely differently). > > > > > > > Now I believe we can be very much in sync here. I've thought for a > > while that there's no reason for Avocado to cooperate or be compatible > > with Meson. There's no reason why users can't simply pick how the > > test gets run. In fact, with the new Avocado architecture, you don't > > even need to run "avocado" to run an "avocado-instrumented" test. You > > could pretty much run "avocado-runner-avocado-instrumented" with the > > right parameters through Meson. > > Looking back to when we introduced avocado, we still had our original > home grown build system, and multiple different ways to run integrate > testing betwen our unit tests / qtests and block iotests. Adding > avocado to the mix introduced a third way to run tests. All round it > was rather a mess of inconsistent approaches to testing, but that was > a fact of life with our old home grown build/test process. > Agreed. > Today we have adopted a much more standardized build system via meson. > While we do still have the block iotests harness, we now expose every > test to meson individually to reduce what the intermediate harness > needs to be involved in doing. Thomas' series here is bringing the same > simplified integration to our functional tests, such that each functional > test is known to meson individually. Overall we'll now have a consistent > approach to integrating tests into meson, without intermediate harnesses > adding their own logic. > > I think this will make it easier for QEMU maintainers to understand how > the tests are working, and make it easier to maintain them and troubleshoot > failures. > I think this is a very good way forward. I don't think anything (simple tests here) should depend on what they don't need to depend on. > So overall I'm not seeing a compelling reason for us to offer users a > choice between meson & avocado how they run tests. Such choice is a maint > burden over the long term, as we need to validate that both options keep > working. Focusing on a single option reduces the complexity and maint > work and gives us more flexiblity to evolve the code over time. > > If meson is capable of doing all that can be done right now, then I believe you're right. No reason to use duplicate software. Now, If you mean that having "single option" may imply duplicating code that already exists and is maintained elsewhere, into the QEMU repo, then you're wrong. This can be easily judged by how easy it is to convert all current tests and their features (timeout, download management, etc). Another level of judgement that I believe would be helpful to the decision making process here is how this impacts growing the test coverage. Tests that manipulate both QEMU and the guest are still very very simple compared to what can be found on what I'll call "real life integration testing" of QEMU (read Avocado-VT like tests). But they present a glimpse of the problem. Having such tests requires a good amount of code to either be used from a project/repo that addresses that specifically, or be duplicated inside QEMU. > > > So instead of trying to update the python-based test suite in QEMU > > > to a newer version of Avocado, we should maybe try to better integrate > > > it with the meson test runner instead. Indeed most tests work quite > > > nicely without the Avocado framework already, as you can see with > > > this patch series - it does not convert all tests, just a subset so > > > far, but this already proves that many tests only need small modifi- > > > cations to work without Avocado. > > > > > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn > > > classes (e.g. based on cloud-init images or using SSH) really depend > > > on the Avocado framework, so we'd need a solution for those if we > > > want to continue using them. One solution might be to simply use the > > > required functions from avocado.utils for these tests, and still run > > > them via the meson test runner instead, but that needs some further > > > investigation that will be done later. > > > > > > > So, I believe this type of higher level testing is something that > > needs to remain, and even grow. Speaking for Red Hat, I see the > > movement of QE contributing more Avocado-VT style tests into QEMU > > itself. This means way more libraries and features that go into a > > common set of utilities and features (more on that later) than it > > currently exists in avocado.utils. > > The series here is showing how simple it is to adapt our existing > tests to the new execution approach, so shouldn't have a significant > impact on ability of people to contribute futher tests in future. > I am on the side of the unified execution approach. But, will it really be simple to convert all tests? "Real life integration" tests too? I would love to see that, but I can't see how that can be done without the woes mentioned before (duplicating other code or limiting the further toolset and capabilities of test writers). Best regards, - Cleber.
The Avocado v88 that we use in QEMU is already on a life support system: It is not supported by upstream anymore, and with the latest versions of Python, it won't work anymore since it depends on the "imp" module that has been removed in Python 3.12. There have been several attempts to update the test suite in QEMU to a newer version of Avocado, but so far no attempt has successfully been merged yet. Additionally, the whole "make check" test suite in QEMU is using the meson test runner nowadays, so running the python-based tests via the Avocodo test runner looks and feels quite like an oddball, requiring the users to deal with the knowledge of multiple test runners in parallel (e.g. the timeout settings work completely differently). So instead of trying to update the python-based test suite in QEMU to a newer version of Avocado, we should maybe try to better integrate it with the meson test runner instead. Indeed most tests work quite nicely without the Avocado framework already, as you can see with this patch series - it does not convert all tests, just a subset so far, but this already proves that many tests only need small modifi- cations to work without Avocado. Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn classes (e.g. based on cloud-init images or using SSH) really depend on the Avocado framework, so we'd need a solution for those if we want to continue using them. One solution might be to simply use the required functions from avocado.utils for these tests, and still run them via the meson test runner instead, but that needs some further investigation that will be done later. Now if you want to try out these patches: Apply the patches, then recompile and then run: make check-functional You can also run single targets e.g. with: make check-functional-ppc You can also run the tests without any test runner now by setting the PYTHONPATH environment variable to the "python" folder of your source tree, and by specifying the build directory via QEMU_BUILD_ROOT (if autodetection fails) and by specifying the QEMU binary via QEMU_TEST_QEMU_BINARY. For example: export PYTHONPATH=$HOME/qemu/python export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 export PYTHONPATH=$HOME/qemu/build ~/qemu/tests/functional/test_virtio_version.py The logs of the tests can be found in the build directory under tests/functional/<testname> - console log and general logs will be put in separate files there. Still to be done: Update the documentation for this new test framework. RFC -> v1: - Now using pycotap for running the tests instead of "pytest" - Change the name from "tests/pytest" to "tests/functional" accordingly - Make it possible to run the tests directly - Use Python's urllib instead of wget for downloading - Lots of makefile / meson integration improvements - Converted more tests - Update MAINTAINERS file accordingly - Added a patch to run check-functional in the gitlab-CI - ... lots of other changes I forgot about ... in fact, I changed so many things that I also did not dare to pick up the Reviewed-bys from the RFC Thomas Huth (11): tests/functional: Add base classes for the upcoming pytest-based tests tests/functional: Convert simple avocado tests into standalone python tests tests/functional: Convert avocado tests that just need a small adjustment tests/functional: Add python-based tests to the meson build system tests/functional: Implement fetch_asset() method for downloading assets tests/functional: Convert some tests that download files via fetch_asset() tests/functional: Add a function for extracting files from an archive tests/functional: Convert some avocado tests that needed avocado.utils.archive tests/functional: Set up logging tests/functional: Convert the s390x avocado tests into standalone tests gitlab-ci: Add "check-functional" to the build tests MAINTAINERS | 22 +- .gitlab-ci.d/buildtest-template.yml | 3 +- .gitlab-ci.d/buildtest.yml | 60 +-- pythondeps.toml | 3 +- tests/Makefile.include | 18 +- tests/functional/meson.build | 112 +++++ tests/functional/qemu_test/__init__.py | 384 ++++++++++++++++++ tests/functional/qemu_test/utils.py | 28 ++ .../test_arm_canona1100.py} | 21 +- .../test_arm_n8x0.py} | 25 +- .../test_avr_mega2560.py} | 11 +- .../test_cpu_queries.py} | 7 +- .../test_empty_cpu_model.py} | 7 +- .../test_info_usernet.py} | 11 +- .../test_loongarch64_virt.py} | 16 +- .../test_mem_addr_space.py} | 53 +-- .../test_mips64el_loongson3v.py} | 26 +- .../test_netdev_ethtool.py} | 32 +- .../test_pc_cpu_hotplug_props.py} | 11 +- .../ppc_405.py => functional/test_ppc_405.py} | 19 +- .../test_ppc_74xx.py} | 74 ++-- .../test_ppc_bamboo.py} | 23 +- .../test_s390x_ccw_virtio.py} | 32 +- .../test_s390x_topology.py} | 70 ++-- .../test_sparc64_sun4u.py} | 25 +- .../version.py => functional/test_version.py} | 13 +- .../test_virtio_version.py} | 8 +- tests/meson.build | 1 + 28 files changed, 795 insertions(+), 320 deletions(-) create mode 100644 tests/functional/meson.build create mode 100644 tests/functional/qemu_test/__init__.py create mode 100644 tests/functional/qemu_test/utils.py rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%) mode change 100644 => 100755 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%) mode change 100644 => 100755 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%) mode change 100644 => 100755 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%) mode change 100644 => 100755 rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%) mode change 100644 => 100755 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%) mode change 100644 => 100755 rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%) mode change 100644 => 100755 rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%) mode change 100644 => 100755 rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%) mode change 100644 => 100755 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%) mode change 100644 => 100755 rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%) mode change 100644 => 100755 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%) mode change 100644 => 100755 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%) mode change 100644 => 100755 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%) mode change 100644 => 100755 rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%) mode change 100644 => 100755 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%) mode change 100644 => 100755 rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%) mode change 100644 => 100755 rename tests/{avocado/version.py => functional/test_version.py} (78%) mode change 100644 => 100755 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%) mode change 100644 => 100755