| Message ID | 20240821082748.65853-12-thuth@redhat.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Convert avocado tests to normal Python unittests | expand |
On 21/8/24 10:27, Thomas Huth wrote: > Provide a meson.build file for the upcoming python-based functional > tests, and add some wrapper glue targets to the tests/Makefile.include > file. We are going to use two "speed" modes for the functional tests: > The "quick" tests can be run at any time (i.e. also during "make check"), > while the "thorough" tests should only be run when running a > "make check-functional" test run (since these tests might download > additional assets from the internet). > > The changes to the meson.build files are partly based on an earlier > patch by Ani Sinha. > > Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> > Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > tests/Makefile.include | 11 ++++++ > tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++ > tests/meson.build | 1 + > 3 files changed, 78 insertions(+) > create mode 100644 tests/functional/meson.build > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index fbb1a4b211..66c8cc3123 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -9,6 +9,8 @@ check-help: > @echo "Individual test suites:" > @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target" > @echo " $(MAKE) check-qtest Run qtest tests" > + @echo " $(MAKE) check-functional Run python-based functional tests" > + @echo " $(MAKE) check-functional-TARGET Run functional tests for a given target" > @echo " $(MAKE) check-unit Run qobject tests" > @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" > @echo " $(MAKE) check-block Run block tests" > @@ -152,6 +154,15 @@ check-acceptance-deprecated-warning: > > check-acceptance: check-acceptance-deprecated-warning | check-avocado > > +FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS))) This only handles system binaries. > +.PHONY: $(FUNCTIONAL_TARGETS) > +$(FUNCTIONAL_TARGETS): > + @$(MAKE) SPEED=thorough $(subst -functional,-func,$@) > + > +.PHONY: check-functional > +check-functional: > + @$(MAKE) SPEED=thorough check-func check-func-quick > + > # Consolidated targets > > .PHONY: check check-clean get-vm-images > diff --git a/tests/functional/meson.build b/tests/functional/meson.build > new file mode 100644 > index 0000000000..7fc9c4e513 > --- /dev/null > +++ b/tests/functional/meson.build > @@ -0,0 +1,66 @@ > +# QEMU functional tests: > +# Tests that are put in the 'quick' category are run by default during > +# 'make check'. Everything that should not be run during 'make check' > +# (e.g. tests that fetch assets from the internet) should be put into > +# the 'thorough' category instead. > + > +# Most tests run too slow with TCI enabled, so skip the functional tests there > +if get_option('tcg_interpreter') > + subdir_done() > +endif > + > +# Timeouts for individual tests that can be slow e.g. with debugging enabled > +test_timeouts = { > +} > + > +tests_generic = [ > +] > + > +tests_x86_64_quick = [ Hard to distinct system/user tests. > +] > + > +tests_x86_64_thorough = [ It is a bit unfortunate to have to modify meson.build to add a test: this triggers meson to regenerate build files. Good enough for now, hopefully we will improve. > +] > + > +foreach speed : ['quick', 'thorough'] > + foreach dir : target_dirs > + if not dir.endswith('-softmmu') > + continue > + endif > + > + target_base = dir.split('-')[0] > + test_emulator = emulators['qemu-system-' + target_base] > + > + if speed == 'quick' > + suites = ['func-quick', 'func-' + target_base] > + target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic > + else > + suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed] > + target_tests = get_variable('tests_' + target_base + '_' + speed, []) > + endif > + > + test_deps = roms > + test_env = environment() > + if have_tools > + test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img') > + test_deps += [qemu_img] > + endif > + test_env.set('QEMU_TEST_QEMU_BINARY', > + meson.global_build_root() / 'qemu-system-' + target_base) > + test_env.set('QEMU_BUILD_ROOT', meson.project_build_root()) > + test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' + > + meson.current_source_dir()) > + > + foreach test : target_tests > + test('func-@0@/@1@'.format(target_base, test), > + python, > + depends: [test_deps, test_emulator, emulator_modules], > + env: test_env, > + args: [meson.current_source_dir() / 'test_' + test + '.py'], > + protocol: 'tap', > + timeout: test_timeouts.get(test, 60), > + priority: test_timeouts.get(test, 60), > + suite: suites) > + endforeach > + endforeach > +endforeach > diff --git a/tests/meson.build b/tests/meson.build > index 80dd3029cf..907a4c1c98 100644 > --- a/tests/meson.build > +++ b/tests/meson.build > @@ -85,3 +85,4 @@ subdir('unit') > subdir('qapi-schema') > subdir('qtest') > subdir('migration') > +subdir('functional')
On 21/8/24 10:27, Thomas Huth wrote: > Provide a meson.build file for the upcoming python-based functional > tests, and add some wrapper glue targets to the tests/Makefile.include > file. We are going to use two "speed" modes for the functional tests: > The "quick" tests can be run at any time (i.e. also during "make check"), > while the "thorough" tests should only be run when running a > "make check-functional" test run (since these tests might download > additional assets from the internet). > > The changes to the meson.build files are partly based on an earlier > patch by Ani Sinha. > > Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> > Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > tests/Makefile.include | 11 ++++++ > tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++ > tests/meson.build | 1 + > 3 files changed, 78 insertions(+) > create mode 100644 tests/functional/meson.build > +# Timeouts for individual tests that can be slow e.g. with debugging enabled > +test_timeouts = { > +} > + foreach test : target_tests > + test('func-@0@/@1@'.format(target_base, test), > + python, > + depends: [test_deps, test_emulator, emulator_modules], > + env: test_env, > + args: [meson.current_source_dir() / 'test_' + test + '.py'], > + protocol: 'tap', > + timeout: test_timeouts.get(test, 60), > + priority: test_timeouts.get(test, 60), IIUC with Avocado the timeout was for each test_func in a TestClass. Now this is only per TestClass. Hopefully I'm wrong... > + suite: suites) > + endforeach > + endforeach > +endforeach > diff --git a/tests/meson.build b/tests/meson.build > index 80dd3029cf..907a4c1c98 100644 > --- a/tests/meson.build > +++ b/tests/meson.build > @@ -85,3 +85,4 @@ subdir('unit') > subdir('qapi-schema') > subdir('qtest') > subdir('migration') > +subdir('functional')
On 23/08/2024 14.54, Philippe Mathieu-Daudé wrote: > On 21/8/24 10:27, Thomas Huth wrote: >> Provide a meson.build file for the upcoming python-based functional >> tests, and add some wrapper glue targets to the tests/Makefile.include >> file. We are going to use two "speed" modes for the functional tests: >> The "quick" tests can be run at any time (i.e. also during "make check"), >> while the "thorough" tests should only be run when running a >> "make check-functional" test run (since these tests might download >> additional assets from the internet). >> >> The changes to the meson.build files are partly based on an earlier >> patch by Ani Sinha. >> >> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> >> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> tests/Makefile.include | 11 ++++++ >> tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++ >> tests/meson.build | 1 + >> 3 files changed, 78 insertions(+) >> create mode 100644 tests/functional/meson.build > > >> +# Timeouts for individual tests that can be slow e.g. with debugging enabled >> +test_timeouts = { >> +} > > >> + foreach test : target_tests >> + test('func-@0@/@1@'.format(target_base, test), >> + python, >> + depends: [test_deps, test_emulator, emulator_modules], >> + env: test_env, >> + args: [meson.current_source_dir() / 'test_' + test + '.py'], >> + protocol: 'tap', >> + timeout: test_timeouts.get(test, 60), >> + priority: test_timeouts.get(test, 60), > > IIUC with Avocado the timeout was for each test_func in a TestClass. > Now this is only per TestClass. Hopefully I'm wrong... I think you're right ... we might need to adjust the meson timeouts here and there in case they are causing problems, but that's just business as usual (we had to do the same when enabling the meson timeouts for the qtests for example). Thomas
On Mon, Aug 26, 2024 at 10:18:44AM +0200, Thomas Huth wrote: > On 23/08/2024 14.54, Philippe Mathieu-Daudé wrote: > > On 21/8/24 10:27, Thomas Huth wrote: > > > Provide a meson.build file for the upcoming python-based functional > > > tests, and add some wrapper glue targets to the tests/Makefile.include > > > file. We are going to use two "speed" modes for the functional tests: > > > The "quick" tests can be run at any time (i.e. also during "make check"), > > > while the "thorough" tests should only be run when running a > > > "make check-functional" test run (since these tests might download > > > additional assets from the internet). > > > > > > The changes to the meson.build files are partly based on an earlier > > > patch by Ani Sinha. > > > > > > Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> > > > Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> > > > Signed-off-by: Thomas Huth <thuth@redhat.com> > > > --- > > > tests/Makefile.include | 11 ++++++ > > > tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++ > > > tests/meson.build | 1 + > > > 3 files changed, 78 insertions(+) > > > create mode 100644 tests/functional/meson.build > > > > > > > +# Timeouts for individual tests that can be slow e.g. with debugging enabled > > > +test_timeouts = { > > > +} > > > > > > > + foreach test : target_tests > > > + test('func-@0@/@1@'.format(target_base, test), > > > + python, > > > + depends: [test_deps, test_emulator, emulator_modules], > > > + env: test_env, > > > + args: [meson.current_source_dir() / 'test_' + test + '.py'], > > > + protocol: 'tap', > > > + timeout: test_timeouts.get(test, 60), > > > + priority: test_timeouts.get(test, 60), > > > > IIUC with Avocado the timeout was for each test_func in a TestClass. > > Now this is only per TestClass. Hopefully I'm wrong... > > I think you're right ... we might need to adjust the meson timeouts here and > there in case they are causing problems, but that's just business as usual > (we had to do the same when enabling the meson timeouts for the qtests for > example). Meson timeouts are around the lifetime of whatever child process we tell it to execute. So, yes, the timeout is for the .py file as a whole, not the individual TestClass within it. This is different from Avocado, but not a big problem. Worst case we'll adjust some meson level timeouts, or split overly long running .py test files into multiple test .py files. With regards, Daniel
diff --git a/tests/Makefile.include b/tests/Makefile.include index fbb1a4b211..66c8cc3123 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -9,6 +9,8 @@ check-help: @echo "Individual test suites:" @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target" @echo " $(MAKE) check-qtest Run qtest tests" + @echo " $(MAKE) check-functional Run python-based functional tests" + @echo " $(MAKE) check-functional-TARGET Run functional tests for a given target" @echo " $(MAKE) check-unit Run qobject tests" @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-block Run block tests" @@ -152,6 +154,15 @@ check-acceptance-deprecated-warning: check-acceptance: check-acceptance-deprecated-warning | check-avocado +FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS))) +.PHONY: $(FUNCTIONAL_TARGETS) +$(FUNCTIONAL_TARGETS): + @$(MAKE) SPEED=thorough $(subst -functional,-func,$@) + +.PHONY: check-functional +check-functional: + @$(MAKE) SPEED=thorough check-func check-func-quick + # Consolidated targets .PHONY: check check-clean get-vm-images diff --git a/tests/functional/meson.build b/tests/functional/meson.build new file mode 100644 index 0000000000..7fc9c4e513 --- /dev/null +++ b/tests/functional/meson.build @@ -0,0 +1,66 @@ +# QEMU functional tests: +# Tests that are put in the 'quick' category are run by default during +# 'make check'. Everything that should not be run during 'make check' +# (e.g. tests that fetch assets from the internet) should be put into +# the 'thorough' category instead. + +# Most tests run too slow with TCI enabled, so skip the functional tests there +if get_option('tcg_interpreter') + subdir_done() +endif + +# Timeouts for individual tests that can be slow e.g. with debugging enabled +test_timeouts = { +} + +tests_generic = [ +] + +tests_x86_64_quick = [ +] + +tests_x86_64_thorough = [ +] + +foreach speed : ['quick', 'thorough'] + foreach dir : target_dirs + if not dir.endswith('-softmmu') + continue + endif + + target_base = dir.split('-')[0] + test_emulator = emulators['qemu-system-' + target_base] + + if speed == 'quick' + suites = ['func-quick', 'func-' + target_base] + target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic + else + suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed] + target_tests = get_variable('tests_' + target_base + '_' + speed, []) + endif + + test_deps = roms + test_env = environment() + if have_tools + test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img') + test_deps += [qemu_img] + endif + test_env.set('QEMU_TEST_QEMU_BINARY', + meson.global_build_root() / 'qemu-system-' + target_base) + test_env.set('QEMU_BUILD_ROOT', meson.project_build_root()) + test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' + + meson.current_source_dir()) + + foreach test : target_tests + test('func-@0@/@1@'.format(target_base, test), + python, + depends: [test_deps, test_emulator, emulator_modules], + env: test_env, + args: [meson.current_source_dir() / 'test_' + test + '.py'], + protocol: 'tap', + timeout: test_timeouts.get(test, 60), + priority: test_timeouts.get(test, 60), + suite: suites) + endforeach + endforeach +endforeach diff --git a/tests/meson.build b/tests/meson.build index 80dd3029cf..907a4c1c98 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -85,3 +85,4 @@ subdir('unit') subdir('qapi-schema') subdir('qtest') subdir('migration') +subdir('functional')