| Message ID | 20200129212345.20547-5-philmd@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | tests/acceptance/virtio_seg_max_adjust: Restrict it to Linux/X86 | expand |
On Wed, 29 Jan 2020 22:23:20 +0100 Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > Introduce the 'version-min' tag. > Tests can set it to the minimum version of QEMU they require. > If QEMU is older, the tests will be cancelled (skipped): > > $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py > (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > ... > > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> > --- > tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) Hm... this might cause headaches if someone tries to backport tests on distro versions; but we probably won't be able to test for the feature in many cases anyway?
On 1/30/20 1:26 PM, Cornelia Huck wrote: > On Wed, 29 Jan 2020 22:23:20 +0100 > Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > >> Introduce the 'version-min' tag. >> Tests can set it to the minimum version of QEMU they require. >> If QEMU is older, the tests will be cancelled (skipped): >> >> $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py >> (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >> (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >> (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >> ... >> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> >> --- >> tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ >> 1 file changed, 18 insertions(+) > > Hm... this might cause headaches if someone tries to backport tests on > distro versions; but we probably won't be able to test for the feature > in many cases anyway? > Are you saying a distro version might not match upstream version when a feature is introduced? Isn't it the point of versioning? If you are thinking of some case you point me at it so I can improve this patch? Thanks, Phil.
On Thu, 30 Jan 2020 14:28:19 +0100 Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > On 1/30/20 1:26 PM, Cornelia Huck wrote: > > On Wed, 29 Jan 2020 22:23:20 +0100 > > Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > > > >> Introduce the 'version-min' tag. > >> Tests can set it to the minimum version of QEMU they require. > >> If QEMU is older, the tests will be cancelled (skipped): > >> > >> $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py > >> (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > >> (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > >> (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > >> ... > >> > >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> > >> --- > >> tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ > >> 1 file changed, 18 insertions(+) > > > > Hm... this might cause headaches if someone tries to backport tests on > > distro versions; but we probably won't be able to test for the feature > > in many cases anyway? > > > > Are you saying a distro version might not match upstream version when a > feature is introduced? Isn't it the point of versioning? If you are > thinking of some case you point me at it so I can improve this patch? Something like the following: - a feature together with a test is introduced in QEMU version n - a distro is using m (where m < n), but wants the new feature and backports it If that distro now wants to include the acceptance test as well (to run in their internal testing), they need to adapt the version check as well, or the test simply won't run. Depending on how this is reported, it might not be easily noticed. If the test depends on another feature (but the feature does not), it might get a bit more complicated. It's probably not a big problem, but I wanted to point it out.
Cc'ing Markus On 1/30/20 6:08 PM, Cornelia Huck wrote: > On Thu, 30 Jan 2020 14:28:19 +0100 > Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > >> On 1/30/20 1:26 PM, Cornelia Huck wrote: >>> On Wed, 29 Jan 2020 22:23:20 +0100 >>> Philippe Mathieu-Daudé <philmd@redhat.com> wrote: >>> >>>> Introduce the 'version-min' tag. >>>> Tests can set it to the minimum version of QEMU they require. >>>> If QEMU is older, the tests will be cancelled (skipped): >>>> >>>> $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py >>>> (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >>>> (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >>>> (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) >>>> ... >>>> >>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> >>>> --- >>>> tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ >>>> 1 file changed, 18 insertions(+) >>> >>> Hm... this might cause headaches if someone tries to backport tests on >>> distro versions; but we probably won't be able to test for the feature >>> in many cases anyway? >>> >> >> Are you saying a distro version might not match upstream version when a >> feature is introduced? Isn't it the point of versioning? If you are >> thinking of some case you point me at it so I can improve this patch? > > Something like the following: > > - a feature together with a test is introduced in QEMU version n > - a distro is using m (where m < n), but wants the new feature and > backports it > > If that distro now wants to include the acceptance test as well (to run > in their internal testing), they need to adapt the version check as > well, or the test simply won't run. Depending on how this is reported, > it might not be easily noticed. If the test depends on another feature > (but the feature does not), it might get a bit more complicated. > > It's probably not a big problem, but I wanted to point it out. OK now I understand better. I wrote that patch having QAPI versioned features in mind, which are versioned/documented. For example: ## # @qom-list-types: # ... # # Since: 1.1 Now it is true that we add a new feature to a device, we do not document that in QAPI, and I don't see how we can test it via QMP.
On Thu, Jan 30, 2020 at 06:18:22PM +0100, Philippe Mathieu-Daudé wrote: > Cc'ing Markus > > On 1/30/20 6:08 PM, Cornelia Huck wrote: > > On Thu, 30 Jan 2020 14:28:19 +0100 > > Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > > > > > On 1/30/20 1:26 PM, Cornelia Huck wrote: > > > > On Wed, 29 Jan 2020 22:23:20 +0100 > > > > Philippe Mathieu-Daudé <philmd@redhat.com> wrote: > > > > > Introduce the 'version-min' tag. > > > > > Tests can set it to the minimum version of QEMU they require. > > > > > If QEMU is older, the tests will be cancelled (skipped): > > > > > > > > > > $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py > > > > > (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > > > > > (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > > > > > (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) > > > > > ... > > > > > > > > > > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> > > > > > --- > > > > > tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ > > > > > 1 file changed, 18 insertions(+) > > > > > > > > Hm... this might cause headaches if someone tries to backport tests on > > > > distro versions; but we probably won't be able to test for the feature > > > > in many cases anyway? > > > > > > Are you saying a distro version might not match upstream version when a > > > feature is introduced? Isn't it the point of versioning? If you are > > > thinking of some case you point me at it so I can improve this patch? > > > > Something like the following: > > > > - a feature together with a test is introduced in QEMU version n > > - a distro is using m (where m < n), but wants the new feature and > > backports it > > > > If that distro now wants to include the acceptance test as well (to run > > in their internal testing), they need to adapt the version check as > > well, or the test simply won't run. Depending on how this is reported, > > it might not be easily noticed. If the test depends on another feature > > (but the feature does not), it might get a bit more complicated. > > > > It's probably not a big problem, but I wanted to point it out. > > OK now I understand better. > > I wrote that patch having QAPI versioned features in mind, which are > versioned/documented. > For example: > > ## > # @qom-list-types: > # > ... > # > # Since: 1.1 > > Now it is true that we add a new feature to a device, we do not document > that in QAPI, and I don't see how we can test it via QMP. The features are documented as introduced in specific QEMU versions to help humans, but the best way to test for their existence is to check the QAPI schema.
diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py index 6618ea67c1..67c75822d5 100644 --- a/tests/acceptance/avocado_qemu/__init__.py +++ b/tests/acceptance/avocado_qemu/__init__.py @@ -19,6 +19,7 @@ import avocado SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..') sys.path.append(os.path.join(SRC_ROOT_DIR, 'python')) +from qemu.binutils import binary_get_version from qemu.machine import QEMUMachine def is_readable_executable_file(path): @@ -111,6 +112,7 @@ class Test(avocado.Test): def setUp(self): self._vms = {} + logger = logging.getLogger('core') self.arch = self.params.get('arch', default=self._get_unique_tag_val('arch')) @@ -118,12 +120,28 @@ class Test(avocado.Test): self.machine = self.params.get('machine', default=self._get_unique_tag_val('machine')) + # Verify qemu_bin default_qemu_bin = pick_default_qemu_bin(arch=self.arch) self.qemu_bin = self.params.get('qemu_bin', default=default_qemu_bin) if self.qemu_bin is None: self.cancel("No QEMU binary defined or found in the source tree") + # Verify version + min_version = self._get_unique_tag_val('version-min') + logger.debug('version-min: {}'.format(min_version)) + if min_version: + min_v = [int(n) for n in min_version.split('.')] + if len(min_v) < 3: + min_v.append(0) + logger.debug(min_v) + bin_v = binary_get_version(self.qemu_bin) + bin_v = (bin_v['major'], bin_v['minor'], bin_v['micro']) + logger.debug(bin_v) + fmt = "Test expects version '{}' but QEMU binary is '{}.{}.{}'" + if bin_v[0] < min_v[0] or bin_v[1] < min_v[1] or bin_v[2] < min_v[2]: + self.cancel(fmt.format(min_version, bin_v[0], bin_v[1], bin_v[2])) + def _new_vm(self, *args): vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp()) if args:
Introduce the 'version-min' tag. Tests can set it to the minimum version of QEMU they require. If QEMU is older, the tests will be cancelled (skipped): $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s) ... Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> --- tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)