| Message ID | 20201202215747.2719507-2-crosa@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Acceptance tests: bump Fedora to 32 | expand |
On 12/2/20 10:57 PM, Cleber Rosa wrote: > Currently in use Fedora 31 has been moved out of the standard download > locations that are supported by the functionality provided by > avocado.utils.vmimage. So right now, the boot_linux.py tests will get > canceled by not being able to find those specific images. > > Ideally, this would be bumped to version 33. But, I've found issues > with the aarch64 images, with various systemd services failing to > start. So to keep all archs consistent, I've settled on 32. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/acceptance/boot_linux.py | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py > index 1da4a53d6a..0824de008e 100644 > --- a/tests/acceptance/boot_linux.py > +++ b/tests/acceptance/boot_linux.py > @@ -42,13 +42,13 @@ class BootLinuxBase(Test): > vmimage.QEMU_IMG = qemu_img > > self.log.info('Downloading/preparing boot image') > - # Fedora 31 only provides ppc64le images > + # Fedora 32 only provides ppc64le images > image_arch = self.arch > if image_arch == 'ppc64': > image_arch = 'ppc64le' > try: > boot = vmimage.get( > - 'fedora', arch=image_arch, version='31', > + 'fedora', arch=image_arch, version='32', I already expressed my view on this (latest QEMU should be able to use at least f31 - which was tested - and eventually f33 - which is coverage extension). I'm not going to vouch this change. If other maintainers are happy with it, I don't mind this gets merged. BTW I don't see why this is urgent for 5.2. Phil.
On Thu, Dec 03, 2020 at 10:37:01AM +0100, Philippe Mathieu-Daudé wrote: > On 12/2/20 10:57 PM, Cleber Rosa wrote: > > Currently in use Fedora 31 has been moved out of the standard download > > locations that are supported by the functionality provided by > > avocado.utils.vmimage. So right now, the boot_linux.py tests will get > > canceled by not being able to find those specific images. > > > > Ideally, this would be bumped to version 33. But, I've found issues > > with the aarch64 images, with various systemd services failing to > > start. So to keep all archs consistent, I've settled on 32. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > --- > > tests/acceptance/boot_linux.py | 12 ++++++------ > > 1 file changed, 6 insertions(+), 6 deletions(-) > > > > diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py > > index 1da4a53d6a..0824de008e 100644 > > --- a/tests/acceptance/boot_linux.py > > +++ b/tests/acceptance/boot_linux.py > > @@ -42,13 +42,13 @@ class BootLinuxBase(Test): > > vmimage.QEMU_IMG = qemu_img > > > > self.log.info('Downloading/preparing boot image') > > - # Fedora 31 only provides ppc64le images > > + # Fedora 32 only provides ppc64le images > > image_arch = self.arch > > if image_arch == 'ppc64': > > image_arch = 'ppc64le' > > try: > > boot = vmimage.get( > > - 'fedora', arch=image_arch, version='31', > > + 'fedora', arch=image_arch, version='32', > > I already expressed my view on this (latest QEMU should be > able to use at least f31 - which was tested - and eventually > f33 - which is coverage extension). I'm not going to vouch > this change. If other maintainers are happy with it, I don't > mind this gets merged. > > BTW I don't see why this is urgent for 5.2. > > Phil. > Hi Phil, Are you implying that, in your opinion, QEMU (say 5.2) should somehow provide compatibility with Fedora 31 because it was used during the entire cycle? I sympathize with that, but, QEMU is not really advertising compatibility support with specific Linux Distros, is it? And, assuming that the issues I found on the Fedora 33 aarch64 image can not be worked around, would you suggest not moving to 32? I mean, I don't see a reason why QEMU shouldn't be able to use at least Fedora 32, which is a currently *active* version (different from 31). Thanks, - Cleber.
On Thu, Dec 03, 2020 at 11:50:33AM -0500, Cleber Rosa wrote: > On Thu, Dec 03, 2020 at 10:37:01AM +0100, Philippe Mathieu-Daudé wrote: > > On 12/2/20 10:57 PM, Cleber Rosa wrote: > > > Currently in use Fedora 31 has been moved out of the standard download > > > locations that are supported by the functionality provided by > > > avocado.utils.vmimage. So right now, the boot_linux.py tests will get > > > canceled by not being able to find those specific images. > > > > > > Ideally, this would be bumped to version 33. But, I've found issues > > > with the aarch64 images, with various systemd services failing to > > > start. So to keep all archs consistent, I've settled on 32. > > > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > > --- > > > tests/acceptance/boot_linux.py | 12 ++++++------ > > > 1 file changed, 6 insertions(+), 6 deletions(-) > > > > > > diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py > > > index 1da4a53d6a..0824de008e 100644 > > > --- a/tests/acceptance/boot_linux.py > > > +++ b/tests/acceptance/boot_linux.py > > > @@ -42,13 +42,13 @@ class BootLinuxBase(Test): > > > vmimage.QEMU_IMG = qemu_img > > > > > > self.log.info('Downloading/preparing boot image') > > > - # Fedora 31 only provides ppc64le images > > > + # Fedora 32 only provides ppc64le images > > > image_arch = self.arch > > > if image_arch == 'ppc64': > > > image_arch = 'ppc64le' > > > try: > > > boot = vmimage.get( > > > - 'fedora', arch=image_arch, version='31', > > > + 'fedora', arch=image_arch, version='32', > > > > I already expressed my view on this (latest QEMU should be > > able to use at least f31 - which was tested - and eventually > > f33 - which is coverage extension). I'm not going to vouch > > this change. If other maintainers are happy with it, I don't > > mind this gets merged. > > > > BTW I don't see why this is urgent for 5.2. > > > > Phil. > > > > Hi Phil, > > Are you implying that, in your opinion, QEMU (say 5.2) should somehow > provide compatibility with Fedora 31 because it was used during the > entire cycle? I sympathize with that, but, QEMU is not really > advertising compatibility support with specific Linux Distros, is it? > > And, assuming that the issues I found on the Fedora 33 aarch64 image > can not be worked around, would you suggest not moving to 32? I mean, > I don't see a reason why QEMU shouldn't be able to use at least Fedora > 32, which is a currently *active* version (different from 31). I think the problem with the Fedora acceptance is that we'll be constantly chasing a moving target. Every URL we pick will go away 6-12 months later. IOW, while the acceptance test pass today, in 6 months time they'll be failing. IOW, switching to F32 doesn't solve the root cause, it just pushs the problem down the road for 6 months until F32 is EOL and hits the same URL change problem. One way to avoid this is to *not* actually test a current Fedora. Instead intentionally point at an EOL Fedora release whose URL has already moved to the archive site which is long term stable. Regards, Daniel
On Thu, Dec 03, 2020 at 05:02:33PM +0000, Daniel P. Berrangé wrote: > I think the problem with the Fedora acceptance is that we'll be constantly > chasing a moving target. Every URL we pick will go away 6-12 months later. > IOW, while the acceptance test pass today, in 6 months time they'll be > failing. IOW, switching to F32 doesn't solve the root cause, it just > pushs the problem down the road for 6 months until F32 is EOL and hits > the same URL change problem. > Just FIY, the tests will not FAIL when the images are removed from the official locations. This is what happens Today: JOB ID : e85527a9d75023070f15b833eac0f91f803afc83 JOB LOG : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/job.log (1/1) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm: CANCEL: Failed to download/prepare boot image (0.33 s) RESULTS : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1 JOB HTML : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/results.html JOB TIME : 0.76 s And *normally*, we'd have 12+ months between updates, that is from Fedora 31 -> 33, 33 -> 35, etc. > One way to avoid this is to *not* actually test a current Fedora. > Instead intentionally point at an EOL Fedora release whose URL has > already moved to the archive site which is long term stable. > So the tradeoff is, a patch every 6 or 12 months, versus using a more modern guest. With other tests, such as virtiofs_submounts.py, already depending on the same decision (to avoid multiple guest images downloaded), I think this tradeoff decision needs more visibility. IMO, the cost of such a simple patch every 6 or 12 months is very low provided we'll benefit from the newer guests. Cheers, - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Thu, Dec 03, 2020 at 12:29:59PM -0500, Cleber Rosa wrote: > On Thu, Dec 03, 2020 at 05:02:33PM +0000, Daniel P. Berrangé wrote: > > I think the problem with the Fedora acceptance is that we'll be constantly > > chasing a moving target. Every URL we pick will go away 6-12 months later. > > IOW, while the acceptance test pass today, in 6 months time they'll be > > failing. IOW, switching to F32 doesn't solve the root cause, it just > > pushs the problem down the road for 6 months until F32 is EOL and hits > > the same URL change problem. > > > > Just FIY, the tests will not FAIL when the images are removed from the > official locations. This is what happens Today: > > JOB ID : e85527a9d75023070f15b833eac0f91f803afc83 > JOB LOG : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/job.log > (1/1) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm: CANCEL: Failed to download/prepare boot image (0.33 s) > RESULTS : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1 > JOB HTML : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/results.html > JOB TIME : 0.76 s > > And *normally*, we'd have 12+ months between updates, that is from > Fedora 31 -> 33, 33 -> 35, etc. > > > One way to avoid this is to *not* actually test a current Fedora. > > Instead intentionally point at an EOL Fedora release whose URL has > > already moved to the archive site which is long term stable. > > > > So the tradeoff is, a patch every 6 or 12 months, versus using a more > modern guest. With other tests, such as virtiofs_submounts.py, > already depending on the same decision (to avoid multiple guest images > downloaded), I think this tradeoff decision needs more visibility. > > IMO, the cost of such a simple patch every 6 or 12 months is very low > provided we'll benefit from the newer guests. I don't think changing the OS version typically changes the level of coverage in aggregate. The new OS may exercise new code paths, but it will stop exercising old code paths, so most of the time it'll be net-zero. The ideal would be to test a representative selection of both old and new versions but capacity limits. The only time there's probably a notable difference is if we need to access to a new type of device that the old OS doesn't have, but that's relatively rare. Regards, Daniel
On Thu, Dec 3, 2020 at 2:30 PM Cleber Rosa <crosa@redhat.com> wrote: > > On Thu, Dec 03, 2020 at 05:02:33PM +0000, Daniel P. Berrangé wrote: > > I think the problem with the Fedora acceptance is that we'll be constantly > > chasing a moving target. Every URL we pick will go away 6-12 months later. > > IOW, while the acceptance test pass today, in 6 months time they'll be > > failing. IOW, switching to F32 doesn't solve the root cause, it just > > pushs the problem down the road for 6 months until F32 is EOL and hits > > the same URL change problem. > > > > Just FIY, the tests will not FAIL when the images are removed from the > official locations. This is what happens Today: > > JOB ID : e85527a9d75023070f15b833eac0f91f803afc83 > JOB LOG : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/job.log > (1/1) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm: CANCEL: Failed to download/prepare boot image (0.33 s) > RESULTS : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1 > JOB HTML : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/results.html > JOB TIME : 0.76 s > In the CI, enabling the GitLab cache will help, but will also hide the URL change problem when it happens. It is also true if the person ran the test and it successfully downloaded the image once. For people running the test for the first time, it will look like your output.
On 12/3/20 6:02 PM, Daniel P. Berrangé wrote: > On Thu, Dec 03, 2020 at 11:50:33AM -0500, Cleber Rosa wrote: >> On Thu, Dec 03, 2020 at 10:37:01AM +0100, Philippe Mathieu-Daudé wrote: >>> On 12/2/20 10:57 PM, Cleber Rosa wrote: >>>> Currently in use Fedora 31 has been moved out of the standard download >>>> locations that are supported by the functionality provided by >>>> avocado.utils.vmimage. So right now, the boot_linux.py tests will get >>>> canceled by not being able to find those specific images. >>>> >>>> Ideally, this would be bumped to version 33. But, I've found issues >>>> with the aarch64 images, with various systemd services failing to >>>> start. So to keep all archs consistent, I've settled on 32. >>>> >>>> Signed-off-by: Cleber Rosa <crosa@redhat.com> >>>> --- >>>> tests/acceptance/boot_linux.py | 12 ++++++------ >>>> 1 file changed, 6 insertions(+), 6 deletions(-) >>>> >>>> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py >>>> index 1da4a53d6a..0824de008e 100644 >>>> --- a/tests/acceptance/boot_linux.py >>>> +++ b/tests/acceptance/boot_linux.py >>>> @@ -42,13 +42,13 @@ class BootLinuxBase(Test): >>>> vmimage.QEMU_IMG = qemu_img >>>> >>>> self.log.info('Downloading/preparing boot image') >>>> - # Fedora 31 only provides ppc64le images >>>> + # Fedora 32 only provides ppc64le images >>>> image_arch = self.arch >>>> if image_arch == 'ppc64': >>>> image_arch = 'ppc64le' >>>> try: >>>> boot = vmimage.get( >>>> - 'fedora', arch=image_arch, version='31', >>>> + 'fedora', arch=image_arch, version='32', >>> >>> I already expressed my view on this (latest QEMU should be >>> able to use at least f31 - which was tested - and eventually >>> f33 - which is coverage extension). I'm not going to vouch >>> this change. If other maintainers are happy with it, I don't >>> mind this gets merged. >>> >>> BTW I don't see why this is urgent for 5.2. >>> >>> Phil. >>> >> >> Hi Phil, >> >> Are you implying that, in your opinion, QEMU (say 5.2) should somehow >> provide compatibility with Fedora 31 because it was used during the >> entire cycle? I sympathize with that, but, QEMU is not really >> advertising compatibility support with specific Linux Distros, is it? What I don't understand is why you remove F31 and not simply add a F32 test. Why should I stop testing F31 if I have it cached? >> >> And, assuming that the issues I found on the Fedora 33 aarch64 image >> can not be worked around, would you suggest not moving to 32? I mean, >> I don't see a reason why QEMU shouldn't be able to use at least Fedora >> 32, which is a currently *active* version (different from 31). > > I think the problem with the Fedora acceptance is that we'll be constantly > chasing a moving target. Every URL we pick will go away 6-12 months later. > IOW, while the acceptance test pass today, in 6 months time they'll be > failing. IOW, switching to F32 doesn't solve the root cause, it just > pushs the problem down the road for 6 months until F32 is EOL and hits > the same URL change problem. > > One way to avoid this is to *not* actually test a current Fedora. > Instead intentionally point at an EOL Fedora release whose URL has > already moved to the archive site which is long term stable. I agree with Daniel, 'acceptance' test must not fail. Having an archived image allow us to not rely on external disappearing storage. > > Regards, > Daniel >
On Fri, Dec 04, 2020 at 03:08:52PM +0100, Philippe Mathieu-Daudé wrote: > On 12/3/20 6:02 PM, Daniel P. Berrangé wrote: > > On Thu, Dec 03, 2020 at 11:50:33AM -0500, Cleber Rosa wrote: > >> On Thu, Dec 03, 2020 at 10:37:01AM +0100, Philippe Mathieu-Daudé wrote: > >>> On 12/2/20 10:57 PM, Cleber Rosa wrote: > >>>> Currently in use Fedora 31 has been moved out of the standard download > >>>> locations that are supported by the functionality provided by > >>>> avocado.utils.vmimage. So right now, the boot_linux.py tests will get > >>>> canceled by not being able to find those specific images. > >>>> > >>>> Ideally, this would be bumped to version 33. But, I've found issues > >>>> with the aarch64 images, with various systemd services failing to > >>>> start. So to keep all archs consistent, I've settled on 32. > >>>> > >>>> Signed-off-by: Cleber Rosa <crosa@redhat.com> > >>>> --- > >>>> tests/acceptance/boot_linux.py | 12 ++++++------ > >>>> 1 file changed, 6 insertions(+), 6 deletions(-) > >>>> > >>>> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py > >>>> index 1da4a53d6a..0824de008e 100644 > >>>> --- a/tests/acceptance/boot_linux.py > >>>> +++ b/tests/acceptance/boot_linux.py > >>>> @@ -42,13 +42,13 @@ class BootLinuxBase(Test): > >>>> vmimage.QEMU_IMG = qemu_img > >>>> > >>>> self.log.info('Downloading/preparing boot image') > >>>> - # Fedora 31 only provides ppc64le images > >>>> + # Fedora 32 only provides ppc64le images > >>>> image_arch = self.arch > >>>> if image_arch == 'ppc64': > >>>> image_arch = 'ppc64le' > >>>> try: > >>>> boot = vmimage.get( > >>>> - 'fedora', arch=image_arch, version='31', > >>>> + 'fedora', arch=image_arch, version='32', > >>> > >>> I already expressed my view on this (latest QEMU should be > >>> able to use at least f31 - which was tested - and eventually > >>> f33 - which is coverage extension). I'm not going to vouch > >>> this change. If other maintainers are happy with it, I don't > >>> mind this gets merged. > >>> > >>> BTW I don't see why this is urgent for 5.2. > >>> > >>> Phil. > >>> > >> > >> Hi Phil, > >> > >> Are you implying that, in your opinion, QEMU (say 5.2) should somehow > >> provide compatibility with Fedora 31 because it was used during the > >> entire cycle? I sympathize with that, but, QEMU is not really > >> advertising compatibility support with specific Linux Distros, is it? > > What I don't understand is why you remove F31 and not simply add a F32 > test. Why should I stop testing F31 if I have it cached? Well we can't let the list of OS increase unbounded. IMHO there's not enough changes between each Fedora release to justify keeping them all. Same for any other short life distro. If we're trying to get a variety of coverage, I'd suggest we focus on the long life distros for old stuff. eg RHEL-5, RHEL-6, RHEL-7, etc all have large enough differences in their virtual hardware to justify having a test for each major version. We could then just have 1 recent-ish Fedora as a way to get coverage of the cutting edge. Regards, Daniel
diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py index 1da4a53d6a..0824de008e 100644 --- a/tests/acceptance/boot_linux.py +++ b/tests/acceptance/boot_linux.py @@ -42,13 +42,13 @@ class BootLinuxBase(Test): vmimage.QEMU_IMG = qemu_img self.log.info('Downloading/preparing boot image') - # Fedora 31 only provides ppc64le images + # Fedora 32 only provides ppc64le images image_arch = self.arch if image_arch == 'ppc64': image_arch = 'ppc64le' try: boot = vmimage.get( - 'fedora', arch=image_arch, version='31', + 'fedora', arch=image_arch, version='32', checksum=self.chksum, algorithm='sha256', cache_dir=self.cache_dirs[0], @@ -111,7 +111,7 @@ class BootLinuxX8664(BootLinux): :avocado: tags=arch:x86_64 """ - chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + chksum = '423a4ce32fa32c50c11e3d3ff392db97a762533b81bef9d00599de518a7469c8' def test_pc_i440fx_tcg(self): """ @@ -161,7 +161,7 @@ class BootLinuxAarch64(BootLinux): :avocado: tags=machine:gic-version=2 """ - chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + chksum = 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967' def add_common_args(self): self.vm.add_args('-bios', @@ -217,7 +217,7 @@ class BootLinuxPPC64(BootLinux): :avocado: tags=arch:ppc64 """ - chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + chksum = 'dd989a078d641713c55720ba3e4320b204ade6954e2bfe4570c8058dc36e2e5d' def test_pseries_tcg(self): """ @@ -235,7 +235,7 @@ class BootLinuxS390X(BootLinux): :avocado: tags=arch:s390x """ - chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + chksum = '93e49b98fa016797a6864a95cbb7beaec86ffd61dbcd42a951158ae732dae1ec' @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_s390_ccw_virtio_tcg(self):
Currently in use Fedora 31 has been moved out of the standard download locations that are supported by the functionality provided by avocado.utils.vmimage. So right now, the boot_linux.py tests will get canceled by not being able to find those specific images. Ideally, this would be bumped to version 33. But, I've found issues with the aarch64 images, with various systemd services failing to start. So to keep all archs consistent, I've settled on 32. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- tests/acceptance/boot_linux.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)