| Message ID | 20210127010946.17370-1-dbuono@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tests/acceptance/boot_linux: Switch to Fedora 32 | expand |
Hi, On 1/26/21 10:09 PM, Daniele Buono wrote: > Local acceptance tests run with "make check-acceptance" are now > showing some cases canceled like the following: > > (01/39) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: CANCEL: Failed to download/prepare boot image (0.25 s) > > Turns out, every full-vm test in boot_linux.py is trying to use a > Fedora 31 cloud image and is failing, with Avocado refusing to download > it, presumably because Fedora 31 is EOL. > > This patch moves to Fedora 32, which is still supported. And seem to > work fine While ago it was discussed about updating the Fedora version which, in my opinion, ended up without a conclusion. Please see the complete thread in: https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the discussion. Thanks! - Wainer > > The script has the image checksums hardcoded. I downloaded and verified > the checksums with the Fedora 32 GPG key, but I would feel more > confident if someone else wants to verify it too. > > The checksum files are here: > https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/ppc64le/images/Fedora-Cloud-32-1.6-ppc64le-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/s390x/images/Fedora-Cloud-32-1.6-s390x-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/aarch64/images/Fedora-Cloud-32-1.6-aarch64-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM > and the GPG keys are available here: > https://getfedora.org/en/security/ > > NOTE: I tried moving to Fedora 33, but the aarch64 VM cannot boot > properly. May be worth investigating but I have no experience with ARM > so I'll leave that to someone else, if interested. > > Daniele Buono (1): > tests/acceptance/boot_linux: Switch to Fedora 32 > > tests/acceptance/boot_linux.py | 11 +++++------ > 1 file changed, 5 insertions(+), 6 deletions(-) >
On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: > Hi, > > On 1/26/21 10:09 PM, Daniele Buono wrote: >> Local acceptance tests run with "make check-acceptance" are now >> showing some cases canceled like the following: >> >> (01/39) >> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: >> CANCEL: Failed to download/prepare boot image (0.25 s) >> >> Turns out, every full-vm test in boot_linux.py is trying to use a >> Fedora 31 cloud image and is failing, with Avocado refusing to download >> it, presumably because Fedora 31 is EOL. >> >> This patch moves to Fedora 32, which is still supported. And seem to >> work fine > > While ago it was discussed about updating the Fedora version which, in > my opinion, ended up without a conclusion. Please see the complete > thread in: > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html Oops, didn't notice the previous thread. Apologies for the duplicate! > > I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the > discussion. > > Thanks! > > - Wainer > >> >> The script has the image checksums hardcoded. I downloaded and verified >> the checksums with the Fedora 32 GPG key, but I would feel more >> confident if someone else wants to verify it too. >> >> The checksum files are here: >> https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/ppc64le/images/Fedora-Cloud-32-1.6-ppc64le-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/s390x/images/Fedora-Cloud-32-1.6-s390x-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/aarch64/images/Fedora-Cloud-32-1.6-aarch64-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM >> >> and the GPG keys are available here: >> https://getfedora.org/en/security/ >> >> NOTE: I tried moving to Fedora 33, but the aarch64 VM cannot boot >> properly. May be worth investigating but I have no experience with ARM >> so I'll leave that to someone else, if interested. >> >> Daniele Buono (1): >> tests/acceptance/boot_linux: Switch to Fedora 32 >> >> tests/acceptance/boot_linux.py | 11 +++++------ >> 1 file changed, 5 insertions(+), 6 deletions(-) >> >
Hi Wainer, On 1/28/21 11:06 PM, Daniele Buono wrote: > On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: >> Hi, >> >> On 1/26/21 10:09 PM, Daniele Buono wrote: >>> Local acceptance tests run with "make check-acceptance" are now >>> showing some cases canceled like the following: >>> >>> (01/39) >>> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: >>> CANCEL: Failed to download/prepare boot image (0.25 s) >>> >>> Turns out, every full-vm test in boot_linux.py is trying to use a >>> Fedora 31 cloud image and is failing, with Avocado refusing to download >>> it, presumably because Fedora 31 is EOL. >>> >>> This patch moves to Fedora 32, which is still supported. And seem to >>> work fine >> >> While ago it was discussed about updating the Fedora version which, in >> my opinion, ended up without a conclusion. Please see the complete >> thread in: >> >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html > > Oops, didn't notice the previous thread. Apologies for the duplicate! > >> >> I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the >> discussion. The first question I'd like to figure out is how/where can we archive the artifacts being tested by the project. As we maintain stable tags, I'm more worried about regressions affecting LTS use rather than recently added features which get more coverage and activity. Is this too conservative for acceptance testing? Regards, Phil.
On Fri, Feb 05, 2021 at 05:54:24PM +0100, Philippe Mathieu-Daudé wrote: > Hi Wainer, > > On 1/28/21 11:06 PM, Daniele Buono wrote: > > On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: > >> Hi, > >> > >> On 1/26/21 10:09 PM, Daniele Buono wrote: > >>> Local acceptance tests run with "make check-acceptance" are now > >>> showing some cases canceled like the following: > >>> > >>> (01/39) > >>> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: > >>> CANCEL: Failed to download/prepare boot image (0.25 s) > >>> > >>> Turns out, every full-vm test in boot_linux.py is trying to use a > >>> Fedora 31 cloud image and is failing, with Avocado refusing to download > >>> it, presumably because Fedora 31 is EOL. > >>> > >>> This patch moves to Fedora 32, which is still supported. And seem to > >>> work fine > >> > >> While ago it was discussed about updating the Fedora version which, in > >> my opinion, ended up without a conclusion. Please see the complete > >> thread in: > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html > > > > Oops, didn't notice the previous thread. Apologies for the duplicate! > > > >> > >> I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the > >> discussion. > > The first question I'd like to figure out is how/where can we archive > the artifacts being tested by the project. As we maintain stable tags, > I'm more worried about regressions affecting LTS use rather than > recently added features which get more coverage and activity. > Is this too conservative for acceptance testing? I think there's multiple issues to worry about, one short term, the others long term. The most immediate issue is that we are pointing to an EOL Fedora release with broken download URL. While we could update to F 32, that doesn't solve the problem. It will burn us again in just a few months when F34 comes out making F32 EOL. We need a long term solution. The broader long term question is what our goals are for the acceptance tests. I think the primary goal is detect regressions in QEMU where we break something which used to work. To achieve this we don't need to be chasing distro releases. It is fine for us to be testing an EOL distro. The fact that it is EOL doesn't invalidate the test behaviour. The only problem switch EOL distros is the URL breakge. That is trivially solved in the Fedora case by downloading from the archive.fedoraproject.org server instead of main server. Problem solved forever. The more important long term question is whether the selection of distros we are testing gives us coverage which exercises all the scenarios that we care about. For example, considering virtio devices, we need - Guest OS which *only* has virtio legacy mode implemented - pc for PCI - q35 for PCI-e Proves deices work in legacy mode on both PCI and PCIe - Guest OS which has both legacy and modern mode implemented - pc for PCI - q35 for PCI-e Proves devices work in transitional mode (pc) and modern-only mode (PCI-e) So for virtio coverage, we'll need 2 guest OS, each with two scenarios as a starting point. RHEL-6 is an example of a distro that was legacy only. Anything newer does the other case. But then we need to expand the axis to non-x86 too, most especially for ppc64 and s390, which have non-PCI based options as well as PCI. Periodically a new virtio device will arrive (eg virtiofs) and that might mean we need to add another test scenario, or upgrade the more modern OS in an existing test scenario. This is reasonably infrequent though, so in general we won't need to be chasing the bleeding edge distros. The matrix can grow pretty damn fast, and we need to keep it under control. One way to deal with that is to have one test case cover multiple features. eg don't run a separate test for each of virtio-net, virtio-blk, virtio-scsi. Launch a single VM in which we can test all of them at once. Also we don't need to test all of Ubuntu, Fedora, RHEL, SUSE if all those distros are basically exercising the same set of QEMU device code paths. ie just becuase the distro is different, doesn't mean it adds value to the test coverage. We should be strongly driven by whether the distro exercises a feature that isn't already covered by an existing test. So the next question is what needs to be in our matrix ? Most downstream vendors will say it needs to be all KVM related features. QEMU as a project though also cares more broadly about many more devices for emulating older platforms. There's some balance to be struck here. On many previous occassions we've talked about classifying QEMU features into support tiers. If we had a view of what our support tiers were for each feature, this would in turn show us where to spend effort in building up testing coverage. I made an abortive start at trying to define what tiers might mean but didn't get into really classifying features https://wiki.qemu.org/Support_Tiers don't pay attention to the host/guest classifications I did there - I was mostly just playing with how to represent it. Regards, Daniel