Message ID | 20221022170350.936685-1-danielhb413@gmail.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | avocado: use sha1 for fc31 imgs to avoid first time re-download | expand |
On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: > 'make check-avocado' will download any images that aren't present in the > cache via 'get-vm-images' in tests/Makefile.include. The target that > downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado > vmimage get --distro=fedora --distro-version=31 --arch=(...)' to > download the image for each arch. Note that this command does not > support any argument to set the hash algorithm used and, based on the > avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The > sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM > in the cache. > For now, in QEMU, let's use sha1 for all Fedora 31 images. This will > immediately spares us at least one extra download for each Fedora 31 > image that we're doing in all our CI runs. > > [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 > [2] https://github.com/avocado-framework/avocado/issues/5496 Can we just ask Avocado maintainers to fix this problem on their side to allow use of a modern hash alg as a priority item. We've already had this problem in QEMU for over a year AFAICT, so doesn't seem like we need to urgently do a workaround on QEMU side, so we can get Avocado devs to commit to fixing it in the next month. With regards, Daniel
On 10/24/22 06:02, Daniel P. Berrangé wrote: > On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: >> 'make check-avocado' will download any images that aren't present in the >> cache via 'get-vm-images' in tests/Makefile.include. The target that >> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >> download the image for each arch. Note that this command does not >> support any argument to set the hash algorithm used and, based on the >> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >> sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >> in the cache. > >> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >> immediately spares us at least one extra download for each Fedora 31 >> image that we're doing in all our CI runs. >> >> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >> [2] https://github.com/avocado-framework/avocado/issues/5496 > > Can we just ask Avocado maintainers to fix this problem on their > side to allow use of a modern hash alg as a priority item. We've > already had this problem in QEMU for over a year AFAICT, so doesn't > seem like we need to urgently do a workaround on QEMU side, so we > can get Avocado devs to commit to fixing it in the next month. Sure, I think it would be better if we can get an Avocado side fix to make the current QEMU usage viable. For now I'll use this patch locally because this issue is 100% reproducible on my home machine :( apparently my internet is capable of 4k streaming but, for some reason, it doesn't manage to download 350Mb from the Fedora mirror in less that 180 seconds. 'make check-avocado' always gives me a timeout in 'test_pseries_tcg'. Thanks, Daniel > > > With regards, > Daniel
On 24/10/2022 11.02, Daniel P. Berrangé wrote: > On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: >> 'make check-avocado' will download any images that aren't present in the >> cache via 'get-vm-images' in tests/Makefile.include. The target that >> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >> download the image for each arch. Note that this command does not >> support any argument to set the hash algorithm used and, based on the >> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >> sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >> in the cache. > >> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >> immediately spares us at least one extra download for each Fedora 31 >> image that we're doing in all our CI runs. >> >> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >> [2] https://github.com/avocado-framework/avocado/issues/5496 > > Can we just ask Avocado maintainers to fix this problem on their > side to allow use of a modern hash alg as a priority item. We've > already had this problem in QEMU for over a year AFAICT, so doesn't > seem like we need to urgently do a workaround on QEMU side, so we > can get Avocado devs to commit to fixing it in the next month. Do we have such a commitment? ... The avocado version in QEMU is completely backlevel these days, it's still using version 88.1 from May 2021, i.e. there hasn't been any update since more than a year. I recently tried to bump it to a newer version on my own (since I'm still suffering from the problem that find_free_port() does not work if you don't have a local IPv6 address), but it's not that straight forward since the recent versions of avocado changed a lot of things (e.g. the new nrunner - do we want to run tests in parallel? If so it breaks a lot of the timeout settings, I think), so an update needs a lot of careful testing... So unless someone is really committing to spend a lot of time on updating Avocado in QEMU in the near future, I don't think that such a fix for the hash algorithm will happen any time soon, and thus I think we should consider to include this work-around for the time being. Thomas
On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: > On 24/10/2022 11.02, Daniel P. Berrangé wrote: > > On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: > > > 'make check-avocado' will download any images that aren't present in the > > > cache via 'get-vm-images' in tests/Makefile.include. The target that > > > downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado > > > vmimage get --distro=fedora --distro-version=31 --arch=(...)' to > > > download the image for each arch. Note that this command does not > > > support any argument to set the hash algorithm used and, based on the > > > avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The > > > sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM > > > in the cache. > > > > > For now, in QEMU, let's use sha1 for all Fedora 31 images. This will > > > immediately spares us at least one extra download for each Fedora 31 > > > image that we're doing in all our CI runs. > > > > > > [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 > > > [2] https://github.com/avocado-framework/avocado/issues/5496 > > > > Can we just ask Avocado maintainers to fix this problem on their > > side to allow use of a modern hash alg as a priority item. We've > > already had this problem in QEMU for over a year AFAICT, so doesn't > > seem like we need to urgently do a workaround on QEMU side, so we > > can get Avocado devs to commit to fixing it in the next month. > > Do we have such a commitment? ... The avocado version in QEMU is completely > backlevel these days, it's still using version 88.1 from May 2021, i.e. > there hasn't been any update since more than a year. I recently tried to > bump it to a newer version on my own (since I'm still suffering from the > problem that find_free_port() does not work if you don't have a local IPv6 > address), but it's not that straight forward since the recent versions of > avocado changed a lot of things (e.g. the new nrunner - do we want to run > tests in parallel? If so it breaks a lot of the timeout settings, I think), > so an update needs a lot of careful testing... That it is so difficult to update Avocado after barely more than 1 year is not exactly a strong vote of confidence in our continued use of Avocado long term :-( With regards, Daniel
On 10/27/22 06:01, Daniel P. Berrangé wrote: > On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: >> On 24/10/2022 11.02, Daniel P. Berrangé wrote: >>> On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: >>>> 'make check-avocado' will download any images that aren't present in the >>>> cache via 'get-vm-images' in tests/Makefile.include. The target that >>>> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >>>> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >>>> download the image for each arch. Note that this command does not >>>> support any argument to set the hash algorithm used and, based on the >>>> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >>>> sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >>>> in the cache. >>> >>>> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >>>> immediately spares us at least one extra download for each Fedora 31 >>>> image that we're doing in all our CI runs. >>>> >>>> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >>>> [2] https://github.com/avocado-framework/avocado/issues/5496 >>> >>> Can we just ask Avocado maintainers to fix this problem on their >>> side to allow use of a modern hash alg as a priority item. We've >>> already had this problem in QEMU for over a year AFAICT, so doesn't >>> seem like we need to urgently do a workaround on QEMU side, so we >>> can get Avocado devs to commit to fixing it in the next month. >> >> Do we have such a commitment? ... The avocado version in QEMU is completely >> backlevel these days, it's still using version 88.1 from May 2021, i.e. >> there hasn't been any update since more than a year. I recently tried to >> bump it to a newer version on my own (since I'm still suffering from the >> problem that find_free_port() does not work if you don't have a local IPv6 >> address), but it's not that straight forward since the recent versions of >> avocado changed a lot of things (e.g. the new nrunner - do we want to run >> tests in parallel? If so it breaks a lot of the timeout settings, I think), >> so an update needs a lot of careful testing... > > That it is so difficult to update Avocado after barely more than > 1 year is not exactly a strong vote of confidence in our continued > use of Avocado long term :-( By the way, Avocado just provided a fix for the problem this patch is trying to amend: https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846 Is there an easy way to plug upstream Avocado into QEMU? I would like to test tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem is fixed by Avocado upstream. Thanks, Daniel > > With regards, > Daniel
On 9/11/22 16:39, Daniel Henrique Barboza wrote: > On 10/27/22 06:01, Daniel P. Berrangé wrote: >> On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: >>> On 24/10/2022 11.02, Daniel P. Berrangé wrote: >>>> On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza >>>> wrote: >>>>> 'make check-avocado' will download any images that aren't present >>>>> in the >>>>> cache via 'get-vm-images' in tests/Makefile.include. The target that >>>>> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >>>>> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >>>>> download the image for each arch. Note that this command does not >>>>> support any argument to set the hash algorithm used and, based on the >>>>> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >>>>> sha1 hash is stored in a >>>>> Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >>>>> in the cache. >>>> >>>>> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >>>>> immediately spares us at least one extra download for each Fedora 31 >>>>> image that we're doing in all our CI runs. >>>>> >>>>> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >>>>> [2] https://github.com/avocado-framework/avocado/issues/5496 >>>> >>>> Can we just ask Avocado maintainers to fix this problem on their >>>> side to allow use of a modern hash alg as a priority item. We've >>>> already had this problem in QEMU for over a year AFAICT, so doesn't >>>> seem like we need to urgently do a workaround on QEMU side, so we >>>> can get Avocado devs to commit to fixing it in the next month. >>> >>> Do we have such a commitment? ... The avocado version in QEMU is >>> completely >>> backlevel these days, it's still using version 88.1 from May 2021, i.e. >>> there hasn't been any update since more than a year. I recently tried to >>> bump it to a newer version on my own (since I'm still suffering from the >>> problem that find_free_port() does not work if you don't have a local >>> IPv6 >>> address), but it's not that straight forward since the recent >>> versions of >>> avocado changed a lot of things (e.g. the new nrunner - do we want to >>> run >>> tests in parallel? If so it breaks a lot of the timeout settings, I >>> think), >>> so an update needs a lot of careful testing... >> >> That it is so difficult to update Avocado after barely more than >> 1 year is not exactly a strong vote of confidence in our continued >> use of Avocado long term :-( > > > By the way, Avocado just provided a fix for the problem this patch is > trying > to amend: > > https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846 Thanks Jan! > Is there an easy way to plug upstream Avocado into QEMU? I would like to > test > tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if > the problem > is fixed by Avocado upstream. See https://lore.kernel.org/qemu-devel/20200403172919.24621-9-philmd@redhat.com/ For your case: -- >8 -- diff --git a/tests/requirements.txt b/tests/requirements.txt index 0ba561b6bd..e17bc3972c 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -4,3 +4,3 @@ # Note that qemu.git/python/ is always implicitly installed. -avocado-framework==88.1 +-e git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework pycdlib==1.11.0 --- Regards, Phil.
On 11/10/22 00:26, Philippe Mathieu-Daudé wrote: > On 9/11/22 16:39, Daniel Henrique Barboza wrote: >> On 10/27/22 06:01, Daniel P. Berrangé wrote: >>> On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: >>>> On 24/10/2022 11.02, Daniel P. Berrangé wrote: >>>>> On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza >>>>> wrote: >>>>>> 'make check-avocado' will download any images that aren't present >>>>>> in the >>>>>> cache via 'get-vm-images' in tests/Makefile.include. The target that >>>>>> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >>>>>> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >>>>>> download the image for each arch. Note that this command does not >>>>>> support any argument to set the hash algorithm used and, based on the >>>>>> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >>>>>> sha1 hash is stored in a >>>>>> Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >>>>>> in the cache. >>>>> >>>>>> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >>>>>> immediately spares us at least one extra download for each Fedora 31 >>>>>> image that we're doing in all our CI runs. >>>>>> >>>>>> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >>>>>> [2] https://github.com/avocado-framework/avocado/issues/5496 >>>>> >>>>> Can we just ask Avocado maintainers to fix this problem on their >>>>> side to allow use of a modern hash alg as a priority item. We've >>>>> already had this problem in QEMU for over a year AFAICT, so doesn't >>>>> seem like we need to urgently do a workaround on QEMU side, so we >>>>> can get Avocado devs to commit to fixing it in the next month. >>>> >>>> Do we have such a commitment? ... The avocado version in QEMU is >>>> completely >>>> backlevel these days, it's still using version 88.1 from May 2021, i.e. >>>> there hasn't been any update since more than a year. I recently >>>> tried to >>>> bump it to a newer version on my own (since I'm still suffering from >>>> the >>>> problem that find_free_port() does not work if you don't have a >>>> local IPv6 >>>> address), but it's not that straight forward since the recent >>>> versions of >>>> avocado changed a lot of things (e.g. the new nrunner - do we want >>>> to run >>>> tests in parallel? If so it breaks a lot of the timeout settings, I >>>> think), >>>> so an update needs a lot of careful testing... Hi Daniel, if the problem of migrating avocado to latest version on qemu is only in parallel run, I would suggest to disable it with `nrunner.max_parallel_tasks` [1]. Even that the differences between avocado legacy runner and nrunner is huge, the migration should be straight forward. So if you have more issues with migration to the nrunner, I would be happy to help you with that. [1] https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks - Jan >>> >>> That it is so difficult to update Avocado after barely more than >>> 1 year is not exactly a strong vote of confidence in our continued >>> use of Avocado long term :-( >> >> >> By the way, Avocado just provided a fix for the problem this patch is >> trying >> to amend: >> >> https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846 > > Thanks Jan! > >> Is there an easy way to plug upstream Avocado into QEMU? I would like >> to test >> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if >> the problem >> is fixed by Avocado upstream. > > See > https://lore.kernel.org/qemu-devel/20200403172919.24621-9-philmd@redhat.com/ > > For your case: > > -- >8 -- > diff --git a/tests/requirements.txt b/tests/requirements.txt > index 0ba561b6bd..e17bc3972c 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -4,3 +4,3 @@ > # Note that qemu.git/python/ is always implicitly installed. > -avocado-framework==88.1 > +-e > git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework > pycdlib==1.11.0 > --- > > Regards, > > Phil. >
On 11/10/22 11:57, Jan Richter wrote: > > > On 11/10/22 00:26, Philippe Mathieu-Daudé wrote: >> On 9/11/22 16:39, Daniel Henrique Barboza wrote: >>> On 10/27/22 06:01, Daniel P. Berrangé wrote: >>>> On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: >>>>> On 24/10/2022 11.02, Daniel P. Berrangé wrote: >>>>>> On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza wrote: >>>>>>> 'make check-avocado' will download any images that aren't present in the >>>>>>> cache via 'get-vm-images' in tests/Makefile.include. The target that >>>>>>> downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado >>>>>>> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >>>>>>> download the image for each arch. Note that this command does not >>>>>>> support any argument to set the hash algorithm used and, based on the >>>>>>> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The >>>>>>> sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >>>>>>> in the cache. >>>>>> >>>>>>> For now, in QEMU, let's use sha1 for all Fedora 31 images. This will >>>>>>> immediately spares us at least one extra download for each Fedora 31 >>>>>>> image that we're doing in all our CI runs. >>>>>>> >>>>>>> [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 >>>>>>> [2] https://github.com/avocado-framework/avocado/issues/5496 >>>>>> >>>>>> Can we just ask Avocado maintainers to fix this problem on their >>>>>> side to allow use of a modern hash alg as a priority item. We've >>>>>> already had this problem in QEMU for over a year AFAICT, so doesn't >>>>>> seem like we need to urgently do a workaround on QEMU side, so we >>>>>> can get Avocado devs to commit to fixing it in the next month. >>>>> >>>>> Do we have such a commitment? ... The avocado version in QEMU is completely >>>>> backlevel these days, it's still using version 88.1 from May 2021, i.e. >>>>> there hasn't been any update since more than a year. I recently tried to >>>>> bump it to a newer version on my own (since I'm still suffering from the >>>>> problem that find_free_port() does not work if you don't have a local IPv6 >>>>> address), but it's not that straight forward since the recent versions of >>>>> avocado changed a lot of things (e.g. the new nrunner - do we want to run >>>>> tests in parallel? If so it breaks a lot of the timeout settings, I think), >>>>> so an update needs a lot of careful testing... > > Hi Daniel, > > if the problem of migrating avocado to latest version on qemu is only in parallel run, I would suggest to disable it with `nrunner.max_parallel_tasks` [1]. Even that the differences between avocado legacy runner and nrunner is huge, the migration should be straight forward. So if you have more issues with migration to the nrunner, I would be happy to help you with that. > > [1] https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks Thanks Jan and Phil for the infos. I didn't manage to do a successful Avocado run with the QEMU test though. What I did, aside from the changes that Phil mentioned in tests/requirements.txt: - created a /etc/avocado/avocado.conf to store the settings - copied python/avocado.cfg from QEMU to avocado.conf - added the following in avocado.conf: [nrunner] max_parallel_tasks = 1 This allowed me to set Avocado as it would be if running with QEMU avocado, but even then I had no success. The test get stuck indefinitely at this point: (...) 2022-11-10 16:00:20,758 avocado.test INFO | Temporary dir: /var/tmp/avocado_tmp_znhvpbh0/avocado_job_ywyf7v30 2022-11-10 16:00:20,758 avocado.test INFO | 2022-11-10 16:00:20,758 avocado.test INFO | Job ID: 4bb3e2a12c05d84a0a06849ecef435d547a198a0 2022-11-10 16:00:20,758 avocado.test INFO | 2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| <Worker spawner="<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>" max_triaging=12 max_running=1 task_timeout=None> has been initialized 2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requested -> triaging 2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": requirements OK (will proceed to check dependencies) 2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": about to be spawned with "<avocado.plugins.spawners.process.ProcessSpawner object at 0x7fdad5da5840>" 2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully No error is provided. Tried to run the test for 10+ minutes and nothing happens. Hitting CTRL+C aborts the test: $ make check-avocado AVOCADO_TESTS='tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg' GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc AVOCADO Downloading avocado tests VM image for ppc64le The image was downloaded: Provider Version Architecture File fedora 31 ppc64le /home/danielhb/avocado/data/cache/by_location/d73d707673a941f5892e4bb0fa7fabaf0d41fd58/Fedora-Cloud-Base-31-1.9.ppc64le.qcow2 AVOCADO tests/avocado JOB ID : 4bb3e2a12c05d84a0a06849ecef435d547a198a0 JOB LOG : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/job.log ^C RESULTS : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/results.html JOB TIME : 396.78 s make: *** [/home/danielhb/kvm-project/qemu/tests/Makefile.include:142: check-avocado] Error 8 And the log registers the following: 2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned successfully 2022-11-10 16:06:54,071 avocado.test INFO | 2022-11-10 16:06:54,123 avocado.test INFO | Test results available in /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a 2022-11-10 16:06:55,694 avocado.sysinfo DEBUG| Not logging /proc/slabinfo (lack of permissions) 2022-11-10 16:06:55,696 avocado.sysinfo DEBUG| Not logging /proc/pci (file not found) 2022-11-10 16:06:55,712 avocado.sysinfo DEBUG| Not logging /sys/kernel/debug/sched_features (file not found) [---end of log---] I'm probably missing more settings. Daniel > > - Jan > >>>> >>>> That it is so difficult to update Avocado after barely more than >>>> 1 year is not exactly a strong vote of confidence in our continued >>>> use of Avocado long term :-( >>> >>> >>> By the way, Avocado just provided a fix for the problem this patch is trying >>> to amend: >>> >>> https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846 >> >> Thanks Jan! >> >>> Is there an easy way to plug upstream Avocado into QEMU? I would like to test >>> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see if the problem >>> is fixed by Avocado upstream. >> >> See https://lore.kernel.org/qemu-devel/20200403172919.24621-9-philmd@redhat.com/ >> >> For your case: >> >> -- >8 -- >> diff --git a/tests/requirements.txt b/tests/requirements.txt >> index 0ba561b6bd..e17bc3972c 100644 >> --- a/tests/requirements.txt >> +++ b/tests/requirements.txt >> @@ -4,3 +4,3 @@ >> # Note that qemu.git/python/ is always implicitly installed. >> -avocado-framework==88.1 >> +-e git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework >> pycdlib==1.11.0 >> --- >> >> Regards, >> >> Phil. >> >
On 11/10/22 20:29, Daniel Henrique Barboza wrote: > > > On 11/10/22 11:57, Jan Richter wrote: >> >> >> On 11/10/22 00:26, Philippe Mathieu-Daudé wrote: >>> On 9/11/22 16:39, Daniel Henrique Barboza wrote: >>>> On 10/27/22 06:01, Daniel P. Berrangé wrote: >>>>> On Thu, Oct 27, 2022 at 09:46:29AM +0200, Thomas Huth wrote: >>>>>> On 24/10/2022 11.02, Daniel P. Berrangé wrote: >>>>>>> On Sat, Oct 22, 2022 at 02:03:50PM -0300, Daniel Henrique Barboza >>>>>>> wrote: >>>>>>>> 'make check-avocado' will download any images that aren't >>>>>>>> present in the >>>>>>>> cache via 'get-vm-images' in tests/Makefile.include. The target >>>>>>>> that >>>>>>>> downloads fedora 31 images, get-vm-image-fedora-31, will use >>>>>>>> 'avocado >>>>>>>> vmimage get --distro=fedora --distro-version=31 --arch=(...)' to >>>>>>>> download the image for each arch. Note that this command does not >>>>>>>> support any argument to set the hash algorithm used and, based >>>>>>>> on the >>>>>>>> avocado source code [1], DEFAULT_HASH_ALGORITHM is set to >>>>>>>> "sha1". The >>>>>>>> sha1 hash is stored in a >>>>>>>> Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM >>>>>>>> in the cache. >>>>>>> >>>>>>>> For now, in QEMU, let's use sha1 for all Fedora 31 images. This >>>>>>>> will >>>>>>>> immediately spares us at least one extra download for each >>>>>>>> Fedora 31 >>>>>>>> image that we're doing in all our CI runs. >>>>>>>> >>>>>>>> [1] https://github.com/avocado-framework/avocado.git @ >>>>>>>> 942a5d6972906 >>>>>>>> [2] https://github.com/avocado-framework/avocado/issues/5496 >>>>>>> >>>>>>> Can we just ask Avocado maintainers to fix this problem on their >>>>>>> side to allow use of a modern hash alg as a priority item. We've >>>>>>> already had this problem in QEMU for over a year AFAICT, so doesn't >>>>>>> seem like we need to urgently do a workaround on QEMU side, so we >>>>>>> can get Avocado devs to commit to fixing it in the next month. >>>>>> >>>>>> Do we have such a commitment? ... The avocado version in QEMU is >>>>>> completely >>>>>> backlevel these days, it's still using version 88.1 from May 2021, >>>>>> i.e. >>>>>> there hasn't been any update since more than a year. I recently >>>>>> tried to >>>>>> bump it to a newer version on my own (since I'm still suffering >>>>>> from the >>>>>> problem that find_free_port() does not work if you don't have a >>>>>> local IPv6 >>>>>> address), but it's not that straight forward since the recent >>>>>> versions of >>>>>> avocado changed a lot of things (e.g. the new nrunner - do we want >>>>>> to run >>>>>> tests in parallel? If so it breaks a lot of the timeout settings, >>>>>> I think), >>>>>> so an update needs a lot of careful testing... >> >> Hi Daniel, >> >> if the problem of migrating avocado to latest version on qemu is only >> in parallel run, I would suggest to disable it with >> `nrunner.max_parallel_tasks` [1]. Even that the differences between >> avocado legacy runner and nrunner is huge, the migration should be >> straight forward. So if you have more issues with migration to the >> nrunner, I would be happy to help you with that. >> >> [1] >> https://avocado-framework.readthedocs.io/en/latest/config/index.html#nrunner-max-parallel-tasks > > > Thanks Jan and Phil for the infos. I didn't manage to do a successful > Avocado run with the QEMU test though. > > What I did, aside from the changes that Phil mentioned in > tests/requirements.txt: > > - created a /etc/avocado/avocado.conf to store the settings > - copied python/avocado.cfg from QEMU to avocado.conf > - added the following in avocado.conf: > > [nrunner] > max_parallel_tasks = 1 > > > This allowed me to set Avocado as it would be if running with QEMU > avocado, but even then I had no success. The test get stuck indefinitely > at this point: > > (...) > 2022-11-10 16:00:20,758 avocado.test INFO | Temporary dir: > /var/tmp/avocado_tmp_znhvpbh0/avocado_job_ywyf7v30 > 2022-11-10 16:00:20,758 avocado.test INFO | > 2022-11-10 16:00:20,758 avocado.test INFO | Job ID: > 4bb3e2a12c05d84a0a06849ecef435d547a198a0 > 2022-11-10 16:00:20,758 avocado.test INFO | > 2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| <Worker > spawner="<avocado.plugins.spawners.process.ProcessSpawner object at > 0x7fdad5da5840>" max_triaging=12 max_running=1 task_timeout=None> has > been initialized > 2022-11-10 16:00:21,041 avocado.core.task.statemachine DEBUG| Task > "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": > requested -> triaging > 2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task > "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": > requirements OK (will proceed to check dependencies) > 2022-11-10 16:00:21,042 avocado.core.task.statemachine DEBUG| Task > "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": about > to be spawned with "<avocado.plugins.spawners.process.ProcessSpawner > object at 0x7fdad5da5840>" > 2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task > "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned > successfully > > No error is provided. Tried to run the test for 10+ minutes and nothing > happens. Hitting CTRL+C aborts the test: > > $ make check-avocado > AVOCADO_TESTS='tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg' > GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 > tests/fp/berkeley-softfloat-3 dtc > AVOCADO Downloading avocado tests VM image for ppc64le > The image was downloaded: > Provider Version Architecture File > fedora 31 ppc64le > /home/danielhb/avocado/data/cache/by_location/d73d707673a941f5892e4bb0fa7fabaf0d41fd58/Fedora-Cloud-Base-31-1.9.ppc64le.qcow2 > AVOCADO tests/avocado > JOB ID : 4bb3e2a12c05d84a0a06849ecef435d547a198a0 > JOB LOG : > /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/job.log > ^C > RESULTS : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | > CANCEL 0 > JOB HTML : > /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a/results.html > JOB TIME : 396.78 s > make: *** [/home/danielhb/kvm-project/qemu/tests/Makefile.include:142: > check-avocado] Error 8 > > And the log registers the following: > > 2022-11-10 16:00:21,043 avocado.core.task.statemachine DEBUG| Task > "1-tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg": spawned > successfully > 2022-11-10 16:06:54,071 avocado.test INFO | > 2022-11-10 16:06:54,123 avocado.test INFO | Test results available in > /home/danielhb/kvm-project/qemu/build/tests/results/job-2022-11-10T16.00-4bb3e2a > 2022-11-10 16:06:55,694 avocado.sysinfo DEBUG| Not logging > /proc/slabinfo (lack of permissions) > 2022-11-10 16:06:55,696 avocado.sysinfo DEBUG| Not logging /proc/pci > (file not found) > 2022-11-10 16:06:55,712 avocado.sysinfo DEBUG| Not logging > /sys/kernel/debug/sched_features (file not found) > [---end of log---] > > > I'm probably missing more settings. Hi Daniel, I discussed this problem on Avocado meeting today and I got information that Cleber Rosa is working on a patch for qemu to update scripts to Avocado 99.0 where he will fix those things, and he will publish it soon. - Jan > > > Daniel > > >> >> - Jan >> >>>>> >>>>> That it is so difficult to update Avocado after barely more than >>>>> 1 year is not exactly a strong vote of confidence in our continued >>>>> use of Avocado long term :-( >>>> >>>> >>>> By the way, Avocado just provided a fix for the problem this patch >>>> is trying >>>> to amend: >>>> >>>> https://github.com/avocado-framework/avocado/pull/5515#issuecomment-1308872846 >>> >>> Thanks Jan! >>> >>>> Is there an easy way to plug upstream Avocado into QEMU? I would >>>> like to test >>>> tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg to see >>>> if the problem >>>> is fixed by Avocado upstream. >>> >>> See >>> https://lore.kernel.org/qemu-devel/20200403172919.24621-9-philmd@redhat.com/ >>> >>> For your case: >>> >>> -- >8 -- >>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>> index 0ba561b6bd..e17bc3972c 100644 >>> --- a/tests/requirements.txt >>> +++ b/tests/requirements.txt >>> @@ -4,3 +4,3 @@ >>> # Note that qemu.git/python/ is always implicitly installed. >>> -avocado-framework==88.1 >>> +-e >>> git+https://github.com/avocado-framework/avocado.git@b31b868c882d4650d3b7d2fbfc9b8ac0f2c3672b#egg=avocado-framework >>> pycdlib==1.11.0 >>> --- >>> >>> Regards, >>> >>> Phil. >>> >> >
diff --git a/tests/avocado/avocado_qemu/__init__.py b/tests/avocado/avocado_qemu/__init__.py index 910f3ba1ea..68c42948a3 100644 --- a/tests/avocado/avocado_qemu/__init__.py +++ b/tests/avocado/avocado_qemu/__init__.py @@ -423,12 +423,24 @@ class LinuxDistro: Holds information of known distros. """ #: A collection of known distros and their respective image checksum + # + # 'get-vm-image-fedora-31' target from Makefile.include will + # download the images using avocado default 'sha1' algorithm. + # This happens because the command line 'avocado vmimage get' + # from Makefile.include is not able to handle an extra --algorithm + # argument. + # + # To avoid a re-download of a recent image download due to a sha + # mismatch in the first test run, all Fedora 31 images are using + # 'sha1' instead of 'sha256'. + # + # FIXME: revisit this change if/when avocado learns to do CHECKSUM + # matches using different algorithms. KNOWN_DISTROS = { 'fedora': { '31': { 'x86_64': - {'checksum': ('e3c1b309d9203604922d6e255c2c5d09' - '8a309c2d46215d8fc026954f3c5c27a0'), + {'checksum': ('f62f8eabbf3687ea610c495bd59551a0025f99b7'), 'pxeboot_url': ('https://archives.fedoraproject.org/' 'pub/archive/fedora/linux/releases/31/' 'Everything/x86_64/os/images/pxeboot/'), @@ -438,8 +450,7 @@ class LinuxDistro: 'console=ttyS0,115200n8'), }, 'aarch64': - {'checksum': ('1e18d9c0cf734940c4b5d5ec592facae' - 'd2af0ad0329383d5639c997fdf16fe49'), + {'checksum': ('39d1fe099cdecacae894480d421be57d18b4e854'), 'pxeboot_url': 'https://archives.fedoraproject.org/' 'pub/archive/fedora/linux/releases/31/' 'Everything/aarch64/os/images/pxeboot/', @@ -450,11 +461,9 @@ class LinuxDistro: ' console=ttyAMA0'), }, 'ppc64': - {'checksum': ('7c3528b85a3df4b2306e892199a9e1e4' - '3f991c506f2cc390dc4efa2026ad2f58')}, + {'checksum': ('9993dc28e7c49ceb52125f9513130dfe2ace026c')}, 's390x': - {'checksum': ('4caaab5a434fd4d1079149a072fdc789' - '1e354f834d355069ca982fdcaf5a122d')}, + {'checksum': ('455f017b82decf32f366e06e7a7d0f6da86f96a7')}, }, '32': { 'aarch64': @@ -595,15 +604,18 @@ def download_boot(self): self.log.info('Downloading/preparing boot image') # Fedora 31 only provides ppc64le images image_arch = self.arch + hash_algorithm = 'sha256' if self.distro.name == 'fedora': if image_arch == 'ppc64': image_arch = 'ppc64le' + if self.distro.version == '31': + hash_algorithm = 'sha1' try: boot = vmimage.get( self.distro.name, arch=image_arch, version=self.distro.version, checksum=self.distro.checksum, - algorithm='sha256', + algorithm=hash_algorithm, cache_dir=self.cache_dirs[0], snapshot_dir=self.workdir) except:
'make check-avocado' will download any images that aren't present in the cache via 'get-vm-images' in tests/Makefile.include. The target that downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado vmimage get --distro=fedora --distro-version=31 --arch=(...)' to download the image for each arch. Note that this command does not support any argument to set the hash algorithm used and, based on the avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM in the cache. Back in QEMU, the LinuxDistro class defines the images it is going to use in the KNOWN_DISTROS attribute. All images have the 'checksum' attribute, defined using sha256, that will make avocado check if it has the right asset in the cache. So this means that we're downloading a sha1 image during check-avocado, then trying to fetch it using a sha256 hash. This doesn't work. Avocado will match the provided checksum, a sha256 hash, with the sha1 hash from the existing CHECKSUM file for the asset, and will re-download the image again. Avocado doesn't take into account that we're using a different hash algorithm than the one used to fetch the image via get-vm-image-fedora31 (in avocado: utils/asset.py, self.find_asset_file() and fetch(self, timeout=None). The new download will take into account the chosen hash algorithm, and subsequent avocado runs will not re-download the image again because now it's comparing sha256 with sha256. All of this, of course, assuming that the system was able to re-download the image before the avocado test timeouts. A more "fun" scenario happens if the network is not fast enough to download the image during the avocado timeout, the test will fail and the sha1 image will remain in the cache. And the test will keep trying to re-fetch it, and will keep failing with a timeout. There's something to be said about avocado matching different sha algorithms instead of re-calculating the hash with the chosen algorithm during fetch(), especially considering that the CHECKSUM file contains the name of the algorithm used. An avocado feature request [2] was opened to see if this is a feasible change to do. For now, in QEMU, let's use sha1 for all Fedora 31 images. This will immediately spares us at least one extra download for each Fedora 31 image that we're doing in all our CI runs. [1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906 [2] https://github.com/avocado-framework/avocado/issues/5496 Cc: Alex Bennée <alex.bennee@linaro.org> Cc: Thomas Huth <thuth@redhat.com> Cc: John Snow <jsnow@redhat.com> Cc: Cleber Rosa <crosa@redhat.com> Cc: Wainer dos Santos Moschetta <wainersm@redhat.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/avocado/avocado_qemu/__init__.py | 30 ++++++++++++++++++-------- 1 file changed, 21 insertions(+), 9 deletions(-)