| Message ID | 20221020123506.26363-1-ani@anisinha.ca (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > Please see the rst doc file in patch 9 for more details. > Sample runs are as follows: > > Passed tests: > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : e95c7413e996bfb59389839e5ca5105464ef098f > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.08 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.18 s > > > $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py > Fetching asset from ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : c6e588594bceb2c5fdbdf45095559c58e4030f9c > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log > (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (32.29 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.27 s > > When PSS tests are enabled: > > $ git diff > diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > index f818a9cce6..18dc818d62 100644 > --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > @@ -40,8 +40,8 @@ import time > > def register_tests(): > testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests") > -# testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") > -# testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") > + testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") > + testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") > testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests") > testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests") > testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests") > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check the test logs for more info. (33.01 s) > RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.17 s > > > Changelog: > v6: > - skip test when dependencies (xorriso for example) are not installed. > - skip test when run on a platform other than x86_64. Hmm why is that btw? > - move README as an rst file under docs/devel. > - updated tags. > v5: > - fixed bits so that grub-mkrescue is built as a static binary. > - updated the test so that it points to the new bits artifact. > - V=1 now spits out more meaningful information on failure. > - added a check to see if xorriso is installed. If not, the test skips with > a meaningful message instructing the user to install it. > - fixed README by removing references to stale instructions and adding > updated instructions. > - tags updated. > v4: > - renamed the bits test files as .py2. > - fixed a bug with missing log file parsing. dump bits logs automatically > for test failures (no need to re-run with V=1). > - fixed issues with README file and enhanced it. > - cosmetic comment updates. > v3: > - converted the existing test to avocado tests as per the popular > recommendation. Added appropriate tags. > - artifact download URL modified based on gitlab-CI URL. > > For biosbits repo: > - moved to a gitlab repo with me being the maintainer. > - added .gitlab-ci.yml file to generate the artifacts. > v2: > - a new class of python based tests introduced that is separate from avocado > tests or qtests. Can be run by using "make check-pytest". > - acpi biosbits tests are the first tests to use pytest environment. > - bios bits tests now download the bits binary archives from a remote > repository if they are not found locally. The test skips if download > fails. > - A new environment variable is introduced that can be passed by the tester > to specify the location of the bits archives locally. test skips if the > bits binaries are not found in that location. > - if pip install of python module fails for whatever reaoson, the test skips. > - misc code fixes including spell check of the README doc. README has been > updated as well. > - addition of SPDX license headers to bits test files. > - update MAINTAINERS to reflect the new pytest test class. > > For biosbits repo: > - added Dockerfile and build script. Made bios bits build on gcc 11. > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > The build script generates the zip archive and tarball used by the test. > > v1: initial patchset. uses qtest to implement the bios bits tests. > > Cc: Qemu Devel <qemu-devel@nongnu.org> > Cc: Daniel P. Berrangé <berrange@redhat.com> > Cc: Paolo Bonzini <pbonzini@redhat.com> > Cc: Maydell Peter <peter.maydell@linaro.org> > Cc: John Snow <jsnow@redhat.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Alex Bennée <alex.bennee@linaro.org> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Michael Tsirkin <mst@redhat.com> > > Ani Sinha (10): > acpi/tests/avocado/bits: initial commit of test scripts that are run > by biosbits > acpi/tests/avocado/bits: add SPDX license identifiers for bios bits > tests > acpi/tests/avocado/bits: disable acpi PSS tests that are failing in > biosbits > acpi/tests/avocado/bits: add smilatency test suite from bits in order > to disable it > acpi/tests/avocado/bits: add SPDX license identifiers for bios bits > smilatency tests > acpi/tests/avocado/bits: disable smilatency test since it does not > pass everytime > acpi/tests/avocado/bits: add biosbits config file for running bios > tests > acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses > biosbits > acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits > test > MAINTAINERS: add myself as the maintainer for acpi biosbits avocado > tests > > MAINTAINERS | 6 + > docs/devel/acpi-bits.rst | 148 + > tests/avocado/acpi-bits.py | 389 +++ > .../acpi-bits/bits-config/bits-cfg.txt | 18 + > tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++ > .../acpi-bits/bits-tests/smilatency.py2 | 107 + > .../avocado/acpi-bits/bits-tests/testacpi.py2 | 287 ++ > .../acpi-bits/bits-tests/testcpuid.py2 | 87 + > 8 files changed, 3476 insertions(+) > create mode 100644 docs/devel/acpi-bits.rst > create mode 100644 tests/avocado/acpi-bits.py > create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt > create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2 > > -- > 2.34.1
On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > Changelog: > > v6: > > - skip test when dependencies (xorriso for example) are not installed. > > - skip test when run on a platform other than x86_64. > > Hmm why is that btw? The bits binaries that generate the iso (grub-mkrescue etc) are built for and are known to work only on x86_64 platform. They might also work on amd64 but I do not have one at my disposal at the moment to check. On other platforms, for example 32 bit x86 and non-x86, those binaries will likely not work. The test will fail.
On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > Changelog: > > > v6: > > > - skip test when dependencies (xorriso for example) are not installed. > > > - skip test when run on a platform other than x86_64. > > > > Hmm why is that btw? > > The bits binaries that generate the iso (grub-mkrescue etc) are built > for and are known to work only on x86_64 platform. They might also > work on amd64 but I do not have one at my disposal at the moment to > check. > On other platforms, for example 32 bit x86 and non-x86, those binaries > will likely not work. The test will fail. confused. I thought we are distributing the iso?
On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > Changelog: > > > > v6: > > > > - skip test when dependencies (xorriso for example) are not installed. > > > > - skip test when run on a platform other than x86_64. > > > > > > Hmm why is that btw? > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > for and are known to work only on x86_64 platform. They might also > > work on amd64 but I do not have one at my disposal at the moment to > > check. > > On other platforms, for example 32 bit x86 and non-x86, those binaries > > will likely not work. The test will fail. > > confused. I thought we are distributing the iso? No, the test builds the iso after adding the modified test scripts and then spawns the vm with it. It is all part of the test itself. We need to do that so that the iso contains the newly added tests etc.
On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > Changelog: > > > > > v6: > > > > > - skip test when dependencies (xorriso for example) are not installed. > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > Hmm why is that btw? > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > for and are known to work only on x86_64 platform. They might also > > > work on amd64 but I do not have one at my disposal at the moment to > > > check. > > > On other platforms, for example 32 bit x86 and non-x86, those binaries > > > will likely not work. The test will fail. > > > > confused. I thought we are distributing the iso? > > No, the test builds the iso after adding the modified test scripts and > then spawns the vm with it. It is all part of the test itself. > We need to do that so that the iso contains the newly added tests etc. It's good to have for people developing tests, but for most qemu developers please just have a ready iso and have avocado fetch it. It's important to make tests run on all platforms. We can think about moving iso generation into a VM too but as a first step I guess we can live with a container.
On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > Changelog: > > > > > > v6: > > > > > > - skip test when dependencies (xorriso for example) are not > installed. > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > Hmm why is that btw? > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > > for and are known to work only on x86_64 platform. They might also > > > > work on amd64 but I do not have one at my disposal at the moment to > > > > check. > > > > On other platforms, for example 32 bit x86 and non-x86, those > binaries > > > > will likely not work. The test will fail. > > > > > > confused. I thought we are distributing the iso? > > > > No, the test builds the iso after adding the modified test scripts and > > then spawns the vm with it. It is all part of the test itself. > > We need to do that so that the iso contains the newly added tests etc. > > It's good to have for people developing tests, but for most qemu > developers please just have a ready iso and have avocado fetch it. > It's important to make tests run on all platforms. This changes things a lot and goes down the path of where do we check in test code changes? Do we deal with multiple repos? Where do we keep iso? Etc. having a static iso limits us also. If we can think through and come up with a consensus, I can write a second test. Else let's just go with this first. If we are really keen on running tests on multiple platforms, we can have binaries built for those. > > We can think about moving iso generation into a VM too > but as a first step I guess we can live with a container. > > -- > MST > >
On Fri, Oct 21, 2022 at 5:10 AM Ani Sinha <ani@anisinha.ca> wrote: > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: >> > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> > > >> > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: >> > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> > > > > >> > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: >> > > > > > >> > > > > > >> > > > > > Changelog: >> > > > > > v6: >> > > > > > - skip test when dependencies (xorriso for example) are not >> installed. >> > > > > > - skip test when run on a platform other than x86_64. >> > > > > >> > > > > Hmm why is that btw? >> > > > >> > > > The bits binaries that generate the iso (grub-mkrescue etc) are >> built >> > > > for and are known to work only on x86_64 platform. They might also >> > > > work on amd64 but I do not have one at my disposal at the moment to >> > > > check. >> > > > On other platforms, for example 32 bit x86 and non-x86, those >> binaries >> > > > will likely not work. The test will fail. >> > > >> > > confused. I thought we are distributing the iso? >> > >> > No, the test builds the iso after adding the modified test scripts and >> > then spawns the vm with it. It is all part of the test itself. >> > We need to do that so that the iso contains the newly added tests etc. >> >> It's good to have for people developing tests, but for most qemu >> developers please just have a ready iso and have avocado fetch it. >> It's important to make tests run on all platforms. > > > This changes things a lot and goes down the path of where do we check in > test code changes? Do we deal with multiple repos? Where do we keep iso? > Etc. having a static iso limits us also. > > If we can think through and come up with a consensus, I can write a second > test. Else let's just go with this first. If we are really keen on running > tests on multiple platforms, we can have binaries built for those. > And have multiple platform specific branches in bits that have fixes for those platforms so that bits can run there. Plus the existing test can be enhanced to pull in binaries from those branches based on the platform on which it is being run.
On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > Changelog: > > > > > > v6: > > > > > > - skip test when dependencies (xorriso for example) are not > installed. > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > Hmm why is that btw? > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > > for and are known to work only on x86_64 platform. They might also > > > > work on amd64 but I do not have one at my disposal at the moment to > > > > check. > > > > On other platforms, for example 32 bit x86 and non-x86, those > binaries > > > > will likely not work. The test will fail. > > > > > > confused. I thought we are distributing the iso? > > > > No, the test builds the iso after adding the modified test scripts and > > then spawns the vm with it. It is all part of the test itself. > > We need to do that so that the iso contains the newly added tests etc. > > It's good to have for people developing tests, but for most qemu > developers please just have a ready iso and have avocado fetch it. > It's important to make tests run on all platforms. > > > This changes things a lot and goes down the path of where do we check in test > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. having > a static iso limits us also. It's the same as any firmware really. I don't see much of a difference. > If we can think through and come up with a consensus, I can write a second > test. Else let's just go with this first. We can go ahead for now. > If we are really keen on running > tests on multiple platforms, we can have binaries built for those. > Oh my. Not dealing with this is *exactly* what people use virtualization for. > > > We can think about moving iso generation into a VM too > but as a first step I guess we can live with a container. > > -- > MST > >
On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > And have multiple platform specific branches in bits that have fixes for those > platforms so that bits can run there. Plus the existing test can be enhanced to > pull in binaries from those branches based on the platform on which it is being > run. > What a mess. Who is going to be testing all these million platforms? All this does nothing at all to help developers avoid bugs and when they do trigger debug the issue. Which is after all why we have testing. Yes once in a very long while we are going to tweak something in the tests, and for that rare occurence it makes sense to periodically rebuild everything, otherwise code bitrots. But the test is supposed to run within a VM anyway, let's have an image and be done with it.
On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > And have multiple platform specific branches in bits that have fixes for those > > platforms so that bits can run there. Plus the existing test can be enhanced to > > pull in binaries from those branches based on the platform on which it is being > > run. > > > > What a mess. > Who is going to be testing all these million platforms? I am not talking about branches in QEMU but branches in bits. If you are going to test multiple platforms, you do need to build bits binaries for them. There is no way around it. bits is not all platform independent python. It does have binary executables. Currently bits is built only for the x86 platform. Other platforms are not tested. I doubt if anyone even tried building bits for arm or mips. It makes sense to try things incrementally once we have something going. Lets discuss this on a separate thread. > All this does nothing at all to help developers avoid > bugs and when they do trigger debug the issue. Which is > after all why we have testing. > Yes once in a very long while we are going to tweak > something in the tests, and for that rare occurence > it makes sense to periodically rebuild everything, > otherwise code bitrots. > > But the test is supposed to run within a VM anyway, let's > have an image and be done with it. > > -- > MST >
Ani Sinha <ani@anisinha.ca> writes: > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> > And have multiple platform specific branches in bits that have fixes for those >> > platforms so that bits can run there. Plus the existing test can be enhanced to >> > pull in binaries from those branches based on the platform on which it is being >> > run. >> > >> >> What a mess. >> Who is going to be testing all these million platforms? > > I am not talking about branches in QEMU but branches in bits. > If you are going to test multiple platforms, you do need to build bits > binaries for them. There is no way around it. > bits is not all platform independent python. It does have binary executables. > > Currently bits is built only for the x86 platform. Other platforms are > not tested. I doubt if anyone even tried building bits for arm or > mips. I'm not worried about test bits on other targets, but we do run x86 targets on a number of hosts. The current reliance on a special patched host build tool for only one architecture is the problem. If we just download the iso that problem goes away. > It makes sense to try things incrementally once we have something going. > > Lets discuss this on a separate thread. > >> All this does nothing at all to help developers avoid >> bugs and when they do trigger debug the issue. Which is >> after all why we have testing. >> Yes once in a very long while we are going to tweak >> something in the tests, and for that rare occurence >> it makes sense to periodically rebuild everything, >> otherwise code bitrots. >> >> But the test is supposed to run within a VM anyway, let's >> have an image and be done with it. >> >> -- >> MST >>
On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > And have multiple platform specific branches in bits that have fixes for those > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > >> > pull in binaries from those branches based on the platform on which it is being > >> > run. > >> > > >> > >> What a mess. > >> Who is going to be testing all these million platforms? > > > > I am not talking about branches in QEMU but branches in bits. > > If you are going to test multiple platforms, you do need to build bits > > binaries for them. There is no way around it. > > bits is not all platform independent python. It does have binary executables. > > > > Currently bits is built only for the x86 platform. Other platforms are > > not tested. I doubt if anyone even tried building bits for arm or > > mips. > > I'm not worried about test bits on other targets, but we do run x86 > targets on a number of hosts. The current reliance on a special patched > host build tool for only one architecture is the problem. If we just > download the iso that problem goes away.
On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > >> > > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > And have multiple platform specific branches in bits that have fixes for those > > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > > >> > pull in binaries from those branches based on the platform on which it is being > > >> > run. > > >> > > > >> > > >> What a mess. > > >> Who is going to be testing all these million platforms? > > > > > > I am not talking about branches in QEMU but branches in bits. > > > If you are going to test multiple platforms, you do need to build bits > > > binaries for them. There is no way around it. > > > bits is not all platform independent python. It does have binary executables. > > > > > > Currently bits is built only for the x86 platform. Other platforms are > > > not tested. I doubt if anyone even tried building bits for arm or > > > mips. > > > > I'm not worried about test bits on other targets, but we do run x86 > > targets on a number of hosts. The current reliance on a special patched > > host build tool for only one architecture is the problem. If we just > > download the iso that problem goes away. > >
On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com > > > > wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > v6: > > > > > > > - skip test when dependencies (xorriso for example) are > not > > installed. > > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) > are built > > > > > for and are known to work only on x86_64 platform. They might > also > > > > > work on amd64 but I do not have one at my disposal at the > moment to > > > > > check. > > > > > On other platforms, for example 32 bit x86 and non-x86, those > > binaries > > > > > will likely not work. The test will fail. > > > > > > > > confused. I thought we are distributing the iso? > > > > > > No, the test builds the iso after adding the modified test scripts > and > > > then spawns the vm with it. It is all part of the test itself. > > > We need to do that so that the iso contains the newly added tests > etc. > > > > It's good to have for people developing tests, but for most qemu > > developers please just have a ready iso and have avocado fetch it. > > It's important to make tests run on all platforms. > > > > > > This changes things a lot and goes down the path of where do we check in > test > > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. > having > > a static iso limits us also. > > It's the same as any firmware really. I don't see much of a difference. Except that the iso is much larger than the current edk2 binaries and there are serious objections to keeping them along with other bios binary blobs in qemu repo.
Ani Sinha <ani@anisinha.ca> writes: > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: >> > >> > Ani Sinha <ani@anisinha.ca> writes: >> > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> > >> >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> > >> > And have multiple platform specific branches in bits that have fixes for those >> > >> > platforms so that bits can run there. Plus the existing test can be enhanced to >> > >> > pull in binaries from those branches based on the platform on which it is being >> > >> > run. >> > >> > >> > >> >> > >> What a mess. >> > >> Who is going to be testing all these million platforms? >> > > >> > > I am not talking about branches in QEMU but branches in bits. >> > > If you are going to test multiple platforms, you do need to build bits >> > > binaries for them. There is no way around it. >> > > bits is not all platform independent python. It does have binary executables. >> > > >> > > Currently bits is built only for the x86 platform. Other platforms are >> > > not tested. I doubt if anyone even tried building bits for arm or >> > > mips. >> > >> > I'm not worried about test bits on other targets, but we do run x86 >> > targets on a number of hosts. The current reliance on a special patched >> > host build tool for only one architecture is the problem. If we just >> > download the iso that problem goes away. >> >>
On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > >> > > >> > Ani Sinha <ani@anisinha.ca> writes: > >> > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> > >> > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > >> > And have multiple platform specific branches in bits that have > fixes for those > >> > >> > platforms so that bits can run there. Plus the existing test can > be enhanced to > >> > >> > pull in binaries from those branches based on the platform on > which it is being > >> > >> > run. > >> > >> > > >> > >> > >> > >> What a mess. > >> > >> Who is going to be testing all these million platforms? > >> > > > >> > > I am not talking about branches in QEMU but branches in bits. > >> > > If you are going to test multiple platforms, you do need to build > bits > >> > > binaries for them. There is no way around it. > >> > > bits is not all platform independent python. It does have binary > executables. > >> > > > >> > > Currently bits is built only for the x86 platform. Other platforms > are > >> > > not tested. I doubt if anyone even tried building bits for arm or > >> > > mips. > >> > > >> > I'm not worried about test bits on other targets, but we do run x86 > >> > targets on a number of hosts. The current reliance on a special > patched > >> > host build tool for only one architecture is the problem. If we just > >> > download the iso that problem goes away. > >> > >>
On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > >> >> Ani Sinha <ani@anisinha.ca> writes: >> >> > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> >> >> >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: >> >> > >> >> > Ani Sinha <ani@anisinha.ca> writes: >> >> > >> >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> >> > >> >> >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> >> > >> > And have multiple platform specific branches in bits that have >> fixes for those >> >> > >> > platforms so that bits can run there. Plus the existing test >> can be enhanced to >> >> > >> > pull in binaries from those branches based on the platform on >> which it is being >> >> > >> > run. >> >> > >> > >> >> > >> >> >> > >> What a mess. >> >> > >> Who is going to be testing all these million platforms? >> >> > > >> >> > > I am not talking about branches in QEMU but branches in bits. >> >> > > If you are going to test multiple platforms, you do need to build >> bits >> >> > > binaries for them. There is no way around it. >> >> > > bits is not all platform independent python. It does have binary >> executables. >> >> > > >> >> > > Currently bits is built only for the x86 platform. Other platforms >> are >> >> > > not tested. I doubt if anyone even tried building bits for arm or >> >> > > mips. >> >> > >> >> > I'm not worried about test bits on other targets, but we do run x86 >> >> > targets on a number of hosts. The current reliance on a special >> patched >> >> > host build tool for only one architecture is the problem. If we just >> >> > download the iso that problem goes away. >> >> >> >>
Ani Sinha <ani@anisinha.ca> writes: > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > >> > > >> > Ani Sinha <ani@anisinha.ca> writes: > >> > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > >> > And have multiple platform specific branches in bits that have fixes for those > >> > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > >> > >> > pull in binaries from those branches based on the platform on which it is being > >> > >> > run. > >> > >> > > >> > >> > >> > >> What a mess. > >> > >> Who is going to be testing all these million platforms? > >> > > > >> > > I am not talking about branches in QEMU but branches in bits. > >> > > If you are going to test multiple platforms, you do need to build bits > >> > > binaries for them. There is no way around it. > >> > > bits is not all platform independent python. It does have binary executables. > >> > > > >> > > Currently bits is built only for the x86 platform. Other platforms are > >> > > not tested. I doubt if anyone even tried building bits for arm or > >> > > mips. > >> > > >> > I'm not worried about test bits on other targets, but we do run x86 > >> > targets on a number of hosts. The current reliance on a special patched > >> > host build tool for only one architecture is the problem. If we just > >> > download the iso that problem goes away. > >> > >>
Ani Sinha <ani@anisinha.ca> writes: <snip> > > We have added a mirror of biosbits to the QEMU project so there is no > reason why we can't track changes and modifications there (we do this > for TestFloat which is forked from the upstream SoftFloat code). > > The whole idea was that say an acpi developer added support for a new table in QEMU, he should write a corresponding > test for bits so that the same table is exercised during run time. Making those changes from a single repo (either directly > or through a submodule) makes things lit simpler and also keeps things in sync with each other. If we use separate > repos for acpi bits test, it will be another mess when comes to developers adding changes and keeping things in sync. > > Anyways these things should have been brought up earlier. I'm out of the debate. > > I've sent v7 , incremental work over the last 6 months in my spare time without getting any pay. So take it or scrap it. > >
On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >>
On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > <snip> > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a corresponding > > test for bits so that the same table is exercised during run time. > Making those changes from a single repo (either directly > > or through a submodule) makes things lit simpler and also keeps things > in sync with each other. If we use separate > > repos for acpi bits test, it will be another mess when comes to > developers adding changes and keeping things in sync. > > > > Anyways these things should have been brought up earlier. I'm out of the > debate. > > > > I've sent v7 , incremental work over the last 6 months in my spare time > without getting any pay. So take it or scrap it. > > > >
On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >>
On Fri, Oct 21, 2022 at 9:32 PM Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >>
On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that have > fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to build > bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >>
On Fri, Oct 21, 2022 at 05:12:07PM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > > wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > v6: > > > > > > > - skip test when dependencies (xorriso for example) are > not > > installed. > > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are > built > > > > > for and are known to work only on x86_64 platform. They might > also > > > > > work on amd64 but I do not have one at my disposal at the > moment to > > > > > check. > > > > > On other platforms, for example 32 bit x86 and non-x86, those > > binaries > > > > > will likely not work. The test will fail. > > > > > > > > confused. I thought we are distributing the iso? > > > > > > No, the test builds the iso after adding the modified test scripts > and > > > then spawns the vm with it. It is all part of the test itself. > > > We need to do that so that the iso contains the newly added tests > etc. > > > > It's good to have for people developing tests, but for most qemu > > developers please just have a ready iso and have avocado fetch it. > > It's important to make tests run on all platforms. > > > > > > This changes things a lot and goes down the path of where do we check in > test > > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. > having > > a static iso limits us also. > > It's the same as any firmware really. I don't see much of a difference. > > > Except that the iso is much larger than the current edk2 binaries and there are > serious objections to keeping them along with other bios binary blobs in qemu > repo. > Right, I get that. So avocado was supposed to know how to download them automatically, no?
On Sat, Oct 22, 2022 at 22:05 Michael S. Tsirkin <mst@redhat.com> wrote: > On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, < > alex.bennee@linaro.org> > > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > >> > > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > >> > > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > > >> > > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > > mst@redhat.com> wrote: > > > >> > >> > > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > > >> > >> > And have multiple platform specific branches in bits > that have > > fixes for those > > > >> > >> > platforms so that bits can run there. Plus the existing > test > > can be enhanced to > > > >> > >> > pull in binaries from those branches based on the > platform on > > which it is being > > > >> > >> > run. > > > >> > >> > > > > >> > >> > > > >> > >> What a mess. > > > >> > >> Who is going to be testing all these million platforms? > > > >> > > > > > >> > > I am not talking about branches in QEMU but branches in > bits. > > > >> > > If you are going to test multiple platforms, you do need > to build > > bits > > > >> > > binaries for them. There is no way around it. > > > >> > > bits is not all platform independent python. It does have > binary > > executables. > > > >> > > > > > >> > > Currently bits is built only for the x86 platform. Other > > platforms are > > > >> > > not tested. I doubt if anyone even tried building bits for > arm or > > > >> > > mips. > > > >> > > > > >> > I'm not worried about test bits on other targets, but we do > run x86 > > > >> > targets on a number of hosts. The current reliance on a > special > > patched > > > >> > host build tool for only one architecture is the problem. > If we > > just > > > >> > download the iso that problem goes away. > > > >> > > > >>
On Sat, Oct 22, 2022 at 22:11 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 05:12:07PM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin < > mst@redhat.com> > > > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > > mst@redhat.com> > > > wrote: > > > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > > v6: > > > > > > > > - skip test when dependencies (xorriso for > example) are > > not > > > installed. > > > > > > > > - skip test when run on a platform other than > x86_64. > > > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue > etc) are > > built > > > > > > for and are known to work only on x86_64 platform. They > might > > also > > > > > > work on amd64 but I do not have one at my disposal at the > > moment to > > > > > > check. > > > > > > On other platforms, for example 32 bit x86 and non-x86, > those > > > binaries > > > > > > will likely not work. The test will fail. > > > > > > > > > > confused. I thought we are distributing the iso? > > > > > > > > No, the test builds the iso after adding the modified test > scripts > > and > > > > then spawns the vm with it. It is all part of the test > itself. > > > > We need to do that so that the iso contains the newly added > tests > > etc. > > > > > > It's good to have for people developing tests, but for most > qemu > > > developers please just have a ready iso and have avocado fetch > it. > > > It's important to make tests run on all platforms. > > > > > > > > > This changes things a lot and goes down the path of where do we > check in > > test > > > code changes? Do we deal with multiple repos? Where do we keep > iso? Etc. > > having > > > a static iso limits us also. > > > > It's the same as any firmware really. I don't see much of a > difference. > > > > > > Except that the iso is much larger than the current edk2 binaries and > there are > > serious objections to keeping them along with other bios binary blobs in > qemu > > repo. > > > > Right, I get that. So avocado was supposed to know how to download them > automatically, no? Yea that’s the reason why we are using CI artifacts and telling avocado to download them from a specific URL. > > -- > MST > >
On Fri, Oct 21, 2022 at 09:47:04PM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> wrote: > > > Ani Sinha <ani@anisinha.ca> writes: > > <snip> > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a corresponding > > test for bits so that the same table is exercised during run time. Making > those changes from a single repo (either directly > > or through a submodule) makes things lit simpler and also keeps things > in sync with each other. If we use separate > > repos for acpi bits test, it will be another mess when comes to > developers adding changes and keeping things in sync. > > > > Anyways these things should have been brought up earlier. I'm out of the > debate. > > > > I've sent v7 , incremental work over the last 6 months in my spare time > without getting any pay. So take it or scrap it. > > > >
On Sat, Oct 22, 2022 at 10:13:13PM +0530, Ani Sinha wrote: > > > On Sat, Oct 22, 2022 at 22:05 Michael S. Tsirkin <mst@redhat.com> wrote: > > On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org > > > > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > >> > > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > >> > > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > > >> > > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > > mst@redhat.com> wrote: > > > >> > >> > > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > > >> > >> > And have multiple platform specific branches in bits > that have > > fixes for those > > > >> > >> > platforms so that bits can run there. Plus the existing > test > > can be enhanced to > > > >> > >> > pull in binaries from those branches based on the > platform on > > which it is being > > > >> > >> > run. > > > >> > >> > > > > >> > >> > > > >> > >> What a mess. > > > >> > >> Who is going to be testing all these million platforms? > > > >> > > > > > >> > > I am not talking about branches in QEMU but branches in > bits. > > > >> > > If you are going to test multiple platforms, you do need to > build > > bits > > > >> > > binaries for them. There is no way around it. > > > >> > > bits is not all platform independent python. It does have > binary > > executables. > > > >> > > > > > >> > > Currently bits is built only for the x86 platform. Other > > platforms are > > > >> > > not tested. I doubt if anyone even tried building bits for > arm or > > > >> > > mips. > > > >> > > > > >> > I'm not worried about test bits on other targets, but we do > run x86 > > > >> > targets on a number of hosts. The current reliance on a > special > > patched > > > >> > host build tool for only one architecture is the problem. If > we > > just > > > >> > download the iso that problem goes away. > > > >> > > > >>
On Sat, Oct 22, 2022 at 22:15 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 09:47:04PM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > <snip> > > > > > > We have added a mirror of biosbits to the QEMU project so there > is no > > > reason why we can't track changes and modifications there (we do > this > > > for TestFloat which is forked from the upstream SoftFloat code). > > > > > > The whole idea was that say an acpi developer added support for a > new > > table in QEMU, he should write a corresponding > > > test for bits so that the same table is exercised during run time. > Making > > those changes from a single repo (either directly > > > or through a submodule) makes things lit simpler and also keeps > things > > in sync with each other. If we use separate > > > repos for acpi bits test, it will be another mess when comes to > > developers adding changes and keeping things in sync. > > > > > > Anyways these things should have been brought up earlier. I'm out > of the > > debate. > > > > > > I've sent v7 , incremental work over the last 6 months in my spare > time > > without getting any pay. So take it or scrap it. > > > > > >
diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 index f818a9cce6..18dc818d62 100644 --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 @@ -40,8 +40,8 @@ import time def register_tests(): testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests") -# testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") -# testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") + testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") + testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests") testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests") testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
Please see the rst doc file in patch 9 for more details. Sample runs are as follows: Passed tests: $ ./tests/venv/bin/avocado run -t acpi tests/avocado Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : e95c7413e996bfb59389839e5ca5105464ef098f JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.08 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.18 s $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py Fetching asset from ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : c6e588594bceb2c5fdbdf45095559c58e4030f9c JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (32.29 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.27 s When PSS tests are enabled: $ git diff $ ./tests/venv/bin/avocado run -t acpi tests/avocado Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check the test logs for more info. (33.01 s) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.17 s Changelog: v6: - skip test when dependencies (xorriso for example) are not installed. - skip test when run on a platform other than x86_64. - move README as an rst file under docs/devel. - updated tags. v5: - fixed bits so that grub-mkrescue is built as a static binary. - updated the test so that it points to the new bits artifact. - V=1 now spits out more meaningful information on failure. - added a check to see if xorriso is installed. If not, the test skips with a meaningful message instructing the user to install it. - fixed README by removing references to stale instructions and adding updated instructions. - tags updated. v4: - renamed the bits test files as .py2. - fixed a bug with missing log file parsing. dump bits logs automatically for test failures (no need to re-run with V=1). - fixed issues with README file and enhanced it. - cosmetic comment updates. v3: - converted the existing test to avocado tests as per the popular recommendation. Added appropriate tags. - artifact download URL modified based on gitlab-CI URL. For biosbits repo: - moved to a gitlab repo with me being the maintainer. - added .gitlab-ci.yml file to generate the artifacts. v2: - a new class of python based tests introduced that is separate from avocado tests or qtests. Can be run by using "make check-pytest". - acpi biosbits tests are the first tests to use pytest environment. - bios bits tests now download the bits binary archives from a remote repository if they are not found locally. The test skips if download fails. - A new environment variable is introduced that can be passed by the tester to specify the location of the bits archives locally. test skips if the bits binaries are not found in that location. - if pip install of python module fails for whatever reaoson, the test skips. - misc code fixes including spell check of the README doc. README has been updated as well. - addition of SPDX license headers to bits test files. - update MAINTAINERS to reflect the new pytest test class. For biosbits repo: - added Dockerfile and build script. Made bios bits build on gcc 11. https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh The build script generates the zip archive and tarball used by the test. v1: initial patchset. uses qtest to implement the bios bits tests. Cc: Qemu Devel <qemu-devel@nongnu.org> Cc: Daniel P. Berrangé <berrange@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Maydell Peter <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Cc: Alex Bennée <alex.bennee@linaro.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Michael Tsirkin <mst@redhat.com> Ani Sinha (10): acpi/tests/avocado/bits: initial commit of test scripts that are run by biosbits acpi/tests/avocado/bits: add SPDX license identifiers for bios bits tests acpi/tests/avocado/bits: disable acpi PSS tests that are failing in biosbits acpi/tests/avocado/bits: add smilatency test suite from bits in order to disable it acpi/tests/avocado/bits: add SPDX license identifiers for bios bits smilatency tests acpi/tests/avocado/bits: disable smilatency test since it does not pass everytime acpi/tests/avocado/bits: add biosbits config file for running bios tests acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses biosbits acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits test MAINTAINERS: add myself as the maintainer for acpi biosbits avocado tests MAINTAINERS | 6 + docs/devel/acpi-bits.rst | 148 + tests/avocado/acpi-bits.py | 389 +++ .../acpi-bits/bits-config/bits-cfg.txt | 18 + tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++ .../acpi-bits/bits-tests/smilatency.py2 | 107 + .../avocado/acpi-bits/bits-tests/testacpi.py2 | 287 ++ .../acpi-bits/bits-tests/testcpuid.py2 | 87 + 8 files changed, 3476 insertions(+) create mode 100644 docs/devel/acpi-bits.rst create mode 100644 tests/avocado/acpi-bits.py create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2