| Message ID | 20230327115030.3418323-3-kconsul@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Re-enabling tests/avocado/boot_linux.py for PPC64 | expand |
Kautuk Consul <kconsul@linux.vnet.ibm.com> writes: > Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI"). > > Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case > for PPC64. On investigation, this turns out to be an issue with the > time taken for downloading the Fedora 31 qcow2 image being included > within the test-case timeout. > Re-enable this test-case by setting the timeout to 360 seconds just > before launching the downloaded VM image. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Reported-by: Alex Bennée <alex.bennee@linaro.org> > Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com It doesn't really address the principle problem that the boot_linux.py:BootLinuxPPC64.test_pseries_tcg is super heavyweight for only 2% extra coverage of the executed lines. What we really need is a script so we can compare the output between the two jsons: gcovr --json --exclude-unreachable-branches --print-summary -o coverage.json --root ../../ . *.p because I suspect we could make up that missing few % noodling the baseline test a bit more. > --- > tests/avocado/boot_linux.py | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py > index be30dcbd58..c3869a987c 100644 > --- a/tests/avocado/boot_linux.py > +++ b/tests/avocado/boot_linux.py > @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest): > :avocado: tags=arch:ppc64 > """ > > + # timeout for downloading new VM image. > timeout = 360 > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > def test_pseries_tcg(self): > """ > :avocado: tags=machine:pseries > @@ -101,6 +101,10 @@ def test_pseries_tcg(self): > """ > self.require_accelerator("tcg") > self.vm.add_args("-accel", "tcg") > + > + # timeout for actual Linux PPC boot test > + self.timeout = 360 > + > self.launch_and_wait(set_up_ssh_connection=False)
On 2023-03-27 17:07:30, Alex Bennée wrote: > > Kautuk Consul <kconsul@linux.vnet.ibm.com> writes: > > > Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI"). > > > > Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case > > for PPC64. On investigation, this turns out to be an issue with the > > time taken for downloading the Fedora 31 qcow2 image being included > > within the test-case timeout. > > Re-enable this test-case by setting the timeout to 360 seconds just > > before launching the downloaded VM image. > > > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Reported-by: Alex Bennée <alex.bennee@linaro.org> > > Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com > > It doesn't really address the principle problem that the > boot_linux.py:BootLinuxPPC64.test_pseries_tcg is super heavyweight for > only 2% extra coverage of the executed lines. By re-enabling this test-case we will ensure that PPC64 part of qemu works okay in terms of basic linux boot. Without this we will have a regression in the sense that there won't be any way to test out basic linux boot for PPC64. > > What we really need is a script so we can compare the output between the > two jsons: > > gcovr --json --exclude-unreachable-branches --print-summary -o coverage.json --root ../../ . *.p > > because I suspect we could make up that missing few % noodling the > baseline test a bit more. Can you tell me how you check code coverage with and without this test-case ? I am kind of new to qemu so it would be nice to know how you do this. And I am trying to increase the code coverage by improving the baseline test by including more devices in the qemu-system-ppc64 command line so I would appreciate any tips on how to do that also. > > > --- > > tests/avocado/boot_linux.py | 6 +++++- > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py > > index be30dcbd58..c3869a987c 100644 > > --- a/tests/avocado/boot_linux.py > > +++ b/tests/avocado/boot_linux.py > > @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest): > > :avocado: tags=arch:ppc64 > > """ > > > > + # timeout for downloading new VM image. > > timeout = 360 > > > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > > def test_pseries_tcg(self): > > """ > > :avocado: tags=machine:pseries > > @@ -101,6 +101,10 @@ def test_pseries_tcg(self): > > """ > > self.require_accelerator("tcg") > > self.vm.add_args("-accel", "tcg") > > + > > + # timeout for actual Linux PPC boot test > > + self.timeout = 360 > > + > > self.launch_and_wait(set_up_ssh_connection=False) > > > -- > Alex Bennée > Virtualisation Tech Lead @ Linaro >
Kautuk Consul <kconsul@linux.vnet.ibm.com> writes: > On 2023-03-27 17:07:30, Alex Bennée wrote: >> >> Kautuk Consul <kconsul@linux.vnet.ibm.com> writes: >> >> > Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI"). >> > >> > Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case >> > for PPC64. On investigation, this turns out to be an issue with the >> > time taken for downloading the Fedora 31 qcow2 image being included >> > within the test-case timeout. >> > Re-enable this test-case by setting the timeout to 360 seconds just >> > before launching the downloaded VM image. >> > >> > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> > Reported-by: Alex Bennée <alex.bennee@linaro.org> >> > Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com >> >> It doesn't really address the principle problem that the >> boot_linux.py:BootLinuxPPC64.test_pseries_tcg is super heavyweight for >> only 2% extra coverage of the executed lines. > By re-enabling this test-case we will ensure that PPC64 part of qemu > works okay in terms of basic linux boot. Without this we will have > a regression in the sense that there won't be any way to test out > basic linux boot for PPC64. Sure we do: ➜ ./tests/venv/bin/avocado list ./tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_p INSTRUMENTED ./tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc32 INSTRUMENTED ./tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64 INSTRUMENTED ./tests/avocado/tuxrun_baselines.py:TuxRunBaselineTest.test_ppc64le boot 3 different ppc configurations. >> >> What we really need is a script so we can compare the output between the >> two jsons: >> >> gcovr --json --exclude-unreachable-branches --print-summary -o coverage.json --root ../../ . *.p >> >> because I suspect we could make up that missing few % noodling the >> baseline test a bit more. > Can you tell me how you check code coverage with and without this > test-case ? I use two build directories, both configured with --enable-gcov. e.g.: ../../configure' '--disable-docs' '--enable-gcov' '--target-list=ppc64-softmmu' and run a different set of tests in each build dir. You can then run: make coverage-html V=1 for the initial report. See: https://qemu.readthedocs.io/en/latest/devel/testing.html#gcc-gcov-support > I am kind of new to qemu so it would be nice to know how you > do this. And I am trying to increase the code coverage by improving > the baseline test by including more devices in the qemu-system-ppc64 > command line so I would appreciate any tips on how to do that also. The only problem is eyeballing the html reports is a very fuzzy way of comparing coverage. However the gcovr report generates some useful machine readable json which could be compared with a script. >> >> > --- >> > tests/avocado/boot_linux.py | 6 +++++- >> > 1 file changed, 5 insertions(+), 1 deletion(-) >> > >> > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py >> > index be30dcbd58..c3869a987c 100644 >> > --- a/tests/avocado/boot_linux.py >> > +++ b/tests/avocado/boot_linux.py >> > @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest): >> > :avocado: tags=arch:ppc64 >> > """ >> > >> > + # timeout for downloading new VM image. >> > timeout = 360 >> > >> > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') >> > def test_pseries_tcg(self): >> > """ >> > :avocado: tags=machine:pseries >> > @@ -101,6 +101,10 @@ def test_pseries_tcg(self): >> > """ >> > self.require_accelerator("tcg") >> > self.vm.add_args("-accel", "tcg") >> > + >> > + # timeout for actual Linux PPC boot test >> > + self.timeout = 360 >> > + >> > self.launch_and_wait(set_up_ssh_connection=False) >> >> >> -- >> Alex Bennée >> Virtualisation Tech Lead @ Linaro >>
Hello, [ Copying qemu-ppc@ and Daniel ] On 3/28/23 13:24, Kautuk Consul wrote: > On 2023-03-27 17:07:30, Alex Bennée wrote: >> >> Kautuk Consul <kconsul@linux.vnet.ibm.com> writes: >> >>> Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI"). >>> >>> Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case >>> for PPC64. On investigation, this turns out to be an issue with the >>> time taken for downloading the Fedora 31 qcow2 image being included >>> within the test-case timeout. >>> Re-enable this test-case by setting the timeout to 360 seconds just >>> before launching the downloaded VM image. >>> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> Reported-by: Alex Bennée <alex.bennee@linaro.org> >>> Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com >> >> It doesn't really address the principle problem that the >> boot_linux.py:BootLinuxPPC64.test_pseries_tcg is super heavyweight for >> only 2% extra coverage of the executed lines. > By re-enabling this test-case we will ensure that PPC64 part of qemu > works okay in terms of basic linux boot. Without this we will have > a regression in the sense that there won't be any way to test out > basic linux boot for PPC64. There are ways and pseries is not only PPC64 machine. There is more to it. See : https://github.com/legoater/qemu-ppc-boot/tree/main/buildroot https://github.com/legoater/buildroot/tree/qemu-ppc/board/qemu QEMU PPC maintainers have external tools for regressions which are run regularly, at least before sending a PR for upstream. Thanks, C. >> >> What we really need is a script so we can compare the output between the >> two jsons: >> >> gcovr --json --exclude-unreachable-branches --print-summary -o coverage.json --root ../../ . *.p >> >> because I suspect we could make up that missing few % noodling the >> baseline test a bit more. > Can you tell me how you check code coverage with and without this > test-case ? I am kind of new to qemu so it would be nice to know how you > do this. And I am trying to increase the code coverage by improving > the baseline test by including more devices in the qemu-system-ppc64 > command line so I would appreciate any tips on how to do that also. >> >>> --- >>> tests/avocado/boot_linux.py | 6 +++++- >>> 1 file changed, 5 insertions(+), 1 deletion(-) >>> >>> diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py >>> index be30dcbd58..c3869a987c 100644 >>> --- a/tests/avocado/boot_linux.py >>> +++ b/tests/avocado/boot_linux.py >>> @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest): >>> :avocado: tags=arch:ppc64 >>> """ >>> >>> + # timeout for downloading new VM image. >>> timeout = 360 >>> >>> - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') >>> def test_pseries_tcg(self): >>> """ >>> :avocado: tags=machine:pseries >>> @@ -101,6 +101,10 @@ def test_pseries_tcg(self): >>> """ >>> self.require_accelerator("tcg") >>> self.vm.add_args("-accel", "tcg") >>> + >>> + # timeout for actual Linux PPC boot test >>> + self.timeout = 360 >>> + >>> self.launch_and_wait(set_up_ssh_connection=False) >> >> >> -- >> Alex Bennée >> Virtualisation Tech Lead @ Linaro >> >
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py index be30dcbd58..c3869a987c 100644 --- a/tests/avocado/boot_linux.py +++ b/tests/avocado/boot_linux.py @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest): :avocado: tags=arch:ppc64 """ + # timeout for downloading new VM image. timeout = 360 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_pseries_tcg(self): """ :avocado: tags=machine:pseries @@ -101,6 +101,10 @@ def test_pseries_tcg(self): """ self.require_accelerator("tcg") self.vm.add_args("-accel", "tcg") + + # timeout for actual Linux PPC boot test + self.timeout = 360 + self.launch_and_wait(set_up_ssh_connection=False)
Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI"). Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case for PPC64. On investigation, this turns out to be an issue with the time taken for downloading the Fedora 31 qcow2 image being included within the test-case timeout. Re-enable this test-case by setting the timeout to 360 seconds just before launching the downloaded VM image. Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> Reported-by: Alex Bennée <alex.bennee@linaro.org> Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com --- tests/avocado/boot_linux.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)