[PULL,13/13] tests/avocado: remove skips from replay_kernel

Commit Message

Alex Bennée Jan. 8, 2024, 3:13 p.m. UTC

With the latest fixes for #2010 and #2013 these tests look pretty
stable now. Of course the only way to be really sure is to run it in
the CI infrastructure and see what breaks.

Acked-by: Pavel Dovgalyuk <[e-mail]>
Signed-off-by: Alex Bennée <[e-mail]>
Message-Id: <[e-mail]>

Comments

Thomas Huth Jan. 11, 2024, 5:19 p.m. UTC | #1

On 08/01/2024 16.13, Alex Bennée wrote:
> With the latest fixes for #2010 and #2013 these tests look pretty
> stable now. Of course the only way to be really sure is to run it in
> the CI infrastructure and see what breaks.
> 
> Acked-by: Pavel Dovgalyuk <[e-mail]>
> Signed-off-by: Alex Bennée <[e-mail]>
> Message-Id: <[e-mail]>

The replay tests seem still to be very flaky, I'm now getting:

  https://gitlab.com/thuth/qemu/-/jobs/5910241580#L227
  https://gitlab.com/thuth/qemu/-/jobs/5910241593#L396

I'd suggest to revert this patch to disable them in the CI again.

  Thomas

Alex Bennée Jan. 12, 2024, 9:35 a.m. UTC | #2

Thomas Huth <[e-mail]> writes:

> On 08/01/2024 16.13, Alex Bennée wrote:
>> With the latest fixes for #2010 and #2013 these tests look pretty
>> stable now. Of course the only way to be really sure is to run it in
>> the CI infrastructure and see what breaks.
>> Acked-by: Pavel Dovgalyuk <[e-mail]>
>> Signed-off-by: Alex Bennée <[e-mail]>
>> Message-Id: <[e-mail]>
>
> The replay tests seem still to be very flaky, I'm now getting:
>
>  https://gitlab.com/thuth/qemu/-/jobs/5910241580#L227
>  https://gitlab.com/thuth/qemu/-/jobs/5910241593#L396
>
> I'd suggest to revert this patch to disable them in the CI again.

Ok I'll move it back to flaky. I thought I'd eliminated the problem as I
can do 100 runs without failure locally. Sadly there doesn't seem to be
anything obvious in the logs as to whats going on.

>
>  Thomas

Alex Bennée Jan. 12, 2024, 1:27 p.m. UTC | #3

Thomas Huth <[e-mail]> writes:

> On 08/01/2024 16.13, Alex Bennée wrote:
>> With the latest fixes for #2010 and #2013 these tests look pretty
>> stable now. Of course the only way to be really sure is to run it in
>> the CI infrastructure and see what breaks.
>> Acked-by: Pavel Dovgalyuk <[e-mail]>
>> Signed-off-by: Alex Bennée <[e-mail]>
>> Message-Id: <[e-mail]>
>
> The replay tests seem still to be very flaky, I'm now getting:
>
>  https://gitlab.com/thuth/qemu/-/jobs/5910241580#L227
>  https://gitlab.com/thuth/qemu/-/jobs/5910241593#L396
>
> I'd suggest to revert this patch to disable them in the CI again.

I did but on reconsideration I'm only partially reverting for the x86_64
bits as everything else looks stable (unless you have more examples).

I've raised a new bug to cover it:

    https://gitlab.com/qemu-project/qemu/-/issues/2094

Patch

diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 1eaa36444cb..6fdcbd6ac3d 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -98,13 +98,10 @@  def test_i386_pc(self):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)
 
-    # See https://gitlab.com/qemu-project/qemu/-/issues/2010
-    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
         :avocado: tags=machine:pc
-        :avocado: tags=flaky
         """
         kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
                       '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
@@ -135,8 +132,6 @@  def test_mips_malta(self):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)
 
-    # See https://gitlab.com/qemu-project/qemu/-/issues/2013
-    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
     def test_mips64el_malta(self):
         """
         This test requires the ar tool to extract "data.tar.gz" from
@@ -152,7 +147,6 @@  def test_mips64el_malta(self):
 
         :avocado: tags=arch:mips64el
         :avocado: tags=machine:malta
-        :avocado: tags=flaky
         """
         deb_url = ('http://snapshot.debian.org/archive/debian/'
                    '20130217T032700Z/pool/main/l/linux-2.6/'
@@ -200,13 +194,10 @@  def test_arm_virt(self):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
 
-    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:cubieboard
-        :avocado: tags=flaky
         """
         deb_url = ('https://apt.armbian.com/pool/main/l/'
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
@@ -354,7 +345,6 @@  def test_m68k_mcf5208evb(self):
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
         self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
 
-    @skip("Test currently broken") # Console stuck as of 5.2-rc1
     def test_microblaze_s3adsp1800(self):
         """
         :avocado: tags=arch:microblaze
@@ -389,7 +379,6 @@  def test_or1k_sim(self):
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
         self.do_test_advcal_2018(file_path, 'vmlinux')
 
-    @skip("nios2 emulation is buggy under record/replay")
     def test_nios2_10m50(self):
         """
         :avocado: tags=arch:nios2