[04/16] Acceptance Tests: keep track and disable tests with known issues

Commit Message

Cleber Rosa Sept. 24, 2021, 6:54 p.m. UTC

This introduces a convention, under which tests with known open
issues, will be automatically excluded from execution via "make
check-acceptance".  It should by itself improve the reliability
of test jobs, even those with rare failures.

Those tests can still be run with:

  ./tests/venv/bin/avocado run $TEST_ID

With the test identification are obtainable via:

  ./tests/venv/bin/avocado list tests/acceptance

And for tests with known issues:

  ./tests/venv/bin/avocado list -t issue tests/acceptance

A list of tests that are enabled (that is, match the currently
configured targets, and do not have known issues) can be obtained
with:

  make list-acceptance

As a future improvement, a tool can be added to sync the "issue" tags
with issues that end up being resolved closed.

Signed-off-by: Cleber Rosa <[e-mail]>
---
 docs/devel/testing.rst                 | 28 ++++++++++++++++++++++++++
 tests/Makefile.include                 |  3 ++-
 tests/acceptance/boot_linux_console.py |  3 +++
 tests/acceptance/machine_rx_gdbsim.py  |  3 +++
 4 files changed, 36 insertions(+), 1 deletion(-)

Patch

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 2818c58ff8..f93a9bed3f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1147,6 +1147,34 @@  variable as shown on the code snippet below to skip the test:
   def test(self):
       do_something()
 
+Disabling tests
+---------------
+
+If you want to have a test temporarily disabled (from ``make
+check-acceptance``), you can add an Avocado ``issue`` tag to the test.  Example::
+
+  class Feature(Test):
+
+      def test(self):
+        """
+        :avocado: tags=issue
+        """
+        do_something()
+
+Rather than simply adding an ``issue`` tag, it may be helpful to point
+to an actual issue on the project's issue tracker::
+
+  class Feature(Test):
+
+      def test(self):
+        """
+        Reason for disabling this test is documented and tracked at
+        https://gitlab.com/qemu-project/qemu/-/issues/9999
+
+        :avocado: tags=issue:9999
+        """
+        do_something()
+
 Uninstalling Avocado
 --------------------
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d1f90572a7..eb3e5d2aa6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,7 +94,8 @@  TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
 AVOCADO_RUNNER=runner
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+comma := ,
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=-issue$(comma)arch:%, $(filter %-softmmu,$(TARGETS)))
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 0a49c0e276..4c824bc161 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -485,6 +485,9 @@  def test_arm_raspi2_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:raspi2b
+
+        # https://gitlab.com/qemu-project/qemu/-/issues/636
+        :avocado: tags=issue:636
         """
         deb_url = ('http://archive.raspberrypi.org/debian/'
                    'pool/main/r/raspberrypi-firmware/'
diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
index 32b737b6d8..62893a9989 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -54,6 +54,9 @@  def test_linux_sash(self):
         :avocado: tags=arch:rx
         :avocado: tags=machine:gdbsim-r5f562n7
         :avocado: tags=endian:little
+
+        # https://gitlab.com/qemu-project/qemu/-/issues/507
+        :avocado: tags=issue:507
         """
         dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
         dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'