[0/4] tests/qtest: Move the bulk of migration tests into a separate target

Message

Fabiano Rosas Oct. 17, 2024, 2:32 p.m. UTC

As promised, here are the changes to move most of the migration tests
into a separate check-migration target that is intended to be used by
migration maintainers and contributors.

A few tests will still be kept in another check-migration-quick
target, that will run as part of make check by default. Those are
there to catch changes in generic code that might affect migration and
the rare device code change that affects migration ("rare" because
migration-test runs with only default devices).

The are two main reasons for this:

- Reduce the time of the common 'make check' invocation.

  migration-test is notoriously wasteful in that it runs a migration
  for every test. We don't yet have a comprehensive coverage analysis
  nor the mechanisms for testing isolated parts of the code without
  resorting to a full migration.

  While the migration tests don't account for that much of make check
  time in a parallel run (only about 30s, see table below), I'd like
  to be able to write tests without encumbering every single make
  check run.

- Avoid breaking a developer's make check invocation and also CI when
  only non-migration changes are present.

  The migration tests are subject to races in the code that tend to be
  quite intermittent. Ideally, if a code change does not touch
  migration, it should not break migration-test.

make -j16                   before     after      migration suite included
--------------------------------------------------------------------------
check                       2m28.923s  1m56.178s  migration-quick
check SPEED=slow            6m07.854s  5m59.982s  migration
check SPEED=thorough       10m17.102s 10m02.416s  migration
check-qtest                 2m07.827s  1m24.065s  none
check-qtest SPEED=slow      5m06.149s  4m17.897s  none
check-qtest SPEED=thorough  5m7.601s   4m18.135s  none
check-migration                        3m41.927s  migration
check-migration-quick                  0m15.592s  migration-quick
--------------------------------------------------------------------------

Note that this series could very well be just patch 3 without any new
target, but then to be able to run the full set of migration tests
we'd need to also run all other slow tests from a set, either check or
check-qtest and that would add at least 2 minutest to the migration
run.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1499208871

Fabiano Rosas (4):
  tests/qtest: Add check-migration
  docs: Add migration tests documentation
  tests/qtest/migration: Move tests into g_test_slow()
  ci: Add check-migration-quick to the clang job

 .gitlab-ci.d/buildtest.yml       |   2 +-
 docs/devel/testing/index.rst     |   1 +
 docs/devel/testing/main.rst      |  13 ++
 docs/devel/testing/migration.rst | 275 +++++++++++++++++++++++++++++++
 docs/devel/testing/qtest.rst     |   1 +
 meson.build                      |   6 +-
 tests/Makefile.include           |   2 +
 tests/qtest/meson.build          |  47 +++++-
 tests/qtest/migration-helpers.c  |   9 +
 tests/qtest/migration-helpers.h  |  16 ++
 tests/qtest/migration-test.c     |  67 ++++----
 11 files changed, 392 insertions(+), 47 deletions(-)
 create mode 100644 docs/devel/testing/migration.rst


base-commit: 05adb38839ba656c7383a548b460d95c91e2febe