| Message ID | 20241211-pks-meson-ci-v1-4-28d18b494374@pks.im (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | ci: wire up support for Meson | expand |
Patrick Steinhardt <ps@pks.im> writes: > It is quite easy for the list of integration tests to go out-of-sync > without anybody noticing. Introduce a new configure-time check that > verifies that all tests are wired up properly. > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > --- > t/meson.build | 36 ++++++++++++++++++++++++++++++++++++ > 1 file changed, 36 insertions(+) > > diff --git a/t/meson.build b/t/meson.build > index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644 > --- a/t/meson.build > +++ b/t/meson.build > @@ -1092,6 +1092,42 @@ integration_tests = [ > 't9903-bash-prompt.sh', > ] > > +# Sanity check that we are not missing any tests present in 't/'. This check > +# only runs once at configure time and is thus best-effort, only. It is > +# sufficient to catch missing test suites in our CI though. > +foreach glob, tests : { > + 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests, > + 'unit-tests/t-*.c': unit_test_programs, > + 'unit-tests/u-*.c': clar_test_suites, > +} > + actual_tests = run_command(shell, '-c', 'ls ' + glob, > + check: true, > + env: script_environment, > + ).stdout().strip().split('\n') > + > + if tests != actual_tests > + missing_tests = [ ] > + foreach actual_test : actual_tests > + if actual_test not in tests > + missing_tests += actual_test > + endif > + endforeach > + if missing_tests.length() > 0 > + error('Missing tests:\n\n - ' + '\n - '.join(missing_tests)) This gives nice output: $ touch t/unit-tests/u-bar.c t/unit-tests/u-foo.c $ meson setup builddir --reconfigure The Meson build system Version: 1.4.1 [snip] Configuring update.sample using configuration Configuring exclude using configuration t/meson.build:1116:6: ERROR: Problem encountered: Missing tests: - unit-tests/u-bar.c - unit-tests/u-foo.c A full log can be found at git/builddir/meson-logs/meson-log.txt But I think the error message is a little bit confusing. It sounds like the test file is missing, but it's the configuration of the test that is missing. Now I've realized it hard to write a good error message here. But I would suggest something like "Tests files found, but not configured". > + endif > + > + superfluous_tests = [ ] > + foreach integration_test : tests > + if integration_test not in actual_tests > + superfluous_tests += integration_test > + endif > + endforeach > + if superfluous_tests.length() > 0 > + error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests)) Also here I would suggest different error message, maybe something like "Tests configured, but file not found"
On Fri, Dec 13, 2024 at 10:58:47AM +0100, Toon Claes wrote: > Patrick Steinhardt <ps@pks.im> writes: > > diff --git a/t/meson.build b/t/meson.build > > index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644 > > --- a/t/meson.build > > +++ b/t/meson.build > > @@ -1092,6 +1092,42 @@ integration_tests = [ > > 't9903-bash-prompt.sh', > > ] > > > > +# Sanity check that we are not missing any tests present in 't/'. This check > > +# only runs once at configure time and is thus best-effort, only. It is > > +# sufficient to catch missing test suites in our CI though. > > +foreach glob, tests : { > > + 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests, > > + 'unit-tests/t-*.c': unit_test_programs, > > + 'unit-tests/u-*.c': clar_test_suites, > > +} > > + actual_tests = run_command(shell, '-c', 'ls ' + glob, > > + check: true, > > + env: script_environment, > > + ).stdout().strip().split('\n') > > + > > + if tests != actual_tests > > + missing_tests = [ ] > > + foreach actual_test : actual_tests > > + if actual_test not in tests > > + missing_tests += actual_test > > + endif > > + endforeach > > + if missing_tests.length() > 0 > > + error('Missing tests:\n\n - ' + '\n - '.join(missing_tests)) > > This gives nice output: > > $ touch t/unit-tests/u-bar.c t/unit-tests/u-foo.c > > $ meson setup builddir --reconfigure > > The Meson build system > Version: 1.4.1 > [snip] > Configuring update.sample using configuration > Configuring exclude using configuration > > t/meson.build:1116:6: ERROR: Problem encountered: Missing tests: > > - unit-tests/u-bar.c > - unit-tests/u-foo.c > > A full log can be found at git/builddir/meson-logs/meson-log.txt > > But I think the error message is a little bit confusing. It sounds like > the test file is missing, but it's the configuration of the test that is > missing. > > Now I've realized it hard to write a good error message here. But I > would suggest something like "Tests files found, but not configured". > > > + endif > > + > > + superfluous_tests = [ ] > > + foreach integration_test : tests > > + if integration_test not in actual_tests > > + superfluous_tests += integration_test > > + endif > > + endforeach > > + if superfluous_tests.length() > 0 > > + error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests)) > > Also here I would suggest different error message, maybe something like > "Tests configured, but file not found" Both good suggestions, thanks! Patrick
diff --git a/t/meson.build b/t/meson.build index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644 --- a/t/meson.build +++ b/t/meson.build @@ -1092,6 +1092,42 @@ integration_tests = [ 't9903-bash-prompt.sh', ] +# Sanity check that we are not missing any tests present in 't/'. This check +# only runs once at configure time and is thus best-effort, only. It is +# sufficient to catch missing test suites in our CI though. +foreach glob, tests : { + 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests, + 'unit-tests/t-*.c': unit_test_programs, + 'unit-tests/u-*.c': clar_test_suites, +} + actual_tests = run_command(shell, '-c', 'ls ' + glob, + check: true, + env: script_environment, + ).stdout().strip().split('\n') + + if tests != actual_tests + missing_tests = [ ] + foreach actual_test : actual_tests + if actual_test not in tests + missing_tests += actual_test + endif + endforeach + if missing_tests.length() > 0 + error('Missing tests:\n\n - ' + '\n - '.join(missing_tests)) + endif + + superfluous_tests = [ ] + foreach integration_test : tests + if integration_test not in actual_tests + superfluous_tests += integration_test + endif + endforeach + if superfluous_tests.length() > 0 + error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests)) + endif + endif +endforeach + # GIT_BUILD_DIR needs to be Unix-style without drive prefixes as it get added # to the PATH variable. And given that drive prefixes contain a colon we'd # otherwise end up with a broken PATH if we didn't convert it.
It is quite easy for the list of integration tests to go out-of-sync without anybody noticing. Introduce a new configure-time check that verifies that all tests are wired up properly. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- t/meson.build | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)