[0/3] Annonate fstests with possible reasons for failure

Message

Amir Goldstein April 17, 2022, 5:40 p.m. UTC

In a perfect worlds, tests are expected to pass, but we do not live in a
perfect world.

In reality, fstests are often merged before a fix is merged to Linus'
tree or before a fix to all fs is available [1][2].

We all know that day when we pull fstests upstream branch and get a
bunch of new test failures.

This series [3] adds several new ways to annotate tests to provide some
hints for testers about which tests are expected to pass on which fs
and or which kernel versions:

 _fixed_by_kernel_commit
 _known_issue_before_kernel
 _known_issue_on_fs

And a new run option SKIP_KNOWN_ISSUES=yes.

These annotation and run option are inspired by a similar feature in LTP.

These annotation could be used by testers to auto-generate expunge list
for testing stable kernels from the information in _fixed_by_kernel_commit
and _known_issue_before_kernel. This trivial task is left is an excercise.

Thanks,
Amir.

[1] https://lore.kernel.org/fstests/20220412172853.GG16799@magnolia/
[2] https://lore.kernel.org/fstests/20220414155007.GC17014@magnolia/
[3] https://github.com/amir73il/xfstests/commits/hints

Amir Goldstein (3):
  common: support black listing fs in _supported_fs()
  common: print hints for reasons of test failures
  common: add run option to skip tests for known issues

 README            |  2 ++
 check             | 10 ++++++++-
 common/preamble   |  2 +-
 common/rc         | 54 ++++++++++++++++++++++++++++++++++++++++-------
 tests/generic/500 |  3 +--
 tests/generic/631 |  3 +--
 tests/generic/679 |  3 +--
 tests/overlay/009 |  3 +++
 tests/overlay/010 |  3 +++
 tests/overlay/014 |  2 ++
 tests/overlay/016 |  4 +++-
 tests/overlay/017 |  4 +++-
 tests/overlay/018 |  4 +++-
 tests/overlay/020 |  3 +++
 tests/overlay/022 |  2 ++
 tests/overlay/029 |  2 ++
 tests/overlay/038 |  2 ++
 tests/overlay/041 |  2 ++
 tests/overlay/042 |  3 +++
 tests/overlay/043 |  4 +++-
 tests/overlay/044 |  4 +++-
 tests/overlay/054 |  3 +++
 tests/overlay/055 |  3 +++
 tests/overlay/061 |  2 ++
 tests/overlay/063 |  3 +++
 tests/overlay/065 |  4 ++++
 tests/overlay/067 |  3 +++
 tests/overlay/070 |  4 +++-
 tests/overlay/071 |  4 +++-
 tests/overlay/072 |  2 ++
 tests/overlay/074 |  5 +++++
 tests/overlay/077 |  5 +++++
 tests/overlay/078 |  1 +
 33 files changed, 135 insertions(+), 23 deletions(-)