| Message ID | 20210123210428.27220-1-vsementsov@virtuozzo.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Rework iotests/check | expand |
Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben: > v8: > > about linters: > > I didn't modify 297, as Max already staged 297 modifications to test all files. > > Also, now I have two complains: > +************* Module testenv > +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) > +************* Module testrunner > +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) > Success: no issues found in 5 source files > > And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can > do some refactoring as a follow-up. I don't think these warning are very helpful, I would agree with disabling them (even globally). When testing this with the other image formats, I found some problems. 1. The first one probably means that we have changed the order of some checks: 150 and 178 have reference outputs for raw and qcow2, but no other formats. Previously, the _supported_fmt line in the test would just skip the test: $ build/check -vhdx 150 178 150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx 178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx Now we seem to test first if a reference output exists and fail: 150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out) 2. Test case 146 for vpc passed previously, it fails now. This seems to be because of whitespace that is checked now. 3. Skipped tests display either "..." or "0.1s" as the elapsed time. The old check implementation didn't display any time for them. I don't really mind either of the three ways, but a consistent result would be nice. Kevin
25.01.2021 19:08, Kevin Wolf wrote: > Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben: >> v8: >> >> about linters: >> >> I didn't modify 297, as Max already staged 297 modifications to test all files. >> >> Also, now I have two complains: >> +************* Module testenv >> +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) >> +************* Module testrunner >> +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) >> Success: no issues found in 5 source files >> >> And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can >> do some refactoring as a follow-up. > > I don't think these warning are very helpful, I would agree with > disabling them (even globally). > > When testing this with the other image formats, I found some problems. > > 1. The first one probably means that we have changed the order of some > checks: 150 and 178 have reference outputs for raw and qcow2, but no > other formats. > > Previously, the _supported_fmt line in the test would just skip the test: > > $ build/check -vhdx 150 178 > 150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > 178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > > Now we seem to test first if a reference output exists and fail: > > 150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) > 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out) > > 2. Test case 146 for vpc passed previously, it fails now. This seems to > be because of whitespace that is checked now. > > 3. Skipped tests display either "..." or "0.1s" as the elapsed time. The > old check implementation didn't display any time for them. I don't > really mind either of the three ways, but a consistent result would > be nice. > Thanks for testing! I should have done it myself actually :/ Will fix soon.
25.01.2021 19:08, Kevin Wolf wrote: > Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben: >> v8: >> >> about linters: >> >> I didn't modify 297, as Max already staged 297 modifications to test all files. >> >> Also, now I have two complains: >> +************* Module testenv >> +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) >> +************* Module testrunner >> +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) >> Success: no issues found in 5 source files >> >> And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can >> do some refactoring as a follow-up. > > I don't think these warning are very helpful, I would agree with > disabling them (even globally). > > When testing this with the other image formats, I found some problems. > > 1. The first one probably means that we have changed the order of some > checks: 150 and 178 have reference outputs for raw and qcow2, but no > other formats. > > Previously, the _supported_fmt line in the test would just skip the test: > > $ build/check -vhdx 150 178 > 150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > 178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > > Now we seem to test first if a reference output exists and fail: > > 150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) > 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out) Hmm. Still, I do think that new order is better: no reason to run the test, when we don't have corresponding .out file. So, may be just change it into "not run", with same "No qualified output (expected ..)" message, what do you think? > > 2. Test case 146 for vpc passed previously, it fails now. This seems to > be because of whitespace that is checked now. > > 3. Skipped tests display either "..." or "0.1s" as the elapsed time. The > old check implementation didn't display any time for them. I don't > really mind either of the three ways, but a consistent result would > be nice. > > Kevin >
Am 25.01.2021 um 17:36 hat Vladimir Sementsov-Ogievskiy geschrieben: > 25.01.2021 19:08, Kevin Wolf wrote: > > Am 23.01.2021 um 22:04 hat Vladimir Sementsov-Ogievskiy geschrieben: > > > v8: > > > > > > about linters: > > > > > > I didn't modify 297, as Max already staged 297 modifications to test all files. > > > > > > Also, now I have two complains: > > > +************* Module testenv > > > +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) > > > +************* Module testrunner > > > +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) > > > Success: no issues found in 5 source files > > > > > > And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can > > > do some refactoring as a follow-up. > > > > I don't think these warning are very helpful, I would agree with > > disabling them (even globally). > > > > When testing this with the other image formats, I found some problems. > > > > 1. The first one probably means that we have changed the order of some > > checks: 150 and 178 have reference outputs for raw and qcow2, but no > > other formats. > > > > Previously, the _supported_fmt line in the test would just skip the test: > > > > $ build/check -vhdx 150 178 > > 150 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > > 178 not run [16:45:46] [16:45:46] not suitable for this image format: vhdx > > > > Now we seem to test first if a reference output exists and fail: > > > > 150 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/150.out) > > 178 fail [16:49:18] [16:49:18] ... No qualified output (expected /home/kwolf/source/qemu/tests/qemu-iotests/178.out) > > > Hmm. Still, I do think that new order is better: no reason to run the > test, when we don't have corresponding .out file. So, may be just > change it into "not run", with same "No qualified output (expected > ..)" message, what do you think? Works for me. (There would actually be a reason to run the test, namely for creating the reference output when you add the test. But this didn't leave a .bad file behind before either, and just doing 'touch 123.out' first is easy enough anyway.) Kevin
v8: about linters: I didn't modify 297, as Max already staged 297 modifications to test all files. Also, now I have two complains: +************* Module testenv +testenv.py:158:4: R0915: Too many statements (53/50) (too-many-statements) +************* Module testrunner +testrunner.py:222:4: R0911: Too many return statements (7/6) (too-many-return-statements) Success: no issues found in 5 source files And I feel, I'm tired to refactor it now.. Probably we can ignore them in 297. Probably I can do some refactoring as a follow-up. findtests: grammar, wording, add Eric's r-b testenv: commit msg adjusted wording, grammar missed env_variables added add logic for searching qemu-system-*, if qemu-system-{arch} is not exist handle valgrind and MALLOC_PERTURB_ fix s/vmkd/vmdk/ testrunner: commit msg adjusted wording fix extra new lines in diff output s/0/fd/ use class instead of namedtuple don't handle MALLOC_PERTURB_ declare self._stack in __init__ change thistime to be float handle KeyboardInterrupt casenotrun init to '' fix "output mismatch .. {f_bad}" to be f-string smarter terminal color handling check: wording, grammar, sheepdoG, exClude_groups change valgrind to be simple boolean option and pass it to TestEnv handle group.local check virtio-blk in iotests.py add --color option Vladimir Sementsov-Ogievskiy (5): iotests: add findtests.py iotests: add testenv.py iotests: add testrunner.py iotests: rewrite check into python iotests: rename and move 169 and 199 tests docs/devel/testing.rst | 50 +- Makefile | 1 - tests/check-block.sh | 2 +- tests/qemu-iotests/check | 1095 ++--------------- tests/qemu-iotests/common.env.in | 3 - tests/qemu-iotests/findtests.py | 159 +++ tests/qemu-iotests/group | 321 ----- tests/qemu-iotests/iotests.py | 8 + tests/qemu-iotests/meson.build | 3 - tests/qemu-iotests/testenv.py | 278 +++++ tests/qemu-iotests/testrunner.py | 366 ++++++ .../migrate-bitmaps-postcopy-test} | 0 .../migrate-bitmaps-postcopy-test.out} | 0 .../{169 => tests/migrate-bitmaps-test} | 0 .../migrate-bitmaps-test.out} | 0 15 files changed, 990 insertions(+), 1296 deletions(-) delete mode 100644 tests/qemu-iotests/common.env.in create mode 100644 tests/qemu-iotests/findtests.py delete mode 100644 tests/qemu-iotests/group create mode 100644 tests/qemu-iotests/testenv.py create mode 100644 tests/qemu-iotests/testrunner.py rename tests/qemu-iotests/{199 => tests/migrate-bitmaps-postcopy-test} (100%) rename tests/qemu-iotests/{199.out => tests/migrate-bitmaps-postcopy-test.out} (100%) rename tests/qemu-iotests/{169 => tests/migrate-bitmaps-test} (100%) rename tests/qemu-iotests/{169.out => tests/migrate-bitmaps-test.out} (100%)