| Message ID | 20200325102131.23270-3-vsementsov@virtuozzo.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Rework iotests finding | expand |
On 25.03.20 11:21, Vladimir Sementsov-Ogievskiy wrote: > Add python script with new logic of searching for tests: > > Old behavior: > - tests are named [0-9][0-9][0-9] > - tests must be registered in group file (even if test doesn't belong > to any group, like 142) > > New behavior: > - group file is dropped > - tests are searched by file-name instead of group file, so it's not > needed more to "register the test", just create it with name > test-*. Old names like [0-9][0-9][0-9] are supported too, but not > recommended for new tests > - groups are parsed from '# group: ' line inside test files > - optional file group.local may be used to define some additional > groups for downstreams > - 'disabled' group is used to temporary disable tests. So instead of > commenting tests in old 'group' file you now can add them to > disabled group with help of 'group.local' file > > Benefits: > - no rebase conflicts in group file on patch porting from branch to > branch > - no conflicts in upstream, when different series want to occupy same > test number > - meaningful names for test files > For example, with digital number, when some person wants to add some > test about block-stream, he most probably will just create a new > test. But if there would be test-block-stream test already, he will > at first look at it and may be just add a test-case into it. > And anyway meaningful names are better and test-* notation is > already used in tests directory. > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> > --- > docs/devel/testing.rst | 51 +++++- > tests/qemu-iotests/check | 20 +-- > tests/qemu-iotests/find_tests.py | 72 ++++++++ > tests/qemu-iotests/group | 298 ------------------------------- > 4 files changed, 132 insertions(+), 309 deletions(-) > create mode 100755 tests/qemu-iotests/find_tests.py > delete mode 100644 tests/qemu-iotests/group [...] > diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check > index f7a2d3d6c3..09b2ced2f0 100755 > --- a/tests/qemu-iotests/check > +++ b/tests/qemu-iotests/check > @@ -168,9 +168,7 @@ do > if $group > then > # arg after -g > - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ > -s/ .*//p > -}') > + group_list=$(./find_tests.py "$r") > if [ -z "$group_list" ] > then > echo "Group \"$r\" is empty or not defined?" > @@ -193,10 +191,8 @@ s/ .*//p > then > # arg after -x > # Populate $tmp.list with all tests > - awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null > - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ > -s/ .*//p > -}') > + ./find_tests.py > $tmp.list 2>/dev/null > + group_list=$(./find_tests.py "$r") > if [ -z "$group_list" ] > then > echo "Group \"$r\" is empty or not defined?" > @@ -486,10 +482,14 @@ testlist options > fi > ;; > > - *) > + [0-9]*) > start=$r > end=$r > ;; > + *) Should this be test-*, or do we really want to allow any filename here? > + echo $r >> $tmp.list > + xpand=false > + ;; > > esac > > @@ -504,7 +504,7 @@ testlist options > BEGIN { for (t='$start'; t<='$end'; t++) printf "%03d\n",t }' \ > | while read id > do > - if grep -s "^$id\( \|\$\)" "$source_iotests/group" >/dev/null > + if (./find_tests.py | grep "$id") I... Actually have no idea what this loop is supposed to do, but couldn’t we cache the find_tests.py output? > then > # in group file ... OK > echo $id >>$tmp.list > @@ -566,7 +566,7 @@ else > touch $tmp.list > else > # no test numbers, do everything from group file > - sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p' <"$source_iotests/group" >$tmp.list > + find_tests.py > $tmp.list > fi > fi > The modifications to check seem a bit too tame to me. Can’t we do some more radical changes to drastically reduce the complexity of the check script and move it to the new Python script? Do we need the whole code to handle a number of groups and excluded groups there? Can’t we just give the Python scripts a list of groups to include and a list of groups to exclude and let it handle the rest? > diff --git a/tests/qemu-iotests/find_tests.py b/tests/qemu-iotests/find_tests.py > new file mode 100755 > index 0000000000..5de0615ebc > --- /dev/null > +++ b/tests/qemu-iotests/find_tests.py > @@ -0,0 +1,72 @@ > +#!/usr/bin/env python3 > + > +import os > +import glob > +from collections import defaultdict > + > + > +class TestFinder: > + def __init__(self): > + self.groups = defaultdict(set) > + self.all_tests = glob.glob('[0-9][0-9][0-9]') > + > + self.all_tests += [f for f in glob.iglob('test-*') > + if not f.endswith('.out')] > + > + for t in self.all_tests: > + with open(t) as f: > + for line in f: > + if line.startswith('# group: '): > + for g in line.split()[2:]: > + self.groups[g].add(t) > + > + def add_group_file(self, fname): > + with open(fname) as f: > + for line in f: > + line = line.strip() > + > + if (not line) or line[0] == '#': > + continue > + > + words = line.split() > + test_file = words[0] > + groups = words[1:] > + > + if test_file not in self.all_tests: > + print('Warning: {}: "{}" test is not found. ' > + 'Skip.'.format(fname, test_file)) > + continue > + > + for g in groups: > + self.groups[g].add(test_file)> + > + def find_tests(self, group=None): > + if group is None: > + tests = self.all_tests Should we exclude the disabled group here? > + elif group not in self.groups: > + tests = [] > + elif group != 'disabled' and 'disabled' in self.groups: > + tests = self.groups[group] - self.groups['disabled'] > + else: > + tests = self.groups[group] > + > + return sorted(tests) > + > + > +if __name__ == '__main__': > + import sys > + > + if len(sys.argv) > 2: > + print("Usage ./find_tests.py [group]") > + sys.exit(1) > + > + tf = TestFinder() > + if os.path.isfile('group'): > + tf.add_group_file('group') So is it “group” or ”group.local”? :) Max > + > + if len(sys.argv) == 2: > + tests = tf.find_tests(sys.argv[1]) > + else: > + tests = tf.find_tests() > + > + print('\n'.join(tests))
06.04.2020 16:02, Max Reitz wrote: > On 25.03.20 11:21, Vladimir Sementsov-Ogievskiy wrote: >> Add python script with new logic of searching for tests: >> >> Old behavior: >> - tests are named [0-9][0-9][0-9] >> - tests must be registered in group file (even if test doesn't belong >> to any group, like 142) >> >> New behavior: >> - group file is dropped >> - tests are searched by file-name instead of group file, so it's not >> needed more to "register the test", just create it with name >> test-*. Old names like [0-9][0-9][0-9] are supported too, but not >> recommended for new tests >> - groups are parsed from '# group: ' line inside test files >> - optional file group.local may be used to define some additional >> groups for downstreams >> - 'disabled' group is used to temporary disable tests. So instead of >> commenting tests in old 'group' file you now can add them to >> disabled group with help of 'group.local' file >> >> Benefits: >> - no rebase conflicts in group file on patch porting from branch to >> branch >> - no conflicts in upstream, when different series want to occupy same >> test number >> - meaningful names for test files >> For example, with digital number, when some person wants to add some >> test about block-stream, he most probably will just create a new >> test. But if there would be test-block-stream test already, he will >> at first look at it and may be just add a test-case into it. >> And anyway meaningful names are better and test-* notation is >> already used in tests directory. >> >> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> >> --- >> docs/devel/testing.rst | 51 +++++- >> tests/qemu-iotests/check | 20 +-- >> tests/qemu-iotests/find_tests.py | 72 ++++++++ >> tests/qemu-iotests/group | 298 ------------------------------- >> 4 files changed, 132 insertions(+), 309 deletions(-) >> create mode 100755 tests/qemu-iotests/find_tests.py >> delete mode 100644 tests/qemu-iotests/group > > [...] > >> diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check >> index f7a2d3d6c3..09b2ced2f0 100755 >> --- a/tests/qemu-iotests/check >> +++ b/tests/qemu-iotests/check >> @@ -168,9 +168,7 @@ do >> if $group >> then >> # arg after -g >> - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ >> -s/ .*//p >> -}') >> + group_list=$(./find_tests.py "$r") >> if [ -z "$group_list" ] >> then >> echo "Group \"$r\" is empty or not defined?" >> @@ -193,10 +191,8 @@ s/ .*//p >> then >> # arg after -x >> # Populate $tmp.list with all tests >> - awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null >> - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ >> -s/ .*//p >> -}') >> + ./find_tests.py > $tmp.list 2>/dev/null >> + group_list=$(./find_tests.py "$r") >> if [ -z "$group_list" ] >> then >> echo "Group \"$r\" is empty or not defined?" >> @@ -486,10 +482,14 @@ testlist options >> fi >> ;; >> >> - *) >> + [0-9]*) >> start=$r >> end=$r >> ;; >> + *) > > Should this be test-*, or do we really want to allow any filename here? > >> + echo $r >> $tmp.list >> + xpand=false >> + ;; >> >> esac >> >> @@ -504,7 +504,7 @@ testlist options >> BEGIN { for (t='$start'; t<='$end'; t++) printf "%03d\n",t }' \ >> | while read id >> do >> - if grep -s "^$id\( \|\$\)" "$source_iotests/group" >/dev/null >> + if (./find_tests.py | grep "$id") > > I... Actually have no idea what this loop is supposed to do, but > couldn’t we cache the find_tests.py output? > >> then >> # in group file ... OK >> echo $id >>$tmp.list >> @@ -566,7 +566,7 @@ else >> touch $tmp.list >> else >> # no test numbers, do everything from group file >> - sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p' <"$source_iotests/group" >$tmp.list >> + find_tests.py > $tmp.list >> fi >> fi >> > > The modifications to check seem a bit too tame to me. Can’t we do some > more radical changes to drastically reduce the complexity of the check > script and move it to the new Python script? Do we need the whole code > to handle a number of groups and excluded groups there? Can’t we just > give the Python scripts a list of groups to include and a list of groups > to exclude and let it handle the rest? Okay.. It is inevitable. I will) > >> diff --git a/tests/qemu-iotests/find_tests.py b/tests/qemu-iotests/find_tests.py >> new file mode 100755 >> index 0000000000..5de0615ebc >> --- /dev/null >> +++ b/tests/qemu-iotests/find_tests.py >> @@ -0,0 +1,72 @@ >> +#!/usr/bin/env python3 >> + >> +import os >> +import glob >> +from collections import defaultdict >> + >> + >> +class TestFinder: >> + def __init__(self): >> + self.groups = defaultdict(set) >> + self.all_tests = glob.glob('[0-9][0-9][0-9]') >> + >> + self.all_tests += [f for f in glob.iglob('test-*') >> + if not f.endswith('.out')] >> + >> + for t in self.all_tests: >> + with open(t) as f: >> + for line in f: >> + if line.startswith('# group: '): >> + for g in line.split()[2:]: >> + self.groups[g].add(t) >> + >> + def add_group_file(self, fname): >> + with open(fname) as f: >> + for line in f: >> + line = line.strip() >> + >> + if (not line) or line[0] == '#': >> + continue >> + >> + words = line.split() >> + test_file = words[0] >> + groups = words[1:] >> + >> + if test_file not in self.all_tests: >> + print('Warning: {}: "{}" test is not found. ' >> + 'Skip.'.format(fname, test_file)) >> + continue >> + >> + for g in groups: >> + self.groups[g].add(test_file)> + >> + def find_tests(self, group=None): >> + if group is None: >> + tests = self.all_tests > > Should we exclude the disabled group here? Hmm, yes we should. > >> + elif group not in self.groups: >> + tests = [] >> + elif group != 'disabled' and 'disabled' in self.groups: >> + tests = self.groups[group] - self.groups['disabled'] >> + else: >> + tests = self.groups[group] >> + >> + return sorted(tests) >> + >> + >> +if __name__ == '__main__': >> + import sys >> + >> + if len(sys.argv) > 2: >> + print("Usage ./find_tests.py [group]") >> + sys.exit(1) >> + >> + tf = TestFinder() >> + if os.path.isfile('group'): >> + tf.add_group_file('group') > > So is it “group” or ”group.local”? :) oops. > > Max > >> + >> + if len(sys.argv) == 2: >> + tests = tf.find_tests(sys.argv[1]) >> + else: >> + tests = tf.find_tests() >> + >> + print('\n'.join(tests)) >
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 770a987ea4..5532f1abe4 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -153,7 +153,7 @@ check-block ----------- ``make check-block`` runs a subset of the block layer iotests (the tests that -are in the "auto" group in ``tests/qemu-iotests/group``). +are in the "auto" group). See the "QEMU iotests" section below for more information. GCC gcov support @@ -267,6 +267,55 @@ another application on the host may have locked the file, possibly leading to a test failure. If using such devices are explicitly desired, consider adding ``locking=off`` option to disable image locking. +Test case groups +---------------- + +Test may belong to some groups, you may define it in the comment inside the +test. By convention, test groups are listed in the second line of the test +file, after "#!/..." line, like this: + +.. code:: + + #!/usr/bin/env python3 + # group: auto quick + # + ... + +Additional way of defining groups is creating tests/qemu-iotests/group.local +file. This should be used only for downstream (this file should never appear +in upstream). This file may be used for defining some downstream test groups +or for temporary disable tests, like this: + +.. code:: + + # groups for some company downstream process + # + # ci - tests to run on build + # down - our downstream tests, not for upstream + # + # Format of each line is: + + 013 ci + 210 disabled + 215 disabled + test-our-ugly-workaround down ci + +The following groups are defined: + +- quick : Tests in this group should finish within some few seconds. + +- img : Tests in this group can be used to excercise the qemu-img tool. + +- auto : Tests in this group are used during "make check" and should be + runnable in any case. That means they should run with every QEMU binary + (also non-x86), with every QEMU configuration (i.e. must not fail if + an optional feature is not compiled in - but reporting a "skip" is ok), + work at least with the qcow2 file format, work with all kind of host + filesystems and users (e.g. "nobody" or "root") and must not take too + much memory and disk space (since CI pipelines tend to fail otherwise). + +- disabled : Tests in this group are disabled and ignored by check. + .. _docker-ref: Docker based tests diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check index f7a2d3d6c3..09b2ced2f0 100755 --- a/tests/qemu-iotests/check +++ b/tests/qemu-iotests/check @@ -168,9 +168,7 @@ do if $group then # arg after -g - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ -s/ .*//p -}') + group_list=$(./find_tests.py "$r") if [ -z "$group_list" ] then echo "Group \"$r\" is empty or not defined?" @@ -193,10 +191,8 @@ s/ .*//p then # arg after -x # Populate $tmp.list with all tests - awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null - group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{ -s/ .*//p -}') + ./find_tests.py > $tmp.list 2>/dev/null + group_list=$(./find_tests.py "$r") if [ -z "$group_list" ] then echo "Group \"$r\" is empty or not defined?" @@ -486,10 +482,14 @@ testlist options fi ;; - *) + [0-9]*) start=$r end=$r ;; + *) + echo $r >> $tmp.list + xpand=false + ;; esac @@ -504,7 +504,7 @@ testlist options BEGIN { for (t='$start'; t<='$end'; t++) printf "%03d\n",t }' \ | while read id do - if grep -s "^$id\( \|\$\)" "$source_iotests/group" >/dev/null + if (./find_tests.py | grep "$id") then # in group file ... OK echo $id >>$tmp.list @@ -566,7 +566,7 @@ else touch $tmp.list else # no test numbers, do everything from group file - sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p' <"$source_iotests/group" >$tmp.list + find_tests.py > $tmp.list fi fi diff --git a/tests/qemu-iotests/find_tests.py b/tests/qemu-iotests/find_tests.py new file mode 100755 index 0000000000..5de0615ebc --- /dev/null +++ b/tests/qemu-iotests/find_tests.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 + +import os +import glob +from collections import defaultdict + + +class TestFinder: + def __init__(self): + self.groups = defaultdict(set) + self.all_tests = glob.glob('[0-9][0-9][0-9]') + + self.all_tests += [f for f in glob.iglob('test-*') + if not f.endswith('.out')] + + for t in self.all_tests: + with open(t) as f: + for line in f: + if line.startswith('# group: '): + for g in line.split()[2:]: + self.groups[g].add(t) + + def add_group_file(self, fname): + with open(fname) as f: + for line in f: + line = line.strip() + + if (not line) or line[0] == '#': + continue + + words = line.split() + test_file = words[0] + groups = words[1:] + + if test_file not in self.all_tests: + print('Warning: {}: "{}" test is not found. ' + 'Skip.'.format(fname, test_file)) + continue + + for g in groups: + self.groups[g].add(test_file) + + def find_tests(self, group=None): + if group is None: + tests = self.all_tests + elif group not in self.groups: + tests = [] + elif group != 'disabled' and 'disabled' in self.groups: + tests = self.groups[group] - self.groups['disabled'] + else: + tests = self.groups[group] + + return sorted(tests) + + +if __name__ == '__main__': + import sys + + if len(sys.argv) > 2: + print("Usage ./find_tests.py [group]") + sys.exit(1) + + tf = TestFinder() + if os.path.isfile('group'): + tf.add_group_file('group') + + if len(sys.argv) == 2: + tests = tf.find_tests(sys.argv[1]) + else: + tests = tf.find_tests() + + print('\n'.join(tests)) diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group deleted file mode 100644 index 79c6dfc85d..0000000000 --- a/tests/qemu-iotests/group +++ /dev/null @@ -1,298 +0,0 @@ -# -# QA groups control file -# Defines test groups -# -# Some notes about the groups: -# -# - do not start group names with a digit -# -# - quick : Tests in this group should finish within some few seconds. -# -# - img : Tests in this group can be used to excercise the qemu-img tool. -# -# - auto : Tests in this group are used during "make check" and should be -# runnable in any case. That means they should run with every QEMU binary -# (also non-x86), with every QEMU configuration (i.e. must not fail if -# an optional feature is not compiled in - but reporting a "skip" is ok), -# work at least with the qcow2 file format, work with all kind of host -# filesystems and users (e.g. "nobody" or "root") and must not take too -# much memory and disk space (since CI pipelines tend to fail otherwise). -# - -# -# test-group association ... one line per test -# -001 rw auto quick -002 rw auto quick -003 rw auto -004 rw auto quick -005 img auto quick -# 006 was removed, do not reuse -007 snapshot auto -008 rw auto quick -009 rw auto quick -010 rw auto quick -011 rw auto quick -012 auto quick -013 rw auto -014 rw -015 rw snapshot -# 016 was removed, do not reuse -017 rw backing auto quick -018 rw backing auto quick -019 rw backing auto quick -020 rw backing auto quick -021 io auto quick -022 rw snapshot auto -023 rw -024 rw backing auto quick -025 rw auto quick -026 rw blkdbg -027 rw auto quick -028 rw backing quick -029 rw auto quick -030 rw auto backing -031 rw auto quick -032 rw auto quick -033 rw auto quick -034 rw auto backing quick -035 rw auto quick -036 rw auto quick -037 rw auto backing quick -038 rw auto backing quick -039 rw auto quick -040 rw auto -041 rw auto backing -042 rw auto quick -043 rw auto backing -044 rw -045 rw quick -046 rw auto aio quick -047 rw auto quick -048 img auto quick -049 rw auto -050 rw auto backing quick -051 rw -052 rw auto backing quick -053 rw auto quick -054 rw auto quick -055 rw -056 rw backing -057 rw -058 rw quick -059 rw quick -060 rw auto quick -061 rw auto -062 rw auto quick -063 rw auto quick -064 rw quick -065 rw quick -066 rw auto quick -067 rw quick -068 rw quick -069 rw auto quick -070 rw quick -071 rw auto quick -072 rw auto quick -073 rw auto quick -074 rw auto quick -075 rw quick -076 io -077 rw quick -078 rw quick -079 rw auto -080 rw auto -081 rw quick -082 rw quick -083 rw -084 img quick -085 rw -086 rw auto quick -087 rw quick -088 rw quick -089 rw auto quick -090 rw auto quick -091 rw migration -092 rw quick -093 throttle -094 rw quick -095 rw quick -096 rw quick -097 rw auto backing -098 rw auto backing quick -099 rw auto quick -# 100 was removed, do not reuse -101 rw quick -102 rw quick -103 rw auto quick -104 rw auto -105 rw auto quick -106 rw quick -107 rw auto quick -108 rw auto quick -109 rw -110 rw auto backing quick -111 rw auto quick -112 rw -113 rw quick -114 rw auto quick -115 rw -116 rw quick -117 rw auto -118 rw -119 rw quick -120 rw auto quick -121 rw -122 rw -123 rw quick -124 rw backing -125 rw -126 rw auto backing -127 rw auto backing quick -128 rw quick -129 rw quick -130 rw quick -131 rw quick -132 rw quick -133 auto quick -134 rw auto quick -135 rw -136 rw -137 rw auto -138 rw auto quick -139 rw quick -140 rw auto quick -141 rw auto quick -142 -143 auto quick -144 rw quick -145 quick -146 quick -147 img -148 rw quick -149 rw sudo -150 rw auto quick -151 rw -152 rw quick -153 rw quick -154 rw auto backing quick -155 rw -156 rw auto quick -157 quick -158 rw auto quick -159 rw auto quick -160 rw quick -161 rw auto quick -162 quick -163 rw -165 rw quick -169 rw quick migration -170 rw auto quick -171 rw quick -172 auto -173 rw -174 auto -175 quick -176 rw auto backing -177 rw auto quick -178 img -179 rw auto quick -181 rw auto migration -182 rw quick -183 rw migration -184 rw auto quick -185 rw -186 rw auto -187 rw auto -188 rw quick -189 rw -190 rw auto quick -191 rw auto -192 rw auto quick -194 rw migration quick -195 rw auto quick -196 rw quick migration -197 rw quick -198 rw -199 rw migration -200 rw -201 rw migration -202 rw quick -203 rw auto migration -204 rw quick -205 rw quick -206 rw -207 rw -208 rw quick -209 rw quick -210 rw -211 rw quick -212 rw quick -213 rw quick -214 rw auto -215 rw quick -216 rw quick -217 rw auto quick -218 rw quick -219 rw -220 rw auto -221 rw quick -222 rw quick -223 rw quick -224 rw quick -225 rw quick -226 auto quick -227 quick -228 rw quick -229 auto quick -231 quick -232 quick -233 quick -234 quick migration -235 quick -236 quick -237 rw quick -238 quick -239 rw quick -240 quick -241 rw quick -242 rw quick -243 rw quick -244 rw auto quick -245 rw -246 rw quick -247 rw quick -248 rw quick -249 rw auto quick -250 rw quick -251 rw auto quick -252 rw auto backing quick -253 rw quick -254 rw backing quick -255 rw quick -256 rw auto quick -257 rw -258 rw quick -259 rw auto quick -260 rw quick -261 rw -262 rw quick migration -263 rw quick -264 rw -265 rw auto quick -266 rw quick -267 rw auto quick snapshot -268 rw auto quick -270 rw backing quick -272 rw -273 backing quick -277 rw quick -279 rw backing quick -280 rw migration quick -281 rw quick -282 rw img quick -283 auto quick -284 rw -286 rw quick -288 quick -289 rw quick
Add python script with new logic of searching for tests: Old behavior: - tests are named [0-9][0-9][0-9] - tests must be registered in group file (even if test doesn't belong to any group, like 142) New behavior: - group file is dropped - tests are searched by file-name instead of group file, so it's not needed more to "register the test", just create it with name test-*. Old names like [0-9][0-9][0-9] are supported too, but not recommended for new tests - groups are parsed from '# group: ' line inside test files - optional file group.local may be used to define some additional groups for downstreams - 'disabled' group is used to temporary disable tests. So instead of commenting tests in old 'group' file you now can add them to disabled group with help of 'group.local' file Benefits: - no rebase conflicts in group file on patch porting from branch to branch - no conflicts in upstream, when different series want to occupy same test number - meaningful names for test files For example, with digital number, when some person wants to add some test about block-stream, he most probably will just create a new test. But if there would be test-block-stream test already, he will at first look at it and may be just add a test-case into it. And anyway meaningful names are better and test-* notation is already used in tests directory. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> --- docs/devel/testing.rst | 51 +++++- tests/qemu-iotests/check | 20 +-- tests/qemu-iotests/find_tests.py | 72 ++++++++ tests/qemu-iotests/group | 298 ------------------------------- 4 files changed, 132 insertions(+), 309 deletions(-) create mode 100755 tests/qemu-iotests/find_tests.py delete mode 100644 tests/qemu-iotests/group