| Message ID | 20161013140043.14774-1-chris@chris-wilson.co.uk (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, Oct 13, 2016 at 03:00:43PM +0100, Chris Wilson wrote: > Add the two basic gem_wait tests to the fast list, together they take a > total of 1s (when correctly functioning). > > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> > --- > tests/intel-ci/fast-feedback.testlist | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist > index f59ec98..853b911 100644 > --- a/tests/intel-ci/fast-feedback.testlist > +++ b/tests/intel-ci/fast-feedback.testlist > @@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each > igt@gem_tiled_blits@basic > igt@gem_tiled_fence_blits@basic > igt@gem_tiled_pread_basic > +igt@gem_wait@basic-busy-all > +igt@gem_wait@basic-wait-all > igt@gem_workarounds@basic-read > igt@gvt_basic@invalid-placeholder-test > igt@kms_addfb_basic@addfb25-bad-modifier > -- > 2.9.3 In the future we will test new tests in the CI system. Unfortunately the technology isn't there yet (tm) so this had to be tested manually. Sorry for the delay. For the record, the testing requirements for incoming new tests is converging to: - Tests must be stable. 1000 repeated runs with the new tests alone must give the same results. 10 repeated runs of modified BAT set must give the same results (excluding known issues in other tests). - Tests must be fast. Full modified BAT run must not exceed the time budget. - Tests must pass on current kernels, on some platform. This will be documented when we can get CI do the testing for us on multiple platforms and properly know what kind of requirements can actually be tested. For these tests, manual testing was done on IVB, HSW, and BYT. Stable, fast and passing on all. Acked-by: Petri Latvala <petri.latvala@intel.com> Please push. -- Petri Latvala
On Fri, Oct 28, 2016 at 01:46:51PM +0300, Petri Latvala wrote: > On Thu, Oct 13, 2016 at 03:00:43PM +0100, Chris Wilson wrote: > > Add the two basic gem_wait tests to the fast list, together they take a > > total of 1s (when correctly functioning). > > > > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> > > --- > > tests/intel-ci/fast-feedback.testlist | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist > > index f59ec98..853b911 100644 > > --- a/tests/intel-ci/fast-feedback.testlist > > +++ b/tests/intel-ci/fast-feedback.testlist > > @@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each > > igt@gem_tiled_blits@basic > > igt@gem_tiled_fence_blits@basic > > igt@gem_tiled_pread_basic > > +igt@gem_wait@basic-busy-all > > +igt@gem_wait@basic-wait-all > > igt@gem_workarounds@basic-read > > igt@gvt_basic@invalid-placeholder-test > > igt@kms_addfb_basic@addfb25-bad-modifier > > -- > > 2.9.3 > > > In the future we will test new tests in the CI system. Unfortunately > the technology isn't there yet (tm) so this had to be tested > manually. Sorry for the delay. > > For the record, the testing requirements for incoming new tests is > converging to: > > - Tests must be stable. 1000 repeated runs with the new tests alone > must give the same results. 10 repeated runs of modified BAT set > must give the same results (excluding known issues in other tests). > - Tests must be fast. Full modified BAT run must not exceed the time > budget. > - Tests must pass on current kernels, on some platform. An idea I had in passing... Remove any test from BAT that has not failed in the last 180days of trybot and replace it with a current failing test. BAT is useless if it is not detecting the right class of bugs, those that we typically make mistakes in (i.e. we should optimise BAT for coverage of coding errors). Plus we also look at BAT results pretty frequency and failing tests get fixed faster due to the greater nagging and faster response time of the automated system. (If by the end of the week we haven't fixed the new failing cases, we are slacking!) Some caveats apply, we definitely want to ensure coverage of ABI and the tests must still be selected to fit within a time budget (I hopeful that an expert system backed up by coverage analysis would be extremely invaluable in refining our tests sets). Filtering of trybot results would have to be done, probably with a voting system: select the most frequently failed test cases, and then discount failures in other tests in the same runs (based on the assumption that the same bug was seem by the different tests, given a large enough sample size should ensure good coverage). Just an idea based around tests that always pass are boring (and a waste of time), but tests we occasionally fail are more sensitive to our errors. -Chris
diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist index f59ec98..853b911 100644 --- a/tests/intel-ci/fast-feedback.testlist +++ b/tests/intel-ci/fast-feedback.testlist @@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each igt@gem_tiled_blits@basic igt@gem_tiled_fence_blits@basic igt@gem_tiled_pread_basic +igt@gem_wait@basic-busy-all +igt@gem_wait@basic-wait-all igt@gem_workarounds@basic-read igt@gvt_basic@invalid-placeholder-test igt@kms_addfb_basic@addfb25-bad-modifier
Add the two basic gem_wait tests to the fast list, together they take a total of 1s (when correctly functioning). Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> --- tests/intel-ci/fast-feedback.testlist | 2 ++ 1 file changed, 2 insertions(+)