[i-g-t,3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

Commit Message

Lespiau, Damien March 26, 2013, 3:24 p.m. UTC

There are two ways to run tests, directly with make check/test
or through piglit.

When IGT_SIMULATION is set to '1', we substitute the list of tests in
those two code paths with carefully selected tests. The stress tests and
other horrors are left to torture the real hardware as they don't make
too much sense in simulation.

Signed-off-by: Damien Lespiau <[e-mail]>
---
 tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

Comments

Daniel Vetter March 26, 2013, 3:38 p.m. UTC | #1

On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> There are two ways to run tests, directly with make check/test
> or through piglit.
> 
> When IGT_SIMULATION is set to '1', we substitute the list of tests in
> those two code paths with carefully selected tests. The stress tests and
> other horrors are left to torture the real hardware as they don't make
> too much sense in simulation.
> 
> Signed-off-by: Damien Lespiau <[e-mail]>

This feels a bit fragile since even just now I often fail to put a test
newly converted to the subtest stuff into the right single/multi make
target list. Also, this way we exclude a test by default, which also feels
like the wrong way round (most of the tests we're adding aren't heavy
stress-tests, but more excercise some corner-case).

So what about a sprinkling

drmtest_skip_on_simulation();

over the remaining testcases instead, which just calls exit(77);? Only
caveat is that for subtests we need to put it into the right spot to not
break subtest enumeration in piglit (it's a bit fragile).

Cheers, Daniel

> ---
>  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 40 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 2fddfe8..33db962 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -137,22 +137,59 @@ kernel_tests = \
>  	$(multi_kernel_tests) \
>  	$(NULL)
>  
> +# We maintain here a list of tests to run in our simulation environments
> +single_simulation_tests = \
> +	gem_mmap \
> +	gem_pread_after_blit \
> +	gem_ring_sync_loop \
> +	gem_ctx_basic \
> +	gem_pipe_control_store_loop \
> +	gem_storedw_loop_render \
> +	gem_storedw_loop_blt \
> +	gem_storedw_loop_bsd \
> +	gem_render_linear_blits \
> +	gem_tiled_blits \
> +	gem_cpu_reloc \
> +	$(NULL)
> +
> +multi_simulation_tests = \
> +	gem_exec_nop \
> +	gem_mmap_gtt \
> +	$(NULL)
> +
> +simulation_tests = \
> +	$(single_simulation_tests) \
> +	$(multi_simulation_tests) \
> +	$(NULL)
> +
>  TESTS = \
>  	$(NULL)
>  
>  test:
>  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
>  	@./check_drm_clients
> -	@make TESTS="${kernel_tests}" check
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		make TESTS="${simulation_tests}" check; \
> +	else \
> +		make TESTS="${kernel_tests}" check; \
> +	fi
>  
>  list-single-tests:
>  	@echo TESTLIST
> -	@echo ${single_kernel_tests}
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		echo ${single_simulation_tests}; \
> +	else \
> +		echo ${single_kernel_tests}; \
> +	fi
>  	@echo END TESTLIST
>  
>  list-multi-tests:
>  	@echo TESTLIST
> -	@echo ${multi_kernel_tests}
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		echo ${multi_simulation_tests}; \
> +	else \
> +		echo ${multi_kernel_tests}; \
> +	fi
>  	@echo END TESTLIST
>  
>  HANG = \
> -- 
> 1.7.11.7
> 
> _______________________________________________
> Intel-gfx mailing list
> [e-mail]
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Ben Widawsky March 26, 2013, 5 p.m. UTC | #2

On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > There are two ways to run tests, directly with make check/test
> > or through piglit.
> > 
> > When IGT_SIMULATION is set to '1', we substitute the list of tests in
> > those two code paths with carefully selected tests. The stress tests and
> > other horrors are left to torture the real hardware as they don't make
> > too much sense in simulation.
> > 
> > Signed-off-by: Damien Lespiau <[e-mail]>
> 
> This feels a bit fragile since even just now I often fail to put a test
> newly converted to the subtest stuff into the right single/multi make
> target list. Also, this way we exclude a test by default, which also feels
> like the wrong way round (most of the tests we're adding aren't heavy
> stress-tests, but more excercise some corner-case).
> 
> So what about a sprinkling
> 
> drmtest_skip_on_simulation();
> 
> over the remaining testcases instead, which just calls exit(77);? Only
> caveat is that for subtests we need to put it into the right spot to not
> break subtest enumeration in piglit (it's a bit fragile).
> 
> Cheers, Daniel

I prefer Damien's explicit definitions in a file. It makes it much
easier to see exactly what's run, and add a test as needed. Since
simulation testing will always be a massive subset of the whole i-g-t
suite, and we'll probably only rarely add or remove a test, I think it's
not a big concern that we might miss a test.

> 
> > ---
> >  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 40 insertions(+), 3 deletions(-)
> > 
> > diff --git a/tests/Makefile.am b/tests/Makefile.am
> > index 2fddfe8..33db962 100644
> > --- a/tests/Makefile.am
> > +++ b/tests/Makefile.am
> > @@ -137,22 +137,59 @@ kernel_tests = \
> >  	$(multi_kernel_tests) \
> >  	$(NULL)
> >  
> > +# We maintain here a list of tests to run in our simulation environments
> > +single_simulation_tests = \
> > +	gem_mmap \
> > +	gem_pread_after_blit \
> > +	gem_ring_sync_loop \
> > +	gem_ctx_basic \
> > +	gem_pipe_control_store_loop \
> > +	gem_storedw_loop_render \
> > +	gem_storedw_loop_blt \
> > +	gem_storedw_loop_bsd \
> > +	gem_render_linear_blits \
> > +	gem_tiled_blits \
> > +	gem_cpu_reloc \
> > +	$(NULL)
> > +
> > +multi_simulation_tests = \
> > +	gem_exec_nop \
> > +	gem_mmap_gtt \
> > +	$(NULL)
> > +
> > +simulation_tests = \
> > +	$(single_simulation_tests) \
> > +	$(multi_simulation_tests) \
> > +	$(NULL)
> > +
> >  TESTS = \
> >  	$(NULL)
> >  
> >  test:
> >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> >  	@./check_drm_clients
> > -	@make TESTS="${kernel_tests}" check
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		make TESTS="${simulation_tests}" check; \
> > +	else \
> > +		make TESTS="${kernel_tests}" check; \
> > +	fi
> >  
> >  list-single-tests:
> >  	@echo TESTLIST
> > -	@echo ${single_kernel_tests}
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		echo ${single_simulation_tests}; \
> > +	else \
> > +		echo ${single_kernel_tests}; \
> > +	fi
> >  	@echo END TESTLIST
> >  
> >  list-multi-tests:
> >  	@echo TESTLIST
> > -	@echo ${multi_kernel_tests}
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		echo ${multi_simulation_tests}; \
> > +	else \
> > +		echo ${multi_kernel_tests}; \
> > +	fi
> >  	@echo END TESTLIST
> >  
> >  HANG = \
> > -- 
> > 1.7.11.7
> > 
> > _______________________________________________
> > Intel-gfx mailing list
> > [e-mail]
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> [phone] - http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> [e-mail]
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Daniel Vetter March 27, 2013, 12:24 a.m. UTC | #3

On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > There are two ways to run tests, directly with make check/test
> > > or through piglit.
> > > 
> > > When IGT_SIMULATION is set to '1', we substitute the list of tests in
> > > those two code paths with carefully selected tests. The stress tests and
> > > other horrors are left to torture the real hardware as they don't make
> > > too much sense in simulation.
> > > 
> > > Signed-off-by: Damien Lespiau <[e-mail]>
> > 
> > This feels a bit fragile since even just now I often fail to put a test
> > newly converted to the subtest stuff into the right single/multi make
> > target list. Also, this way we exclude a test by default, which also feels
> > like the wrong way round (most of the tests we're adding aren't heavy
> > stress-tests, but more excercise some corner-case).
> > 
> > So what about a sprinkling
> > 
> > drmtest_skip_on_simulation();
> > 
> > over the remaining testcases instead, which just calls exit(77);? Only
> > caveat is that for subtests we need to put it into the right spot to not
> > break subtest enumeration in piglit (it's a bit fragile).
> > 
> > Cheers, Daniel
> 
> I prefer Damien's explicit definitions in a file. It makes it much
> easier to see exactly what's run, and add a test as needed. Since
> simulation testing will always be a massive subset of the whole i-g-t
> suite, and we'll probably only rarely add or remove a test, I think it's
> not a big concern that we might miss a test.

Imo that's the wrong approach, since most of the tests we've recently
added exercise corner-cases of our code, and in a rather deterministic
way. If we currently have too many tests to get through all of the useful
ones in a day, we need more machines, not fewer tests.

In the end we want to be able to run full piglit on all this stuff in
simulation after all. Cc'ing Yi so he knows where I'm aiming at ;-)
-Daniel

> 
> > 
> > > ---
> > >  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/tests/Makefile.am b/tests/Makefile.am
> > > index 2fddfe8..33db962 100644
> > > --- a/tests/Makefile.am
> > > +++ b/tests/Makefile.am
> > > @@ -137,22 +137,59 @@ kernel_tests = \
> > >  	$(multi_kernel_tests) \
> > >  	$(NULL)
> > >  
> > > +# We maintain here a list of tests to run in our simulation environments
> > > +single_simulation_tests = \
> > > +	gem_mmap \
> > > +	gem_pread_after_blit \
> > > +	gem_ring_sync_loop \
> > > +	gem_ctx_basic \
> > > +	gem_pipe_control_store_loop \
> > > +	gem_storedw_loop_render \
> > > +	gem_storedw_loop_blt \
> > > +	gem_storedw_loop_bsd \
> > > +	gem_render_linear_blits \
> > > +	gem_tiled_blits \
> > > +	gem_cpu_reloc \
> > > +	$(NULL)
> > > +
> > > +multi_simulation_tests = \
> > > +	gem_exec_nop \
> > > +	gem_mmap_gtt \
> > > +	$(NULL)
> > > +
> > > +simulation_tests = \
> > > +	$(single_simulation_tests) \
> > > +	$(multi_simulation_tests) \
> > > +	$(NULL)
> > > +
> > >  TESTS = \
> > >  	$(NULL)
> > >  
> > >  test:
> > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > >  	@./check_drm_clients
> > > -	@make TESTS="${kernel_tests}" check
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		make TESTS="${simulation_tests}" check; \
> > > +	else \
> > > +		make TESTS="${kernel_tests}" check; \
> > > +	fi
> > >  
> > >  list-single-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${single_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${single_simulation_tests}; \
> > > +	else \
> > > +		echo ${single_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  list-multi-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${multi_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${multi_simulation_tests}; \
> > > +	else \
> > > +		echo ${multi_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  HANG = \
> > > -- 
> > > 1.7.11.7
> > > 
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > [e-mail]
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > 
> > -- 
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > [phone] - http://blog.ffwll.ch
> > _______________________________________________
> > Intel-gfx mailing list
> > [e-mail]
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> -- 
> Ben Widawsky, Intel Open Source Technology Center

Yi Sun March 27, 2013, 2:54 a.m. UTC | #4

Hi Daniel, 

Thank you for your notification. This would make it easy to run sub suite on simulation during this stage. I know the goal is to run the whole i-g-t test case. 
So we can run tests/igt.test just like any other real machines. Ok, I'll add it to our nightly ASAP.

Thanks
    --Sun, Yi


> -----Original Message-----
> From: Daniel Vetter [mailto:[e-mail]] On Behalf Of Daniel Vetter
> Sent: Wednesday, March 27, 2013 8:25 AM
> To: Ben Widawsky
> Cc: Daniel Vetter; Lespiau, Damien; [e-mail]; Sun, Yi
> Subject: Re: [Intel-gfx] [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune
> the list of tests to run
> 
> On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > There are two ways to run tests, directly with make check/test or
> > > > through piglit.
> > > >
> > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > in those two code paths with carefully selected tests. The stress
> > > > tests and other horrors are left to torture the real hardware as
> > > > they don't make too much sense in simulation.
> > > >
> > > > Signed-off-by: Damien Lespiau <[e-mail]>
> > >
> > > This feels a bit fragile since even just now I often fail to put a
> > > test newly converted to the subtest stuff into the right
> > > single/multi make target list. Also, this way we exclude a test by
> > > default, which also feels like the wrong way round (most of the
> > > tests we're adding aren't heavy stress-tests, but more excercise some
> corner-case).
> > >
> > > So what about a sprinkling
> > >
> > > drmtest_skip_on_simulation();
> > >
> > > over the remaining testcases instead, which just calls exit(77);?
> > > Only caveat is that for subtests we need to put it into the right
> > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > >
> > > Cheers, Daniel
> >
> > I prefer Damien's explicit definitions in a file. It makes it much
> > easier to see exactly what's run, and add a test as needed. Since
> > simulation testing will always be a massive subset of the whole i-g-t
> > suite, and we'll probably only rarely add or remove a test, I think
> > it's not a big concern that we might miss a test.
> 
> Imo that's the wrong approach, since most of the tests we've recently added
> exercise corner-cases of our code, and in a rather deterministic way. If we
> currently have too many tests to get through all of the useful ones in a day, we
> need more machines, not fewer tests.
> 
> In the end we want to be able to run full piglit on all this stuff in simulation after
> all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> 
> >
> > >
> > > > ---
> > > >  tests/Makefile.am | 43
> > > > ++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/tests/Makefile.am b/tests/Makefile.am index
> > > > 2fddfe8..33db962 100644
> > > > --- a/tests/Makefile.am
> > > > +++ b/tests/Makefile.am
> > > > @@ -137,22 +137,59 @@ kernel_tests = \
> > > >  	$(multi_kernel_tests) \
> > > >  	$(NULL)
> > > >
> > > > +# We maintain here a list of tests to run in our simulation
> > > > +environments single_simulation_tests = \
> > > > +	gem_mmap \
> > > > +	gem_pread_after_blit \
> > > > +	gem_ring_sync_loop \
> > > > +	gem_ctx_basic \
> > > > +	gem_pipe_control_store_loop \
> > > > +	gem_storedw_loop_render \
> > > > +	gem_storedw_loop_blt \
> > > > +	gem_storedw_loop_bsd \
> > > > +	gem_render_linear_blits \
> > > > +	gem_tiled_blits \
> > > > +	gem_cpu_reloc \
> > > > +	$(NULL)
> > > > +
> > > > +multi_simulation_tests = \
> > > > +	gem_exec_nop \
> > > > +	gem_mmap_gtt \
> > > > +	$(NULL)
> > > > +
> > > > +simulation_tests = \
> > > > +	$(single_simulation_tests) \
> > > > +	$(multi_simulation_tests) \
> > > > +	$(NULL)
> > > > +
> > > >  TESTS = \
> > > >  	$(NULL)
> > > >
> > > >  test:
> > > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > > >  	@./check_drm_clients
> > > > -	@make TESTS="${kernel_tests}" check
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		make TESTS="${simulation_tests}" check; \
> > > > +	else \
> > > > +		make TESTS="${kernel_tests}" check; \
> > > > +	fi
> > > >
> > > >  list-single-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${single_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${single_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${single_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  list-multi-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${multi_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${multi_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${multi_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  HANG = \
> > > > --
> > > > 1.7.11.7
> > > >
> > > > _______________________________________________
> > > > Intel-gfx mailing list
> > > > [e-mail]
> > > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > >
> > > --
> > > Daniel Vetter
> > > Software Engineer, Intel Corporation
> > > [phone] - http://blog.ffwll.ch
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > [e-mail]
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> >
> > --
> > Ben Widawsky, Intel Open Source Technology Center
> 
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> [phone] - http://blog.ffwll.ch

Yi Sun April 1, 2013, 7:21 a.m. UTC | #5

> On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > There are two ways to run tests, directly with make check/test or
> > > > through piglit.
> > > >
> > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > in those two code paths with carefully selected tests. The stress
> > > > tests and other horrors are left to torture the real hardware as
> > > > they don't make too much sense in simulation.
> > > >
> > > > Signed-off-by: Damien Lespiau <[e-mail]>
> > >
> > > This feels a bit fragile since even just now I often fail to put a
> > > test newly converted to the subtest stuff into the right
> > > single/multi make target list. Also, this way we exclude a test by
> > > default, which also feels like the wrong way round (most of the
> > > tests we're adding aren't heavy stress-tests, but more excercise some
> corner-case).
> > >
> > > So what about a sprinkling
> > >
> > > drmtest_skip_on_simulation();
> > >
> > > over the remaining testcases instead, which just calls exit(77);?
> > > Only caveat is that for subtests we need to put it into the right
> > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > >
> > > Cheers, Daniel
> >
> > I prefer Damien's explicit definitions in a file. It makes it much
> > easier to see exactly what's run, and add a test as needed. Since
> > simulation testing will always be a massive subset of the whole i-g-t
> > suite, and we'll probably only rarely add or remove a test, I think
> > it's not a big concern that we might miss a test.
> 
> Imo that's the wrong approach, since most of the tests we've recently added
> exercise corner-cases of our code, and in a rather deterministic way. If we
> currently have too many tests to get through all of the useful ones in a day, we
> need more machines, not fewer tests.
> 
> In the end we want to be able to run full piglit on all this stuff in simulation after
> all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> 
[Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
I noticed Damien's patches aren't on the branch yet.
If we want to split all i-g-t test cases to different HAS, how can we split it?


Thanks
	--Yi Sun

Ben Widawsky April 2, 2013, 5:06 p.m. UTC | #6

On Mon, Apr 01, 2013 at 07:21:59AM +0000, Sun, Yi wrote:
> > On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > > There are two ways to run tests, directly with make check/test or
> > > > > through piglit.
> > > > >
> > > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > > in those two code paths with carefully selected tests. The stress
> > > > > tests and other horrors are left to torture the real hardware as
> > > > > they don't make too much sense in simulation.
> > > > >
> > > > > Signed-off-by: Damien Lespiau <[e-mail]>
> > > >
> > > > This feels a bit fragile since even just now I often fail to put a
> > > > test newly converted to the subtest stuff into the right
> > > > single/multi make target list. Also, this way we exclude a test by
> > > > default, which also feels like the wrong way round (most of the
> > > > tests we're adding aren't heavy stress-tests, but more excercise some
> > corner-case).
> > > >
> > > > So what about a sprinkling
> > > >
> > > > drmtest_skip_on_simulation();
> > > >
> > > > over the remaining testcases instead, which just calls exit(77);?
> > > > Only caveat is that for subtests we need to put it into the right
> > > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > > >
> > > > Cheers, Daniel
> > >
> > > I prefer Damien's explicit definitions in a file. It makes it much
> > > easier to see exactly what's run, and add a test as needed. Since
> > > simulation testing will always be a massive subset of the whole i-g-t
> > > suite, and we'll probably only rarely add or remove a test, I think
> > > it's not a big concern that we might miss a test.
> > 
> > Imo that's the wrong approach, since most of the tests we've recently added
> > exercise corner-cases of our code, and in a rather deterministic way. If we
> > currently have too many tests to get through all of the useful ones in a day, we
> > need more machines, not fewer tests.
> > 
> > In the end we want to be able to run full piglit on all this stuff in simulation after
> > all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> > 
> [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> I noticed Damien's patches aren't on the branch yet.
> If we want to split all i-g-t test cases to different HAS, how can we split it?
> 

Daniel, can you please advise Sun Yi since the solution I like has been
shot down?

Lespiau, Damien April 2, 2013, 5:16 p.m. UTC | #7

On Tue, Apr 02, 2013 at 10:06:37AM -0700, Ben Widawsky wrote:
> > [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> > I noticed Damien's patches aren't on the branch yet.
> > If we want to split all i-g-t test cases to different HAS, how can we split it?
> > 
> 
> Daniel, can you please advise Sun Yi since the solution I like has been
> shot down?

It's not unreasonable to want to run test cases by default and black
list the ones we don't want to run. I'm on holidays soon though and
won't be able to rework this before a little while.

Daniel Vetter April 2, 2013, 5:38 p.m. UTC | #8

On Tue, Apr 02, 2013 at 06:16:59PM +0100, Damien Lespiau wrote:
> On Tue, Apr 02, 2013 at 10:06:37AM -0700, Ben Widawsky wrote:
> > > [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> > > I noticed Damien's patches aren't on the branch yet.
> > > If we want to split all i-g-t test cases to different HAS, how can we split it?
> > > 
> > 
> > Daniel, can you please advise Sun Yi since the solution I like has been
> > shot down?
> 
> It's not unreasonable to want to run test cases by default and black
> list the ones we don't want to run. I'm on holidays soon though and
> won't be able to rework this before a little while.

tbh I'm not set in stone on this issue, and blocking enabling on fulsim
feels bad, too. So please go ahead with whatever approach you deem best.
Since for both approaches the interface for QA/testing would be just a
magic enviroment variable, we could even switch without much fuzz. So I
don't mind if you go ahead with the current patches.
-Daniel

Patch

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2fddfe8..33db962 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -137,22 +137,59 @@  kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+# We maintain here a list of tests to run in our simulation environments
+single_simulation_tests = \
+	gem_mmap \
+	gem_pread_after_blit \
+	gem_ring_sync_loop \
+	gem_ctx_basic \
+	gem_pipe_control_store_loop \
+	gem_storedw_loop_render \
+	gem_storedw_loop_blt \
+	gem_storedw_loop_bsd \
+	gem_render_linear_blits \
+	gem_tiled_blits \
+	gem_cpu_reloc \
+	$(NULL)
+
+multi_simulation_tests = \
+	gem_exec_nop \
+	gem_mmap_gtt \
+	$(NULL)
+
+simulation_tests = \
+	$(single_simulation_tests) \
+	$(multi_simulation_tests) \
+	$(NULL)
+
 TESTS = \
 	$(NULL)
 
 test:
 	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
 	@./check_drm_clients
-	@make TESTS="${kernel_tests}" check
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		make TESTS="${simulation_tests}" check; \
+	else \
+		make TESTS="${kernel_tests}" check; \
+	fi
 
 list-single-tests:
 	@echo TESTLIST
-	@echo ${single_kernel_tests}
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		echo ${single_simulation_tests}; \
+	else \
+		echo ${single_kernel_tests}; \
+	fi
 	@echo END TESTLIST
 
 list-multi-tests:
 	@echo TESTLIST
-	@echo ${multi_kernel_tests}
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		echo ${multi_simulation_tests}; \
+	else \
+		echo ${multi_kernel_tests}; \
+	fi
 	@echo END TESTLIST
 
 HANG = \