[VSP-Tests,5/7] vsp-lib: Allow forcing pixel perfect comparisons
diff mbox series

Message ID 20181204155146.9726-6-kieran.bingham@ideasonboard.com
State New
Delegated to: Kieran Bingham
Headers show
Series
  • Reset controls and unloved patches
Related show

Commit Message

Kieran Bingham Dec. 4, 2018, 3:51 p.m. UTC
Provide a means for the tester to request pixel perfect matches on tests.
This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
by passing either '-p' or '--pixel-perfect' on the test command line.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

Comments

Laurent Pinchart Feb. 17, 2019, 11:59 a.m. UTC | #1
Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
> Provide a means for the tester to request pixel perfect matches on tests.
> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
> by passing either '-p' or '--pixel-perfect' on the test command line.

What's the use case for this ?

> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  scripts/vsp-lib.sh | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index cf15a045ea8c..9140254c7459 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -287,7 +287,11 @@ compare_frames() {
>  	params=${params//)/_}
>  	params=$pipe-$in_fmt-$out_fmt-$size$params
>  
> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
> +	# The system can hint when pixel-perfection is not supported,
> +	# however the user can override to force this requirement with
> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
> +	# --pixel-perfect on the commandline.
> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
>  		method=fuzzy
>  	fi
>  
> @@ -1113,10 +1117,15 @@ case $1 in
>  		export VSP_KEEP_FRAMES=1
>  		shift
>  		;;
> +	-p|--pixel-perfect)
> +		export VSP_PIXEL_PERFECT=1
> +		shift
> +		;;
>  	-h|--help)
>  		echo "$(basename $0): VSP Test library"
>  		echo "  -x|--debug          enable shell debug"
>  		echo "  -k|--keep-frames    keep generated and captured frames"
> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
>  		echo "  -h|--help           this help"
>  		exit
>  		shift
Kieran Bingham Feb. 17, 2019, 8:51 p.m. UTC | #2
Hi Laurent,

On 17/02/2019 11:59, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
>> Provide a means for the tester to request pixel perfect matches on tests.
>> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
>> by passing either '-p' or '--pixel-perfect' on the test command line.
> 
> What's the use case for this ?

Some of the tests (i.e. scaling) provide 'fuzzy comparisons' to allow "a
close match" to pass.

It can be desirable to 'turn that off' so that you can generate a set of
frames (using the '-k' option) to diff the expected against the actual
frames.

This helps while improving the system without making changes to the
specific tests.

It could be called 'disable fuzzy matching' if you like?


> 
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  scripts/vsp-lib.sh | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index cf15a045ea8c..9140254c7459 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -287,7 +287,11 @@ compare_frames() {
>>  	params=${params//)/_}
>>  	params=$pipe-$in_fmt-$out_fmt-$size$params
>>  
>> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
>> +	# The system can hint when pixel-perfection is not supported,
>> +	# however the user can override to force this requirement with
>> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
>> +	# --pixel-perfect on the commandline.
>> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
>>  		method=fuzzy
>>  	fi
>>  
>> @@ -1113,10 +1117,15 @@ case $1 in
>>  		export VSP_KEEP_FRAMES=1
>>  		shift
>>  		;;
>> +	-p|--pixel-perfect)
>> +		export VSP_PIXEL_PERFECT=1
>> +		shift
>> +		;;
>>  	-h|--help)
>>  		echo "$(basename $0): VSP Test library"
>>  		echo "  -x|--debug          enable shell debug"
>>  		echo "  -k|--keep-frames    keep generated and captured frames"
>> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
>>  		echo "  -h|--help           this help"
>>  		exit
>>  		shift
>
Laurent Pinchart Feb. 19, 2019, 4:35 p.m. UTC | #3
Hi Kieran,

On Sun, Feb 17, 2019 at 08:51:52PM +0000, Kieran Bingham wrote:
> On 17/02/2019 11:59, Laurent Pinchart wrote:
> > On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
> >> Provide a means for the tester to request pixel perfect matches on tests.
> >> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
> >> by passing either '-p' or '--pixel-perfect' on the test command line.
> > 
> > What's the use case for this ?
> 
> Some of the tests (i.e. scaling) provide 'fuzzy comparisons' to allow "a
> close match" to pass.
> 
> It can be desirable to 'turn that off' so that you can generate a set of
> frames (using the '-k' option) to diff the expected against the actual
> frames.

Ideally we should improve the reference frame generation tool to
generate pixel-perfect frames :-) In the meantime I agree this could be
useful, but I think we should then integrate this better by marking a
test run as reference, saving frames to a good location, and then
reusing them for the subsequent runs. It's a bit too manual in its
proposed form for an automated test suite :-)

> This helps while improving the system without making changes to the
> specific tests.
> 
> It could be called 'disable fuzzy matching' if you like?
> 
> >> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> ---
> >>  scripts/vsp-lib.sh | 11 ++++++++++-
> >>  1 file changed, 10 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> >> index cf15a045ea8c..9140254c7459 100755
> >> --- a/scripts/vsp-lib.sh
> >> +++ b/scripts/vsp-lib.sh
> >> @@ -287,7 +287,11 @@ compare_frames() {
> >>  	params=${params//)/_}
> >>  	params=$pipe-$in_fmt-$out_fmt-$size$params
> >>  
> >> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
> >> +	# The system can hint when pixel-perfection is not supported,
> >> +	# however the user can override to force this requirement with
> >> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
> >> +	# --pixel-perfect on the commandline.
> >> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
> >>  		method=fuzzy
> >>  	fi
> >>  
> >> @@ -1113,10 +1117,15 @@ case $1 in
> >>  		export VSP_KEEP_FRAMES=1
> >>  		shift
> >>  		;;
> >> +	-p|--pixel-perfect)
> >> +		export VSP_PIXEL_PERFECT=1
> >> +		shift
> >> +		;;
> >>  	-h|--help)
> >>  		echo "$(basename $0): VSP Test library"
> >>  		echo "  -x|--debug          enable shell debug"
> >>  		echo "  -k|--keep-frames    keep generated and captured frames"
> >> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
> >>  		echo "  -h|--help           this help"
> >>  		exit
> >>  		shift

Patch
diff mbox series

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index cf15a045ea8c..9140254c7459 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -287,7 +287,11 @@  compare_frames() {
 	params=${params//)/_}
 	params=$pipe-$in_fmt-$out_fmt-$size$params
 
-	if [ x$__vsp_pixel_perfect != xtrue ] ; then
+	# The system can hint when pixel-perfection is not supported,
+	# however the user can override to force this requirement with
+	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
+	# --pixel-perfect on the commandline.
+	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
 		method=fuzzy
 	fi
 
@@ -1113,10 +1117,15 @@  case $1 in
 		export VSP_KEEP_FRAMES=1
 		shift
 		;;
+	-p|--pixel-perfect)
+		export VSP_PIXEL_PERFECT=1
+		shift
+		;;
 	-h|--help)
 		echo "$(basename $0): VSP Test library"
 		echo "  -x|--debug          enable shell debug"
 		echo "  -k|--keep-frames    keep generated and captured frames"
+		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
 		echo "  -h|--help           this help"
 		exit
 		shift