test_cmp: diagnose incorrect arguments
diff mbox series

Message ID 20200809060810.31370-1-sunshine@sunshineco.com
State New
Headers show
Series
  • test_cmp: diagnose incorrect arguments
Related show

Commit Message

Eric Sunshine Aug. 9, 2020, 6:08 a.m. UTC
Under normal circumstances, if a test author misspells a filename passed
to test_cmp(), the error is quickly discovered when the test fails
unexpectedly due to test_cmp() being unable to find the file. However,
if the test is expected to fail, as with test_expect_failure(), a
misspelled filename as argument to test_cmp() will go unnoticed since
the test will indeed fail, but for the wrong reason. Make it easier for
test authors to discover such problems early by sanity-checking the
arguments to test_cmp(). To avoid penalizing all clients of test_cmp()
in the general case, only check for missing files if the comparison
fails.

While at it, make test_cmp_bin() sanity-check its arguments, as well.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
---

Notes:
    This change was motivated by seeing Elijah's patch[1] which fixed
    several cases of bogus test_cmp() invocations which perhaps could
    have been discovered earlier had test_cmp() done some
    sanity-checking of its arguments. It turns out that the tests in
    question[1] fail before test_cmp() is ever called, so this patch
    would not have helped catch those mistakes after all. It also only
    helps catch mistakes in test_expect_failure() cases, which are
    relatively rare compared with test_expect_success() cases, thus may
    not have a lot of value. Nevertheless, it's a relatively small and
    simple change which only kicks in when test_cmp() fails, thus should
    not penalize the typical case; there are a handful of `! test_cmp`
    cases, however, which will trigger the additional argument sanity
    checking but that cost is probably small enough to go unnoticed.
    
    [1]: https://lore.kernel.org/git/7f408b7d4069403b969d334f4940ebf87f1dc797.1596906081.git.gitgitgadget@gmail.com/

 t/test-lib-functions.sh | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

Comments

Shourya Shukla Aug. 9, 2020, 8:32 a.m. UTC | #1
Hello Eric,

>  test_cmp() {
> -	eval "$GIT_TEST_CMP" '"$@"'
> +	test $# -eq 2 || BUG "test_cmp requires two arguments"
> +	if ! eval "$GIT_TEST_CMP" '"$@"'
> +	then
> +		test -e "$1" || BUG "test_cmp 'expect' file missing"
> +		test -e "$2" || BUG "test_cmp 'actual' file missing"
> +		return 1
> +	fi
>  }

I reckon we could be just a little bit more precise here by bugging out
with the exact filename which is missing instead of 'expect' or 'actual'
so that the user has more idea as to what happened. What do you think?

BTW, I looked up the 'test_i18ncmp' function as well and if we have
this small loophole you mentioned above, I think maybe we could make a
similar fix for it too. What I mean is that in case of absence of the
required locale, it should error out kind of like what we did above

    BUG "locale missing"


so that the user it is clear to the user what was the failure point.
Though I will be honest that I have not really encountered a locale related
error or seen what the error looks like, so maybe we can ignore this
suggestion.

Regards,
Shourya Shukla
Eric Sunshine Aug. 9, 2020, 8:49 a.m. UTC | #2
On Sun, Aug 9, 2020 at 4:32 AM Shourya Shukla
<shouryashukla.oo@gmail.com> wrote:
> > +       test -e "$1" || BUG "test_cmp 'expect' file missing"
> > +       test -e "$2" || BUG "test_cmp 'actual' file missing"
>
> I reckon we could be just a little bit more precise here by bugging out
> with the exact filename which is missing instead of 'expect' or 'actual'
> so that the user has more idea as to what happened. What do you think?

Good idea. I'm planning on re-rolling anyhow since Junio pointed out
privately that some callers use "-" (meaning standard input) as one of
the arguments to test_cmp(), so that case ought to be taken into
account, as well.

> BTW, I looked up the 'test_i18ncmp' function as well and if we have
> this small loophole you mentioned above, I think maybe we could make a
> similar fix for it too. What I mean is that in case of absence of the
> required locale, it should error out kind of like what we did above
>
>   BUG "locale missing"
>
> so that the user it is clear to the user what was the failure point.

Sorry, I'm not really sure what you are suggesting, but I'm guessing
that you're misunderstanding of the purpose of test_i18ncmp:

    test_i18ncmp () {
        ! test_have_prereq C_LOCALE_OUTPUT || test_cmp "$@"
    }

which says that test_cmp() should only be called if the current locale
is "C"; in all other cases, we specifically do not want test_cmp() to
be called and instead simply pretend that the test passed.

Patch
diff mbox series

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index b791933ffd..8d77deebd2 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -952,7 +952,13 @@  test_expect_code () {
 # - not all diff versions understand "-u"
 
 test_cmp() {
-	eval "$GIT_TEST_CMP" '"$@"'
+	test $# -eq 2 || BUG "test_cmp requires two arguments"
+	if ! eval "$GIT_TEST_CMP" '"$@"'
+	then
+		test -e "$1" || BUG "test_cmp 'expect' file missing"
+		test -e "$2" || BUG "test_cmp 'actual' file missing"
+		return 1
+	fi
 }
 
 # Check that the given config key has the expected value.
@@ -981,7 +987,13 @@  test_cmp_config() {
 # test_cmp_bin - helper to compare binary files
 
 test_cmp_bin() {
-	cmp "$@"
+	test $# -eq 2 || BUG "test_cmp_bin requires two arguments"
+	if ! cmp "$@"
+	then
+		test -e "$1" || BUG "test_cmp_bin 'expect' file missing"
+		test -e "$2" || BUG "test_cmp_bin 'actual' file missing"
+		return 1
+	fi
 }
 
 # Use this instead of test_cmp to compare files that contain expected and