[v3,1/7] t7063: correct broken test expectation

Commit Message

Linus Arver via GitGitGadget March 25, 2020, 7:31 p.m. UTC

From: Elijah Newren <[e-mail]>

The untracked cache is caching wrong information, resulting in commands
like `git status --porcelain` producing erroneous answers.  The tests in
t7063 actually have a wide enough test to catch a relevant case, in
particular surrounding the directory 'dthree/', but it appears the
answers were not checked quite closely enough and the tests were coded
with the wrong expectation.  Once the wrong info got into the cache in
an early test, since later tests built on it, many others have a wrong
expectation as well.  This affects just over a third of the tests in
t7063.

The error can be seen starting at t7063.12 (the first one switched from
expect_success to expect_failure in this patch).  That test runs in a
directory with the following files present:
  done/one
  dthree/three
  dtwo/two
  four
  .gitignore
  one
  three
  two

Of those files, the following files are tracked:
  done/one
  one
  two

and the contents of .gitignore are:
  four

and the contents of .git/info/exclude are:
  three

And there is no core.excludesfile.  Therefore, the following should be
untracked:
  .gitignore
  dthree/
  dtwo/
Indeed, these three paths are reported if you run
  git ls-files -o --directory --exclude-standard
within this directory.  However, 'git status --porcelain' was reporting
for this test:
  A  done/one
  A  one
  A  two
  ?? .gitignore
  ?? dtwo/
which was clearly wrong -- dthree/ should also be listed as untracked.
This appears to have been broken since the test was introduced with
commit a3ddcefd97 ("t7063: tests for untracked cache", 2015-03-08).
Correct the test to expect the right output, marking the test as failed
for now.  Make the same change throughout the remainder of the testsuite
to reflect that dthree/ remains an untracked directory throughout and
should be recognized as such.

Signed-off-by: Elijah Newren <[e-mail]>
---
 t/t7063-status-untracked-cache.sh | 51 ++++++++++++++++++++-----------
 1 file changed, 33 insertions(+), 18 deletions(-)

Comments

Derrick Stolee March 26, 2020, 1:02 p.m. UTC | #1

On 3/25/2020 3:31 PM, Elijah Newren via GitGitGadget wrote:
> From: Elijah Newren <[e-mail]>
> 
> The untracked cache is caching wrong information, resulting in commands
> like `git status --porcelain` producing erroneous answers.  The tests in
> t7063 actually have a wide enough test to catch a relevant case, in
> particular surrounding the directory 'dthree/', but it appears the
> answers were not checked quite closely enough and the tests were coded
> with the wrong expectation.  Once the wrong info got into the cache in
> an early test, since later tests built on it, many others have a wrong
> expectation as well.  This affects just over a third of the tests in
> t7063.

Wow. Good find.

> The error can be seen starting at t7063.12 (the first one switched from
> expect_success to expect_failure in this patch).  That test runs in a
> directory with the following files present:
>   done/one
>   dthree/three
>   dtwo/two
>   four
>   .gitignore
>   one
>   three
>   two
> 
> Of those files, the following files are tracked:
>   done/one
>   one
>   two
> 
> and the contents of .gitignore are:
>   four
> 
> and the contents of .git/info/exclude are:
>   three
> 
> And there is no core.excludesfile.  Therefore, the following should be
> untracked:
>   .gitignore
>   dthree/
>   dtwo/
> Indeed, these three paths are reported if you run
>   git ls-files -o --directory --exclude-standard
> within this directory.  However, 'git status --porcelain' was reporting
> for this test:
>   A  done/one
>   A  one
>   A  two
>   ?? .gitignore
>   ?? dtwo/
> which was clearly wrong -- dthree/ should also be listed as untracked.
> This appears to have been broken since the test was introduced with
> commit a3ddcefd97 ("t7063: tests for untracked cache", 2015-03-08).
> Correct the test to expect the right output, marking the test as failed
> for now.  Make the same change throughout the remainder of the testsuite
> to reflect that dthree/ remains an untracked directory throughout and
> should be recognized as such.

I wonder if we could simultaneously verify these "expected" results match
using another command without the untracked cache? It's good that we have
the expected outputs explicitly, but perhaps double-checking the command
with `-c core.untrackedCache=false` would help us know these are the correct
expected outputs?

Thanks,
-Stolee

Elijah Newren March 26, 2020, 9:18 p.m. UTC | #2

On Thu, Mar 26, 2020 at 6:02 AM Derrick Stolee <[e-mail]> wrote:
>
> On 3/25/2020 3:31 PM, Elijah Newren via GitGitGadget wrote:
> > From: Elijah Newren <[e-mail]>
> >
> > The untracked cache is caching wrong information, resulting in commands
> > like `git status --porcelain` producing erroneous answers.  The tests in
> > t7063 actually have a wide enough test to catch a relevant case, in
> > particular surrounding the directory 'dthree/', but it appears the
> > answers were not checked quite closely enough and the tests were coded
> > with the wrong expectation.  Once the wrong info got into the cache in
> > an early test, since later tests built on it, many others have a wrong
> > expectation as well.  This affects just over a third of the tests in
> > t7063.
>
> Wow. Good find.

or maybe not...

> > The error can be seen starting at t7063.12 (the first one switched from
> > expect_success to expect_failure in this patch).  That test runs in a
> > directory with the following files present:
> >   done/one
> >   dthree/three
> >   dtwo/two
> >   four
> >   .gitignore
> >   one
> >   three
> >   two
> >
> > Of those files, the following files are tracked:
> >   done/one
> >   one
> >   two
> >
> > and the contents of .gitignore are:
> >   four
> >
> > and the contents of .git/info/exclude are:
> >   three
> >
> > And there is no core.excludesfile.  Therefore, the following should be
> > untracked:
> >   .gitignore
> >   dthree/
> >   dtwo/
> > Indeed, these three paths are reported if you run
> >   git ls-files -o --directory --exclude-standard
> > within this directory.  However, 'git status --porcelain' was reporting
> > for this test:
> >   A  done/one
> >   A  one
> >   A  two
> >   ?? .gitignore
> >   ?? dtwo/
> > which was clearly wrong -- dthree/ should also be listed as untracked.
> > This appears to have been broken since the test was introduced with
> > commit a3ddcefd97 ("t7063: tests for untracked cache", 2015-03-08).
> > Correct the test to expect the right output, marking the test as failed
> > for now.  Make the same change throughout the remainder of the testsuite
> > to reflect that dthree/ remains an untracked directory throughout and
> > should be recognized as such.
>
> I wonder if we could simultaneously verify these "expected" results match
> using another command without the untracked cache? It's good that we have
> the expected outputs explicitly, but perhaps double-checking the command
> with `-c core.untrackedCache=false` would help us know these are the correct
> expected outputs?

This was an *awesome* idea, even if the implementation doesn't quite
work.  It turns out that -c core.untrackedCache=false does not
instruct status to ignore the untracked cache, it instructs status to
delete it. Since we had subsequent tests that depended on the
untrackedCache created in previous tests, this would break a number of
tests.  But I can introduce a helper to workaround that:

# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can
# compare commands side-by-side, e.g.
#    iuc status --porcelain >expect &&
#    git status --porcelain >actual &&
#    test_cmp expect actual
iuc() {
        git ls-files -s >../current-index-entries
        git ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries

        GIT_INDEX_FILE=.git/tmp_index
        export GIT_INDEX_FILE
        git update-index --index-info <../current-index-entries
        git update-index --skip-worktree $(cat ../current-sparse-entries)

        git -c core.untrackedCache=false "$@"
        ret=$?

        rm ../current-index-entries
        rm $GIT_INDEX_FILE
        unset GIT_INDEX_FILE

        return $ret
}


Doing that helped me discover that the test didn't have a wrong
expectation; I did.  When a directory that is not tracked is filled
entirely with files that are ignored, then status --porcelain treats
the directory itself as ignored...and thus doesn't display it.  (`git
status --porcelain --ignored` will show it).  I had seen that
somewhere, but hadn't fully understood the check_only and
stop_at_first_file pieces related to it.  Anyway, with this helpful
hint:

  * I can say that there was not a bug in the untracked cache (at
least not any that I'm aware of)
  * I can update my first patch to do more thorough checking instead
of changing the expectation
  * I found the bug in my final patch that had been evading me
  * I added a huge comment explaining check_only and
stop_at_first_file, how they're used, and what they mean for the
future reader
  * I also no longer need to partially disable the untracked cache in
my changes.

New patches incoming...

Patch

diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh
index 190ae149cf3..41705ec1526 100755
--- a/t/t7063-status-untracked-cache.sh
+++ b/t/t7063-status-untracked-cache.sh
@@ -224,7 +224,7 @@  EOF
 	test_cmp ../expect ../actual
 '
 
-test_expect_success 'new info/exclude invalidates everything' '
+test_expect_failure 'new info/exclude invalidates everything' '
 	avoid_racy &&
 	echo three >>.git/info/exclude &&
 	: >../trace &&
@@ -235,6 +235,7 @@  A  done/one
 A  one
 A  two
 ?? .gitignore
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../actual &&
@@ -247,7 +248,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'verify untracked cache dump' '
+test_expect_failure 'verify untracked cache dump' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -256,6 +257,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 /done/ 0000000000000000000000000000000000000000 recurse valid
 /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid
@@ -282,7 +284,7 @@  EOF
 	test_cmp ../expect ../actual
 '
 
-test_expect_success 'status after the move' '
+test_expect_failure 'status after the move' '
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
 	git status --porcelain >../actual &&
@@ -290,6 +292,7 @@  test_expect_success 'status after the move' '
 A  done/one
 A  one
 ?? .gitignore
+?? dthree/
 ?? dtwo/
 ?? two
 EOF
@@ -303,7 +306,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'verify untracked cache dump' '
+test_expect_failure 'verify untracked cache dump' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -312,6 +315,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 two
 /done/ 0000000000000000000000000000000000000000 recurse valid
@@ -339,7 +343,7 @@  EOF
 	test_cmp ../expect ../actual
 '
 
-test_expect_success 'status after the move' '
+test_expect_failure 'status after the move' '
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
 	git status --porcelain >../actual &&
@@ -348,6 +352,7 @@  A  done/one
 A  one
 A  two
 ?? .gitignore
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../actual &&
@@ -360,7 +365,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'verify untracked cache dump' '
+test_expect_failure 'verify untracked cache dump' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -369,6 +374,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 /done/ 0000000000000000000000000000000000000000 recurse valid
 /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid
@@ -386,12 +392,13 @@  test_expect_success 'set up for sparse checkout testing' '
 	git commit -m "first commit"
 '
 
-test_expect_success 'status after commit' '
+test_expect_failure 'status after commit' '
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
 	git status --porcelain >../actual &&
 	cat >../status.expect <<EOF &&
 ?? .gitignore
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../actual &&
@@ -404,7 +411,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'untracked cache correct after commit' '
+test_expect_failure 'untracked cache correct after commit' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -413,6 +420,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 /done/ 0000000000000000000000000000000000000000 recurse valid
 /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid
@@ -442,7 +450,7 @@  test_expect_success 'create/modify files, some of which are gitignored' '
 	sync_mtime
 '
 
-test_expect_success 'test sparse status with untracked cache' '
+test_expect_failure 'test sparse status with untracked cache' '
 	: >../trace &&
 	avoid_racy &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
@@ -451,6 +459,7 @@  test_expect_success 'test sparse status with untracked cache' '
  M done/two
 ?? .gitignore
 ?? done/five
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../status.actual &&
@@ -463,7 +472,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'untracked cache correct after status' '
+test_expect_failure 'untracked cache correct after status' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -472,6 +481,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 /done/ 1946f0437f90c5005533cbe1736a6451ca301714 recurse valid
 five
@@ -482,7 +492,7 @@  EOF
 	test_cmp ../expect ../actual
 '
 
-test_expect_success 'test sparse status again with untracked cache' '
+test_expect_failure 'test sparse status again with untracked cache' '
 	avoid_racy &&
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
@@ -491,6 +501,7 @@  test_expect_success 'test sparse status again with untracked cache' '
  M done/two
 ?? .gitignore
 ?? done/five
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../status.actual &&
@@ -509,7 +520,7 @@  test_expect_success 'set up for test of subdir and sparse checkouts' '
 	echo "sub" > done/sub/sub/file
 '
 
-test_expect_success 'test sparse status with untracked cache and subdir' '
+test_expect_failure 'test sparse status with untracked cache and subdir' '
 	avoid_racy &&
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
@@ -519,6 +530,7 @@  test_expect_success 'test sparse status with untracked cache and subdir' '
 ?? .gitignore
 ?? done/five
 ?? done/sub/
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../status.actual &&
@@ -531,7 +543,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'verify untracked cache dump (sparse/subdirs)' '
+test_expect_failure 'verify untracked cache dump (sparse/subdirs)' '
 	test-tool dump-untracked-cache >../actual &&
 	cat >../expect-from-test-dump <<EOF &&
 info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0
@@ -540,6 +552,7 @@  exclude_per_dir .gitignore
 flags 00000006
 / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid
 .gitignore
+dthree/
 dtwo/
 /done/ 1946f0437f90c5005533cbe1736a6451ca301714 recurse valid
 five
@@ -555,7 +568,7 @@  EOF
 	test_cmp ../expect-from-test-dump ../actual
 '
 
-test_expect_success 'test sparse status again with untracked cache and subdir' '
+test_expect_failure 'test sparse status again with untracked cache and subdir' '
 	avoid_racy &&
 	: >../trace &&
 	GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \
@@ -570,7 +583,7 @@  EOF
 	test_cmp ../trace.expect ../trace
 '
 
-test_expect_success 'move entry in subdir from untracked to cached' '
+test_expect_failure 'move entry in subdir from untracked to cached' '
 	git add dtwo/two &&
 	git status --porcelain >../status.actual &&
 	cat >../status.expect <<EOF &&
@@ -579,11 +592,12 @@  A  dtwo/two
 ?? .gitignore
 ?? done/five
 ?? done/sub/
+?? dthree/
 EOF
 	test_cmp ../status.expect ../status.actual
 '
 
-test_expect_success 'move entry in subdir from cached to untracked' '
+test_expect_failure 'move entry in subdir from cached to untracked' '
 	git rm --cached dtwo/two &&
 	git status --porcelain >../status.actual &&
 	cat >../status.expect <<EOF &&
@@ -591,6 +605,7 @@  test_expect_success 'move entry in subdir from cached to untracked' '
 ?? .gitignore
 ?? done/five
 ?? done/sub/
+?? dthree/
 ?? dtwo/
 EOF
 	test_cmp ../status.expect ../status.actual
@@ -609,7 +624,7 @@  test_expect_success 'git status does not change anything' '
 	test_cmp ../expect-no-uc ../actual
 '
 
-test_expect_success 'setting core.untrackedCache to true and using git status creates the cache' '
+test_expect_failure 'setting core.untrackedCache to true and using git status creates the cache' '
 	git config core.untrackedCache true &&
 	test-tool dump-untracked-cache >../actual &&
 	test_cmp ../expect-no-uc ../actual &&
@@ -642,7 +657,7 @@  test_expect_success 'using --untracked-cache does not fail when core.untrackedCa
 	test_cmp ../expect-empty ../actual
 '
 
-test_expect_success 'setting core.untrackedCache to keep' '
+test_expect_failure 'setting core.untrackedCache to keep' '
 	git config core.untrackedCache keep &&
 	git update-index --untracked-cache &&
 	test-tool dump-untracked-cache >../actual &&