[v3,1/1] fsmonitor: don't fill bitmap with entries to be removed

Commit Message

Linus Arver via GitGitGadget Oct. 11, 2019, 8:11 p.m. UTC

From: William Baker <[e-mail]>

While doing some testing with fsmonitor enabled I found
that git commands would segfault after staging and
unstaging an untracked file.  Looking at the crash it
appeared that fsmonitor_ewah_callback was attempting to
adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that
the fsmonitor extension must have been written with
more bits than there were entries in the index.  The
root cause ended up being that fill_fsmonitor_bitmap was
populating fsmonitor_dirty with bits for all entries in
the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been
updated to skip entries with the the CE_REMOVE flag set.
With this change the bits written for the fsmonitor
extension will be consistent with the index entries
written by do_write_index.  Additionally, BUG checks
have been added to detect if the number of bits in
fsmonitor_dirty should ever exceed the number of
entries in the index again.

Another option that was considered was moving the call
to fill_fsmonitor_bitmap closer to where the index is
written (and where the fsmonitor extension itself is
written).  However, that did not work as the
fsmonitor_dirty bitmap must be filled before the index
is split during writing.

Signed-off-by: William Baker <[e-mail]>
---
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 17 +++++++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 65 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

Comments

Junio C Hamano Oct. 12, 2019, 1:26 a.m. UTC | #1

"William Baker via GitGitGadget" <[e-mail]> writes:

> +# Test staging/unstaging files that appear at the end of the index.  Test
> +# file names begin with 'z' so that they are sorted to the end of the index. 

Well, the test is now done in a freshly created repository, so the
z* files are the only thing you have in here---technically they are
at the end of the index, but so they are at the beginning, too.

Would it affect the effectiveness of the test that you do not have
any other paths in the working tree or in the index, unlike the test
in the previous rounds that did not use a newly created test
repository?  

This is not a rhetorical question, but purely asking. "no, this
still tests what we want to test and shows breakage when the fix to
the code in the patch gets reverted" is perfectly a good answer, but
in that case, is "the end of" the most important trait of the
condition this test is checking?  Wouldn't the bug be exposed as
long as we remove sufficiently large number of entries (like
"removing more paths than the paths still in the index at the end"
or something like that)?

Thanks.

> +test_expect_success 'status succeeds after staging/unstaging ' '
> +	test_create_repo fsmonitor-stage-unstage &&
> +	(
> +		cd fsmonitor-stage-unstage &&
> +		test_commit initial &&
> +		git update-index --fsmonitor &&
> +		removed=$(test_seq 1 100 | sed "s/^/z/") &&
> +		touch $removed &&
> +		git add $removed &&
> +		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
> +		FSMONITOR_LIST="$removed" git restore -S $removed &&
> +		FSMONITOR_LIST="$removed" git status
> +	)
> +'
> +
>  test_done
> diff --git a/t/t7519/fsmonitor-env b/t/t7519/fsmonitor-env
> new file mode 100755
> index 0000000000..8f1f7ab164
> --- /dev/null
> +++ b/t/t7519/fsmonitor-env
> @@ -0,0 +1,24 @@
> +#!/bin/sh
> +#
> +# An test hook script to integrate with git to test fsmonitor.
> +#
> +# The hook is passed a version (currently 1) and a time in nanoseconds
> +# formatted as a string and outputs to stdout all files that have been
> +# modified since the given time. Paths must be relative to the root of
> +# the working tree and separated by a single NUL.
> +#
> +#echo "$0 $*" >&2
> +
> +if test "$#" -ne 2
> +then
> +	echo "$0: exactly 2 arguments expected" >&2
> +	exit 2
> +fi
> +
> +if test "$1" != 1
> +then
> +	echo "Unsupported core.fsmonitor hook version." >&2
> +	exit 1
> +fi
> +
> +printf '%s\n' $FSMONITOR_LIST

William Baker Oct. 15, 2019, 7:07 p.m. UTC | #2

On 10/11/19 6:26 PM, Junio C Hamano wrote:
> "William Baker via GitGitGadget" <[e-mail]> writes:
> 
>> +# Test staging/unstaging files that appear at the end of the index.  Test
>> +# file names begin with 'z' so that they are sorted to the end of the index. 
> 
> Well, the test is now done in a freshly created repository, so the
> z* files are the only thing you have in here---technically they are
> at the end of the index, but so they are at the beginning, too.
> 

There is one other file in the index created by 'test_commit', however,
the point still stands that there are almost no other entries in the
index now that the test is using its own repository.

> Would it affect the effectiveness of the test that you do not have
> any other paths in the working tree or in the index, unlike the test
> in the previous rounds that did not use a newly created test
> repository?  

The test still validates the scenario that we're concerned about,
namely that the new index that's written has less entries than
the index of the last entry in the old index that's is not flagged
with CE_FSMONITOR_VALID but is flagged for removal (CE_REMOVE).

> This is not a rhetorical question, but purely asking. "no, this
> still tests what we want to test and shows breakage when the fix to
> the code in the patch gets reverted" is perfectly a good answer, but
> in that case, is "the end of" the most important trait of the
> condition this test is checking?  Wouldn't the bug be exposed as
> long as we remove sufficiently large number of entries (like
> "removing more paths than the paths still in the index at the end"
> or something like that)?

This is exactly right.  The most important trait is that the last
entry flagged with CE_REMOVE does not have CE_FSMONITOR_VALID set
and has an index >= the number of entries in the new index being
written.

I will send out a patch on top of 'wb/fsmonitor-bitmap-fix' with
an update to the comment for this test.

Thanks,
William

Patch

diff --git a/fsmonitor.c b/fsmonitor.c
index 231e83a94d..7e27839278 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -14,8 +14,13 @@  struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
 	struct index_state *istate = (struct index_state *)is;
-	struct cache_entry *ce = istate->cache[pos];
+	struct cache_entry *ce;
+	
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
+		    (uintmax_t)pos, istate->cache_nr);
 
+	ce = istate->cache[pos];
 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 
@@ -50,17 +55,24 @@  int read_fsmonitor_extension(struct index_state *istate, const void *data,
 	}
 	istate->fsmonitor_dirty = fsmonitor_dirty;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
@@ -71,6 +83,10 @@  void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
 	uint32_t ewah_size = 0;
 	int fixup = 0;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 
@@ -236,6 +252,9 @@  void tweak_fsmonitor(struct index_state *istate)
 			}
 
 			/* Mark all previously saved entries as dirty */
+			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
 			/* Now mark the untracked cache for fsmonitor usage */
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 81a375fa0f..af3f9951fa 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -354,4 +354,21 @@  test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# Test staging/unstaging files that appear at the end of the index.  Test
+# file names begin with 'z' so that they are sorted to the end of the index. 
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_create_repo fsmonitor-stage-unstage &&
+	(
+		cd fsmonitor-stage-unstage &&
+		test_commit initial &&
+		git update-index --fsmonitor &&
+		removed=$(test_seq 1 100 | sed "s/^/z/") &&
+		touch $removed &&
+		git add $removed &&
+		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+		FSMONITOR_LIST="$removed" git restore -S $removed &&
+		FSMONITOR_LIST="$removed" git status
+	)
+'
+
 test_done
diff --git a/t/t7519/fsmonitor-env b/t/t7519/fsmonitor-env
new file mode 100755
index 0000000000..8f1f7ab164
--- /dev/null
+++ b/t/t7519/fsmonitor-env
@@ -0,0 +1,24 @@ 
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST