[1/3] sparse-checkout: fix OOM error with mixed patterns

Commit Message

Derrick Stolee Sept. 20, 2021, 5:57 p.m. UTC

From: Derrick Stolee <[e-mail]>

Add a test to t1091-sparse-checkout-builtin.sh that would result in an
infinite loop and out-of-memory error before this change. The issue
relies on having non-cone-mode patterns while trying to modify the
patterns in cone-mode.

The fix is simple, allowing us to break from the loop when the input
path does not contain a slash, as the "dir" pattern we added does not.

This is only a fix to the critical out-of-memory error. A better
response to such a strange state will follow in a later change.

Reported-by: Calbabreaker <[e-mail]>
Helped-by: Taylor Blau <[e-mail]>
Signed-off-by: Derrick Stolee <[e-mail]>
---
 builtin/sparse-checkout.c          | 2 +-
 t/t1091-sparse-checkout-builtin.sh | 8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

Comments

Taylor Blau Sept. 20, 2021, 6:24 p.m. UTC | #1

On Mon, Sep 20, 2021 at 05:57:36PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[e-mail]>
>
> Add a test to t1091-sparse-checkout-builtin.sh that would result in an
> infinite loop and out-of-memory error before this change. The issue
> relies on having non-cone-mode patterns while trying to modify the
> patterns in cone-mode.
>
> The fix is simple, allowing us to break from the loop when the input
> path does not contain a slash, as the "dir" pattern we added does not.
>
> This is only a fix to the critical out-of-memory error. A better
> response to such a strange state will follow in a later change.
>
> Reported-by: Calbabreaker <[e-mail]>
> Helped-by: Taylor Blau <[e-mail]>
> Signed-off-by: Derrick Stolee <[e-mail]>
> ---
>  builtin/sparse-checkout.c          | 2 +-
>  t/t1091-sparse-checkout-builtin.sh | 8 ++++++++
>  2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
> index 8ba9f13787b..b45fd97a98b 100644
> --- a/builtin/sparse-checkout.c
> +++ b/builtin/sparse-checkout.c
> @@ -389,7 +389,7 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat
>  		char *oldpattern = e->pattern;
>  		size_t newlen;
>
> -		if (slash == e->pattern)
> +		if (!slash || slash == e->pattern)
>  			break;

If we are preparing to make it so that we do not blindly copy patterns
from a sparse checkout without cone mode enabled, then wouldn't this new
case be a BUG()?

Of course, users could still tweak the contents of their sparse-checkout
file as they wish, but I'd expect that we'd catch those cases, too
(i.e., by validating the contents of the existing sparse-checkout before
calling this function).

> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index 38fc8340f5c..a429d2cc671 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -103,6 +103,14 @@ test_expect_success 'clone --sparse' '
>  	check_files clone a
>  '
>
> +test_expect_success 'switching to cone mode with non-cone mode patterns' '
> +	git init bad-patterns &&
> +	git -C bad-patterns sparse-checkout init &&
> +	git -C bad-patterns sparse-checkout add dir &&
> +	git -C bad-patterns config core.sparseCheckoutCone true &&

Makes sense that we'd want to update the config rather than call "init
--cone" here, since a subsequent patch would change the thing that this
is testing (from "doesn't OOM in the above-described situation" to
"clears the existing contents of your sparse-checkout").

> +	git -C bad-patterns sparse-checkout add dir
> +'
> +

In other series I've suggested a cosmetic change to move all of these to
a sub-shell that begins with "cd bad-patterns &&", but obviously that is
a relatively unimportant suggestion.

Thanks,
Taylor

Derrick Stolee Sept. 21, 2021, 1:06 p.m. UTC | #2

On 9/20/2021 2:24 PM, Taylor Blau wrote:
> On Mon, Sep 20, 2021 at 05:57:36PM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[e-mail]>
>>
>> Add a test to t1091-sparse-checkout-builtin.sh that would result in an
>> infinite loop and out-of-memory error before this change. The issue
>> relies on having non-cone-mode patterns while trying to modify the
>> patterns in cone-mode.
>>
>> The fix is simple, allowing us to break from the loop when the input
>> path does not contain a slash, as the "dir" pattern we added does not.
>>
>> This is only a fix to the critical out-of-memory error. A better
>> response to such a strange state will follow in a later change.
>>
>> Reported-by: Calbabreaker <[e-mail]>
>> Helped-by: Taylor Blau <[e-mail]>
>> Signed-off-by: Derrick Stolee <[e-mail]>
>> ---
>>  builtin/sparse-checkout.c          | 2 +-
>>  t/t1091-sparse-checkout-builtin.sh | 8 ++++++++
>>  2 files changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
>> index 8ba9f13787b..b45fd97a98b 100644
>> --- a/builtin/sparse-checkout.c
>> +++ b/builtin/sparse-checkout.c
>> @@ -389,7 +389,7 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat
>>  		char *oldpattern = e->pattern;
>>  		size_t newlen;
>>
>> -		if (slash == e->pattern)
>> +		if (!slash || slash == e->pattern)
>>  			break;
> 
> If we are preparing to make it so that we do not blindly copy patterns
> from a sparse checkout without cone mode enabled, then wouldn't this new
> case be a BUG()?
> 
> Of course, users could still tweak the contents of their sparse-checkout
> file as they wish, but I'd expect that we'd catch those cases, too
> (i.e., by validating the contents of the existing sparse-checkout before
> calling this function).

If I was more confident that we were catching absolutely every possible
case of non-cone mode patterns in our parsing logic, then I suppose a BUG()
could apply here. At minimum, at this point in time (before fixing the gap
in parsing in patch 3) the test below would not even work with test_must_fail,
since the exit code would be unexpected.
 
>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
>> index 38fc8340f5c..a429d2cc671 100755
>> --- a/t/t1091-sparse-checkout-builtin.sh
>> +++ b/t/t1091-sparse-checkout-builtin.sh
>> @@ -103,6 +103,14 @@ test_expect_success 'clone --sparse' '
>>  	check_files clone a
>>  '
>>
>> +test_expect_success 'switching to cone mode with non-cone mode patterns' '
>> +	git init bad-patterns &&
>> +	git -C bad-patterns sparse-checkout init &&
>> +	git -C bad-patterns sparse-checkout add dir &&
>> +	git -C bad-patterns config core.sparseCheckoutCone true &&
> 
> Makes sense that we'd want to update the config rather than call "init
> --cone" here, since a subsequent patch would change the thing that this
> is testing (from "doesn't OOM in the above-described situation" to
> "clears the existing contents of your sparse-checkout").
>
>> +	git -C bad-patterns sparse-checkout add dir
>> +'
>> +
> 
> In other series I've suggested a cosmetic change to move all of these to
> a sub-shell that begins with "cd bad-patterns &&", but obviously that is
> a relatively unimportant suggestion.

The only defense I have for not using a subshell and 'cd' is that later
I can use an "expect" file in my current directory without it being "in"
the repository. It doesn't really matter for this example, but it has in
the past, causing me to do this by habit. A tab is smaller than the string
" -C bad-patterns", so it's probably worth changing.

Thanks,
-Stolee

Taylor Blau Sept. 21, 2021, 4:35 p.m. UTC | #3

On Tue, Sep 21, 2021 at 09:06:59AM -0400, Derrick Stolee wrote:
> > If we are preparing to make it so that we do not blindly copy patterns
> > from a sparse checkout without cone mode enabled, then wouldn't this new
> > case be a BUG()?
> >
> > Of course, users could still tweak the contents of their sparse-checkout
> > file as they wish, but I'd expect that we'd catch those cases, too
> > (i.e., by validating the contents of the existing sparse-checkout before
> > calling this function).
>
> If I was more confident that we were catching absolutely every possible
> case of non-cone mode patterns in our parsing logic, then I suppose a BUG()
> could apply here. At minimum, at this point in time (before fixing the gap
> in parsing in patch 3) the test below would not even work with test_must_fail,
> since the exit code would be unexpected.

Right, but it could be a test_must_fail after the second patch, no?

Not calling BUG() is fine with me if you think there may be other cases
we haven't discovered. But we should have some way to discover them
instead if a user can generate them organically. Maybe a warning()?

> >> +	git -C bad-patterns sparse-checkout add dir
> >> +'
> >> +
> >
> > In other series I've suggested a cosmetic change to move all of these to
> > a sub-shell that begins with "cd bad-patterns &&", but obviously that is
> > a relatively unimportant suggestion.
>
> The only defense I have for not using a subshell and 'cd' is that later
> I can use an "expect" file in my current directory without it being "in"
> the repository. It doesn't really matter for this example, but it has in
> the past, causing me to do this by habit. A tab is smaller than the string
> " -C bad-patterns", so it's probably worth changing.

Yeah. I admit to hardly caring about this (subshell vs. '-C
bad-patterns') at all. Either is completely fine with me.

Thanks,
Taylor

Patch

diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c
index 8ba9f13787b..b45fd97a98b 100644
--- a/builtin/sparse-checkout.c
+++ b/builtin/sparse-checkout.c
@@ -389,7 +389,7 @@  static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat
 		char *oldpattern = e->pattern;
 		size_t newlen;
 
-		if (slash == e->pattern)
+		if (!slash || slash == e->pattern)
 			break;
 
 		newlen = slash - e->pattern;
diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
index 38fc8340f5c..a429d2cc671 100755
--- a/t/t1091-sparse-checkout-builtin.sh
+++ b/t/t1091-sparse-checkout-builtin.sh
@@ -103,6 +103,14 @@  test_expect_success 'clone --sparse' '
 	check_files clone a
 '
 
+test_expect_success 'switching to cone mode with non-cone mode patterns' '
+	git init bad-patterns &&
+	git -C bad-patterns sparse-checkout init &&
+	git -C bad-patterns sparse-checkout add dir &&
+	git -C bad-patterns config core.sparseCheckoutCone true &&
+	git -C bad-patterns sparse-checkout add dir
+'
+
 test_expect_success 'interaction with clone --no-checkout (unborn index)' '
 	git clone --no-checkout "file://$(pwd)/repo" clone_no_checkout &&
 	git -C clone_no_checkout sparse-checkout init --cone &&