diff mbox series

[01/13] bisect tests: test for v2.30.0 "bisect run" regressions

Message ID 5571bc36b84210e16ab8731fea8bf0ed8d62cca3.1667667460.git.congdanhqx@gmail.com (mailing list archive)
State Superseded
Headers show
Series Turn git-bisect to be builtin | expand

Commit Message

Đoàn Trần Công Danh Nov. 5, 2022, 5:07 p.m. UTC
From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>

Add three failing tests which succeed on v2.29.0, but due to the topic
merged at [1] (specifically [2]) have been failing since then. We'll
address those regressions in subsequent commits.

There was also a "regression" where:

	git bisect run ./missing-script.sh

Would count a non-existing script as "good", as the shell would exit
with 127. That edge case is a bit too insane to preserve, so let's not
add it to these regression tests.

1. 0a4cb1f1f2f (Merge branch 'mr/bisect-in-c-4', 2021-09-23)
2. d1bbbe45df8 (bisect--helper: reimplement `bisect_run` shell
   function in C, 2021-09-13)

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 t/t6030-bisect-porcelain.sh | 79 +++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

Comments

Ævar Arnfjörð Bjarmason Nov. 7, 2022, 9:31 p.m. UTC | #1
On Sun, Nov 06 2022, Đoàn Trần Công Danh wrote:

> From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>
> Add three failing tests which succeed on v2.29.0, but due to the topic
> merged at [1] (specifically [2]) have been failing since then. We'll
> address those regressions in subsequent commits.
>
> There was also a "regression" where:
>
> 	git bisect run ./missing-script.sh
>
> Would count a non-existing script as "good", as the shell would exit
> with 127. That edge case is a bit too insane to preserve, so let's not
> add it to these regression tests.
>
> 1. 0a4cb1f1f2f (Merge branch 'mr/bisect-in-c-4', 2021-09-23)
> 2. d1bbbe45df8 (bisect--helper: reimplement `bisect_run` shell
>    function in C, 2021-09-13)
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
>  t/t6030-bisect-porcelain.sh | 79 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 79 insertions(+)

Looks good, if I do say so myself :) (unchanged from my topic)

But I wonder why your "fix the regression" base topic isn't starting
with this. I.e. our intial report was about that "--log" issue, but now
we know we altered the output in ways we didn't intend.

It's fine if we say "that's less important", but then ... something
should say that .. :)
Đoàn Trần Công Danh Nov. 8, 2022, 1:17 a.m. UTC | #2
On 2022-11-07 22:31:50+0100, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> 
> On Sun, Nov 06 2022, Đoàn Trần Công Danh wrote:
> 
> > From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> >
> > Add three failing tests which succeed on v2.29.0, but due to the topic
> > merged at [1] (specifically [2]) have been failing since then. We'll
> > address those regressions in subsequent commits.
> >
> > There was also a "regression" where:
> >
> > 	git bisect run ./missing-script.sh
> >
> > Would count a non-existing script as "good", as the shell would exit
> > with 127. That edge case is a bit too insane to preserve, so let's not
> > add it to these regression tests.
> >
> > 1. 0a4cb1f1f2f (Merge branch 'mr/bisect-in-c-4', 2021-09-23)
> > 2. d1bbbe45df8 (bisect--helper: reimplement `bisect_run` shell
> >    function in C, 2021-09-13)
> >
> > Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> > Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> > ---
> >  t/t6030-bisect-porcelain.sh | 79 +++++++++++++++++++++++++++++++++++++
> >  1 file changed, 79 insertions(+)
> 
> Looks good, if I do say so myself :) (unchanged from my topic)
> 
> But I wonder why your "fix the regression" base topic isn't starting
> with this. I.e. our intial report was about that "--log" issue, but now
> we know we altered the output in ways we didn't intend.

Well, I would like to keep that topic as much short as possible to
have a higher chance of fixing real regression first. Those changes is
error/warning/log are less important as you said below ...
 
> It's fine if we say "that's less important", but then ... something
> should say that .. :)

Sure, I will add it in the next revision.
diff mbox series

Patch

diff --git a/t/t6030-bisect-porcelain.sh b/t/t6030-bisect-porcelain.sh
index 6dbbe62eb2..6901bbd202 100755
--- a/t/t6030-bisect-porcelain.sh
+++ b/t/t6030-bisect-porcelain.sh
@@ -252,6 +252,85 @@  test_expect_success 'bisect skip: with commit both bad and skipped' '
 	grep $HASH4 my_bisect_log.txt
 '
 
+test_bisect_run_args () {
+	test_when_finished "rm -f run.sh actual" &&
+	>actual &&
+	cat >expect.args &&
+	cat <&6 >expect.out &&
+	cat <&7 >expect.err &&
+	write_script run.sh <<-\EOF &&
+	while test $# != 0
+	do
+		echo "<$1>" &&
+		shift
+	done >actual.args
+	EOF
+
+	test_when_finished "git bisect reset" &&
+	git bisect start &&
+	git bisect good $HASH1 &&
+	git bisect bad $HASH4 &&
+	git bisect run ./run.sh $@ >actual.out.raw 2>actual.err &&
+	# Prune just the log output
+	sed -n \
+		-e '/^Author:/d' \
+		-e '/^Date:/d' \
+		-e '/^$/d' \
+		-e '/^commit /d' \
+		-e '/^ /d' \
+		-e 'p' \
+		<actual.out.raw >actual.out &&
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err &&
+	test_cmp expect.args actual.args
+}
+
+test_expect_failure 'git bisect run: args, stdout and stderr with no arguments' '
+	test_bisect_run_args <<-\EOF_ARGS 6<<-EOF_OUT 7<<-\EOF_ERR
+	EOF_ARGS
+	running ./run.sh
+	$HASH4 is the first bad commit
+	bisect run success
+	EOF_OUT
+	EOF_ERR
+'
+
+test_expect_failure 'git bisect run: args, stdout and stderr: "--" argument' '
+	test_bisect_run_args -- <<-\EOF_ARGS 6<<-EOF_OUT 7<<-\EOF_ERR
+	<-->
+	EOF_ARGS
+	running ./run.sh --
+	$HASH4 is the first bad commit
+	bisect run success
+	EOF_OUT
+	EOF_ERR
+'
+
+test_expect_failure 'git bisect run: args, stdout and stderr: "--log foo --no-log bar" arguments' '
+	test_bisect_run_args --log foo --no-log bar <<-\EOF_ARGS 6<<-EOF_OUT 7<<-\EOF_ERR
+	<--log>
+	<foo>
+	<--no-log>
+	<bar>
+	EOF_ARGS
+	running ./run.sh --log foo --no-log bar
+	$HASH4 is the first bad commit
+	bisect run success
+	EOF_OUT
+	EOF_ERR
+'
+
+test_expect_failure 'git bisect run: args, stdout and stderr: "--bisect-start" argument' '
+	test_bisect_run_args --bisect-start <<-\EOF_ARGS 6<<-EOF_OUT 7<<-\EOF_ERR
+	<--bisect-start>
+	EOF_ARGS
+	running ./run.sh --bisect-start
+	$HASH4 is the first bad commit
+	bisect run success
+	EOF_OUT
+	EOF_ERR
+'
+
 # We want to automatically find the commit that
 # added "Another" into hello.
 test_expect_success '"git bisect run" simple case' '