[v2,0/2] perf-lib.sh: fix 0baf78e7bc regression, refactor & fix bugs

Message

Ævar Arnfjörð Bjarmason May 6, 2019, 11:23 p.m. UTC

On Mon, May 06 2019, Jeff King wrote:

> On Mon, May 06, 2019 at 09:16:11PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> Perhaps there's some better way to fix this, but it seems to me that
>> the best solution is to just make this behavior less magical. We know
>> in run_dirs_helper() that we're about to run performance tests on a
>> given <revision>, so let's just set GIT_TEST_INSTALLED to an absolute
>> path there, and then make getting logging target from a previously
>> relative path less magical, we'll just explicitly pass down the
>> relative path as a variable.
>> 
>> This makes e.g. these cases all work:
>> 
>>     ./run . $PWD/../../ origin/master origin/next HEAD -- <tests>
>> 
>> As well as just a plain one-off:
>> 
>>     ./run <tests>
>
> Doing this naively would break anybody doing:
>
>   GIT_TEST_INSTALLED=some-relative-path ./p1234-foo.sh
>
> but I doubt that actually matters in practice (notably this already does
> not work with non-perf tests, as test-lib.sh does not do any
> normalization).

Yeah this has never worked and GIT_TEST_INSTALLED has been documented
to take an absolute path when used with the normal test suite.

I've noted as much in the 1/2 here.

> I don't think your patch does, because it leaves the extra absolutizing
> in perf-lib.sh. But then it doesn't feel like it's really simplified
> anything. ;)

Yeah, v1 didn't because I was trying for a minimal regression fix
without digging us fully out of that GIT_TEST_INSTALLED hole, but as
2/2 here shows we can fully get rid of it from perf-lib.sh, which I
think makes things much simpler.

>> diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
>> index 169f92eae3..b15ee1d262 100644
>> --- a/t/perf/perf-lib.sh
>> +++ b/t/perf/perf-lib.sh
>> @@ -32,6 +32,10 @@ TEST_NO_MALLOC_CHECK=t
>>  if test -z "$GIT_TEST_INSTALLED"; then
>>  	perf_results_prefix=
>>  else
>> +	if test -n "$GIT_PERF_DIR_MYDIR_REL"
>> +	then
>> +		GIT_TEST_INSTALLED=$GIT_PERF_DIR_MYDIR_REL
>> +	fi
>>  	perf_results_prefix=$(printf "%s" "${GIT_TEST_INSTALLED%/bin-wrappers}" | tr -c "[a-zA-Z0-9]" "[_*]")"."
>>  	GIT_TEST_INSTALLED=$ABSOLUTE_GIT_TEST_INSTALLED
>>  fi
>
> So we reset GIT_TEST_INSTALLED to the relative path here (ignoring
> what's in it!), and then afterwards set it to the absolute path. That
> still seems rather magical. :)
>
> What if instead we:
>
>   - taught test-lib.sh to make GIT_TEST_INSTALLED absolute (since after
>     all it is the one who is planning to chdir and wreck the relative
>     path)
>
>   - let callers pass in $GIT_PERF_RESULTS_PREFIX instead of guessing at
>     it ourselves from the path name. Then the "run" script could quite
>     reasonably just pass in the tree oid it already has instead of us
>     trying to decode it. And nobody would care about whether
>     $GIT_TEST_INSTALLED has been mangled.
>
> I thought about going this route for my original patch, but I wanted to
> fix the regression (which I agree is quite serious and embarrassing) as
> quickly and simply as possible.

It seems simplest after the changes I've made here to just make a
relative GIT_TEST_INSTALLED be an error in test-lib.sh, why bend over
backwards to support it?

Re GIT_PERF_RESULTS_PREFIX: Depending on what you mean we now have
that in 2/2 as PERF_RESULTS_PREFIX.

But if you mean the user can pass it in that doesn't make sense, since
we need to pick a different prefix revision we test, so it's a
many-to-many relationship.

As seen in 2/2 modifying some of the shell & associated Perl code it's
a bit nasty that we need to duplicate the logic for picking these
PERF_RESULTS_PREFIXes in the shell code and Perl code. Ideally the
shell code would pick it, and pass the mapping to the Perl code
somehow. But that's another "has sucked since forever, future TODO"
item.

Ævar Arnfjörð Bjarmason (2):
  perf-lib.sh: make "./run <revisions>" use the correct gits
  perf-lib.sh: remove GIT_TEST_INSTALLED from perf-lib.sh

 t/perf/aggregate.perl | 13 ++++++-------
 t/perf/perf-lib.sh    | 11 +----------
 t/perf/run            | 41 +++++++++++++++++++++++++++++++----------
 3 files changed, 38 insertions(+), 27 deletions(-)

Comments

Jeff King May 7, 2019, 7:06 a.m. UTC | #1

On Tue, May 07, 2019 at 01:23:07AM +0200, Ævar Arnfjörð Bjarmason wrote:

> > What if instead we:
> >
> >   - taught test-lib.sh to make GIT_TEST_INSTALLED absolute (since after
> >     all it is the one who is planning to chdir and wreck the relative
> >     path)
> >
> >   - let callers pass in $GIT_PERF_RESULTS_PREFIX instead of guessing at
> >     it ourselves from the path name. Then the "run" script could quite
> >     reasonably just pass in the tree oid it already has instead of us
> >     trying to decode it. And nobody would care about whether
> >     $GIT_TEST_INSTALLED has been mangled.
> >
> > I thought about going this route for my original patch, but I wanted to
> > fix the regression (which I agree is quite serious and embarrassing) as
> > quickly and simply as possible.
> 
> It seems simplest after the changes I've made here to just make a
> relative GIT_TEST_INSTALLED be an error in test-lib.sh, why bend over
> backwards to support it?

I'm OK with that, though _somebody_ has to generate the full path. My
thinking was that it would be a little nicer if we did at the lowest
level, but it's pretty easy to do it directly in the "run" script.

> Re GIT_PERF_RESULTS_PREFIX: Depending on what you mean we now have
> that in 2/2 as PERF_RESULTS_PREFIX.
> 
> But if you mean the user can pass it in that doesn't make sense, since
> we need to pick a different prefix revision we test, so it's a
> many-to-many relationship.

I couldn't quite parse this. What I meant was that any caller of the
perf scripts would pass in two pieces of data: the path to the git we
want to test (GIT_TEST_INSTALLED) and the unique name under which it
should be saved (GIT_PERF_PREFIX, which would generally be either empty
or the tree oid).

I see that perf-lib.sh also has support for running git out of some
arbitrary installed directory. I'm not sure if anybody actually uses
that, but it's not too hard to support either (the run script just
generates the prefix).

Something like (just for illustration on top of master; this doesn't
have the actual bugfix):

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 169f92eae3..e2b342f216 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -29,10 +29,7 @@ TEST_NO_MALLOC_CHECK=t
 
 . ../test-lib.sh
 
-if test -z "$GIT_TEST_INSTALLED"; then
-	perf_results_prefix=
-else
-	perf_results_prefix=$(printf "%s" "${GIT_TEST_INSTALLED%/bin-wrappers}" | tr -c "[a-zA-Z0-9]" "[_*]")"."
+if test -n "$GIT_TEST_INSTALLED"; then
 	GIT_TEST_INSTALLED=$ABSOLUTE_GIT_TEST_INSTALLED
 fi
 
@@ -179,7 +176,8 @@ test_wrapper_ () {
 		base=$(basename "$0" .sh)
 		echo "$test_count" >>"$perf_results_dir"/$base.subtests
 		echo "$1" >"$perf_results_dir"/$base.$test_count.descr
-		base="$perf_results_dir"/"$perf_results_prefix$(basename "$0" .sh)"."$test_count"
+		prefix=$PERF_RESULTS_PREFIX${PERF_RESULTS_PREFIX:+.}
+		base="$perf_results_dir"/"$prefix$(basename "$0" .sh)"."$test_count"
 		"$test_wrapper_func_" "$@"
 	fi
 
diff --git a/t/perf/run b/t/perf/run
index 9aaa733c77..e5695ec8de 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -87,15 +87,20 @@ run_dirs_helper () {
 		fi
 		build_git_rev $rev "$mydir"
 		mydir=build/$rev
+		PERF_RESULTS_PREFIX=$rev
+	else
+		PERF_RESULTS_PREFIX=$(echo mydir | tr -c "[a-zA-Z0-9]" "[_*]")"."
 	fi
 	if test "$mydir" = .; then
 		unset GIT_TEST_INSTALLED
+		unset PERF_RESULTS_PREFIX
 	else
 		GIT_TEST_INSTALLED="$mydir/bin-wrappers"
 		# Older versions of git lacked bin-wrappers; fallback to the
 		# files in the root.
 		test -d "$GIT_TEST_INSTALLED" || GIT_TEST_INSTALLED=$mydir
 		export GIT_TEST_INSTALLED
+		export PERF_RESULTS_PREFIX
 	fi
 	run_one_dir "$@"
 }

It does mean that anybody calling "GIT_TEST_INSTALLED=whatever
./p1234-foo.sh" has to pass in PERF_RESULTS_PREFIX, but I think that's
OK (they should be using the "run" script anyway).

> As seen in 2/2 modifying some of the shell & associated Perl code it's
> a bit nasty that we need to duplicate the logic for picking these
> PERF_RESULTS_PREFIXes in the shell code and Perl code. Ideally the
> shell code would pick it, and pass the mapping to the Perl code
> somehow. But that's another "has sucked since forever, future TODO"
> item.

Yeah, I didn't realize that aggregate.perl has the duplicate logic.
Gross. But I agree that's not a topic for now. It would continue to
compute the same PERF_RESULTS_PREFIX that "run" does, except it doesn't
need to bother to pass it to anybody else.

(Note that in my patch above, I dropped the pointless "build_" from the
rev-based names, so aggregate.perl would need to do the same).

-Peff