mbox series

[v6,00/14] CI: js/ci-github-workflow-markup rebased on "use $GITHUB_ENV"

Message ID cover-v6-00.14-00000000000-20220525T100743Z-avarab@gmail.com (mailing list archive)
Headers show
Series CI: js/ci-github-workflow-markup rebased on "use $GITHUB_ENV" | expand

Message

Ævar Arnfjörð Bjarmason May 25, 2022, 11:25 a.m. UTC 
This non-RFC re-roll of [1] is based on the just-re-rolled [2]
(hereafter "base") and proposes to replace [3]. It incorporates the
relevant parts of [3] that were new in that re-roll.

Before getting into the details it's helpful to open the following
links:

  master: https://github.com/avar/git/actions/runs/2377697338
  js: https://github.com/avar/git/actions/runs/2377281297
  base: https://github.com/avar/git/actions/runs/2377257487
  this: https://github.com/avar/git/actions/runs/2383110429
  this with 'raw': https://github.com/avar/git/actions/runs/2383134850

These are all versions of the relevant branches in play with the same
commit on top to trigger CI failures. The "master" and "base" are the
same as those noted in [2], the "js" is [3] (but rebased on master),
and "this" is this topic. The "this with 'raw'" is this topic, but
configured to omit the new GitHub CI Markdown output (see 13/14 and
14/14).

Having these open allows us to compare the output and performance of
these different proposed CI topics. Here clicking on e.g. linux-sha256
(which has a verbose test failure) will on these topics on in the
Chrome debugger[4].

  master: https://github.com/avar/git/runs/6573058614?check_suite_focus=true
  262 ms  Loading
  3977 ms  Scripting
  1632 ms  Rendering
  641 ms  Painting
  934 ms  System
  920 ms  Idle
  8366 ms  Total

  js: https://github.com/avar/git/runs/6571972194?check_suite_focus=true
  170 ms  Loading
  12934 ms  Scripting
  7364 ms  Rendering
  6940 ms  Painting
  5489 ms  System
  5940 ms  Idle
  38836 ms  Total

  base: https://github.com/avar/git/runs/6571804381?check_suite_focus=true
  215 ms  Loading
  3270 ms  Scripting
  1171 ms  Rendering
  649 ms  Painting
  941 ms  System
  731 ms  Idle
  6976 ms  Total

  this: https://github.com/avar/git/runs/6588407676?check_suite_focus=true
  201 ms  Loading
  9192 ms  Scripting
  6104 ms  Rendering
  5553 ms  Painting
  4684 ms  System
  6184 ms  Idle
  31917 ms  Total

  this with 'raw': https://github.com/avar/git/runs/6588579493?check_suite_focus=true
  173 ms  Loading
  3036 ms  Scripting
  1110 ms  Rendering
  617 ms  Painting
  1028 ms  System
  714 ms  Idle
  6678 ms  Total

As noted in previous performance testing [5] what these results
indicate is that the more output you put under GitHub CI's "groups"
the slower it will get. With the "js" variant in particular it can be
seen to struggle to render as you're watching it.

It's faster with "this" because there's less rendering to be done. Due
to the changes in the "base" topic to make "make" and "make test"
top-level commands we can run there's less data in the step where we
need to print failures in detail.

So re: [6] this shows how the two serieses are complimentary and make
the overall CI UX better when combined than stand-alone.

Changes since the RFC v5:

 * Incorporated the v3[3] of js/ci-github-workflow-markup.

 * Applied fix-up to get rid of broken
   ci/print-test-failures-github.sh (which didn't exist anymore)

 * The v5 of this series had an issue where we'd "zoom in" on the
   "make test" output, instead of the new GitHub CI markdown
   output. We now combine the best of both worlds. We'll have a "make
   test" and "ci/print-test-failures.sh" steps, both of which fail,
   but we'll "zoom in" on the latter, which contains the new output.

 * As before this series applied some DRY principles to the code [7]
   proposed to add, i.e. the "handle_failed_tests" was a
   near-duplication of the code in
   ci/print-test-failures-github.sh. The 01-03/14 here allow us to
   unify the two & share the code instead.

   My RFC v5 had some options we didn't need (--exit-code and
   --no-exit-code) that I removed. It furthermore defaults to "auto"
   discovering if we should emit the GitHub CI *.markup output based
   on if we find those files in t/test-results, which as we'll see in
   13-14/14 makes configuring the output simpler.

 * Due to the above and other changes here being smaller the "ci: make
   it easier to find failed tests' logs in the GitHub workflow" here
   is now a squashed version of changes done in several steps in
   [3]. We now add the ability to emit GitHub CI markup in one commit,
   and enable it in another.

 * In the latest "js" topic the "ok" tests were skipped, but we'd
   still include "skip" tests, i.e. "not ok .. TODO" tests. These
   should be considered "ok" for the purposes of the output, so a new
   09/14 here excludes them too.

 * An updated commit message for 10/14 notes a subtle and presumably
   unintended side-effect/bug in "ci(github): avoid printing test case
   preamble twice" new in the v3 of [3]. In tweaking the *.markup
   files we also change the output in the *.out files (i.e. the raw
   logs).

   You thus won't get the same output in *.out (the raw logs) with the
   new output.

 * New 13/14 and 14/14 commits here make the CI output type
   configurable using the same "ci-config" method we use for the
   existing "check-ref". As explained in 14/14 this is what I'm using
   to produce "this with 'raw'" output the same commit as "this". I
   push to a branch whose name ends with "_CI_OUT_RAW".

   This allows users (such as me, or per 13/14 probably SZEDER too) to
   configure our output to avoid this GitHub markdown in our
   ci-config.

 * As 13/14 argues for I think a better thing to do would be to avoid
   14/14 for now and take this series without making the new "github"
   output the default.

   But in the interest of not conflating that with this proposed
   re-roll 14/14 changes the default from "raw" to "github" again,
   thus making the output [3] is proposing the default here too.

   We can always revert 14/14 later if we think that was premature,
   which as shown there is now a one-line change.

To continue from the last note above: This series also allows for
combining the new & old output in a way where we can have our cake &
eat it too. If we load the "this" output and "zoom in" on the failed
"make test" step instead of the subsequent "ci/print-test-failures.sh"
step we'll load the page in around 1/6 the time, or:

  this (zoomed in on 'make test'): https://github.com/avar/git/runs/6588407676?check_suite_focus=true#step:7:1437
  149 ms  Loading
  2559 ms  Scripting
  934 ms  Rendering
  591 ms  Painting
  845 ms  System
  605 ms  Idle
  5684 ms  Total

Clicking on the "ci/print-test-failures.sh" will then take just as
long. I haven't implemented this as a configurable feature is just a
matter of making ci/print-test-failures.sh exit with 0 where it now
exits with 1 (as we always "zoom in" on the last failed step, which
then will be "make test").

For Junio:

If you eject js/ci-github-workflow-markup you'll get a minor conflict
(as before) with js/use-builtin-add-i, and a new conflict with
jx/l10n-workflow-change. The correct resolution is to "git rm
ci/run-build-and-tests.sh ci/run-static-analysis.sh" and apply this
diff on top:

	diff --git a/Makefile b/Makefile
	index b602bdaed4a..984b1a1905d 100644
	--- a/Makefile
	+++ b/Makefile
	@@ -3625,3 +3625,4 @@ ci-static-analysis: ci-check-directional-formatting
	 ci-static-analysis: check-builtins
	 ci-static-analysis: check-coccicheck
	 ci-static-analysis: hdr-check
	+ci-static-analysis: check-pot
	diff --git a/ci/lib.sh b/ci/lib.sh
	index b148d3f209f..c6b17b4cb22 100755
	--- a/ci/lib.sh
	+++ b/ci/lib.sh
	@@ -282,7 +282,7 @@ linux-TEST-vars)
	 	setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1
	 	setenv --test GIT_TEST_MULTI_PACK_INDEX 1
	 	setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1
	-	setenv --test GIT_TEST_ADD_I_USE_BUILTIN 1
	+	setenv --test GIT_TEST_ADD_I_USE_BUILTIN 0
	 	setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master
	 	setenv --test GIT_TEST_WRITE_REV_INDEX 1
	 	setenv --test GIT_TEST_CHECKOUT_WORKERS 2


1. https://lore.kernel.org/git/RFC-cover-v5-00.10-00000000000-20220421T183001Z-avarab@gmail.com/
2. https://lore.kernel.org/git/cover-v6-00.29-00000000000-20220525T094123Z-avarab@gmail.com/
3. https://lore.kernel.org/git/pull.1117.v3.git.1653171536.gitgitgadget@gmail.com/
4. Version 101.0.4951.54 (Official Build) (64-bit). Opening the
   debugger in an empty tab (F12), going to "Performance" and
   recording a trace (Ctrl+E) (note: *not* Ctrl+Shift+E, as that would
   end it after network load, but before rendering is done & the page
   is ready to interact with).

   The "Total" times here are "real" and don't very with how fast I am
   to "stop" the trace (tested by leaving it lingering for a long
   time).
5. https://lore.kernel.org/git/220523.86ee0kzjix.gmgdl@evledraar.gmail.com/
6. https://lore.kernel.org/git/nycvar.QRO.7.76.6.2205232040130.352@tvgsbejvaqbjf.bet/
7. https://lore.kernel.org/git/417f702a2454316f0bdc5b66eb16751272ff2989.1653171536.git.gitgitgadget@gmail.com/

Johannes Schindelin (6):
  tests: refactor --write-junit-xml code
  test(junit): avoid line feeds in XML attributes
  ci: optionally mark up output in the GitHub workflow
  ci(github): skip the logs of the successful test cases
  ci(github): mention where the full logs can be found
  ci: call `finalize_test_case_output` a little later

Victoria Dye (1):
  ci(github): avoid printing test case preamble twice

Ævar Arnfjörð Bjarmason (7):
  CI: don't "cd" in ci/print-test-failures.sh
  CI: don't include "test-results/" in ci/print-test-failures.sh output
  CI: stop setting FAILED_TEST_ARTIFACTS N times
  ci: make it easier to find failed tests' logs in the GitHub workflow
  ci(github): skip "skip" tests in --github-workflow-markup
  CI: make --github-workflow-markup ci-config, off by default
  CI: make the --github-workflow-markup "github" output the default

 .github/workflows/main.yml           |  24 +++++
 ci/lib.sh                            |  12 +++
 ci/print-test-failures.sh            | 100 ++++++++++++++++----
 t/test-lib-functions.sh              |   6 +-
 t/test-lib-github-workflow-markup.sh |  56 ++++++++++++
 t/test-lib-junit.sh                  | 132 +++++++++++++++++++++++++++
 t/test-lib.sh                        | 128 ++++----------------------
 7 files changed, 332 insertions(+), 126 deletions(-)
 create mode 100644 t/test-lib-github-workflow-markup.sh
 create mode 100644 t/test-lib-junit.sh

Range-diff against v5:
 1:  56910df26e7 =  1:  c10a11fdae8 CI: don't "cd" in ci/print-test-failures.sh
 3:  577a5be34f4 =  2:  1d07ca58322 CI: don't include "test-results/" in ci/print-test-failures.sh output
 4:  1482f840f64 !  3:  fdc5b534644 CI: stop setting FAILED_TEST_ARTIFACTS N times
    @@ ci/print-test-failures.sh: do
      			cp "t/test-results/${TEST_OUT}" t/failed-test-artifacts/
      			(
      				cd t &&
    -@@ ci/print-test-failures.sh: done
    - 
    - if test -n "$failed"
    - then
    +@@ ci/print-test-failures.sh: do
    + 		esac
    + 	fi
    + done
    ++
    ++if test -n "$failed"
    ++then
     +	if test -n "$GITHUB_ENV"
     +	then
     +		echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
     +	fi
    -+
    - 	if test -n "$exit_code"
    - 	then
    - 		exit 1
    ++fi
 6:  d9b7d5ddf5a =  4:  a75846a12f2 tests: refactor --write-junit-xml code
 7:  2d233bbdc22 =  5:  5b709dde86a test(junit): avoid line feeds in XML attributes
 8:  5dd43a1624b =  6:  f5b39f1660d ci: optionally mark up output in the GitHub workflow
 2:  caec0b1089a !  7:  e713ca6041b CI: add --exit-code to ci/print-test-failures.sh
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    CI: add --exit-code to ci/print-test-failures.sh
    +    ci: make it easier to find failed tests' logs in the GitHub workflow
     
    -    The ci/print-test-failures.sh scripts will exit 0 when there's failed
    -    tests, let's teach it --exit-code to have it exit non-zero in those
    -    cases.
    +    When investigating a test failure, the time that matters most is the
    +    time it takes from getting aware of the failure to displaying the output
    +    of the failing test case.
     
    +    You currently have to know a lot of implementation details when
    +    investigating test failures in the CI runs. The first step is easy: the
    +    failed job is marked quite clearly, but when opening it, the failed step
    +    is expanded, which in our case is the one running
    +    `make` or 'make test`. When failing in the latter we only show a
    +    high-level view of what went wrong: it prints the output of `prove`
    +    which merely tells the reader which test script failed, along with the failed
    +    test numbers.
    +
    +    To diagnose those failures in detail the user has then had to know
    +    that they should expand the subsequent "Run ci/print-test-failures.sh"
    +    step. Because that step always exits successfully it's not expanded by
    +    default (the GitHub CI only expands the last failing step), making it
    +    easy to miss that there is useful information hidden in there.
    +
    +    This change does a couple of things to improve the situation:
    +
    +     * We now make use of the "exit code" feature of
    +       ci/print-test-failures.sh added in a preceding commit, thus if "make
    +       test" fails we'll pick up that we have failing tests, and fail in
    +       ci/print-test-failures.sh too.
    +
    +       Note that we still exit successfully from ci/print-test-failures.sh
    +       if we don't have logs of any failed tests in t/test-results. Thus
    +       if we fail in "make" (i.e. when building), or even in "make test"
    +       in a way that doesn't yield a failed test (e.g. the t/Makefile
    +       itself failing) we'll correctly "zoom in" on the (hopefully
    +       failing) "make test" instead.
    +
    +       As before if there was no previously failed step the
    +       ci/print-test-failures.sh step is skipped entirely (see
    +       .github/workflows/main.yml)
    +
    +       To make the script DWYM the ci/print-test-failures.sh defaults to
    +       an "auto" mode. Thus since we turn on --github-workflow-markup in
    +       ci/lib.sh we'll write *.markup files, and will pick up on that in
    +       ci/print-test-failures.sh.
    +
    +       The "$github_workflow_markup" being "auto" (or "t") on failure is
    +       also what makes us exit non-zero. We could also add
    +       --[no-]exit-code arguments, but at this point we only need this for
    +       the "*.markup" mode, so let's leave that for now.
    +
    +       By having this be "auto" well pick up on whatever the GIT_TEST_OPTS
    +       setting is, which as we'll see in a subsequent commit will be
    +       useful if we'd like to configure the CI to emit it only some of the
    +       time.
    +
    +     * The ci/print-test-failures.sh now makes use of the GitHub markdown
    +       introduced for t/test-lib.sh in the preceding commit. I.e. in GitHub
    +       CI we'll now run with --github-workflow-markup in GIT_TEST_OPTS,
    +       which causes our "--verbose" output in
    +       "t/test-results/$TEST_OUT.markup" to be in the GitHub CI-specific
    +       Markdown format.
    +
    +    The new output is easier to read since it makes use of collapsible grouping.
    +    Initially, readers will see the high-level view of what actually happened.
    +    To drill down, the "group" for the specific failing test(s) can be expanded.
    +
    +    Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    + ## ci/lib.sh ##
    +@@ ci/lib.sh: MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
    + case "$CI_TYPE" in
    + github-actions)
    + 	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
    +-	GIT_TEST_OPTS="--verbose-log -x"
    ++	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
    + 	test Windows != "$RUNNER_OS" ||
    + 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    + 	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
    +
      ## ci/print-test-failures.sh ##
     @@ ci/print-test-failures.sh: set -e
      . ${0%/*}/lib-ci-type.sh
      . ${0%/*}/lib-tput.sh
      
    -+exit_code=
    ++github_workflow_markup=auto
    ++case "$CI_TYPE" in
    ++github-actions)
    ++	github_workflow_markup=t
    ++	;;
    ++esac
    ++
     +while test $# != 0
     +do
     +	case "$1" in
    -+	--exit-code)
    -+		exit_code=t
    ++	--github-workflow-markup)
    ++		github_workflow_markup=t
    ++		;;
    ++	--no-github-workflow-markup)
    ++		github_workflow_markup=
     +		;;
     +	*)
     +		echo "BUG: invalid $0 argument: $1" >&2
    @@ ci/print-test-failures.sh: set -e
      		TEST_NAME="${TEST_EXIT%.exit}"
      		TEST_NAME="${TEST_NAME##*/}"
      		TEST_OUT="${TEST_NAME}.out"
    -@@ ci/print-test-failures.sh: do
    - 		esac
    + 		TEST_MARKUP="${TEST_NAME}.markup"
    + 
    +-		echo "------------------------------------------------------------------------"
    +-		echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"
    +-		echo "------------------------------------------------------------------------"
    +-		cat "t/test-results/${TEST_OUT}"
    ++		do_markup=
    ++		case "$github_workflow_markup" in
    ++		t)
    ++			do_markup=t
    ++			;;
    ++		auto)
    ++			if test -f "t/test-results/$TEST_MARKUP"
    ++			then
    ++				do_markup=t
    ++			fi
    ++			;;
    ++		esac
    ++
    ++		if test -n "$do_markup"
    ++		then
    ++			printf "\\e[33m\\e[1m=== Failed test: ${TEST_NAME} ===\\e[m\\n"
    ++			cat "t/test-results/$TEST_MARKUP"
    ++		else
    ++			echo "------------------------------------------------------------------------"
    ++			echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"
    ++			echo "------------------------------------------------------------------------"
    ++			cat "t/test-results/${TEST_OUT}"
    ++		fi
    + 
    + 		trash_dir="trash directory.$TEST_NAME"
    + 		case "$CI_TYPE" in
    +@@ ci/print-test-failures.sh: then
    + 	then
    + 		echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
      	fi
    - done
     +
    -+if test -n "$failed"
    -+then
    -+	if test -n "$exit_code"
    -+	then
    ++	case "$github_workflow_markup" in
    ++	t|auto)
     +		exit 1
    -+	fi
    -+fi
    ++		;;
    ++	'')
    ++		exit 0
    ++		;;
    ++	esac
    + fi
 -:  ----------- >  8:  9c9fd24fffa ci(github): skip the logs of the successful test cases
 -:  ----------- >  9:  03d7b54d68b ci(github): skip "skip" tests in --github-workflow-markup
 -:  ----------- > 10:  90a152d79f9 ci(github): avoid printing test case preamble twice
 9:  877edff3b4d ! 11:  68eed85162b ci: use `--github-workflow-markup` in the GitHub workflow
    @@ Metadata
     Author: Johannes Schindelin <Johannes.Schindelin@gmx.de>
     
      ## Commit message ##
    -    ci: use `--github-workflow-markup` in the GitHub workflow
    +    ci(github): mention where the full logs can be found
     
    -    [Ævar: due to re-structuring on top of my series the {begin,end}_group
    -    in CI isn't needed at all to get "group" output for the test
    -    suite. This commit includes the now-squashed "ci/run-build-and-tests:
    -    add some structure to the GitHub workflow output":]
    -
    -    The current output of Git's GitHub workflow can be quite confusing,
    -    especially for contributors new to the project.
    -
    -    To make it more helpful, let's introduce some collapsible grouping.
    -    Initially, readers will see the high-level view of what actually
    -    happened (did the build fail, or the test suite?). To drill down, the
    -    respective group can be expanded.
    -
    -    Note: sadly, workflow output currently cannot contain any nested groups
    -    (see https://github.com/actions/runner/issues/802 for details),
    -    therefore we take pains to ensure to end any previous group before
    -    starting a new one.
    -
    -    [Ævar: The above comment isn't true anymore, as that limitation has
    -    been removed by basing this on my patches to run "make" and "make
    -    test" directly from the top-level of main.yml.
    -
    -    Those are now effectively their own "group", effectively giving this
    -    stage another group "level" to use. This means that the equivalent of
    -    "make test" won't be on the same level as an individual test failure.
    -
    -    We no longer take any pains to ensure balanced group output as a
    -    result (which was a caveat the previous ci/lib.sh implementation had
    -    to deal with., We just need to "cat" the generated *.markup]
    +    The full logs are contained in the `failed-tests-*.zip` artifacts that
    +    are attached to the failed CI run. Since this is not immediately
    +    obvious to the well-disposed reader, let's mention it explicitly.
     
    +    Suggested-by: Victoria Dye <vdye@github.com>
         Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    - ## ci/lib.sh ##
    -@@ ci/lib.sh: MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
    - case "$CI_TYPE" in
    - github-actions)
    - 	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
    --	GIT_TEST_OPTS="--verbose-log -x"
    -+	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
    - 	test Windows != "$RUNNER_OS" ||
    - 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    - 	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
    -
      ## ci/print-test-failures.sh ##
     @@ ci/print-test-failures.sh: do
    - 		if test -n "$github_workflow_markup"
    + 		if test -n "$do_markup"
      		then
      			printf "\\e[33m\\e[1m=== Failed test: ${TEST_NAME} ===\\e[m\\n"
    -+			cat "t/test-results/$TEST_MARKUP"
    ++			echo "The full logs are in the artifacts attached to this run."
    + 			cat "t/test-results/$TEST_MARKUP"
      		else
      			echo "------------------------------------------------------------------------"
    - 			echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"
    - 			echo "------------------------------------------------------------------------"
    -+			cat "t/test-results/${TEST_OUT}"
    - 		fi
    --		cat "t/test-results/${TEST_OUT}"
    - 
    - 		trash_dir="trash directory.$TEST_NAME"
    - 		case "$CI_TYPE" in
10:  6b278194f9a = 12:  11dab84b0dc ci: call `finalize_test_case_output` a little later
 5:  4291892fbd3 ! 13:  fbe0d99c6b3 ci: make it easier to find failed tests' logs in the GitHub workflow
    @@
      ## Metadata ##
    -Author: Johannes Schindelin <Johannes.Schindelin@gmx.de>
    +Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    ci: make it easier to find failed tests' logs in the GitHub workflow
    -
    -    When investigating a test failure, the time that matters most is the
    -    time it takes from getting aware of the failure to displaying the output
    -    of the failing test case.
    -
    -    You currently have to know a lot of implementation details when
    -    investigating test failures in the CI runs. The first step is easy: the
    -    failed job is marked quite clearly, but when opening it, the failed step
    -    is expanded, which in our case is the one running
    -    `ci/run-build-and-tests.sh`. This step, most notably, only offers a
    -    high-level view of what went wrong: it prints the output of `prove`
    -    which merely tells the reader which test script failed.
    -
    -    The actually interesting part is in the detailed log of said failed
    -    test script. But that log is shown in the CI run's step that runs
    -    `ci/print-test-failures.sh`. And that step is _not_ expanded in the web
    -    UI by default. It is even marked as "successful", which makes it very
    -    easy to miss that there is useful information hidden in there.
    -
    -    Let's help the reader by showing the failed tests' detailed logs in the
    -    step that is expanded automatically, i.e. directly after the test suite
    -    failed.
    -
    -    This also helps the situation where the _build_ failed and the
    -    `print-test-failures` step was executed under the assumption that the
    -    _test suite_ failed, and consequently failed to find any failed tests.
    -
    -    An alternative way to implement this patch would be to source
    -    `ci/print-test-failures.sh` in the `handle_test_failures` function to
    -    show these logs. However, over the course of the next few commits, we
    -    want to introduce some grouping which would be harder to achieve that
    -    way (for example, we do want a leaner, and colored, preamble for each
    -    failed test script, and it would be trickier to accommodate the lack of
    -    nested groupings in GitHub workflows' output).
    -
    -    Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
    +    CI: make --github-workflow-markup ci-config, off by default
    +
    +    In preceding commits the --github-workflow-markup output was made the
    +    default under GitHub CI.
    +
    +    There's a few outstanding issues with that new output target however:
    +
    +    A. As noted in [1] it's much slower in some common cases, i.e. the
    +       page rendering time (under Firefox & Chrome debugging) goes from ~20s
    +       to on the order of 60-80s (less pronounced when not debugging, but
    +       still in that ballpark). See also [2] for tests on an earlier iteration
    +       of the series.
    +
    +    B. Preceding commits in this series to omit non-failing test output
    +       may have mitigated that somewhat, i.e. skipping "ok" and "skip" tests
    +       in the *.markup files (although [1] is a benchmark with that "ok"
    +       change).
    +
    +       Per [3] there are some CI users that prefer to see only the full
    +       verbose output, i.e. we can't in the general case assume that we
    +       can emit only the trace output for the failing test, and that's
    +       going to give us the information we need to debug the test. It
    +       might be failing because of earlier setup, or from trace output
    +       outside of any test (i.e. in the main body of the file).
    +
    +       Users such as [3] could still just consult the *.out output, but
    +       then we'd make them download the archive instead of viewing the
    +       output in the web UX.
    +
    +    C. Per B above and [3] an earlier change in this series modified
    +       the *.out output to change the *.out output inadvertently while trying
    +       to modify the *.markup output. See the commit message of "ci(github):
    +       avoid printing test case preamble twice" earlier in this series.
    +
    +    It thus make sense for now to make the "ci/print-test-failures.sh"
    +    output be the "raw" one. I.e. the *.out output we'd get before
    +    --github-workflow-markup was implemented.
    +
    +    To make it use ci/print-test-failures.sh we can create a
    +    "ci/config/print-test-failures-output-type" script on the "ci-config"
    +    branch that does e.g.:
    +
    +            #!/bin/sh
    +            echo github
    +
    +    That script will (like the existing "check-ref") get the ref as an
    +    argument, so we can in the future turn this on experimentally for
    +    "seen", and flip the default in the future. The script can also emit
    +    nothing to get whatever our idea is of the default (currently "raw").
    +
    +    Note the corresponding change to ci/print-test-failures.sh, i.e. we'll
    +    do the right thing vis-a-vis what we "zoom in" to depending on the
    +    setting. If it's "raw" we'd like a failed "make test" to "zoom in" to
    +    that step, if it's "github" we'd like the to open the subsequent
    +    "ci/print-test-failures.sh" step instead.
    +
    +    1. https://lore.kernel.org/git/220523.86ee0kzjix.gmgdl@evledraar.gmail.com/
    +    2. https://lore.kernel.org/git/220222.86tucr6kz5.gmgdl@evledraar.gmail.com/
    +    3. https://lore.kernel.org/git/20210309175249.GE3590451@szeder.dev/
    +
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## .github/workflows/main.yml ##
     @@ .github/workflows/main.yml: jobs:
    +     runs-on: ubuntu-latest
    +     outputs:
    +       enabled: ${{ steps.check-ref.outputs.enabled }}${{ steps.skip-if-redundant.outputs.enabled }}
    ++      print-test-failures-output-type: ${{ steps.print-test-failures-output-type.outputs.type }}
    +     steps:
    +       - name: try to clone ci-config branch
    +         run: |
    +@@ .github/workflows/main.yml: jobs:
    +             config-repo &&
    +           cd config-repo &&
    +           git checkout HEAD -- ci/config || : ignore
    ++      - id: print-test-failures-output-type
    ++        name: check what output type ci/print-test-failures.sh uses
    ++        run: |
    ++          type_default=raw
    ++          type=$type_default
    ++
    ++          if test -x config-repo/ci/config/print-test-failures-output-type
    ++          then
    ++            type=$(config-repo/ci/config/print-test-failures-output-type '${{ github.ref }}')
    ++            if test -z "$type"
    ++            then
    ++              type=$type_default
    ++            fi
    ++          fi
    ++          echo "::set-output name=type::$type"
    +       - id: check-ref
    +         name: check whether CI is enabled for ref
    +         run: |
    +@@ .github/workflows/main.yml: jobs:
    +     - uses: git-for-windows/setup-git-for-windows-sdk@v1
    +     - run: ci/lib.sh --test
            shell: bash
    -     - name: test
    ++      env:
    ++        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
    +     - name: select tests
    +       run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10
            shell: bash
    --      run: . /etc/profile && make -C t -e
    --    - name: ci/print-test-failures.sh
    --      if: failure()
    --      shell: bash
    --      run: ci/print-test-failures.sh
    -+      run: . /etc/profile && make -C t -e || ci/print-test-failures.sh
    -     - name: Upload failed tests' directories
    -       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
    -       uses: actions/upload-artifact@v2
     @@ .github/workflows/main.yml: jobs:
    +       run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz
    +     - run: ci/lib.sh --test
            shell: bash
    -     - name: test
    ++      env:
    ++        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
    +     - name: select tests
    +       run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10
            shell: bash
    --      run: . /etc/profile && make -C t -e
    --    - name: ci/print-test-failures.sh
    --      if: failure()
    --      shell: bash
    --      run: ci/print-test-failures.sh
    -+      run: . /etc/profile && make -C t -e || ci/print-test-failures-github.sh
    -     - name: Upload failed tests' directories
    -       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
    -       uses: actions/upload-artifact@v2
     @@ .github/workflows/main.yml: jobs:
          - run: ci/lib.sh --build
          - run: make
          - run: ci/lib.sh --test
    --    - run: make test
    -+    - run: make test || ci/print-test-failures-github.sh
    ++      env:
    ++        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
    +     - run: make test
            if: success()
    --    - run: ci/print-test-failures.sh
    --      if: failure()
    -     - name: Upload failed tests' directories
    -       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
    -       uses: actions/upload-artifact@v2
    +     - run: ci/print-test-failures.sh
     @@ .github/workflows/main.yml: jobs:
          - run: make
          - run: ci/lib.sh --test
            if: success() && matrix.vector.skip-tests != 'yes'
    --    - run: make test
    -+    - run: make test || ci/print-test-failures-github.sh
    ++      env:
    ++        GIT_CI_PTF_OUTPUT_TYPE: ${{needs.ci-config.outputs.print-test-failures-output-type}}
    +     - run: make test
            if: success() && matrix.vector.skip-tests != 'yes'
    --    - run: ci/print-test-failures.sh
    --      if: failure() && matrix.vector.skip-tests != 'yes'
    -     - name: Upload failed tests' directories
    -       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
    -       uses: actions/upload-artifact@v1
    +     - run: ci/print-test-failures.sh
    +
    + ## ci/lib.sh ##
    +@@ ci/lib.sh: if test -n "$GITHUB_ENV"
    + then
    + 	echo "CONFIG: GITHUB_ENV=$GITHUB_ENV" >&2
    + fi
    ++echo "CONFIG: GIT_CI_PTF_OUTPUT_TYPE=$GIT_CI_PTF_OUTPUT_TYPE" >&2
    + echo >&2
    + 
    + # Helper functions
    +@@ ci/lib.sh: MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
    + case "$CI_TYPE" in
    + github-actions)
    + 	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
    +-	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
    ++	GIT_TEST_OPTS="--verbose-log -x"
    ++	if test -n "$GIT_CI_PTF_OUTPUT_TYPE"
    ++	then
    ++		# For later use in ci/print-test-failures.sh
    ++		setenv --test GIT_CI_PTF_OUTPUT_TYPE "$GIT_CI_PTF_OUTPUT_TYPE"
    ++
    ++		case "$GIT_CI_PTF_OUTPUT_TYPE" in
    ++		github)
    ++			GIT_TEST_OPTS="$GIT_TEST_OPTS --github-workflow-markup"
    ++			;;
    ++		esac
    ++	fi
    + 	test Windows != "$RUNNER_OS" ||
    + 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    + 	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
     
      ## ci/print-test-failures.sh ##
     @@ ci/print-test-failures.sh: set -e
    - . ${0%/*}/lib-ci-type.sh
      . ${0%/*}/lib-tput.sh
      
    --exit_code=
    -+case "$CI_TYPE" in
    -+github-actions)
    -+	exit_code=t
    -+	github_workflow_markup=t
    -+	;;
    -+*)
    -+	exit_code=
    + github_workflow_markup=auto
    +-case "$CI_TYPE" in
    +-github-actions)
    ++case "$GIT_CI_PTF_OUTPUT_TYPE" in
    ++github)
    + 	github_workflow_markup=t
    + 	;;
    ++raw)
     +	github_workflow_markup=
     +	;;
    -+esac
    -+
    - while test $# != 0
    - do
    - 	case "$1" in
    - 	--exit-code)
    - 		exit_code=t
    - 		;;
    -+	--no-exit-code)
    -+		exit_code=
    -+		;;
    -+	--github-workflow-markup)
    -+		github_workflow_markup=t
    -+		;;
    -+	--no-github-workflow-markup)
    -+		github_workflow_markup=
    -+		;;
    - 	*)
    - 		echo "BUG: invalid $0 argument: $1" >&2
    - 		exit 1
    -@@ ci/print-test-failures.sh: do
    - 		TEST_OUT="${TEST_NAME}.out"
    - 		TEST_MARKUP="${TEST_NAME}.markup"
    - 
    --		echo "------------------------------------------------------------------------"
    --		echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"
    --		echo "------------------------------------------------------------------------"
    -+		if test -n "$github_workflow_markup"
    -+		then
    -+			printf "\\e[33m\\e[1m=== Failed test: ${TEST_NAME} ===\\e[m\\n"
    -+		else
    -+			echo "------------------------------------------------------------------------"
    -+			echo "$(tput setaf 1)${TEST_OUT}...$(tput sgr0)"
    -+			echo "------------------------------------------------------------------------"
    -+		fi
    - 		cat "t/test-results/${TEST_OUT}"
    + esac
      
    - 		trash_dir="trash directory.$TEST_NAME"
    + while test $# != 0
 -:  ----------- > 14:  0b02b186c87 CI: make the --github-workflow-markup "github" output the default