[v6,00/29] CI: run "make" in CI "steps", improve UX

Message

Ævar Arnfjörð Bjarmason May 25, 2022, 10:03 a.m. UTC

This series changes CI "step" targets that are shellscripts that do N
things to instead be single command invocations at the "step" level,
driven by the CI recipe itself.

To do that we need to pass state that we previously re-setup for every
"step" via $GITHUB_ENV, whose state we then helpfully show (this is just
a standard GitHub CI feature) in a drop-down at the start of every
"step".

I.e. at the tip of this series you can reliably look at that
$GITHUB_ENV view to see what the full and relevant environment was for
that "make", "make test" or whatever.

The only changes since v5 are to rebase this on an updated "master",
which had some small fixes in the ci/ directory. For the last
iteration see:
https://lore.kernel.org/git/[e-mail]/

I'll follow this up with a re-roll of
https://lore.kernel.org/git/[e-mail]/
which is a replacement for
https://lore.kernel.org/git/[e-mail]/
based on this topic.

To view the CI output differences this topic makes see these two runs:

  master: https://github.com/avar/git/actions/runs/2377697338
  this: https://github.com/avar/git/actions/runs/2377257487

These runs have a commit on top to trigger CI build and test failures
in various areas. One example is here:

 master: https://github.com/avar/git/runs/6573059000?check_suite_focus=true
 this: https://github.com/avar/git/runs/6571804790?check_suite_focus=true

This shows that we fail on "make" instead of a monolothic
"ci/run-build-and-test.sh" step. Furthermore we can see that a
preceding "ci/lib.sh --build" step has been added to set & export (via
$GITHUB_ENV) the full set of variables "make" will use in the failing
step. Those are then visible at (click to expand it):

  https://github.com/avar/git/runs/6571804790?check_suite_focus=true#step:5:1

For every step we now do the same, i.e. it's immediately obvious what
parameters go into running that step. The last step in this series
then makes it easy to run builds and tests "like CI" using the same
tooling:

	$ ci/lib.sh 
	error: must set a CI jobname in the environment
	usage: jobname=<job-name> [runs_on_pool=<pool-name>] ci/lib.sh (--build | --test)

	Running ci/lib.sh outside of CI? You can use ci/lib.sh to set up your
	environment like a given CI job. E.g.:

	        # run "make all test" like the "linux-leaks" job
	        (eval $(jobname=linux-leaks ci/lib.sh --all) && make test)

	        # run "make all test" like the "linux-musl" job
	        (eval $(jobname=linux-musl ci/lib.sh --all) && make test)

	        # run "make test" like the "linux-TEST-vars" job (uses various GIT_TEST_* modes)
	        make && (eval $(jobname=linux-TEST-vars ci/lib.sh --test) && make test)

	        # run "make test" like the "linux-sha256" job
	        make && (eval $(jobname=linux-sha256 ci/lib.sh --test) && make test)

	Note that some of these (e.g. the linux-musl one) may not work as
	expected due to the CI job configuring a platform that may not match
	yours.

As noted in the usage this can then be directly eval'd:

	$ jobname=osx-gcc ci/lib.sh --build 2>/dev/null 
	MAKEFLAGS="j8 DEVELOPER=1 SKIP_DASHED_BUILT_INS=YesPlease PYTHON_PATH=/usr/bin/python3 CC=gcc"
	export MAKEFLAGS

Of course YMMV when doing so, running that on my Linux box won't
magically give me an OSX environment. But it's very useful to be able
to to e.g.:

	jobname=linux-TEST-vars ci/lib.sh --test

Which allows myself and others to tweak our local build scripts to
test things locally with setups that mirror these CI setups as far as
GIT_TEST_* etc. parameters are concerned.

Ævar Arnfjörð Bjarmason (29):
  CI: run "set -ex" early in ci/lib.sh
  CI: make "$jobname" explicit, remove fallback
  CI: remove more dead Travis CI support
  CI: remove dead "tree skipping" code
  CI: remove unused Azure ci/* code
  CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS
  CI: don't have "git grep" invoke a pager in tree content check
  CI: have "static-analysis" run a "make ci-static-analysis" target
  CI: have "static-analysis" run "check-builtins", not "documentation"
  CI: move p4 and git-lfs variables to ci/install-dependencies.sh
  CI: consistently use "export" in ci/lib.sh
  CI: export variables via a wrapper
  CI: remove "run-build-and-tests.sh", run "make [test]" directly
  CI: make ci/{lib,install-dependencies}.sh POSIX-compatible
  CI: check ignored unignored build artifacts in "win[+VS] build" too
  CI: invoke "make artifacts-tar" directly in windows-build
  CI: split up and reduce "ci/test-documentation.sh"
  CI: combine ci/install{,-docker}-dependencies.sh
  CI: move "env" definitions into ci/lib.sh
  ci/run-test-slice.sh: replace shelling out with "echo"
  CI: pre-select test slice in Windows & VS tests
  CI: only invoke ci/lib.sh as "steps" in main.yml
  CI: narrow down variable definitions in --build and --test
  CI: add more variables to MAKEFLAGS, except under vs-build
  CI: set CC in MAKEFLAGS directly, don't add it to the environment
  CI: set SANITIZE=leak in MAKEFLAGS directly
  CI: set PYTHON_PATH setting for osx-{clang,gcc} into "$jobname" case
  CI: don't use "set -x" in "ci/lib.sh" output
  CI: make it easy to use ci/*.sh outside of CI

 .github/workflows/main.yml            | 101 +++---
 Makefile                              |  31 +-
 ci/check-directional-formatting.bash  |   2 +-
 ci/check-unignored-build-artifacts.sh |  20 ++
 ci/install-dependencies.sh            |  82 +++--
 ci/install-docker-dependencies.sh     |  22 --
 ci/lib-ci-type.sh                     |   6 +
 ci/lib-online_cpus.sh                 |  26 ++
 ci/lib-tput.sh                        |   5 +
 ci/lib.sh                             | 424 ++++++++++++++++----------
 ci/make-test-artifacts.sh             |  12 -
 ci/mount-fileshare.sh                 |  25 --
 ci/print-test-failures.sh             |  53 +---
 ci/run-build-and-tests.sh             |  54 ----
 ci/run-docker-build.sh                |  66 ----
 ci/run-docker.sh                      |  47 ---
 ci/run-static-analysis.sh             |  32 --
 ci/run-test-slice.sh                  |  17 --
 ci/select-test-slice.sh               |  13 +
 ci/test-documentation.sh              |  37 +--
 ci/util/extract-trash-dirs.sh         |  50 ---
 shared.mak                            |   1 +
 22 files changed, 508 insertions(+), 618 deletions(-)
 create mode 100755 ci/check-unignored-build-artifacts.sh
 delete mode 100755 ci/install-docker-dependencies.sh
 create mode 100644 ci/lib-ci-type.sh
 create mode 100644 ci/lib-online_cpus.sh
 create mode 100644 ci/lib-tput.sh
 delete mode 100755 ci/make-test-artifacts.sh
 delete mode 100755 ci/mount-fileshare.sh
 delete mode 100755 ci/run-build-and-tests.sh
 delete mode 100755 ci/run-docker-build.sh
 delete mode 100755 ci/run-docker.sh
 delete mode 100755 ci/run-static-analysis.sh
 delete mode 100755 ci/run-test-slice.sh
 create mode 100755 ci/select-test-slice.sh
 delete mode 100755 ci/util/extract-trash-dirs.sh

Range-diff against v5:
 1:  84d8098e365 =  1:  25cb08ad06c CI: run "set -ex" early in ci/lib.sh
 2:  3f5c071b0f6 =  2:  1be7910c728 CI: make "$jobname" explicit, remove fallback
 3:  c64845a4efb !  3:  234e248e831 CI: remove more dead Travis CI support
    @@ ci/lib.sh: then
     -	test macos != "$CI_OS_NAME" || CI_OS_NAME=osx
     -	CI_REPO_SLUG="$GITHUB_REPOSITORY"
      	CI_JOB_ID="$GITHUB_RUN_ID"
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
      	DONT_SKIP_TAGS=t
     @@ ci/lib.sh: then
      	export GIT_PROVE_OPTS="--timer --jobs 10"
 4:  45856e67d5d !  4:  36a6287ee9b CI: remove dead "tree skipping" code
    @@ ci/lib.sh: then
     -	CI_BRANCH="$GITHUB_REF"
     -	CI_COMMIT="$GITHUB_SHA"
     -	CI_JOB_ID="$GITHUB_RUN_ID"
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
     -	DONT_SKIP_TAGS=t
     -
     -	cache_dir="$HOME/none"
 5:  d4671148eb7 !  5:  66b0a383daf CI: remove unused Azure ci/* code
    @@ ci/install-dependencies.sh: macos-latest)
     -	test -z "$BREW_INSTALL_PACKAGES" ||
     -	brew install $BREW_INSTALL_PACKAGES
      	brew link --force gettext
    - 	brew install --cask --no-quarantine perforce || {
    - 		# Update the definitions and try again
    + 	mkdir -p $HOME/bin
    + 	(
     @@ ci/install-dependencies.sh: Documentation)
      	sudo apt-get -q update
      	sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make
    @@ ci/lib.sh: export TERM=${TERM:-dumb}
     +if test "$GITHUB_ACTIONS" = "true"
      then
      	CI_TYPE=github-actions
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
     
      ## ci/mount-fileshare.sh (deleted) ##
     @@
 6:  45ea80d0495 !  6:  d96333c1bd8 CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS
    @@ ci/lib.sh: check_unignored_build_artifacts ()
      if test "$GITHUB_ACTIONS" = "true"
      then
      	CI_TYPE=github-actions
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
      
     -	export GIT_PROVE_OPTS="--timer --jobs 10"
     +	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
 7:  a264b40ba0d =  7:  d0780ab4a96 CI: don't have "git grep" invoke a pager in tree content check
 8:  4a58f7be8d4 =  8:  84e368736e2 CI: have "static-analysis" run a "make ci-static-analysis" target
 9:  78c0c81af43 =  9:  1a9c98b909c CI: have "static-analysis" run "check-builtins", not "documentation"
10:  07e36035a1e ! 10:  e208a9ab1e2 CI: move p4 and git-lfs variables to ci/install-dependencies.sh
    @@ ci/install-dependencies.sh
      
      . ${0%/*}/lib.sh
      
    --P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION
    +-P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION
     -LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
      UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev
       tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl
    @@ ci/install-dependencies.sh
     +	GIT_LFS_PATH="$HOME/custom/git-lfs"
     +	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
     +
    -+	P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION
    ++	P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION
     +	LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
     +
      	sudo apt-get -q update
11:  272bd590a95 ! 11:  9655bae9e21 CI: consistently use "export" in ci/lib.sh
    @@ ci/lib.sh: export TERM=${TERM:-dumb}
      if test "$GITHUB_ACTIONS" = "true"
      then
     @@ ci/lib.sh: then
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
      
      	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
     -	export GIT_TEST_OPTS="--verbose-log -x"
12:  ea35327e24c ! 12:  2da0aa1d40e CI: export variables via a wrapper
    @@ ci/lib.sh: check_unignored_build_artifacts ()
      NPROC=10
     @@ ci/lib.sh: then
      	CI_TYPE=github-actions
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
      
     -	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
     +	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
13:  f6ff81f48ab ! 13:  6e97633652e CI: remove "run-build-and-tests.sh", run "make [test]" directly
    @@ ci/install-dependencies.sh: ubuntu-latest)
     +		echo "$PATH" >>"$GITHUB_PATH"
     +	fi
      
    - 	P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION
    + 	P4WHENCE=https://cdist2.perforce.com/perforce/r$LINUX_P4_VERSION
      	LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
     
      ## ci/lib.sh ##
14:  533dbc50c4f = 14:  0488e29859b CI: make ci/{lib,install-dependencies}.sh POSIX-compatible
15:  6044c2b383f = 15:  f7ac6c33044 CI: check ignored unignored build artifacts in "win[+VS] build" too
16:  4d1a9f88c32 = 16:  f89346f11eb CI: invoke "make artifacts-tar" directly in windows-build
17:  5c8f464a60f = 17:  7fa9c69e3ca CI: split up and reduce "ci/test-documentation.sh"
18:  a218be76819 ! 18:  7925b2610fc CI: combine ci/install{,-docker}-dependencies.sh
    @@ ci/install-dependencies.sh: linux-gcc-default)
     +	;;
      esac
      
    - if type p4d >/dev/null && type p4 >/dev/null
    + if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1
     
      ## ci/install-docker-dependencies.sh (deleted) ##
     @@
19:  83e21b4f501 = 19:  b00abc07637 CI: move "env" definitions into ci/lib.sh
20:  fd0e1610577 = 20:  bb5960355fe ci/run-test-slice.sh: replace shelling out with "echo"
21:  0c7cd9d64ba = 21:  1eeb2e8e7f0 CI: pre-select test slice in Windows & VS tests
22:  1b077665352 ! 22:  39979d9887d CI: only invoke ci/lib.sh as "steps" in main.yml
    @@ ci/lib.sh: setenv () {
     -	CI_TYPE=github-actions
     +case "$CI_TYPE" in
     +github-actions)
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
      
      	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
     @@ ci/lib.sh: then
23:  1867da7eddb = 23:  e2b3bf032d6 CI: narrow down variable definitions in --build and --test
24:  a4e3a9d2c28 ! 24:  c6f47d52823 CI: add more variables to MAKEFLAGS, except under vs-build
    @@ ci/lib.sh: COMMON_MAKEFLAGS=--jobs=$NPROC
     +
      case "$CI_TYPE" in
      github-actions)
    - 	CC="${CC:-gcc}"
    + 	CC="${CC_PACKAGE:-${CC:-gcc}}"
     @@ ci/lib.sh: github-actions)
      	;;
      esac
25:  48d99cee809 ! 25:  377460e8024 CI: set CC in MAKEFLAGS directly, don't add it to the environment
    @@ ci/install-dependencies.sh: UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl
      ubuntu-latest)
      	# The Linux build installs the defined dependency versions below.
     @@ ci/install-dependencies.sh: macos-latest)
    - 	} ||
    - 	brew install homebrew/cask/perforce
    + 	PATH="$PATH:${HOME}/bin"
    + 	export PATH
      
     -	if test -n "$CC_PACKAGE"
     +	if test -n "$BREW_CC_PACKAGE"
    @@ ci/lib.sh: MAKEFLAGS="$MAKEFLAGS SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS"
      
      case "$CI_TYPE" in
      github-actions)
    --	CC="${CC:-gcc}"
    +-	CC="${CC_PACKAGE:-${CC:-gcc}}"
     -
      	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
      	GIT_TEST_OPTS="--verbose-log -x"
26:  07a6c087297 = 26:  98e320d5e67 CI: set SANITIZE=leak in MAKEFLAGS directly
27:  50f21ffdffe = 27:  282b7c89a4f CI: set PYTHON_PATH setting for osx-{clang,gcc} into "$jobname" case
28:  e35dff2bef3 = 28:  7a208927819 CI: don't use "set -x" in "ci/lib.sh" output
29:  13d8c6997c3 = 29:  4d8a8f3be07 CI: make it easy to use ci/*.sh outside of CI