Message ID | f3f2c98a0dc6042b7ed5eab9c10bee4f64858f02.1698742590.git.ps@pks.im (mailing list archive) |
---|---|
State | Superseded |
Headers | show |
Series | ci: add GitLab CI definition | expand |
Patrick Steinhardt wrote:> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh > index 6e845283680..48cb2e735b5 100755 > --- a/ci/install-docker-dependencies.sh > +++ b/ci/install-docker-dependencies.sh > @@ -7,6 +7,9 @@ > > begin_group "Install dependencies" > > +# Required so that apt doesn't wait for user input on certain packages. > +export DEBIAN_FRONTEND=noninteractive > + > case "$jobname" in > linux32) > linux32 --32bit i386 sh -c ' > @@ -16,11 +19,19 @@ linux32) > ' > ;; > linux-musl) > - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ > + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ > pcre2-dev python3 musl-libintl perl-utils ncurses \ > apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ > bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null > ;; > +linux-*) > + apt update -q && > + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ > + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ > + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ > + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ > + apache2 cvs cvsps gnupg libcgi-pm-perl subversion > + ;; > pedantic) > dnf -yq update >/dev/null && > dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ... > diff --git a/ci/lib.sh b/ci/lib.sh > index e14b1029fad..6e3d64004ec 100755 > --- a/ci/lib.sh > +++ b/ci/lib.sh > @@ -208,6 +224,7 @@ then > cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" > > GIT_TEST_OPTS="--write-junit-xml" > + JOBS=10 > elif test true = "$GITHUB_ACTIONS" > then > CI_TYPE=github-actions ... > -MAKEFLAGS="$MAKEFLAGS --jobs=10" > -GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" > +MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}" > +GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save" > Organizationally, this commit seems to be doing two things at once: - Adding GitLab-specific CI setup (either in the new .gitlab-ci.yml or in conditions gated on "gitlab-ci"). - Updating the common CI scripts with things that are needed for GitLab CI, but aren't conditioned on it (i.e. the patch excerpts I've included above). I'd prefer these being separated into two patches, mainly to isolate "things that affect all CI" from "things that affect only GitLab CI". This is ultimately a pretty minor nit, though; if you're not planning on re-rolling (or just disagree with what I'm suggesting :) ), I'm okay with leaving it as-is. Otherwise, I can't comment on the correctness of the GitLab CI definition (I assume you've tested it anyway), but AFAICT the changes above shouldn't break GitHub CI.
On Tue, Oct 31, 2023 at 10:47:44AM -0700, Victoria Dye wrote: > Patrick Steinhardt wrote:> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh > > index 6e845283680..48cb2e735b5 100755 > > --- a/ci/install-docker-dependencies.sh > > +++ b/ci/install-docker-dependencies.sh > > @@ -7,6 +7,9 @@ > > > > begin_group "Install dependencies" > > > > +# Required so that apt doesn't wait for user input on certain packages. > > +export DEBIAN_FRONTEND=noninteractive > > + > > case "$jobname" in > > linux32) > > linux32 --32bit i386 sh -c ' > > @@ -16,11 +19,19 @@ linux32) > > ' > > ;; > > linux-musl) > > - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ > > + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ > > pcre2-dev python3 musl-libintl perl-utils ncurses \ > > apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ > > bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null > > ;; > > +linux-*) > > + apt update -q && > > + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ > > + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ > > + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ > > + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ > > + apache2 cvs cvsps gnupg libcgi-pm-perl subversion > > + ;; > > pedantic) > > dnf -yq update >/dev/null && > > dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null > > ... > > > diff --git a/ci/lib.sh b/ci/lib.sh > > index e14b1029fad..6e3d64004ec 100755 > > --- a/ci/lib.sh > > +++ b/ci/lib.sh > > @@ -208,6 +224,7 @@ then > > cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" > > > > GIT_TEST_OPTS="--write-junit-xml" > > + JOBS=10 > > elif test true = "$GITHUB_ACTIONS" > > then > > CI_TYPE=github-actions > > ... > > > -MAKEFLAGS="$MAKEFLAGS --jobs=10" > > -GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" > > +MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}" > > +GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save" > > > > Organizationally, this commit seems to be doing two things at once: > > - Adding GitLab-specific CI setup (either in the new .gitlab-ci.yml or in > conditions gated on "gitlab-ci"). > - Updating the common CI scripts with things that are needed for GitLab CI, > but aren't conditioned on it (i.e. the patch excerpts I've included > above). > > I'd prefer these being separated into two patches, mainly to isolate "things > that affect all CI" from "things that affect only GitLab CI". This is > ultimately a pretty minor nit, though; if you're not planning on re-rolling > (or just disagree with what I'm suggesting :) ), I'm okay with leaving it > as-is. Yeah, the JOBS refactoring can certainly be split out into a preparatory commit where we unify the envvars (currently patch 5). But for the other changes it makes a bit less sense to do so, in my opinion: - The DEBIAN_FRONTEND variable isn't needed before as the there are no Docker-based CI jobs that use apt. - Adding the shadow and sudo packages to the linux-musl job wouldn't be needed either as there are no cases yet where we run unprivileged CI builds via Docker. - Adding the apt packages as a preparatory step doesn't make much sense either as there is no Docker job using it. But anyway. I will: - Move around the JOBS variable refactoring to a preparatory patch, which feels sensible to me. - Move the `DEBIAN_FRONTEND` varible into the "linux-*" case, which should further clarify that this only impacts the newly added and thus GitLab-specific infrastructure. With these changes, the only thing left in this commit that is not guarded by a GitLab CI specific condition is the change to the "linux-musl" case where we install shadow and sudo now. But I don't feel like it makes sense to move them into a standalone preparatory commit. Thanks! Patrick > Otherwise, I can't comment on the correctness of the GitLab CI definition (I > assume you've tested it anyway), but AFAICT the changes above shouldn't break > GitHub CI.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 00000000000..cd98bcb18aa --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,53 @@ +default: + timeout: 2h + +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED == "true" + +test: + image: $image + before_script: + - ./ci/install-docker-dependencies.sh + script: + - useradd builder --create-home + - chown -R builder "${CI_PROJECT_DIR}" + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh + after_script: + - | + if test "$CI_JOB_STATUS" != 'success' + then + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh + fi + parallel: + matrix: + - jobname: linux-sha256 + image: ubuntu:latest + CC: clang + - jobname: linux-gcc + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-TEST-vars + image: ubuntu:20.04 + CC: gcc + CC_PACKAGE: gcc-8 + - jobname: linux-gcc-default + image: ubuntu:latest + CC: gcc + - jobname: linux-leaks + image: ubuntu:latest + CC: gcc + - jobname: linux-asan-ubsan + image: ubuntu:latest + CC: clang + - jobname: pedantic + image: fedora:latest + - jobname: linux-musl + image: alpine:latest + artifacts: + paths: + - t/failed-test-artifacts + when: on_failure diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh index 6e845283680..48cb2e735b5 100755 --- a/ci/install-docker-dependencies.sh +++ b/ci/install-docker-dependencies.sh @@ -7,6 +7,9 @@ begin_group "Install dependencies" +# Required so that apt doesn't wait for user input on certain packages. +export DEBIAN_FRONTEND=noninteractive + case "$jobname" in linux32) linux32 --32bit i386 sh -c ' @@ -16,11 +19,19 @@ linux32) ' ;; linux-musl) - apk add --update build-base curl-dev openssl-dev expat-dev gettext \ + apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \ pcre2-dev python3 musl-libintl perl-utils ncurses \ apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \ bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null ;; +linux-*) + apt update -q && + apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \ + libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \ + perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \ + libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \ + apache2 cvs cvsps gnupg libcgi-pm-perl subversion + ;; pedantic) dnf -yq update >/dev/null && dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null diff --git a/ci/lib.sh b/ci/lib.sh index e14b1029fad..6e3d64004ec 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -14,6 +14,22 @@ then need_to_end_group= echo '::endgroup::' >&2 } +elif test true = "$GITLAB_CI" +then + begin_group () { + need_to_end_group=t + printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n" + trap "end_group '$1'" EXIT + set -x + } + + end_group () { + test -n "$need_to_end_group" || return 0 + set +x + need_to_end_group= + printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n" + trap - EXIT + } else begin_group () { :; } end_group () { :; } @@ -208,6 +224,7 @@ then cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME" GIT_TEST_OPTS="--write-junit-xml" + JOBS=10 elif test true = "$GITHUB_ACTIONS" then CI_TYPE=github-actions @@ -227,14 +244,43 @@ then cache_dir="$HOME/none" GIT_TEST_OPTS="--github-workflow-markup" + JOBS=10 +elif test true = "$GITLAB_CI" +then + CI_TYPE=gitlab-ci + CI_BRANCH="$CI_COMMIT_REF_NAME" + CI_COMMIT="$CI_COMMIT_SHA" + case "$CI_JOB_IMAGE" in + macos-*) + CI_OS_NAME=osx;; + alpine:*|fedora:*|ubuntu:*) + CI_OS_NAME=linux;; + *) + echo "Could not identify OS image" >&2 + env >&2 + exit 1 + ;; + esac + CI_REPO_SLUG="$CI_PROJECT_PATH" + CI_JOB_ID="$CI_JOB_ID" + CC="${CC_PACKAGE:-${CC:-gcc}}" + DONT_SKIP_TAGS=t + handle_failed_tests () { + create_failed_test_artifacts + } + + cache_dir="$HOME/none" + + runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -) + JOBS=$(nproc) else echo "Could not identify CI type" >&2 env >&2 exit 1 fi -MAKEFLAGS="$MAKEFLAGS --jobs=10" -GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save" +MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}" +GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save" GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x" if test windows = "$CI_OS_NAME" diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh index 57277eefcd0..c33ad4e3a22 100755 --- a/ci/print-test-failures.sh +++ b/ci/print-test-failures.sh @@ -51,6 +51,12 @@ do tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" continue ;; + gitlab-ci) + mkdir -p failed-test-artifacts + cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/ + tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir" + continue + ;; *) echo "Unhandled CI type: $CI_TYPE" >&2 exit 1
We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. The integration does not necessarily have to be a first-class citizen, which would in practice only add to the fallout that pipeline failures have for the maintainer. That being said, we are happy to maintain this alternative CI setup for the Git project and will make test results available as part of our own mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt <ps@pks.im> --- .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++ ci/install-docker-dependencies.sh | 13 +++++++- ci/lib.sh | 50 +++++++++++++++++++++++++++-- ci/print-test-failures.sh | 6 ++++ 4 files changed, 119 insertions(+), 3 deletions(-) create mode 100644 .gitlab-ci.yml