diff mbox series

[4/5] gitlab: convert build/container jobs to .base_job_template

Message ID 20220526110705.59952-5-berrange@redhat.com (mailing list archive)
State New
Headers show
Series gitlab: restrict running jobs in forks and upstream master | expand

Commit Message

Daniel P. Berrangé May 26, 2022, 11:07 a.m. UTC
This converts the main build and container jobs to use the
base job rules, defining the following new variables

 - QEMU_JOB_SKIPPED - jobs that are known to be currently
   broken and should not be run. Can still be manually
   launched if desired.

 - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
   test harness.

 - QEMU_JOB_PUBLISH - jobs that publish content after the
   branch is merged upstream

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
 .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
 .gitlab-ci.d/container-cross.yml     |  6 ++----
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +++
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
 8 files changed, 65 insertions(+), 31 deletions(-)

Comments

Thomas Huth June 2, 2022, 5:41 p.m. UTC | #1
On 26/05/2022 13.07, Daniel P. Berrangé wrote:
> This converts the main build and container jobs to use the
> base job rules, defining the following new variables
> 
>   - QEMU_JOB_SKIPPED - jobs that are known to be currently
>     broken and should not be run. Can still be manually
>     launched if desired.
> 
>   - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
>     test harness.
> 
>   - QEMU_JOB_PUBLISH - jobs that publish content after the
>     branch is merged upstream
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
>   .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
>   .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
>   .gitlab-ci.d/container-cross.yml     |  6 ++----
>   .gitlab-ci.d/container-template.yml  |  1 +
>   .gitlab-ci.d/crossbuild-template.yml |  3 +++
>   .gitlab-ci.d/windows.yml             |  1 +
>   docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
>   8 files changed, 65 insertions(+), 31 deletions(-)
...
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index e9620c3074..ecac3ec50c 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -360,12 +360,11 @@ build-cfi-aarch64:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> +  variables:
>       # FIXME: This job is often failing, likely due to out-of-memory problems in
>       # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
> +    # skipped until the situation has been solved.
> +    QEMU_JOB_SKIPPED: 1
>   
>   check-cfi-aarch64:
>     extends: .native_test_job_template
> @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> +  variables:
>       # FIXME: This job is often failing, likely due to out-of-memory problems in
>       # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
> +    # skipped until the situation has been solved.
> +    QEMU_JOB_SKIPPED: 1

FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs 
since they've now got two "variables:" sections and apparently only the 
second one is taken into account...

  Thomas
Daniel P. Berrangé June 6, 2022, 10:50 a.m. UTC | #2
On Thu, Jun 02, 2022 at 07:41:59PM +0200, Thomas Huth wrote:
> On 26/05/2022 13.07, Daniel P. Berrangé wrote:
> > This converts the main build and container jobs to use the
> > base job rules, defining the following new variables
> > 
> >   - QEMU_JOB_SKIPPED - jobs that are known to be currently
> >     broken and should not be run. Can still be manually
> >     launched if desired.
> > 
> >   - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
> >     test harness.
> > 
> >   - QEMU_JOB_PUBLISH - jobs that publish content after the
> >     branch is merged upstream
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
> >   .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
> >   .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
> >   .gitlab-ci.d/container-cross.yml     |  6 ++----
> >   .gitlab-ci.d/container-template.yml  |  1 +
> >   .gitlab-ci.d/crossbuild-template.yml |  3 +++
> >   .gitlab-ci.d/windows.yml             |  1 +
> >   docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
> >   8 files changed, 65 insertions(+), 31 deletions(-)
> ...
> > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> > index e9620c3074..ecac3ec50c 100644
> > --- a/.gitlab-ci.d/buildtest.yml
> > +++ b/.gitlab-ci.d/buildtest.yml
> > @@ -360,12 +360,11 @@ build-cfi-aarch64:
> >       expire_in: 2 days
> >       paths:
> >         - build
> > -  rules:
> > +  variables:
> >       # FIXME: This job is often failing, likely due to out-of-memory problems in
> >       # the constrained containers of the shared runners. Thus this is marked as
> > -    # manual until the situation has been solved.
> > -    - when: manual
> > -      allow_failure: true
> > +    # skipped until the situation has been solved.
> > +    QEMU_JOB_SKIPPED: 1
> >   check-cfi-aarch64:
> >     extends: .native_test_job_template
> > @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
> >       expire_in: 2 days
> >       paths:
> >         - build
> > -  rules:
> > +  variables:
> >       # FIXME: This job is often failing, likely due to out-of-memory problems in
> >       # the constrained containers of the shared runners. Thus this is marked as
> > -    # manual until the situation has been solved.
> > -    - when: manual
> > -      allow_failure: true
> > +    # skipped until the situation has been solved.
> > +    QEMU_JOB_SKIPPED: 1
> 
> FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs
> since they've now got two "variables:" sections and apparently only the
> second one is taken into account...

Opps, sorry about that, and of course I didn't notice as it is a skipped
job and I forgot to try running it :-(

With regards,
Daniel
diff mbox series

Patch

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 9a0b8d7f97..6a918abbda 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -16,10 +16,22 @@ 
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Publishing jobs should only run on the default branch in upstream
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+      when: never
+
+    # Non-publishing jobs should only run on staging branches in upstream
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+      when: never
+
     # Jobs only intended for forks should always be skipped on upstram
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
@@ -31,6 +43,16 @@ 
       when: manual
       allow_failure: true
 
+    # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED'
+      when: manual
+      allow_failure: true
+
+    # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index dc6d67aacf..1874c07a4b 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,4 +1,5 @@ 
 .native_build_job_template:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +28,7 @@ 
       fi
 
 .common_test_job_template:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -75,15 +77,5 @@ 
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_AVOCADO: 1
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index e9620c3074..ecac3ec50c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -360,12 +360,11 @@  build-cfi-aarch64:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -402,12 +401,11 @@  build-cfi-ppc64-s390x:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
@@ -579,6 +577,7 @@  build-without-default-features:
     MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
@@ -595,10 +594,13 @@  build-tools-and-docs-debian:
   extends: .native_build_job_template
   needs:
     job: amd64-debian-container
+    # when running on 'master' we use pre-existing container
+    optional: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+    QEMU_JOB_PUBLISH: 1
   artifacts:
     expire_in: 2 days
     paths:
@@ -618,6 +620,7 @@  build-tools-and-docs-debian:
 # that users can see the results of their commits, regardless
 # of what topic branch they're currently using
 pages:
+  extends: .base_job_template
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
@@ -635,10 +638,5 @@  pages:
   artifacts:
     paths:
       - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
+  variables:
+    QEMU_JOB_PUBLISH: 1
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index e622ac2d21..90ecfd8da0 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -42,15 +42,13 @@  armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
+    QEMU_JOB_ONLY_FORKS: 1
   services:
     - docker:dind
   before_script:
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 1baecd9460..c434b9c8f3 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,4 +1,5 @@ 
 .container_job_template:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
   services:
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 29c3c2b826..28b2142ec2 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,4 +1,5 @@ 
 .cross_system_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -24,6 +25,7 @@ 
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -36,6 +38,7 @@ 
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index cf7724b8e5..1b2ede49e1 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -1,4 +1,5 @@ 
 .shared_msys2_builder:
+  extends: .base_job_template
   tags:
   - shared-windows
   - windows
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 4c7e30ab08..0b4926e537 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -73,6 +73,25 @@  The job results are only of interest to contributors prior to
 submitting code. They are not required as part of the gating
 CI pipeline.
 
+QEMU_JOB_SKIPPED
+~~~~~~~~~~~~~~~~
+
+The job is not reliably successsful in general, so is not
+currently suitable to be run by default. Ideally this should
+be a temporary marker until the problems can be addressed, or
+the job permanently removed.
+
+QEMU_JOB_PUBLISH
+~~~~~~~~~~~~~~~~
+
+The job is for publishing content after a branch has been
+merged into the upstream default branch.
+
+QEMU_JOB_AVOCADO
+~~~~~~~~~~~~~~~~
+
+The job runs the Avocado integration test suite
+
 Contributor controlled runtime variables
 ----------------------------------------