From patchwork Thu May 26 11:07:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 12862285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61F92C433EF for ; Thu, 26 May 2022 11:19:21 +0000 (UTC) Received: from localhost ([::1]:51740 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nuBWS-0000AP-3z for qemu-devel@archiver.kernel.org; Thu, 26 May 2022 07:19:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39170) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKi-0007oq-Um for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22417) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKg-0003LZ-Vq for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653563230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dsBD/TtWA9TENON+NIB8vN8zw4Jh4ObFyQIpvFEd9bo=; b=JyUZNfz51W6q83HdXsA4FZt7jEq2yN3tXO2mSsCzoKR8mDuemNA3yuMBqKAxTpaYQY9mqn 2PCjrmFS0Gds0VvYsENI6Veucv1VJqxFRn8eYMkTSamWOPKvIe5Z2CPEZhgi+AbD73knP0 oUEbwSNytJSQI6NuA5Z+9smqXVf0Lo4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-222-9vRTYJibNTu2PVvq1ppg5g-1; Thu, 26 May 2022 07:07:07 -0400 X-MC-Unique: 9vRTYJibNTu2PVvq1ppg5g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C81B13802B89; Thu, 26 May 2022 11:07:06 +0000 (UTC) Received: from virtlab420.virt.lab.eng.bos.redhat.com (virtlab420.virt.lab.eng.bos.redhat.com [10.19.152.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9477482872; Thu, 26 May 2022 11:07:06 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Beraldo Leal , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 1/5] gitlab: introduce a common base job template Date: Thu, 26 May 2022 07:07:01 -0400 Message-Id: <20220526110705.59952-2-berrange@redhat.com> In-Reply-To: <20220526110705.59952-1-berrange@redhat.com> References: <20220526110705.59952-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Currently job rules are spread across the various templates and jobs, making it hard to understand exactly what runs in what scenario. This leads to inconsistency in the rules and increased maint burden. The intent is that we introduce a common '.base_job_template' which will have a general purpose 'rules:' block. No other template or job should define 'rules:', but instead they must rely on the inherited rules. To allow behaviour to be tweaked, rules will be influenced by a number of variables with the naming scheme 'QEMU_JOB_nnnn'. Signed-off-by: Daniel P. Berrangé --- .gitlab-ci.d/base.yml | 28 +++++++++++++++++++++++++++ .gitlab-ci.d/qemu-project.yml | 1 + docs/devel/ci-jobs.rst.inc | 36 ++++++++++++++++++++++++++++++++++- 3 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 .gitlab-ci.d/base.yml diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml new file mode 100644 index 0000000000..10eb6ab8bc --- /dev/null +++ b/.gitlab-ci.d/base.yml @@ -0,0 +1,28 @@ + +# The order of rules defined here is critically important. +# They are evaluated in order and first match wins. +# +# Thus we group them into a number of stages, ordered from +# most restrictive to least restrictive +# +.base_job_template: + rules: + ############################################################# + # Stage 1: exclude scenarios where we definitely don't + # want jobs to run + ############################################################# + + + ############################################################# + # Stage 2: fine tune execution of jobs in specific scenarios + # where the catch all logic is inapprorpaite + ############################################################# + + + ############################################################# + # Stage 3: catch all logic applying to any job not matching + # an earlier criteria + ############################################################# + + # Jobs can run if any jobs they depend on were successfull + - when: on_success diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml index 871262fe0e..691d9bf5dc 100644 --- a/.gitlab-ci.d/qemu-project.yml +++ b/.gitlab-ci.d/qemu-project.yml @@ -2,6 +2,7 @@ # https://gitlab.com/qemu-project/qemu/-/pipelines include: + - local: '/.gitlab-ci.d/base.yml' - local: '/.gitlab-ci.d/stages.yml' - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 92e25872aa..eb6a9e6122 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -28,7 +28,35 @@ For further information about how to set these variables, please refer to:: https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd -Here is a list of the most used variables: +Variable naming and grouping +---------------------------- + +The variables used by QEMU's CI configuration are grouped together +in a handful of namespaces + + * QEMU_JOB_nnnn - variables to be defined in individual jobs + or templates, to influence the shared rules defined in the + .base_job_template. + + * QEMU_CI_nnn - variables to be set by contributors in their + repository CI settings, or as git push variables, to influence + which jobs get run in a pipeline + + * nnn - other misc variables not falling into the above + categories, or using different names for historical reasons + and not yet converted. + +Maintainer controlled job variables +----------------------------------- + +The following variables may be set when defining a job in the +CI configuration file. + +Contributor controlled runtime variables +---------------------------------------- + +The following variables may be set by contributors to control +job execution QEMU_CI_AVOCADO_TESTING ~~~~~~~~~~~~~~~~~~~~~~~ @@ -38,6 +66,12 @@ these artifacts are not already cached, downloading them make the jobs reach the timeout limit). Set this variable to have the tests using the Avocado framework run automatically. +Other misc variables +-------------------- + +These variables are primarily to control execution of jobs on +private runners + AARCH64_RUNNER_AVAILABLE ~~~~~~~~~~~~~~~~~~~~~~~~ If you've got access to an aarch64 host that can be used as a gitlab-CI From patchwork Thu May 26 11:07:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 12862307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9687FC433EF for ; Thu, 26 May 2022 11:23:39 +0000 (UTC) Received: from localhost ([::1]:54664 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nuBac-0002WH-I9 for qemu-devel@archiver.kernel.org; Thu, 26 May 2022 07:23:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKi-0007os-Vw for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:27990) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKh-0003Lf-5B for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653563230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zk78IgLksLw5ElFhkPV9WfJH/qiEb/4v5lLpdjzLdGE=; b=V1/OoTXYCFidBtKyiOX79gET881XPpTJVP5yOf78j5x3j5hz72zVDzbRRFReE+kPA0wvWl trHD0nwqG0yqSqSJOzqA7W9/k/9N0w9VZDGm49179Xi5LMCNIURG9BxwQwu1dvit+UH9Zt D+ulNXc2jOxE5LOPAevCbWifZxtiOrc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-317-kPdwJwkKN8ObBmqEwl1MmQ-1; Thu, 26 May 2022 07:07:07 -0400 X-MC-Unique: kPdwJwkKN8ObBmqEwl1MmQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 108F280088A; Thu, 26 May 2022 11:07:07 +0000 (UTC) Received: from virtlab420.virt.lab.eng.bos.redhat.com (virtlab420.virt.lab.eng.bos.redhat.com [10.19.152.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id D1BFB1731B; Thu, 26 May 2022 11:07:06 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Beraldo Leal , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 2/5] gitlab: convert Cirrus jobs to .base_job_template Date: Thu, 26 May 2022 07:07:02 -0400 Message-Id: <20220526110705.59952-3-berrange@redhat.com> In-Reply-To: <20220526110705.59952-1-berrange@redhat.com> References: <20220526110705.59952-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass client-ip=170.10.129.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This folds the Cirrus job rules into the base job template, introducing two new variables - QEMU_JOB_CIRRUS - identifies the job as making use of Cirrus CI via cirrus-run - QEMU_JOB_OPTIONAL - identifies the job as one that is not run by default, primarily due to resource constraints. It can be manually invoked by users if they wish to validate that scenario. Signed-off-by: Daniel P. Berrangé --- .gitlab-ci.d/base.yml | 9 +++++++++ .gitlab-ci.d/cirrus.yml | 16 ++++++++-------- docs/devel/ci-jobs.rst.inc | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 10eb6ab8bc..5734caf9fe 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -12,12 +12,21 @@ # want jobs to run ############################################################# + # Cirrus jobs can't run unless the creds / target repo are set + - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios # where the catch all logic is inapprorpaite ############################################################# + # Optional jobs should not be run unless manually triggered + - if: '$QEMU_JOB_OPTIONAL' + when: manual + allow_failure: true + ############################################################# # Stage 3: catch all logic applying to any job not matching diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml index b96b22e269..609c364308 100644 --- a/.gitlab-ci.d/cirrus.yml +++ b/.gitlab-ci.d/cirrus.yml @@ -11,6 +11,7 @@ # special care, because we can't just override it at the GitLab CI job # definition level or we risk breaking it completely. .cirrus_build_job: + extends: .base_job_template stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] @@ -40,11 +41,8 @@ <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml - cat .gitlab-ci.d/cirrus/$NAME.yml - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml - rules: - # Allow on 'staging' branch and 'stable-X.Y-staging' branches only - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/' - when: never - - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" + variables: + QEMU_JOB_CIRRUS: 1 x64-freebsd-12-build: extends: .cirrus_build_job @@ -90,11 +88,11 @@ x64-macos-11-base-build: # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job .cirrus_kvm_job: + extends: .base_job_template stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] timeout: 80m - allow_failure: true script: - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" @@ -105,8 +103,10 @@ x64-macos-11-base-build: <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml - cat .gitlab-ci.d/cirrus/$NAME.yml - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml - rules: - - when: manual + variables: + QEMU_JOB_CIRRUS: 1 + QEMU_JOB_OPTIONAL: 1 + x86-netbsd: extends: .cirrus_kvm_job diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index eb6a9e6122..a539f502da 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -52,6 +52,20 @@ Maintainer controlled job variables The following variables may be set when defining a job in the CI configuration file. +QEMU_JOB_CIRRUS +~~~~~~~~~~~~~~~ + +The job makes use of Cirrus CI infrastructure, requiring the +configuration setup for cirrus-run to be present in the repository + +QEMU_JOB_OPTIONAL +~~~~~~~~~~~~~~~~~ + +The job is expected to be successful in general, but is not run +by default due to need to conserve limited CI resources. It is +available to be started manually by the contributor in the CI +pipelines UI. + Contributor controlled runtime variables ---------------------------------------- From patchwork Thu May 26 11:07:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 12862254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 762B1C433EF for ; Thu, 26 May 2022 11:13:34 +0000 (UTC) Received: from localhost ([::1]:46048 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nuBQr-0003ra-9x for qemu-devel@archiver.kernel.org; Thu, 26 May 2022 07:13:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39118) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKh-0007ni-A5 for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26697) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKf-0003LJ-MT for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653563229; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YmvpvgkpF/lfko9RrJeQLBK/k0kptVig6260OiXg/sg=; b=HOweBHSjNaGMDbJICv7wq0akDbgB2Id8WFgK1pF7/OuR++NxVLYApO4Yo9Ktwy8tKU3qJ4 8S9EiB60KslJpysx07TSqDzPPZgStxEF+x3/3WfZRxROmxkgYRdaMNycgg62G8TvLazJZY BNJtKUW3O+foUoZuhUa7mN7nMlqLywo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-488-HTkgh6BbMiOqtfnJDu2Mcw-1; Thu, 26 May 2022 07:07:07 -0400 X-MC-Unique: HTkgh6BbMiOqtfnJDu2Mcw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4D6AB185A7A4; Thu, 26 May 2022 11:07:07 +0000 (UTC) Received: from virtlab420.virt.lab.eng.bos.redhat.com (virtlab420.virt.lab.eng.bos.redhat.com [10.19.152.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1A43082872; Thu, 26 May 2022 11:07:07 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Beraldo Leal , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 3/5] gitlab: convert static checks to .base_job_template Date: Thu, 26 May 2022 07:07:03 -0400 Message-Id: <20220526110705.59952-4-berrange@redhat.com> In-Reply-To: <20220526110705.59952-1-berrange@redhat.com> References: <20220526110705.59952-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This folds the static checks into using the base job template rules, introducing one new variable - QEMU_JOB_ONLY_FORKS - a job that should never run on an upstream pipeline. The information it reports is only applicable to contributors in a pre-submission scenario, not time of merge. Signed-off-by: Daniel P. Berrangé --- .gitlab-ci.d/base.yml | 4 ++++ .gitlab-ci.d/static_checks.yml | 19 +++++++------------ docs/devel/ci-jobs.rst.inc | 7 +++++++ 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 5734caf9fe..9a0b8d7f97 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -16,6 +16,10 @@ - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' when: never + # Jobs only intended for forks should always be skipped on upstram + - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml index 94858e3272..289ad1359e 100644 --- a/.gitlab-ci.d/static_checks.yml +++ b/.gitlab-ci.d/static_checks.yml @@ -1,4 +1,5 @@ check-patch: + extends: .base_job_template stage: build image: python:3.10-alpine needs: [] @@ -6,15 +7,13 @@ check-patch: - .gitlab-ci.d/check-patch.py variables: GIT_DEPTH: 1000 + QEMU_JOB_ONLY_FORKS: 1 before_script: - apk -U add git perl - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - when: on_success - allow_failure: true + allow_failure: true check-dco: + extends: .base_job_template stage: build image: python:3.10-alpine needs: [] @@ -23,12 +22,9 @@ check-dco: GIT_DEPTH: 1000 before_script: - apk -U add git - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' - when: never - - when: on_success check-python-pipenv: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/python:latest script: @@ -39,6 +35,7 @@ check-python-pipenv: job: python-container check-python-tox: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/python:latest script: @@ -46,8 +43,6 @@ check-python-tox: variables: GIT_DEPTH: 1 QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false + QEMU_JOB_OPTIONAL: 1 needs: job: python-container - rules: - - when: manual - allow_failure: true diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index a539f502da..4c7e30ab08 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -66,6 +66,13 @@ by default due to need to conserve limited CI resources. It is available to be started manually by the contributor in the CI pipelines UI. +QEMU_JOB_ONLY_FORKS +~~~~~~~~~~~~~~~~~~~ + +The job results are only of interest to contributors prior to +submitting code. They are not required as part of the gating +CI pipeline. + Contributor controlled runtime variables ---------------------------------------- From patchwork Thu May 26 11:07:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 12862308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4048BC433EF for ; Thu, 26 May 2022 11:26:06 +0000 (UTC) Received: from localhost ([::1]:56804 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nuBcz-00047t-6n for qemu-devel@archiver.kernel.org; Thu, 26 May 2022 07:26:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39202) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKm-0007s0-99 for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26494) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKk-0003MF-5a for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653563233; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ju5ZyeQNPJAUj0I2nP+HadNRw85D3ptqaovIHW+KnIQ=; b=J6y7kOdLOhDS1niUlnJPNJ8vxusGF4/O/S60kg9aS8eTv5QOLlZKMoSC5tNJS8M+nbA/MW q3UPv6udpfwNXPiMTGVIIyu9GPTlo5zg+GcBz2Q1EaRunndGelG+yVAn9rmz1t6ks1YHr+ ZaQ/m0MtrmC6O1UecTWk729uSNLQX+8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-60-14__0n3zP36kiP7aNmzHAg-1; Thu, 26 May 2022 07:07:07 -0400 X-MC-Unique: 14__0n3zP36kiP7aNmzHAg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8A5503802B8B; Thu, 26 May 2022 11:07:07 +0000 (UTC) Received: from virtlab420.virt.lab.eng.bos.redhat.com (virtlab420.virt.lab.eng.bos.redhat.com [10.19.152.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5747C1731B; Thu, 26 May 2022 11:07:07 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Beraldo Leal , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 4/5] gitlab: convert build/container jobs to .base_job_template Date: Thu, 26 May 2022 07:07:04 -0400 Message-Id: <20220526110705.59952-5-berrange@redhat.com> In-Reply-To: <20220526110705.59952-1-berrange@redhat.com> References: <20220526110705.59952-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This converts the main build and container jobs to use the base job rules, defining the following new variables - QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired. - QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness. - QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream Signed-off-by: Daniel P. Berrangé --- .gitlab-ci.d/base.yml | 22 ++++++++++++++++++++++ .gitlab-ci.d/buildtest-template.yml | 16 ++++------------ .gitlab-ci.d/buildtest.yml | 28 +++++++++++++--------------- .gitlab-ci.d/container-cross.yml | 6 ++---- .gitlab-ci.d/container-template.yml | 1 + .gitlab-ci.d/crossbuild-template.yml | 3 +++ .gitlab-ci.d/windows.yml | 1 + docs/devel/ci-jobs.rst.inc | 19 +++++++++++++++++++ 8 files changed, 65 insertions(+), 31 deletions(-) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 9a0b8d7f97..6a918abbda 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -16,10 +16,22 @@ - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")' when: never + # Publishing jobs should only run on the default branch in upstream + - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH' + when: never + + # Non-publishing jobs should only run on staging branches in upstream + - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/' + when: never + # Jobs only intended for forks should always be skipped on upstram - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' when: never + # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set + - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: never + ############################################################# # Stage 2: fine tune execution of jobs in specific scenarios @@ -31,6 +43,16 @@ when: manual allow_failure: true + # Skipped jobs should not be run unless manually triggered + - if: '$QEMU_JOB_SKIPPED' + when: manual + allow_failure: true + + # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset + - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: manual + allow_failure: true + ############################################################# # Stage 3: catch all logic applying to any job not matching diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index dc6d67aacf..1874c07a4b 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -1,4 +1,5 @@ .native_build_job_template: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script: @@ -27,6 +28,7 @@ fi .common_test_job_template: + extends: .base_job_template stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: @@ -75,15 +77,5 @@ after_script: - cd build - du -chs ${CI_PROJECT_DIR}/avocado-cache - rules: - # Only run these jobs if running on the mainstream namespace, - # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either - # in its namespace setting or via git-push option, see documentation - # in /.gitlab-ci.yml of this repository). - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: on_success - - if: '$QEMU_CI_AVOCADO_TESTING' - when: on_success - # Otherwise, set to manual (the jobs are created but not run). - - when: manual - allow_failure: true + variables: + QEMU_JOB_AVOCADO: 1 diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index e9620c3074..ecac3ec50c 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -360,12 +360,11 @@ build-cfi-aarch64: expire_in: 2 days paths: - build - rules: + variables: # FIXME: This job is often failing, likely due to out-of-memory problems in # the constrained containers of the shared runners. Thus this is marked as - # manual until the situation has been solved. - - when: manual - allow_failure: true + # skipped until the situation has been solved. + QEMU_JOB_SKIPPED: 1 check-cfi-aarch64: extends: .native_test_job_template @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x: expire_in: 2 days paths: - build - rules: + variables: # FIXME: This job is often failing, likely due to out-of-memory problems in # the constrained containers of the shared runners. Thus this is marked as - # manual until the situation has been solved. - - when: manual - allow_failure: true + # skipped until the situation has been solved. + QEMU_JOB_SKIPPED: 1 check-cfi-ppc64-s390x: extends: .native_test_job_template @@ -579,6 +577,7 @@ build-without-default-features: MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow build-libvhost-user: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/fedora:latest needs: @@ -595,10 +594,13 @@ build-tools-and-docs-debian: extends: .native_build_job_template needs: job: amd64-debian-container + # when running on 'master' we use pre-existing container + optional: true variables: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools + QEMU_JOB_PUBLISH: 1 artifacts: expire_in: 2 days paths: @@ -618,6 +620,7 @@ build-tools-and-docs-debian: # that users can see the results of their commits, regardless # of what topic branch they're currently using pages: + extends: .base_job_template image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest stage: test needs: @@ -635,10 +638,5 @@ pages: artifacts: paths: - public - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' - when: on_success - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - if: '$CI_PROJECT_NAMESPACE != "qemu-project"' - when: on_success + variables: + QEMU_JOB_PUBLISH: 1 diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml index e622ac2d21..90ecfd8da0 100644 --- a/.gitlab-ci.d/container-cross.yml +++ b/.gitlab-ci.d/container-cross.yml @@ -42,15 +42,13 @@ armhf-debian-cross-container: # We never want to build hexagon in the CI system and by default we # always want to refer to the master registry where it lives. hexagon-cross-container: + extends: .base_job_template image: docker:stable stage: containers - rules: - - if: '$CI_PROJECT_NAMESPACE == "qemu-project"' - when: never - - when: always variables: NAME: debian-hexagon-cross GIT_DEPTH: 1 + QEMU_JOB_ONLY_FORKS: 1 services: - docker:dind before_script: diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 1baecd9460..c434b9c8f3 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,4 +1,5 @@ .container_job_template: + extends: .base_job_template image: docker:stable stage: containers services: diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml index 29c3c2b826..28b2142ec2 100644 --- a/.gitlab-ci.d/crossbuild-template.yml +++ b/.gitlab-ci.d/crossbuild-template.yml @@ -1,4 +1,5 @@ .cross_system_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 80m @@ -24,6 +25,7 @@ # KVM), and set extra options (such disabling other accelerators) via the # $EXTRA_CONFIGURE_OPTS variable. .cross_accel_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest timeout: 30m @@ -36,6 +38,7 @@ - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS .cross_user_build_job: + extends: .base_job_template stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml index cf7724b8e5..1b2ede49e1 100644 --- a/.gitlab-ci.d/windows.yml +++ b/.gitlab-ci.d/windows.yml @@ -1,4 +1,5 @@ .shared_msys2_builder: + extends: .base_job_template tags: - shared-windows - windows diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 4c7e30ab08..0b4926e537 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -73,6 +73,25 @@ The job results are only of interest to contributors prior to submitting code. They are not required as part of the gating CI pipeline. +QEMU_JOB_SKIPPED +~~~~~~~~~~~~~~~~ + +The job is not reliably successsful in general, so is not +currently suitable to be run by default. Ideally this should +be a temporary marker until the problems can be addressed, or +the job permanently removed. + +QEMU_JOB_PUBLISH +~~~~~~~~~~~~~~~~ + +The job is for publishing content after a branch has been +merged into the upstream default branch. + +QEMU_JOB_AVOCADO +~~~~~~~~~~~~~~~~ + +The job runs the Avocado integration test suite + Contributor controlled runtime variables ---------------------------------------- From patchwork Thu May 26 11:07:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= X-Patchwork-Id: 12862319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B357EC433F5 for ; Thu, 26 May 2022 11:28:39 +0000 (UTC) Received: from localhost ([::1]:60032 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nuBfS-0006cB-Pz for qemu-devel@archiver.kernel.org; Thu, 26 May 2022 07:28:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39190) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKk-0007pV-Ed for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:51173) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nuBKi-0003Lw-QG for qemu-devel@nongnu.org; Thu, 26 May 2022 07:07:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1653563232; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C6P3TOnwGTv8ggo6E9OqOCanLAJZzK8SrWfUEHmbEOc=; b=GMr8XeujRST5XKJyIC2syZ0JyRba9Dc5x6dkgu7mI0l3ClT7RSdwqB//UHbzqejr3VwSwp Elz1f84A1/6IaYGJlLOdzLRf5x7O1/2NoHHy+PEi+FVueGSP9fahCywfXDJenbroLiW0Aw ux7asvO4EhMu9qKTeLHiP/NrOSZ50Yo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-28-NMikiaHlMcG2Pa5mOw7DRw-1; Thu, 26 May 2022 07:07:08 -0400 X-MC-Unique: NMikiaHlMcG2Pa5mOw7DRw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C72CD801228; Thu, 26 May 2022 11:07:07 +0000 (UTC) Received: from virtlab420.virt.lab.eng.bos.redhat.com (virtlab420.virt.lab.eng.bos.redhat.com [10.19.152.148]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9404382872; Thu, 26 May 2022 11:07:07 +0000 (UTC) From: =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= To: qemu-devel@nongnu.org Cc: Thomas Huth , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , =?utf-8?q?Alex_Benn=C3=A9e?= , Beraldo Leal , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= Subject: [PATCH 5/5] gitlab: don't run CI jobs in forks by default Date: Thu, 26 May 2022 07:07:05 -0400 Message-Id: <20220526110705.59952-6-berrange@redhat.com> In-Reply-To: <20220526110705.59952-1-berrange@redhat.com> References: <20220526110705.59952-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Received-SPF: pass client-ip=170.10.133.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" To preserve CI shared runner credits we don't want to run pipelines on every push. This sets up the config to that pipelines are never created for contributors by default. To override this the QEMU_CI variable can be set to a non-zero value. If set to 1, the pipeline will be created but all jobs will remain manually started. The contributor can selectively run jobs that they care about. If set to 2, the pipeline will be created and all jobs will immediately start. This behavior can be controlled using push variables git push -o ci.variable=QEMU_CI=1 To make this more convenient define an alias git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1" git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2" Which lets you run git push-ci to create the pipeline, or git push-ci-now to create and run the pipeline Signed-off-by: Daniel P. Berrangé --- .gitlab-ci.d/base.yml | 9 +++++++++ docs/devel/ci-jobs.rst.inc | 12 ++++++++++++ 2 files changed, 21 insertions(+) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 6a918abbda..62f2a850c3 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -28,6 +28,10 @@ - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"' when: never + # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set + - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: never + # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"' when: never @@ -59,5 +63,10 @@ # an earlier criteria ############################################################# + # Forks pipeline jobs don't start automatically unless + # QEMU_CI=2 is set + - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"' + when: manual + # Jobs can run if any jobs they depend on were successfull - when: on_success diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 0b4926e537..e36c8cc68d 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -98,6 +98,18 @@ Contributor controlled runtime variables The following variables may be set by contributors to control job execution +QEMU_CI +~~~~~~~ + +By default, no pipelines will be created on contributor forks +in order to preserve CI credits + +Set this variable to 1 to create the pipelines, but leave all +the jobs to be manually started from the UI + +Set this variable to 2 to create the pipelines and run all +the jobs immediately, as was historicaly behaviour + QEMU_CI_AVOCADO_TESTING ~~~~~~~~~~~~~~~~~~~~~~~ By default, tests using the Avocado framework are not run automatically in