From patchwork Sun Apr 18 23:34:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= X-Patchwork-Id: 12210521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68107C433B4 for ; Sun, 18 Apr 2021 23:36:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE5276100B for ; Sun, 18 Apr 2021 23:36:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE5276100B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amsat.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43738 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lYGxL-0000st-Ml for qemu-devel@archiver.kernel.org; Sun, 18 Apr 2021 19:35:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44258) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYGwI-0008Db-Q0 for qemu-devel@nongnu.org; Sun, 18 Apr 2021 19:34:54 -0400 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]:52962) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lYGwH-0002wU-5w for qemu-devel@nongnu.org; Sun, 18 Apr 2021 19:34:54 -0400 Received: by mail-wm1-x335.google.com with SMTP id y204so15786791wmg.2 for ; Sun, 18 Apr 2021 16:34:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=r01iM2KbID8qjwwpuUA9Z/vootSEp7qdc6/UZqHFLOI=; b=NA0++XsM9e5B0KBQnEns6pbruuFFpR5/A6D3GjAdY8uBBBptG8++GmMGDwenpIDiWC PpBe5y3SDrKWyXy/rLtlnEgjA2QrmvH0dBEIKDy8ulflvMMi2gURH/n/dqvkuzXC/tp9 ZcHLdFwVOrm6EV7HJO/xIYXvoX98Mts30bRkxGqSmbufSxCqs3i9ztzPr2ZEPmB1bRDV N9xXmjayahpnxj882jiPd/1m05aAaIJqRpyTnnGXtuabrmNOmJpUhioaa18vd8ZkRIYj mSceQvGQ9F3bqI538iVhEBrf52S0xq/FgGTvZo9j1uZTBFLYAWsByQmrNDwRLGPk47E4 RjOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; bh=r01iM2KbID8qjwwpuUA9Z/vootSEp7qdc6/UZqHFLOI=; b=fYwLZbO7NdmDZm3XyDTM6SbFGCnQO/ozlDKGI8jHK4kYxk1MVrNAWA0tfXr2nwAgIe xw0bInBnBS0du7mZ5yWrcMxC3qkJTUDha3FyCQrZ36JT82WehGs/835iJ0mz3/ckrqjY qJNY1xh8saI7s+xc0U8CMn6833HIHRP0mIVdGEncbjCaW5GdI1aFgKHsoUF0MoN82i8y KMvKkZaw6ApoxkJNip3ARS6ySgT8O8yfVdyN37ZsqjCJxdsU8LOA3hykPsVsmkCkaUeD 3K9kvHFaOY2RfB+yAN2aAoEcSY/3EUzmhUFBUGlaq2K0/uME2zlBtsU4oQB3Wuw2sV7f qP6A== X-Gm-Message-State: AOAM530rRNXwfLm2ufWD0Ujw05XZLNV9KJkNqvLZYk/KHn9ipGLkrNAy bGGIlxlwSaoMvOzXg2dihRoj9QE9lXuvzQ== X-Google-Smtp-Source: ABdhPJw0mVmrVh4wopjziETzckyqST1vw9zQSeKVdS8N/Iddix+XApDbEdWrU3CaMq1qf7nEOp85Zw== X-Received: by 2002:a1c:4089:: with SMTP id n131mr18883138wma.77.1618788891391; Sun, 18 Apr 2021 16:34:51 -0700 (PDT) Received: from localhost.localdomain (39.red-81-40-121.staticip.rima-tde.net. [81.40.121.39]) by smtp.gmail.com with ESMTPSA id z15sm19104521wrv.39.2021.04.18.16.34.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Apr 2021 16:34:50 -0700 (PDT) From: =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= To: qemu-devel@nongnu.org Subject: [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Date: Mon, 19 Apr 2021 01:34:33 +0200 Message-Id: <20210418233448.1267991-1-f4bug@amsat.org> X-Mailer: git-send-email 2.26.3 MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::335; envelope-from=philippe.mathieu.daude@gmail.com; helo=mail-wm1-x335.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Erik Skultety , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Willian Rampazzo , Cleber Rosa , Miroslav Rezanina , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" TL;DR: Patch 13 is an experiment to dynamically generate pipelines. Hi, This series is an intent to allow git forks to use different set of jobs than the ones used by the mainstream repository. Currently, a fork gets the mainstream CI pipeline YAML configuration and runs its set of jobs by default. Mainstream tests many cases which might not be always useful to forks. Beside, one mainstream pipeline might be a waste of ressources when forks provide their own runners to their gitlab namespace, or use private instances. This is not a new / unique problem to our community, and Gitlab already figured it out. They describe this case as Upstream (our project mainline) / Downstream (a fork). The pipeline is divided in 2 steps. First the "upstream" part, which does some evaluation, and depending on the result, triggers the "downstream" part. The "upstream" is committed into mainline, and dynamically generate the "downstream" pipeline. This allows forks to easily rebase with mainstream, avoiding merge conflicts. This might not be adapted to the QEMU use case, but I wanted to give it a try. Gitlab recommends to use job templates. To be able to re-use / include templates, they can not use YAML anchors. All current jobs has first to be converted to use the 'extends' keyworkd. This is done by patches 1-5 (already reviewed). Patches 6-12 extract various generic jobs (which can be reused by forks) as template. Patches 13-14 follow Gitlab guideline to generate dynamic pipelines Patch 15 is an example of how a fork could use mainstream templates to maintain his own set of jobs. The PRO is forks can contribute to templates. Regards, Phil. Philippe Mathieu-Daudé (15): gitlab-ci: Replace YAML anchors by extends (container_job) gitlab-ci: Replace YAML anchors by extends (native_build_job) gitlab-ci: Replace YAML anchors by extends (native_test_job) gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) gitlab-ci: Rename acceptance_test_job -> integration_test_job gitlab-ci: Extract container job template to container-template.yml gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml gitlab-ci: Extract DCO/style check jobs to checks.yml gitlab-ci: Extract build stages to stages.yml gitlab-ci: Extract all default build/test jobs to buildtest.yml gitlab-ci: Extract core container jobs to container-core.yml gitlab-ci: Move current job set to qemu-project.yml gitlab-ci: Switch to dynamically generated pipelines gitlab-ci: Allow forks to use different set of jobs gitlab-ci: Use my own set of jobs for CI pipeline .gitlab-ci.d/buildtest-template.yml | 80 +++ .gitlab-ci.d/buildtest.yml | 744 +++++++++++++++++++++++ .gitlab-ci.d/checks.yml | 24 + .gitlab-ci.d/container-core.yml | 17 + .gitlab-ci.d/container-template.yml | 22 + .gitlab-ci.d/containers.yml | 108 ++-- .gitlab-ci.d/crossbuild-template.yml | 41 ++ .gitlab-ci.d/crossbuilds.yml | 43 +- .gitlab-ci.d/philmd.yml | 33 ++ .gitlab-ci.d/qemu-project.yml | 8 + .gitlab-ci.d/stages.yml | 8 + .gitlab-ci.yml | 845 +-------------------------- 12 files changed, 1033 insertions(+), 940 deletions(-) create mode 100644 .gitlab-ci.d/buildtest-template.yml create mode 100644 .gitlab-ci.d/buildtest.yml create mode 100644 .gitlab-ci.d/checks.yml create mode 100644 .gitlab-ci.d/container-core.yml create mode 100644 .gitlab-ci.d/container-template.yml create mode 100644 .gitlab-ci.d/crossbuild-template.yml create mode 100644 .gitlab-ci.d/philmd.yml create mode 100644 .gitlab-ci.d/qemu-project.yml create mode 100644 .gitlab-ci.d/stages.yml