From patchwork Tue Mar 5 19:09:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 13582901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2214FC54E41 for ; Tue, 5 Mar 2024 19:09:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rhaA7-0001lZ-Tp; Tue, 05 Mar 2024 14:09:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhaA6-0001lO-34 for qemu-devel@nongnu.org; Tue, 05 Mar 2024 14:09:14 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhaA4-0002Bv-Bm for qemu-devel@nongnu.org; Tue, 05 Mar 2024 14:09:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709665751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WNBwNyP93UNETTCQt7HxqHOcuB/TSrwLC8yZ1GgpcbY=; b=SwCQia0wRLwf0P9xBguGSRZdBbTBX97GQjSO1M2YVwZqP2sIViEtEaxgIWXAjOgCwKczoS Ncke6B4QSM3Pxsozzsndh+RvyWEKvfS28DLfIIr+7cxcLdxxuQhGwYhNvqkjYJSJfW4D8M D/HVpyM45xAVj9sshsU7LN/ZZCIMb9k= Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-173-pQQPJwSmNGGsL-XZJQ1plw-1; Tue, 05 Mar 2024 14:09:10 -0500 X-MC-Unique: pQQPJwSmNGGsL-XZJQ1plw-1 Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-a4533388b03so153277066b.0 for ; Tue, 05 Mar 2024 11:09:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709665747; x=1710270547; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WNBwNyP93UNETTCQt7HxqHOcuB/TSrwLC8yZ1GgpcbY=; b=wyEutdtcrA6qB/NejydKnlZ8CvHK3JhC0vw+xw9Vc06dC3ZPCq6wmMXimWyPEZ7yH5 o4z6DEQLFUwrz6EM++FGTG9NYEORgBh7FfayuEvLzY1ilMh619lNTymHpstUnCpEYYuD 222n9mPkkR2xnWdLNQD2X03BGHq17OzZootDTy3NUvtsA7vzBG95yC12k5RXC59Q6gkl nZPFm/NATa4ol49Z5T/RbQKtUWeJze0cEG8DyUFRALpIKmFO+mCGIogt2abHK7v7IxW4 2/63+835tKlomiIhgJMAxr7WsDYTOLF8YLdtaoGfHPlC2DV/cEputTmJjlJtvhDcnfeQ up3g== X-Gm-Message-State: AOJu0YyxXm4At7p7l8C5HjtoTIxm7qniVzaQkAKoDNYgDytCEKPzYCFB RYFLBDkldLMjiA/g+oR6EqWAYjnKBVaLO/9ZSC7Xu3t+9eDuhlT6BTiRqDooFk2S2MLLYvBxo0c p2PL2/vTrAdbopK1JOpomcvlDLAPBHtQUP2NBF8++Gq4bbiIjfV6YYOGekrLplOYT176EklK/GK ShFl8W28s80jq4sQRPcx4GQULju/WtuiIVWV+X X-Received: by 2002:a17:906:29cd:b0:a45:b10a:9fe9 with SMTP id y13-20020a17090629cd00b00a45b10a9fe9mr861102eje.2.1709665747285; Tue, 05 Mar 2024 11:09:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IHFGFL6uxVaA/dv+UkVyjfHJb3PJeUb7XnGRpkDqrj/ksiL0pLdIJ9gSM44Sgrs37ow7PT3Pg== X-Received: by 2002:a17:906:29cd:b0:a45:b10a:9fe9 with SMTP id y13-20020a17090629cd00b00a45b10a9fe9mr861088eje.2.1709665746877; Tue, 05 Mar 2024 11:09:06 -0800 (PST) Received: from [192.168.10.118] ([151.49.77.21]) by smtp.gmail.com with ESMTPSA id o4-20020a17090611c400b00a458d85f9d9sm1515008eja.142.2024.03.05.11.09.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 11:09:04 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, berrange@redhat.com Subject: [PATCH 1/2] run-coverity-scan: add --check-upload-only option Date: Tue, 5 Mar 2024 20:09:01 +0100 Message-ID: <20240305190902.364753-2-pbonzini@redhat.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240305190902.364753-1-pbonzini@redhat.com> References: <20240305190902.364753-1-pbonzini@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=pbonzini@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.568, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add an option to check if upload is permitted without actually attempting a build. This can be useful to add a third outcome beyond success and failure---namely, a CI job can self-cancel if the uploading quota has been reached. Signed-off-by: Paolo Bonzini --- scripts/coverity-scan/run-coverity-scan | 51 ++++++++++++++++++------- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/scripts/coverity-scan/run-coverity-scan b/scripts/coverity-scan/run-coverity-scan index d56c9b66776..4bc991f70fd 100755 --- a/scripts/coverity-scan/run-coverity-scan +++ b/scripts/coverity-scan/run-coverity-scan @@ -28,6 +28,7 @@ # project settings, if you have maintainer access there. # Command line options: +# --check-upload-only : return success if upload is possible # --dry-run : run the tools, but don't actually do the upload # --docker : create and work inside a container # --docker-engine : specify the container engine to use (docker/podman/auto); @@ -57,18 +58,18 @@ # putting it in a file and using --tokenfile. Everything else has # a reasonable default if this is run from a git tree. -check_upload_permissions() { +upload_permitted() { # Check whether we can do an upload to the server; will exit the script # with status 1 if the check failed (usually a bad token); # will exit the script with status 0 if the check indicated that we # can't upload yet (ie we are at quota) - # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized. + # Assumes that COVERITY_TOKEN and PROJNAME have been initialized. echo "Checking upload permissions..." if ! up_perm="$(wget https://scan.coverity.com/api/upload_permitted --post-data "token=$COVERITY_TOKEN&project=$PROJNAME" -q -O -)"; then echo "Coverity Scan API access denied: bad token?" - exit 1 + exit 99 fi # Really up_perm is a JSON response with either @@ -76,25 +77,40 @@ check_upload_permissions() { # We do some hacky string parsing instead of properly parsing it. case "$up_perm" in *upload_permitted*true*) - echo "Coverity Scan: upload permitted" + return 0 ;; *next_upload_permitted_at*) - if [ "$DRYRUN" = yes ]; then - echo "Coverity Scan: upload quota reached, continuing dry run" - else - echo "Coverity Scan: upload quota reached; stopping here" - # Exit success as this isn't a build error. - exit 0 - fi + return 1 ;; *) echo "Coverity Scan upload check: unexpected result $up_perm" - exit 1 + exit 99 ;; esac } +check_upload_permissions() { + # Check whether we can do an upload to the server; will exit the script + # with status 1 if the check failed (usually a bad token); + # will exit the script with status 0 if the check indicated that we + # can't upload yet (ie we are at quota) + # Assumes that COVERITY_TOKEN, PROJNAME and DRYRUN have been initialized. + + if upload_permitted; then + echo "Coverity Scan: upload permitted" + else + if [ "$DRYRUN" = yes ]; then + echo "Coverity Scan: upload quota reached, continuing dry run" + else + echo "Coverity Scan: upload quota reached; stopping here" + # Exit success as this isn't a build error. + exit 0 + fi + fi +} + + build_docker_image() { # build docker container including the coverity-scan tools echo "Building docker container..." @@ -152,9 +168,14 @@ update_coverity_tools () { DRYRUN=no UPDATE=yes DOCKER=no +PROJNAME=QEMU while [ "$#" -ge 1 ]; do case "$1" in + --check-upload-only) + shift + DRYRUN=check + ;; --dry-run) shift DRYRUN=yes @@ -251,6 +272,11 @@ if [ -z "$COVERITY_TOKEN" ]; then exit 1 fi +if [ "$DRYRUN" = check ]; then + upload_permitted + exit $? +fi + if [ -z "$COVERITY_BUILD_CMD" ]; then NPROC=$(nproc) COVERITY_BUILD_CMD="make -j$NPROC" @@ -266,7 +292,6 @@ if [ -z "$SRCDIR" ]; then SRCDIR="$PWD" fi -PROJNAME=QEMU TARBALL=cov-int.tar.xz if [ "$UPDATE" = only ]; then From patchwork Tue Mar 5 19:09:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 13582902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 74673C54798 for ; Tue, 5 Mar 2024 19:10:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rhaAE-0001mM-UZ; Tue, 05 Mar 2024 14:09:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhaAD-0001lx-IL for qemu-devel@nongnu.org; Tue, 05 Mar 2024 14:09:21 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rhaAB-0002CH-T1 for qemu-devel@nongnu.org; Tue, 05 Mar 2024 14:09:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709665759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tAn3BPw9Rj/HBNoW+GN/JnNR0QESXgEweD9Y1fNZnNA=; b=JmFY92SLXZkZ06jm0qohT/FIreN5zfaLM7HViszdBVrEOubEcQvbMWYwuVttNEuId9wO5T Iin6n3tso6+pkBIZa9eAafENnIlfonQeaXgw3Y6a6IGQWPXtN0qeSi1jSvySQH39A2xNnP b6izE08dB64WrPEOXOPLh7tfNjFdWGM= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-441-k6Oo7C5OPRCfDuOkArxl4Q-1; Tue, 05 Mar 2024 14:09:15 -0500 X-MC-Unique: k6Oo7C5OPRCfDuOkArxl4Q-1 Received: by mail-lf1-f71.google.com with SMTP id 2adb3069b0e04-51337da375dso3612780e87.3 for ; Tue, 05 Mar 2024 11:09:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709665750; x=1710270550; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tAn3BPw9Rj/HBNoW+GN/JnNR0QESXgEweD9Y1fNZnNA=; b=tpyS8tbhsgpNaVCHgzmeoNA1fiw7QV9+QZqoyu+WsZomHytMOEiCGf+3K2ekcH99Vj y4s1Pm2H7G/Rc6ZLTEhsomM6QdFIpuJqyyEJkcLUa5opsH1S9IcZvE9bZyOZ56u5TN54 /c0HE9PuFryXOJSQj6SdJ5iQQ6vDIOhhMhJV0jJPrET1KH/UmKchZLQlYmuHRHfkjm6F uSrKuobB9lrBTkSaPc9nRbdfeuBUDnL9Yioc3/PFev/ONF54fOJ73RMT6SP4sCRNLRVL oCCjqXAZwY+tUKB/oxK6AZjRwhKzlXkKOFZvOiYYrt0I1OCFSvug2SWGdThBRU8hDxg0 OgCg== X-Gm-Message-State: AOJu0YxroEbCON1s6xZUE3V2pTqKX41SPDoBi5/pIgCr1OyBJqn9h1rD qsYHAqOmSiuxySfUWYru2zcyhyRajlOGEwayaS+gpfcLpVhrQtoYyqhPbgP8W4gaHbsXZPRxXBr ZWI2HgmJLL5LAGAIsdBcFiDXAScw+mwek/8Ta/Byq9sySAlih00Od55gHi1tZ0jyvfdThJkumWk kNGQRmDsF1g6Q/bPJeiQLsz/O5qCmaKccn6VdF X-Received: by 2002:a05:6512:4db:b0:513:26e7:440c with SMTP id w27-20020a05651204db00b0051326e7440cmr1970179lfq.61.1709665750687; Tue, 05 Mar 2024 11:09:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6I+sSTq3iLMaIAU45h5vw+RAc7+Flqqnj7ZyBAC9MPWEF1AIJQxadCh1TaELbDrt4xcR4iw== X-Received: by 2002:a05:6512:4db:b0:513:26e7:440c with SMTP id w27-20020a05651204db00b0051326e7440cmr1970162lfq.61.1709665750227; Tue, 05 Mar 2024 11:09:10 -0800 (PST) Received: from [192.168.10.118] ([151.49.77.21]) by smtp.gmail.com with ESMTPSA id kw11-20020a170907770b00b00a43f170ad9asm6270079ejc.152.2024.03.05.11.09.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 11:09:07 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, berrange@redhat.com Subject: [PATCH 2/2] gitlab-ci: add manual job to run Coverity Date: Tue, 5 Mar 2024 20:09:02 +0100 Message-ID: <20240305190902.364753-3-pbonzini@redhat.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240305190902.364753-1-pbonzini@redhat.com> References: <20240305190902.364753-1-pbonzini@redhat.com> MIME-Version: 1.0 Received-SPF: pass client-ip=170.10.129.124; envelope-from=pbonzini@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.568, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a job that can be run, either manually or on a schedule, to upload a build to Coverity Scan. The job uses the run-coverity-scan script in multiple phases of check, download tools and upload, in order to avoid both wasting time (skip everything if you are above the upload quota) and avoid filling the log with the progress of downloading the tools. The job is intended to run on a scheduled pipeline run, and scheduled runs will not get any other job. It requires two variables to be in GitLab CI, COVERITY_TOKEN and COVERITY_EMAIL. Those are already set up in qemu-project's configuration as protected and masked variables. Signed-off-by: Paolo Bonzini --- RFC->v1: - disable opensbi job in scheduled pipelines - fix exit codes by using {} compound statements instead of () subshells - do not limit to master branch since anyway we have only one schedule (i.e. make it the problem of whoever creates a second schedule's) .gitlab-ci.d/base.yml | 4 ++++ .gitlab-ci.d/buildtest.yml | 38 ++++++++++++++++++++++++++++++++++++++ .gitlab-ci.d/opensbi.yml | 4 ++++ 3 files changed, 46 insertions(+) diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index ef173a34e63..2dd8a9b57cb 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -41,6 +41,10 @@ variables: - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG' when: never + # Scheduled runs on mainline don't get pipelines except for the special Coverity job + - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"' + when: never + # Cirrus jobs can't run unless the creds / target repo are set - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)' when: never diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index 901265af95d..7832c7ff3a8 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -729,3 +729,40 @@ pages: - public variables: QEMU_JOB_PUBLISH: 1 + +coverity: + image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG + stage: build + allow_failure: true + timeout: 3h + needs: + - job: amd64-fedora-container + optional: true + before_script: + - dnf install -y curl wget + script: + # would be nice to cancel the job if over quota (https://gitlab.com/gitlab-org/gitlab/-/issues/256089) + # for example: + # curl --request POST --header "PRIVATE-TOKEN: $CI_JOB_TOKEN" "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/jobs/${CI_JOB_ID}/cancel + - 'scripts/coverity-scan/run-coverity-scan --check-upload-only || { exitcode=$?; if test $exitcode = 1; then + exit 0; + else + exit $exitcode; + fi; }; + scripts/coverity-scan/run-coverity-scan --update-tools-only > update-tools.log 2>&1 || { cat update-tools.log; exit 1; }; + scripts/coverity-scan/run-coverity-scan --no-update-tools' + rules: + - if: '$COVERITY_TOKEN == null' + when: never + - if: '$COVERITY_EMAIL == null' + when: never + # Never included on upstream pipelines, except for schedules + - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"' + when: on_success + - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM' + when: never + # Forks don't get any pipeline unless QEMU_CI=1 or QEMU_CI=2 is set + - if: '$QEMU_CI != "1" && $QEMU_CI != "2"' + when: never + # Always manual on forks even if $QEMU_CI == "2" + - when: manual diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml index fd293e6c317..42f137d624e 100644 --- a/.gitlab-ci.d/opensbi.yml +++ b/.gitlab-ci.d/opensbi.yml @@ -24,6 +24,10 @@ - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /opensbi/i' when: manual + # Scheduled runs on mainline don't get pipelines except for the special Coverity job + - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_PIPELINE_SOURCE == "schedule"' + when: never + # Run if any files affecting the build output are touched - changes: - .gitlab-ci.d/opensbi.yml