From patchwork Wed Feb 3 09:46:15 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 8199601 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E9C749F37A for ; Wed, 3 Feb 2016 09:49:01 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7AF1420218 for ; Wed, 3 Feb 2016 09:49:00 +0000 (UTC) Received: from lists.xen.org (lists.xenproject.org [50.57.142.19]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9D0E20221 for ; Wed, 3 Feb 2016 09:48:58 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aQu12-00029M-9x; Wed, 03 Feb 2016 09:46:24 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aQu10-000292-J4 for xen-devel@lists.xen.org; Wed, 03 Feb 2016 09:46:22 +0000 Received: from [85.158.139.211] by server-7.bemta-5.messagelabs.com id 17/37-29293-D6CC1B65; Wed, 03 Feb 2016 09:46:21 +0000 X-Env-Sender: prvs=8347a21d9=Ian.Campbell@citrix.com X-Msg-Ref: server-16.tower-206.messagelabs.com!1454492778!19690958!2 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 7.35.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 56061 invoked from network); 3 Feb 2016 09:46:20 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-16.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 3 Feb 2016 09:46:20 -0000 X-IronPort-AV: E=Sophos;i="5.22,389,1449532800"; d="scan'208";a="335762756" From: Ian Campbell To: , Date: Wed, 3 Feb 2016 09:46:15 +0000 Message-ID: <1454492776-23788-2-git-send-email-ian.campbell@citrix.com> X-Mailer: git-send-email 2.6.1 In-Reply-To: <1454492776-23788-1-git-send-email-ian.campbell@citrix.com> References: <1454492776-23788-1-git-send-email-ian.campbell@citrix.com> MIME-Version: 1.0 X-DLP: MIA2 Cc: Andrew Cooper , Ian Campbell Subject: [Xen-devel] [PATCH 2/3] Add a weekly coverity flight X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This primarily consists of ts-coverity-scan and make-coverity-flight which constructs the sole job. The most recently scanned revision is pushed to a new coverity-scanned branch in the usual xen.git, tests are run on the master branch. For the cr-* integration we treat branch=coverity as a special case of tree=xen. I didn't think tree=coverity made much sense, and would probably reach tendrils into lots of other places (such as the invocations of check_tested). I initially thoughts that $c{CoverityEmail} would need to be an actual account registered with scan, however a manual experiment using email=security@xen.org was accepted by the service. An "analysis complete" message was sent to security@ while individual results mails were sent to each member of the coverity project who was configured to receive them. I think this is what we want. The "analysis complete" mail contained no sensitive data, but also no real information other than "success" (or presumably "failure" if that were to be the case). I think going to security@ is probably OK. I have run this in non-uploading mode on the production infra and then run the curl manually, adjusting the CLI until it works and updated the script to match. I have not yet run in uploading mode but will do so once another upload is allowed by the service, In my experiments the curl command took ~35 minutes to complete (rate in the 100-200k range). Not sure if this is a problem. Note that curl is run on the controller (via system_checked) and consequently has no timeout etc. Note that the token must be supplied with Cc: Andrew Cooper --- v2: - Split move of collect_xen_built_versions() into separate patch - Implemented support for coverity_upload = true (but don't yet set it) - Add host_hostflags to the job so it can actually run somewhere. - Call tsreadconfig() before referencing $r{coverity_upload} so that $r is actually populated. - use token=. + + +set -e -o posix + +branch=$1 +xenbranch=$2 +blessing=$3 +buildflight=$4 + +flight=`./cs-flight-create $blessing $branch` + +. ./cri-common +. ./ap-common +. ./mfi-common + +defsuite=`getconfig DebianSuite` + +arch=amd64 +suite=$defsuite + +build_hostflags=share-build-$suite-$arch,arch-$arch,suite-$suite,purpose-build + +./cs-job-create $flight coverity-$arch coverity \ + arch=$arch host_hostflags=$build_hostflags \ + tree_xen=$TREE_COVERITY \ + revision_xen=$REVISION_COVERITY \ + coverity_upload=false + +echo $flight + +# Local variables: +# mode: sh +# sh-basic-offset: 2 +# indent-tabs-mode: nil +# End: diff --git a/production-config b/production-config index f2f0584..e67a253 100644 --- a/production-config +++ b/production-config @@ -100,6 +100,10 @@ TftpGrubVersion XXXX-XX-XX XenUsePath /usr/groups/xencore/systems/bin/xenuse XenUseUser osstest +# Results might include potential vulnerabilities. +CoverityEmail security@xen.org +CoverityTools cov-analysis-linux64-7.7.0.4.tar.gz + # We use the IP address because Citrix can't manage reliable nameservice #DebianMirrorHost debian.uk.xensource.com #DebianMirrorHost 10.80.16.196 diff --git a/sg-run-job b/sg-run-job index 20ebb64..7e592dd 100755 --- a/sg-run-job +++ b/sg-run-job @@ -445,6 +445,11 @@ proc prepare-build-host {} { run-ts . host-build-prep ts-xen-build-prep } +proc need-hosts/coverity {} { return BUILD } +proc run-job/coverity {} { + run-ts . = ts-coverity-scan + host +} + #---------- main program ---------- jobdb::set-flight diff --git a/ts-coverity-scan b/ts-coverity-scan new file mode 100755 index 0000000..f3cc497 --- /dev/null +++ b/ts-coverity-scan @@ -0,0 +1,110 @@ +#!/usr/bin/perl -w +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2015 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +use strict qw(vars); +use DBI; +use Osstest; +use File::Path; +use POSIX; +use Osstest::TestSupport; +use Osstest::BuildSupport; + +tsreadconfig(); +selectbuildhost(\@ARGV); +# remaining arguments are passed as targets to "make" +builddirsprops(); + +# Require explicit opt in from flight construction +my $coverity_upload = ($r{coverity_upload}//'false') =~ m/true/ ? 1 : 0; + +# This must contain exactly and only the token, for example there must +# be no trailing "\n", otherwise it is included in the literal token, +# which is then invalid. +my $tokenfile = "$ENV{HOME}/.xen-osstest/coverity-secret"; +my $submit_url = "https://scan.coverity.com/builds?project=XenProject"; + +sub checkout () { + prepbuilddirs(); + + build_clone($ho, 'xen', $builddir, 'xen'); +} + +sub covtools () { + target_putfile($ho, 100, "$c{Images}/$c{CoverityTools}", "$builddir/covtools.tar.gz"); + target_cmd($ho, <{Suite} =~ m/wheezy|squeeze|lenny/) { push(@packages, "libnl-dev");