From patchwork Wed Nov 22 07:01:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13464123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 143BFC61D99 for ; Wed, 22 Nov 2023 07:01:36 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.135]) by mx.groups.io with SMTP id smtpd.web10.13995.1700636488039514159 for ; Tue, 21 Nov 2023 23:01:28 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.135, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1801) id 3AM71PFf2844676; Wed, 22 Nov 2023 16:01:25 +0900 X-Iguazu-Qid: 2yAam8fxz1CJtYUqbi X-Iguazu-QSIG: v=2; s=0; t=1700636485; q=2yAam8fxz1CJtYUqbi; m=pk073jCR3WbqK+h94uayJV3bMo4htugf/A3mXIloMqA= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1801) id 3AM71OZW850005 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 22 Nov 2023 16:01:25 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v1 1/3] .gitlab-ci.yml: Add cve-check stage in CI Date: Wed, 22 Nov 2023 12:31:19 +0530 X-TSB-HOP2: ON Message-Id: <20231122070121.326276-2-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 22 Nov 2023 07:01:21.0248 (UTC) FILETIME=[B2D00E00:01DA1D11] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Nov 2023 07:01:36 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/13609 From: Sai Sathujoda The cve-check uses the dpkg-status file to get the latest CVE report using the cve_checker.py script in debian-cve-checker container. The dpkg-status file will also be deployed to aws s3 bucket along with the targets whose deployment is enabled. Signed-off-by: Sai Sathujoda --- .gitlab-ci.yml | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1de6570..c838a81 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -29,6 +29,7 @@ variables: stages: - build - test + - cve-check default: before_script: @@ -48,6 +49,7 @@ default: - large variables: base_yaml: "kas-cip.yml:kas/board/${target}.yml" + dpkg_status: "cip-core-image-*.dpkg_status" script: - if [ "${use_rt}" = "enable" ]; then base_yaml="${base_yaml}:kas/opt/rt.yml"; fi - if [ "${extension}" != "none" ]; then base_yaml="${base_yaml}:kas/opt/${extension}.yml"; fi @@ -60,9 +62,17 @@ default: - if [ "${encrypt}" = "enable" ]; then base_yaml="${base_yaml}:kas/opt/encrypt-partitions.yml"; fi - echo "Building ${base_yaml}" - kas build ${base_yaml} - - if [ "${deploy}" = "enable" ]; then scripts/deploy-cip-core.sh ${release} ${target} ${extension} ${dtb} ${CI_COMMIT_REF_SLUG}; fi + - if [ "${deploy}" = "enable" ]; then scripts/deploy-cip-core.sh ${release} ${target} ${extension} ${dtb} ${CI_COMMIT_REF_SLUG} ${dpkg_status}; fi - if [ "${deploy_kernelci}" = "enable" ]; then scripts/deploy-kernelci.py ${release} ${target} ${extension} ${dtb}; fi +cve-checks: + stage: cve-check + image: registry.gitlab.com/cip-playground/debian-cve-checker:latest + script: + - scripts/run-cve-checks.sh + rules: + - if: $CI_PIPELINE_SOURCE == "schedule" && $CI_JOB_NAME == "cve-checks" + # base image build:simatic-ipc227e-base: extends: From patchwork Wed Nov 22 07:01:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13464121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07394C072A2 for ; Wed, 22 Nov 2023 07:01:36 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.158]) by mx.groups.io with SMTP id smtpd.web11.14083.1700636490669955266 for ; Tue, 21 Nov 2023 23:01:31 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.158, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1122) id 3AM71RC83873515; Wed, 22 Nov 2023 16:01:27 +0900 X-Iguazu-Qid: 2rWgM08Ffz8SXgp1s8 X-Iguazu-QSIG: v=2; s=0; t=1700636487; q=2rWgM08Ffz8SXgp1s8; m=CtRhubI77m1DKRIkT3uwDguFdrMNXNI7AlkPilrDtYU= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1121) id 3AM71PVk3403006 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 22 Nov 2023 16:01:26 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v1 2/3] scripts/deploy-cip-core.sh: Upload dpkg-status file to aws s3 bucket Date: Wed, 22 Nov 2023 12:31:20 +0530 X-TSB-HOP2: ON Message-Id: <20231122070121.326276-3-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 22 Nov 2023 07:01:21.0326 (UTC) FILETIME=[B2DBF4E0:01DA1D11] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Nov 2023 07:01:36 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/13611 From: Sai Sathujoda For every build job executed in CI, the dpkg-status file shall be uploaded to a respective directory in aws s3 bucket. Signed-off-by: Sai Sathujoda --- scripts/deploy-cip-core.sh | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/deploy-cip-core.sh b/scripts/deploy-cip-core.sh index 5855acb..834ba8e 100755 --- a/scripts/deploy-cip-core.sh +++ b/scripts/deploy-cip-core.sh @@ -15,6 +15,7 @@ TARGET=$2 EXTENSION=$3 DTB=$4 REF=$5 +DPKG_STATUS=$6 BASE_FILENAME=cip-core-image-cip-core-$RELEASE-$TARGET if [ "${EXTENSION}" != "none" ]; then @@ -53,3 +54,17 @@ else aws s3 cp --no-progress --acl public-read build/tmp/deploy/images/*/"$DTB" "${S3_TARGET}" fi fi + +# Deploy the dpkg status file (/var/lib/dpkg/status) to AWS for the CIP CVE checker (cip-core-sec) +# to periodically extract and report pending CVEs. +# * CI builds will override the previous dpkg status file +# * Released images will contain the commit_id in the file name, and will not be automatically overriden +# * they have to be signed and released separately from the CI +# * CVE checks will only scan the latest builds, and the released images +# * AWS S3 bucket structure +# * download.cip-project.org/cip-core/cve-checks/ +# * dpkg-status: folder to store the dpkg status files generated by the CI and released images +# * cve-reports: folder to store the cve-reports generated from the dpkg status files +# * they can be linked from the release website +DPKG_STATUS_FILENAME=${CI_JOB_NAME#build:}.dpkg_status +aws s3 cp --no-progress build/tmp/deploy/images/$TARGET/$DPKG_STATUS s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/$DPKG_STATUS_FILENAME From patchwork Wed Nov 22 07:01:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13464122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0D69C61D92 for ; Wed, 22 Nov 2023 07:01:35 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.134]) by mx.groups.io with SMTP id smtpd.web10.13996.1700636490162519672 for ; Tue, 21 Nov 2023 23:01:30 -0800 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.134, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1800) id 3AM71SCU2306656; Wed, 22 Nov 2023 16:01:28 +0900 X-Iguazu-Qid: 2yAaLcf4ODe1IlQeY4 X-Iguazu-QSIG: v=2; s=0; t=1700636488; q=2yAaLcf4ODe1IlQeY4; m=iKE85ISlhpgGwLcOxU5CCUNO3oVbLSeywMmUNCu/uwU= Received: from imx12-a.toshiba.co.jp ([38.106.60.135]) by relay.securemx.jp (mx-mr1803) id 3AM71RCM889045 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 22 Nov 2023 16:01:28 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: Sai Sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v1 3/3] scripts/run-cve-checks.sh: Add script to generate CVE report Date: Wed, 22 Nov 2023 12:31:21 +0530 X-TSB-HOP2: ON Message-Id: <20231122070121.326276-4-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> References: <20231122070121.326276-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 22 Nov 2023 07:01:21.0404 (UTC) FILETIME=[B2E7DBC0:01DA1D11] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 22 Nov 2023 07:01:35 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/13610 From: Sai Sathujoda This script will extract latest dpkg-status files for all the deployed targets and generate their CVE reports using the cve_checker.py script in [1] and these report shall be uploaded back to cve-reports sub-directory under cip-project.org in the s3 bucket. [1] https://gitlab.com/cip-playground/debian-cve-checker Signed-off-by: Sai Sathujoda --- scripts/run-cve-checks.sh | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 scripts/run-cve-checks.sh diff --git a/scripts/run-cve-checks.sh b/scripts/run-cve-checks.sh new file mode 100755 index 0000000..a655151 --- /dev/null +++ b/scripts/run-cve-checks.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# +# CIP Core, generic profile +# +# Copyright (c) Toshiba Corp., 2020 +# +# Authors: +# Daniel Sangorrin +# +# SPDX-License-Identifier: MIT +# + +# This script is used in .gitlab-ci.yml to create +# CVE reports in CSV format for each build target. +# It uses the dpkg status files generated during +# the build stages and saved as gitlab-ci artifacts. + +set -e + +# Install AWS CLI +if ! which aws 2>&1 >/dev/null; then + echo "Installing awscli..." + apt update + apt install -y python3-wheel + apt install -y awscli +fi + +# Retrieve the latest dpkg status files from AWS +aws s3 cp --no-progress --recursive s3://download.cip-project.org/cip-core/cve-checks/dpkg-status/ ./ + +# Create new CVE reports +mkdir cve-reports +for i in *.dpkg_status; do + echo "Checking $i" + filename=${i%.dpkg_status} + cve_checker.py --status $i --output ./cve-reports/$filename.csv +done + +# Synchronize the CVE reports to AWS (it will delete old reports) +aws s3 sync --no-progress --delete --acl public-read cve-reports s3://download.cip-project.org/cip-core/cve-checks/cve-reports