From patchwork Wed Jun 5 15:40:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13687057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2687DC27C53 for ; Wed, 5 Jun 2024 15:40:18 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.152]) by mx.groups.io with SMTP id smtpd.web10.18788.1717602016263161794 for ; Wed, 05 Jun 2024 08:40:16 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.152, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1802) id 455FeDYp1030211; Thu, 6 Jun 2024 00:40:14 +0900 X-Iguazu-Qid: 2yAbfjWaliZdjCYbfm X-Iguazu-QSIG: v=2; s=0; t=1717602013; q=2yAbfjWaliZdjCYbfm; m=or8h3YWafoG3QrxonZSlIHegl7Cx37IxTBPeUT4Pm3A= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1800) id 455FeCph2365437 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 6 Jun 2024 00:40:13 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: sai ashrith sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 1/4] tests/templates: Include functionality based templates Date: Wed, 5 Jun 2024 21:10:07 +0530 X-TSB-HOP2: ON Message-Id: <20240605154010.3446832-2-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 05 Jun 2024 15:40:09.0582 (UTC) FILETIME=[A5B5E0E0:01DAB75E] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Jun 2024 15:40:18 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/16116 From: sai ashrith sathujoda These templates shall be extended to form architecture specific LAVA job definitions Signed-off-by: sai ashrith sathujoda --- tests/templates/IEC_template.yml | 74 ++++++++++++++ tests/templates/secureboot_template.yml | 65 ++++++++++++ tests/templates/swupdate_template.yml | 125 ++++++++++++++++++++++++ 3 files changed, 264 insertions(+) create mode 100644 tests/templates/IEC_template.yml create mode 100644 tests/templates/secureboot_template.yml create mode 100644 tests/templates/swupdate_template.yml diff --git a/tests/templates/IEC_template.yml b/tests/templates/IEC_template.yml new file mode 100644 index 0000000..d85a9c0 --- /dev/null +++ b/tests/templates/IEC_template.yml @@ -0,0 +1,74 @@ +device_type: qemu +job_name: qemu #architecture# IEC layer testing +timeouts: + job: + minutes: 30 + action: + minutes: 20 + actions: + power-off: + seconds: 60 +priority: medium +tags: +- swtpm-jobs +visibility: public +notify: + criteria: + status: finished + recipients: + - to: + method: email + email: cip-testing-results@lists.cip-project.org + +# ACTION BLOCK +actions: +- command: + name: start_tpm + timeout: + minutes: 20 + +# DEPLOY BLOCK +- deploy: + timeout: + minutes: 15 + to: tmpfs + images: + system: + image_arg: '-drive file={system},discard=unmap,if=none,id=disk,format=raw -m 1G -serial mon:stdio -smp 4 + -nographic -netdev user,id=net,hostfwd=tcp:127.0.0.1:22222-:22 -chardev socket,id=chrtpm,path=/tmp/qemu-swtpm.sock + -tpmdev emulator,id=tpm0,chardev=chrtpm #imageargs#' + url: #project_url#/#branch#/qemu-#architecture#/cip-core-image-security-cip-core-#distribution#-qemu-#architecture#.wic.xz + compression: xz + + #Firmware# + #Firmware_args# + #Firmware_url# + +# BOOT BLOCK +- boot: + timeout: + minutes: 15 + method: qemu + media: tmpfs + prompts: ["root@demo:~#"] + auto_login: + login_prompt: "demo login:" + username: "root" + password_prompt: "Password:" + password: "CIPsecurity@123" + +# TEST_BLOCK +- test: + timeout: + minutes: 30 + definitions: + - repository: https://gitlab.com/cip-project/cip-testing/cip-security-tests.git + from: git + branch: master + path: iec-security-tests/Singlenode-TestDefinition.yaml + name: Security-Testcases + +context: + arch: #architecture# + guestfs_interface: virtio + lava_test_results_dir: '/home/lava-%s' diff --git a/tests/templates/secureboot_template.yml b/tests/templates/secureboot_template.yml new file mode 100644 index 0000000..5996dbd --- /dev/null +++ b/tests/templates/secureboot_template.yml @@ -0,0 +1,65 @@ +device_type: qemu +job_name: qemu #architecture# secure boot testing +timeouts: + job: + minutes: 30 + action: + minutes: 20 + actions: + power-off: + seconds: 60 +priority: medium +tags: +- swtpm-jobs +visibility: public +notify: + criteria: + status: finished + recipients: + - to: + method: email + email: cip-testing-results@lists.cip-project.org + +# ACTION BLOCK +actions: +- command: + name: start_tpm + timeout: + minutes: 20 + +# DEPLOY BLOCK +- deploy: + timeout: + minutes: 15 + to: tmpfs + images: + system: + image_arg: '-drive file={system},discard=unmap,if=none,id=disk,format=raw -m 1G -serial mon:stdio -smp 4 + -nographic -netdev user,id=net,hostfwd=tcp:127.0.0.1:22222-:22 -chardev socket,id=chrtpm,path=/tmp/qemu-swtpm.sock + -tpmdev emulator,id=tpm0,chardev=chrtpm #imageargs#' + url: #project_url#/#branch#/qemu-#architecture#/cip-core-image-security-cip-core-#distribution#-qemu-#architecture#.wic.xz + compression: xz + + #Firmware# + #Firmware_args# + #Firmware_url# + +# BOOT BLOCK +- boot: + timeout: + minutes: 15 + method: qemu + media: tmpfs + prompts: ["root@demo:~#"] + auto_login: + login_prompt: "demo login:" + username: "root" + password_prompt: "Password:" + password: "CIPsecurity@123" + parameters: + kernel-start-message: "EFI stub: UEFI Secure Boot is enabled." + +context: + arch: #architecture# + guestfs_interface: virtio + lava_test_results_dir: '/home/lava-%s' diff --git a/tests/templates/swupdate_template.yml b/tests/templates/swupdate_template.yml new file mode 100644 index 0000000..3812118 --- /dev/null +++ b/tests/templates/swupdate_template.yml @@ -0,0 +1,125 @@ +device_type: qemu +job_name: qemu #architecture# software update testing +timeouts: + job: + minutes: 30 + action: + minutes: 20 + actions: + power-off: + seconds: 60 +tags: +- swtpm-jobs +priority: medium +visibility: public +notify: + criteria: + status: finished + recipients: + - to: + method: email + email: cip-testing-results@lists.cip-project.org + +# ACTION BLOCK +actions: +- command: + name: start_tpm + timeout: + minutes: 20 + +# DEPLOY BLOCK +- deploy: + timeout: + minutes: 15 + to: tmpfs + images: + system: + image_arg: '-drive file={system},discard=unmap,if=none,id=disk,format=raw -m 1G -serial mon:stdio -smp 4 + -nographic -netdev user,id=net,hostfwd=tcp:127.0.0.1:22222-:22 -chardev socket,id=chrtpm,path=/tmp/qemu-swtpm.sock + -tpmdev emulator,id=tpm0,chardev=chrtpm #imageargs#' + url: #project_url#/#branch#/qemu-#architecture#/cip-core-image-security-cip-core-#distribution#-qemu-#architecture#.wic.xz + compression: xz + + #Firmware# + #Firmware_args# + #Firmware_url# + +# BOOT BLOCK +- boot: + timeout: + minutes: 15 + method: qemu + media: tmpfs + prompts: ["root@demo:~#"] + auto_login: + login_prompt: "demo login:" + username: "root" + password_prompt: "Password:" + password: "CIPsecurity@123" + +# TEST_BLOCK +# Sleep 200s is added to make the job incomplete with a timeout if software update application failed +- test: + timeout: + minutes: 5 + definitions: + - repository: + metadata: + format: Lava-Test Test Definition 1.0 + name: sample-test + description: "Test software update" + run: + steps: + - if swupdate -d "-u #project_url#/#branch#/qemu-#architecture#/cip-core-image-security-cip-core-#distribution#-qemu-#architecture#.swu"; then echo software update is successful!!; else sleep 200s; fi + from: inline + name: sample-test-1 + path: inline/sample-test.yaml + +# qemu-swtpm.sock will be gone after soft reboot. +# So the swtpm socket need to be started again for proper reboot +# To start the swtpm daemon, first the existing one should be killed +- command: + name: manual_kill + timeout: + minutes: 1 +# Start the swtpm daemon +- command: + name: start_tpm + timeout: + minutes: 1 + +- boot: + timeout: + minutes: 5 + method: qemu + media: tmpfs + prompts: ["root@demo:~#"] + auto_login: + login_prompt: "demo login:" + username: "root" + password_prompt: "Password:" + password: "CIPsecurity@123" + parameters: + kernel-start-message: "kernel: C:BOOT1:linux.efi" + +# sleep 180s is added to make the job incomplete if ustate is not 2 in the updated partition +- test: + timeout: + minutes: 5 + definitions: + - repository: + metadata: + format: Lava-Test Test Definition 1.0 + name: sample-test + description: "check boot loader environment variables" + run: + steps: + - if [ $(bg_printenv | grep ustate | awk 'FNR == 2{print $2}') = 2 ]; then bg_setenv -c; else sleep 180s; fi + from: inline + name: sample-test-2 + path: inline/sample-test.yaml + +context: + arch: #architecture# + guestfs_interface: virtio + lava_test_results_dir: '/home/lava-%s' From patchwork Wed Jun 5 15:40:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13687056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 259D7C25B76 for ; Wed, 5 Jun 2024 15:40:18 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.131]) by mx.groups.io with SMTP id smtpd.web10.18789.1717602016927456169 for ; Wed, 05 Jun 2024 08:40:17 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.131, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1120) id 455FeE7U2022957; Thu, 6 Jun 2024 00:40:15 +0900 X-Iguazu-Qid: 2rWhTy7tCk8YDIx4BU X-Iguazu-QSIG: v=2; s=0; t=1717602014; q=2rWhTy7tCk8YDIx4BU; m=5gAqTJsUgkffFA1mvJffhUYlZqiHACKQVWGDlxvkqfg= Received: from imx12-a.toshiba.co.jp ([38.106.60.135]) by relay.securemx.jp (mx-mr1123) id 455FeDUM2243115 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Thu, 6 Jun 2024 00:40:14 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: sai ashrith sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 2/4] submit_lava.sh: Add script to submit the jobs prepared from templates Date: Wed, 5 Jun 2024 21:10:08 +0530 X-TSB-HOP2: ON Message-Id: <20240605154010.3446832-3-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 05 Jun 2024 15:40:09.0676 (UTC) FILETIME=[A5C438C0:01DAB75E] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Jun 2024 15:40:18 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/16117 From: sai ashrith sathujoda This script creates job definitions from the template files included in the test directory and shall submit these jobs to CIP LAVA platform. This script will wait until all the results of all the submitted jobs are obtained. Signed-off-by: sai ashrith sathujoda --- scripts/submit_lava.sh | 464 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 464 insertions(+) create mode 100755 scripts/submit_lava.sh diff --git a/scripts/submit_lava.sh b/scripts/submit_lava.sh new file mode 100755 index 0000000..e9d89dc --- /dev/null +++ b/scripts/submit_lava.sh @@ -0,0 +1,464 @@ +#!/bin/bash +# Copyright (C) 2024, Renesas Electronics Europe GmbH +# Chris Paterson +# Sai Ashrith +################################################################################ + +set -e + +################################################################################ +LAVA_TEMPLATES="tests/templates" +LAVA_JOBS_URL="https://${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/scheduler/job" +LAVA_API_URL="https://${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/api/v0.2" +LAVACLI_ARGS="--uri https://$CIP_LAVA_LAB_USER:$CIP_LAVA_LAB_TOKEN@${CIP_LAVA_LAB_SERVER:-lava.ciplatform.org}/RPC2" +SQUAD_GROUP="cip-core" +SQUAD_WATCH_JOBS_URL="${CIP_SQUAD_URL}/api/watchjob" +SQUAD_LAVA_BACKEND="${CIP_SQUAD_LAVA_BACKEND:-cip}" +PROJECT_URL="https://s3.eu-central-1.amazonaws.com/download2.cip-project.org/cip-core" +WORK_DIR=$(pwd) +RESULTS_DIR="$WORK_DIR/results" +ERROR=false +TEST=$1 +COMMIT_REF=$2 +RELEASE=$3 +COMMIT_BRANCH=$4 + +if [ -z "$SUBMIT_ONLY" ]; then SUBMIT_ONLY=false; fi + +# Create a dictionary to handle image arguments based on architecture +declare -A image_args +image_args[amd64]="-cpu qemu64 -machine q35,accel=tcg -global ICH9-LPC.noreboot=off -device ide-hd,drive=disk -drive if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.secboot.fd -device virtio-net-pci,netdev=net -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_VARS_4M.snakeoil.fd -global ICH9-LPC.disable_s3=1 -global isa-fdc.driveA= -device tpm-tis,tpmdev=tpm0" +image_args[arm64]="-cpu cortex-a57 -machine virt -device virtio-serial-device -device virtconsole,chardev=con -chardev vc,id=con -device virtio-blk-device,drive=disk -device virtio-net-device,netdev=net -device tpm-tis-device,tpmdev=tpm0" +image_args[arm]="-cpu cortex-a15 -machine virt -device virtio-serial-device -device virtconsole,chardev=con -chardev vc,id=con -device virtio-blk-device,drive=disk -device virtio-net-device,netdev=net -device tpm-tis-device,tpmdev=tpm0" + +set_up (){ + echo "Installing dependencies to run this script..." + sudo apt update && sudo apt install -y --no-install-recommends lavacli curl + job_dir="$(mktemp -d)" +} + +clean_up () { + rm -rf "$job_dir" +} + +# This method is called only for arm64 and arm targets while building job definitions +add_firmware_artifacts () { + sed -i "s@#Firmware#@firmware:@g" "$2" + sed -i "s@#Firmware_args#@image_arg: '-bios {firmware}'@g" "$2" + sed -i "s@#Firmware_url#@url: ${PROJECT_URL}/${COMMIT_BRANCH}/qemu-${1}/firmware.bin@g" "$2" +} + +# This method creates LAVA job definitions for QEMU amd64, arm64 and armhf +# The created job definitions test SWUpdate, Secureboot and IEC layer +create_jobs () { + if [ "$1" = "IEC_Layer_test" ]; then + for arch in amd64 arm64 arm + do + cp $LAVA_TEMPLATES/IEC_template.yml "${job_dir}"/IEC_${arch}.yml + + if [ $arch != amd64 ]; then + add_firmware_artifacts $arch "${job_dir}"/IEC_${arch}.yml + fi + done + + elif [ "$1" = "software_update_test" ]; then + if [ -z "$2" ]; then + for arch in amd64 arm64 arm + do + cp $LAVA_TEMPLATES/swupdate_template.yml "${job_dir}"/swupdate_${arch}.yml + + if [ $arch != amd64 ]; then + add_firmware_artifacts $arch "${job_dir}"/swupdate_${arch}.yml + fi + done + else + cp $LAVA_TEMPLATES/swupdate_template.yml "${job_dir}"/"${2}"_amd64.yml + sed -i "s@software update testing@${2}@g" "${job_dir}"/"${2}"_amd64.yml + sed -i "s@) = 2@) = 0@g" "${job_dir}"/"${2}"_amd64.yml + if [ "$2" = "kernel_panic" ]; then + sed -i "s@kernel: C:BOOT1:linux.efi@Kernel panic - not syncing: sysrq triggered crash@g" "${job_dir}"/"${2}"_amd64.yml + else + sed -i "s@kernel: C:BOOT1:linux.efi@Can't open verity rootfs - continuing will lead to a broken trust chain!@g" "${job_dir}"/"${2}"_amd64.yml + sed -i "s@echo software update is successful!!@dd if=/dev/urandom of=/dev/sda5 bs=512 count=1@g" "${job_dir}"/"${2}"_amd64.yml + fi + fi + else + for arch in amd64 arm64 arm + do + cp $LAVA_TEMPLATES/secureboot_template.yml "${job_dir}"/secureboot_${arch}.yml + + if [ $arch != amd64 ]; then + add_firmware_artifacts $arch "${job_dir}"/secureboot_${arch}.yml + fi + done + fi + + if [ "$2" = "kernel_panic" ]; then + sed -i "s@#branch#@maintain-lava-artifact@g" "${job_dir}"/"${2}"_amd64.yml + elif [ "$2" = "kernel_panic" ]; then + sed -i "s@#branch#@${COMMIT_BRANCH}@g" "${job_dir}"/"${2}"_amd64.yml + else + sed -i "s@#branch#@${COMMIT_BRANCH}@g" "${job_dir}"/*.yml + fi + sed -i "s@#distribution#@${release}@g" "${job_dir}"/*.yml + sed -i "s@#project_url#@${PROJECT_URL}@g" "${job_dir}"/*.yml + + for arch in amd64 arm64 arm + do + sed -i "s@#architecture#@${arch}@g" "${job_dir}"/*${arch}.yml + sed -i "s@#imageargs#@${image_args[$arch]}@g" "${job_dir}"/*${arch}.yml + done +} + +create_cip_core_jobs () { + if [ "$TEST" = "IEC" ]; then + create_jobs IEC_Layer_test + elif [ "$TEST" = "swupdate" ]; then + create_jobs software_update_test + create_jobs software_update_test kernel_panic + create_jobs software_update_test initramfs_crash + else + create_jobs secure_boot_test + fi +} + +# This method attaches SQUAD watch job to the submitted LAVA job +# $1: LAVA Job ID +submit_squad_watch_job(){ +# SQUAD watch job submission + local ret + if [ -z ${CIP_SQUAD_LAB_TOKEN+x} ]; then + echo "SQUAD_LAB_TOKEN not found, omitting SQUAD results reporting!" + return 0 + fi + + if [ "$TEST" = "swupdate" ]; then + squad_project="swupdate-testing" + elif [ "$TEST" = "secure-boot" ]; then + squad_project="secure-boot-testing" + else + squad_project="iec-layer-testing" + fi + + local DEVICE=$2 + local ENV="${DEVICE}_${squad_project}" + local squad_url="$SQUAD_WATCH_JOBS_URL/${SQUAD_GROUP}/${squad_project}/${COMMIT_REF}/${ENV}" + ret=$(curl -s \ + --header "Authorization: token $CIP_SQUAD_LAB_TOKEN" \ + --form backend="$SQUAD_LAVA_BACKEND" \ + --form testjob_id="$1" \ + --form metadata='{"device": "'${DEVICE}'", "CI pipeline": "'${CI_PIPELINE_URL}'", "CI job": "'${CI_JOB_URL}'"}' \ + "$squad_url") + + if [[ $ret != [0-9]* ]] + then + echo "Something went wrong with SQUAD watch job submission. SQUAD returned:" + echo "${ret}" + echo "SQUAD URL: ${squad_url}" + echo "SQUAD Backend: ${SQUAD_LAVA_BACKEND}" + echo "LAVA Job Id: $1" + else + echo "SQUAD watch job submitted successfully as #${ret}." + fi +} + +# $1: Job definition file +submit_job() { + # Make sure yaml file exists + if [ -f "$1" ]; then + echo "Submitting $1 to LAVA master..." + # Catch error that occurs if invalid yaml file is submitted + local ret=$(lavacli $LAVACLI_ARGS jobs submit "$1") || error=true + + if [[ $ret != [0-9]* ]] + then + echo "Something went wrong with job submission. LAVA returned:" + echo "${ret}" + else + echo "Job submitted successfully as #${ret}." + + local lavacli_output=${job_dir}/lavacli_output + lavacli $LAVACLI_ARGS jobs show "${ret}" \ + > "$lavacli_output" + + local status=$(cat "$lavacli_output" \ + | grep "state" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + STATUS[${ret}]=$status + + local health=$(cat "$lavacli_output" \ + | grep "Health" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + HEALTH[${ret}]=$health + + local device_type=$(cat "$lavacli_output" \ + | grep "device-type" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + DEVICE_TYPE[${ret}]=$device_type + + local device=$(cat "$lavacli_output" \ + | grep "device :" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + DEVICE[${ret}]=$device + + local test=$(cat "$lavacli_output" \ + | grep "description" \ + | rev | cut -d "_" -f 1 | rev) + TEST[${ret}]=$test + + submit_squad_watch_job "${ret}" "${device}" + + JOBS+=("${ret}") + + fi + fi +} + +# $1: Device-type to search for +is_device_online () { + local lavacli_output=${job_dir}/lavacli_output + + # Get list of all devices + lavacli $LAVACLI_ARGS devices list > "$lavacli_output" + + # Count the number of online devices + local count=$(grep "(${1})" "$lavacli_output" | grep -c "Good") + echo "There are currently $count \"${1}\" devices online." + + if [ "$count" -gt 0 ]; then + return 0 + fi + return 1 +} + +submit_jobs () { + local ret=0 + for JOB in "${job_dir}"/*.yml; do + local device=$(grep device_type "$JOB" | cut -d ":" -f 2 | awk '{$1=$1};1') + if is_device_online "$device"; then + submit_job "$JOB" + else + echo "Refusing to submit test job as there are no suitable devices available." + ret=1 + fi + done + return $ret +} + +# This method is added with the intention to check if all the jobs are valid before submit +# If even a single definition is found to be invalid, then no job shall be submitted until +# it is fixed by the maintainer +validate_jobs () { + local ret=0 + for JOB in "${job_dir}"/*.yml; do + if lavacli $LAVACLI_ARGS jobs validate "$JOB"; then + echo "$JOB is a valid definition" + else + echo "$JOB is not a valid definition" + ret=1 + fi + done + return $ret +} + +check_if_all_finished () { + for i in "${JOBS[@]}"; do + if [ "${STATUS[$i]}" != "Finished" ]; then + return 1 + fi + done + return 0 +} + +check_for_test_error () { + for i in "${JOBS[@]}"; do + if [ "${HEALTH[$i]}" != "Complete" ]; then + return 0 + fi + done + return 1 +} + +# $1: LAVA job ID to show results for +get_test_result () { + if [ -n "${1}" ]; then + lavacli "$LAVACLI_ARGS" results "${1}" + fi +} + +get_junit_test_results () { + mkdir -p "${RESULTS_DIR}" + for i in "${JOBS[@]}"; do + curl -s -o "${RESULTS_DIR}"/results_"$i".xml "${LAVA_API_URL}"/jobs/"$i"/junit/ + done +} + +# $1: Test to print before job summaries +# $2: Set to true to print results for each job +print_status () { + if [ -z "${1}" ]; then + # Set default text + local message="Current job status:" + else + local message="${1}" + fi + + echo "------------------------------" + echo "${message}" + echo "------------------------------" + for i in "${JOBS[@]}"; do + echo "Job #$i: ${STATUS[$i]}" + echo "Health: ${HEALTH[$i]}" + echo "Device Type: ${DEVICE_TYPE[$i]}" + echo "Device: ${DEVICE[$i]}" + echo "Test: ${TEST[$i]}" + echo "URL: ${LAVA_JOBS_URL}/$i" + if [ -n "${2}" ]; then + get_test_result "$i" + fi + echo " " + done +} + +print_summary () { + echo "------------------------------" + echo "Job Summary" + echo "------------------------------" + for i in "${JOBS[@]}" + do + echo "Job #${i} ${STATUS[$i]}. Job health: ${HEALTH[$i]}. URL: ${LAVA_JOBS_URL}/${i}" + done +} + +check_status () { + if [ -n "$TEST_TIMEOUT" ]; then + # Current time + timeout time + local end_time=$(date +%s -d "+ $TEST_TIMEOUT min") + fi + + local error=false + + if [ ${#JOBS[@]} -ne 0 ] + then + + print_status "Current job status:" + while true + do + # Get latest status + for i in "${JOBS[@]}" + do + if [ "${STATUS[$i]}" != "Finished" ] + then + local lavacli_output=${job_dir}/lavacli_output + lavacli $LAVACLI_ARGS jobs show "$i" \ + > "$lavacli_output" + + local status=$(cat "$lavacli_output" \ + | grep "state" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + + local health=$(cat "$lavacli_output" \ + | grep "Health" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + HEALTH[$i]=$health + + local device_type=$(cat "$lavacli_output" \ + | grep "device-type" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + DEVICE_TYPE[$i]=$device_type + + local device=$(cat "$lavacli_output" \ + | grep "device :" \ + | cut -d ":" -f 2 \ + | awk '{$1=$1};1') + DEVICE[$i]=$device + + if [ "${STATUS[$i]}" != "$status" ]; then + STATUS[$i]=$status + + # Something has changed + print_status "Current job status:" + else + STATUS[$i]=$status + fi + fi + done + + if check_if_all_finished; then + break + fi + + if [ -n "$TEST_TIMEOUT" ]; then + # Check timeout + local now=$(date +%s) + if [ "$now" -ge "$end_time" ]; then + echo "Timed out waiting for test jobs to complete" + error=true + break + fi + fi + + # Wait to avoid spamming the server too hard + sleep 60 + done + + if check_if_all_finished; then + # Print job outcome + print_status "Final job status:" true + + if check_for_test_error; then + error=true + fi + fi + fi + + if $error; then + echo "---------------------" + echo "Errors during testing" + echo "---------------------" + print_summary + clean_up + return 1 + fi + + echo "-----------------------------------" + echo "All submitted tests were successful" + echo "-----------------------------------" + print_summary + return 0 +} + +set_up +create_cip_core_jobs + +if ! validate_jobs; then + clean_up + exit 1 +fi + +if ! submit_jobs; then + clean_up + exit 1 +fi + +if ! $SUBMIT_ONLY; then + if ! check_status; then + ERROR=true + fi + + get_junit_test_results +fi + +clean_up + +if $ERROR; then + exit 1 +fi -- 2.30.2 From patchwork Wed Jun 5 15:40:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13687058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BEAC25B76 for ; Wed, 5 Jun 2024 15:40:28 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.131]) by mx.groups.io with SMTP id smtpd.web11.18958.1717602018781854014 for ; Wed, 05 Jun 2024 08:40:19 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.131, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1120) id 455FeHI12022979; Thu, 6 Jun 2024 00:40:17 +0900 X-Iguazu-Qid: 2rWhGbqYjCI7V9o1bc X-Iguazu-QSIG: v=2; s=0; t=1717602016; q=2rWhGbqYjCI7V9o1bc; m=RWdhrQoj9sJyXyKbOcC+wCQl6S+tEfZmO6tItm+atOg= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1123) id 455FeF542243142 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 6 Jun 2024 00:40:16 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: sai ashrith sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 3/4] .gitlab-ci.yml: Add test targets specific for CIP core functionalities Date: Wed, 5 Jun 2024 21:10:09 +0530 X-TSB-HOP2: ON Message-Id: <20240605154010.3446832-4-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 05 Jun 2024 15:40:09.0832 (UTC) FILETIME=[A5DC0680:01DAB75E] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Jun 2024 15:40:28 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/16118 From: sai ashrith sathujoda Signed-off-by: sai ashrith sathujoda --- .gitlab-ci.yml | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index aab55be..80f8659 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -358,6 +358,39 @@ build:qemu-riscv64: deploy: disable allow_failure: true +.test-cip-core: + stage: test + needs: ["build:qemu-amd64-base","build:qemu-arm64-base","build:qemu-arm-base"] + tags: + - small + allow_failure: true + artifacts: + name: "$CI_JOB_NAME" + when: always + expire_in: 1 day + paths: + - results + reports: + junit: results/results*.xml + +test-qemu-secure-boot: + extends: + - .test-cip-core + script: + - scripts/submit_lava.sh secure-boot ${CI_COMMIT_SHA} ${release} ${CI_COMMIT_REF_SLUG} + +test-qemu-swupdate: + extends: + - .test-cip-core + script: + - scripts/submit_lava.sh swupdate ${CI_COMMIT_SHA} ${release} ${CI_COMMIT_REF_SLUG} + +test-qemu-IEC: + extends: + - .test-cip-core + script: + - scripts/submit_lava.sh IEC ${CI_COMMIT_SHA} ${release} ${CI_COMMIT_REF_SLUG} + cve-checks: stage: cve-check needs: [] From patchwork Wed Jun 5 15:40:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sai.Sathujoda@toshiba-tsip.com X-Patchwork-Id: 13687059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 080C7C41513 for ; Wed, 5 Jun 2024 15:40:28 +0000 (UTC) Received: from mo-csw.securemx.jp (mo-csw.securemx.jp [210.130.202.152]) by mx.groups.io with SMTP id smtpd.web11.18960.1717602019359612221 for ; Wed, 05 Jun 2024 08:40:19 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: toshiba-tsip.com, ip: 210.130.202.152, mailfrom: sai.sathujoda@toshiba-tsip.com) Received: by mo-csw.securemx.jp (mx-mo-csw1802) id 455FeH3f1030382; Thu, 6 Jun 2024 00:40:17 +0900 X-Iguazu-Qid: 2yAbxjwjwIb6YE2E9B X-Iguazu-QSIG: v=2; s=0; t=1717602017; q=2yAbxjwjwIb6YE2E9B; m=EL9IXmz3Uxotggg0QQga+CPHh3QkXGXe55xOzcUYDGE= Received: from imx2-a.toshiba.co.jp (imx2-a.toshiba.co.jp [106.186.93.35]) by relay.securemx.jp (mx-mr1801) id 455FeGeb2989019 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 6 Jun 2024 00:40:17 +0900 From: Sai.Sathujoda@toshiba-tsip.com To: cip-dev@lists.cip-project.org, jan.kiszka@siemens.com Cc: sai ashrith sathujoda , dinesh.kumar@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp Subject: [isar-cip-core v2 4/4] doc/README.lava-testing.md: Add README to explain CIP Core testing Date: Wed, 5 Jun 2024 21:10:10 +0530 X-TSB-HOP2: ON Message-Id: <20240605154010.3446832-5-Sai.Sathujoda@toshiba-tsip.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> References: <20240605154010.3446832-1-Sai.Sathujoda@toshiba-tsip.com> MIME-Version: 1.0 X-OriginalArrivalTime: 05 Jun 2024 15:40:09.0848 (UTC) FILETIME=[A5DE7780:01DAB75E] List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 05 Jun 2024 15:40:28 -0000 X-Groupsio-URL: https://lists.cip-project.org/g/cip-dev/message/16119 From: sai ashrith sathujoda This README explains the kind of testing done by the LAVA job definitions created from the template files. Signed-off-by: sai ashrith sathujoda --- doc/README.lava-testing.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 doc/README.lava-testing.md diff --git a/doc/README.lava-testing.md b/doc/README.lava-testing.md new file mode 100644 index 0000000..85beaa7 --- /dev/null +++ b/doc/README.lava-testing.md @@ -0,0 +1,30 @@ +# CIP Core functionality testing with LAVA + + +This document explains how SWUpdate, Secure boot and IEC layer are being tested using LAVA. The tests/templates directory contains functionality based templates which are extended to LAVA job definitions which tests the respective functionality on QEMU amd64, arm64 and arm architectures. CI jobs run until the final health of all the submitted jobs is reported. Test results are also pushed to [CIP SQUAD platform](https://squad.ciplatform.org/cip-core/). + +## Templates + +#### IEC_template.yml: +This template is extended to three LAVA job definitions which runs [IEC layer test cases](https://gitlab.com/cip-project/cip-testing/cip-security-tests) on the QEMU security target. Here is a [reference](https://lava.ciplatform.org/scheduler/job/1143475/definition) to the IEC job definition. + +#### secureboot_template.yml: +This template is extended to three LAVA job definitions which checks whether secure boot is enabled on the QEMU security target. Here is a [reference](https://lava.ciplatform.org/scheduler/job/1143474/definition) to the secure boot job definition. + +#### swupdate_template.yml: +This template is extended to five LAVA job definitions in which three jobs verify successful software update, partition switch after reboot and checking whether bootloader environment variables(ustate) are updated. Here is a [reference](https://lava.ciplatform.org/scheduler/job/1143538/definition) to the software update job definition. + +The other two jobs i.e [kernel_panic.yml](https://lava.ciplatform.org/scheduler/job/1143642/definition) and [initramfs_crash.yml](https://lava.ciplatform.org/scheduler/job/1143643/definition) verify the roll back feature during a fail case scenarios. + +## LAVA Setup + +The above mentioned job definitions shall be sent to CIP LAVA Lab for testing. For local testing please follow the steps mentioned in [CIP LAVA Docker](https://gitlab.com/cip-project/cip-testing/lava-docker/-/tree/cip-lava-docker?ref_type=heads#linaros-automated-validation-architecture-lava-docker-container) to have your own setup. + +## LAVA template variables + +* `architecture` : This variable represents architecture of the QEMU security target on which the test is done. +* `project_url` : Default value is `https://s3.eu-central-1.amazonaws.com/download2.cip-project.org/cip-core`. +* `branch` : This variable represents the branch on which the pipeline is triggered. +* `distribution` : This variable is assigned with the `release` assigned in gitlab CI configuration file. +* `Firmware_args` : This variable is used only for `arm64` and `armhf` architectures and it has boot parameters for the U-boot firmware binary. +* `Firmware_url` : This variable represents the firmware binary artifact uploaded in cip-project s3 bucket.