[isar-cip-core] ci: Only allow IEC tests to still fail

Commit Message

Jan Kiszka July 1, 2024, 5:08 a.m. UTC

From: Jan Kiszka <[e-mail]>

Secure boot and SWUpdate tests are passing now and are supposed to keep
that on future changes as well.

Signed-off-by: Jan Kiszka <[e-mail]>
---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comments

Jan Kiszka July 2, 2024, 4:04 p.m. UTC | #1

On 01.07.24 07:08, Jan Kiszka wrote:
> From: Jan Kiszka <[e-mail]>
> 
> Secure boot and SWUpdate tests are passing now and are supposed to keep
> that on future changes as well.
> 

And... it fails:

https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397870
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397883

I tried to find where that swtpm_start.sh is defined but I also failed.
Can someone explain to me where we maintain these - as it looks like -
lava user-commands?

As I said during our cip-core WG meeting today, I would like to ensure
that all those definitions are pinned via isar-cip-core when its
pipeline runs so that this top-level controls when a test change is
imported. Any suggestions how to achieve that?

Thanks,
Jan

[e-mail] July 2, 2024, 4:58 p.m. UTC | #2

Hi Jan,

The reason for these failures is because the swtpm package is not present in "lab-cip-siemens-muc". swtpm_start.sh, swtpm_finish.sh etc. are maintained here -> https://gitlab.com/cip-project/cip-testing/lava-docker

The maintainer of "lab-cip-siemens-muc" should update to get the below mentioned changes.

https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/35
https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/38
https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/39

We added a tag named "swtpm-jobs" to specific QEMU devices which has the necessary changes in their device template which allow them to run some user-defined commands before booting the CIP security image on its host.

I guess today someone might have added the "swtpm-jobs" tag to qemu-cip-siemens-muc but as the lab is not updated, jobs assigned to mentioned device were unable to start. So either the swtpm-jobs tag can be removed or the respective lab should be updated.

Regarding IEC layer tests maintained in separate  repository https://gitlab.com/cip-project/cip-testing/cip-security-tests, I will send a patch to include config revision (commit id) to use in IEC job definitions. We can pass the commit reference to the 'test action' in the job definitions through submit_lava.sh. I am waiting for Stefan's review on one of our merge requests which fix most of the failed test cases. Once it is merged, I will send a patch to include config revision of cip-security-tests in isar-cip-core.

Thanks and regards,
Sai Ashrith Sathujoda (T S I P)

-----Original Message-----
From: [e-mail] <[e-mail]> On Behalf Of Jan Kiszka via lists.cip-project.org
Sent: Tuesday, July 2, 2024 9:34 PM
To: cip-dev <[e-mail]>; ashrith sai(ＴＳＩＰ) <[e-mail]>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <[e-mail]>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <[e-mail]>
Subject: Re: [cip-dev] [isar-cip-core][PATCH] ci: Only allow IEC tests to still fail

On 01.07.24 07:08, Jan Kiszka wrote:
> From: Jan Kiszka <[e-mail]>
> 
> Secure boot and SWUpdate tests are passing now and are supposed to 
> keep that on future changes as well.
> 

And... it fails:

https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397870
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397883

I tried to find where that swtpm_start.sh is defined but I also failed.
Can someone explain to me where we maintain these - as it looks like - lava user-commands?

As I said during our cip-core WG meeting today, I would like to ensure that all those definitions are pinned via isar-cip-core when its pipeline runs so that this top-level controls when a test change is imported. Any suggestions how to achieve that?

Thanks,
Jan

--
Siemens AG, Technology
Linux Expert Center

Jan Kiszka July 2, 2024, 8:01 p.m. UTC | #3

On 02.07.24 18:58, [e-mail] wrote:
> Hi Jan,
> 
> The reason for these failures is because the swtpm package is not present in "lab-cip-siemens-muc". swtpm_start.sh, swtpm_finish.sh etc. are maintained here -> https://gitlab.com/cip-project/cip-testing/lava-docker
> 
> The maintainer of "lab-cip-siemens-muc" should update to get the below mentioned changes.
> 
> https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/35
> https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/38
> https://gitlab.com/cip-project/cip-testing/lava-docker/-/merge_requests/39
> 
> We added a tag named "swtpm-jobs" to specific QEMU devices which has the necessary changes in their device template which allow them to run some user-defined commands before booting the CIP security image on its host.
> 
> I guess today someone might have added the "swtpm-jobs" tag to qemu-cip-siemens-muc but as the lab is not updated, jobs assigned to mentioned device were unable to start. So either the swtpm-jobs tag can be removed or the respective lab should be updated.
> 

Ok, thanks for the explanation. Quirin, any idea why that tag could
already be assigned to our lab? But more important is likely to update
the container.

> Regarding IEC layer tests maintained in separate  repository https://gitlab.com/cip-project/cip-testing/cip-security-tests, I will send a patch to include config revision (commit id) to use in IEC job definitions. We can pass the commit reference to the 'test action' in the job definitions through submit_lava.sh. I am waiting for Stefan's review on one of our merge requests which fix most of the failed test cases. Once it is merged, I will send a patch to include config revision of cip-security-tests in isar-cip-core.
> 

Perfect, thanks in advance!

Jan

> Thanks and regards,
> Sai Ashrith Sathujoda (T S I P)
> 
> -----Original Message-----
> From: [e-mail] <[e-mail]> On Behalf Of Jan Kiszka via lists.cip-project.org
> Sent: Tuesday, July 2, 2024 9:34 PM
> To: cip-dev <[e-mail]>; ashrith sai(ＴＳＩＰ) <[e-mail]>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <[e-mail]>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <[e-mail]>
> Subject: Re: [cip-dev] [isar-cip-core][PATCH] ci: Only allow IEC tests to still fail
> 
> On 01.07.24 07:08, Jan Kiszka wrote:
>> From: Jan Kiszka <[e-mail]>
>>
>> Secure boot and SWUpdate tests are passing now and are supposed to 
>> keep that on future changes as well.
>>
> 
> And... it fails:
> 
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397870
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/jobs/7246397883
> 
> I tried to find where that swtpm_start.sh is defined but I also failed.
> Can someone explain to me where we maintain these - as it looks like - lava user-commands?
> 
> As I said during our cip-core WG meeting today, I would like to ensure that all those definitions are pinned via isar-cip-core when its pipeline runs so that this top-level controls when a test change is imported. Any suggestions how to achieve that?
> 
> Thanks,
> Jan
> 
> --
> Siemens AG, Technology
> Linux Expert Center
>

Patch

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3cb98f7f..683790f4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -354,7 +354,6 @@  build:qemu-riscv64:
     - if: $CI_COMMIT_BRANCH != "master"
   tags:
     - small
-  allow_failure: true
   script:
     - scripts/submit_lava.sh ${test_function} ${target} ${CI_COMMIT_SHORT_SHA} ${release} ${CI_COMMIT_REF_SLUG}
   artifacts:
@@ -437,6 +436,7 @@  test:qemu-amd64-IEC:
   variables:
     target: qemu-amd64
     test_function: IEC
+  allow_failure: true
 
 test:qemu-arm64-IEC:
   extends:
@@ -445,6 +445,7 @@  test:qemu-arm64-IEC:
   variables:
     target: qemu-arm64
     test_function: IEC
+  allow_failure: true
 
 test:qemu-arm-IEC:
   extends:
@@ -453,6 +454,7 @@  test:qemu-arm-IEC:
   variables:
     target: qemu-arm
     test_function: IEC
+  allow_failure: true
 
 cve-checks:
   stage: cve-check