| Message ID | 90a256242870d1772bade171a7171d4e889f4e02.1739409822.git-series.marmarek@invisiblethingslab.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Few CI improvements | expand |
On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote: > Debugging sometimes involves running specific jobs on different > versions. It's useful to easily avoid running all of the not interesting > ones (for given case) to save both time and CI resources. Doing so used > to require changing the yaml files, usually in several places. > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. > Note that one needs to satisfy job dependencies on their own (for > example if a test job needs a build job, that specific build job > needs to be included too). > > The variable can be specified via Gitlab web UI when scheduling a > pipeline, but it can be also set when doing git push directly: > > git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html > > The variable needs to include regex for selecting jobs, including > enclosing slashes. > A coma/space separated list of jobs to select would be friendlier UX, > but unfortunately that is not supported: > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed > workaround doesn't work for job-level CI_JOB_NAME). > On the other hand, the regex is more flexible (one can select for > example all arm32 jobs). I was trying to find workarounds so that we could also support the simple list of comma-separated jobs you mentioned, but I couldn't find an easy way to do that. However, one thing we can do is to support writing SELECTED_JOBS_ONLY in .gitlab-ci.yml as a commit in xen.git, for people that don't know or don't remember how to set ci.variable using the git command line. Given that this is for testing, I think it would be no problem to adding a special commit on top of your tree. We are just trying to make it easier compared to having to manually delete the list of jobs we don't need. But even with the special commit, I couldn't think of an easy way to achieve the nicer comma-separated list of jobs... > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > This probably wants documenting beyond this commit message. I don't > think we have any CI-related docs anywhere, do we? Some new file in > docs/misc? Yes please :-) It would be also worth documenting how to create a simple pipeline by removing everything that you don't need for a single test > And also, it's possible to extend web ui for starting pipelines to > include pre-defined variables. I use it in qubes here if you want to > see: > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new I don't have access to this > Does it make sense to include SELECTED_JOBS_ONLY this way too? > Personally, I'll probably use it via cmdline push only anyway, but I > don't know what workflows other people have. > --- > automation/gitlab-ci/build.yaml | 6 ++++++ > automation/gitlab-ci/test.yaml | 14 ++++++++++++++ > 2 files changed, 20 insertions(+) > > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index 35e224366f62..f12de00a164a 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -12,6 +12,12 @@ > - '*/*.log' > when: always > needs: [] > + rules: > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > + when: always > + - if: $SELECTED_JOBS_ONLY > + when: never > + - when: on_success > > .gcc-tmpl: > variables: &gcc > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index c21a37933881..93632f1f9204 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -1,6 +1,11 @@ > .test-jobs-common: > stage: test > image: ${XEN_REGISTRY}/${CONTAINER} > + rules: > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > + - if: $SELECTED_JOBS_ONLY > + when: never > + - when: on_success > > .arm64-test-needs: &arm64-test-needs > - alpine-3.18-arm64-rootfs-export > @@ -99,6 +104,9 @@ > - '*.dtb' > when: always > rules: > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > + - if: $SELECTED_JOBS_ONLY > + when: never > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > @@ -117,6 +125,9 @@ > - '*.log' > when: always > rules: > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > + - if: $SELECTED_JOBS_ONLY > + when: never > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > @@ -137,6 +148,9 @@ > - '*.log' > when: always > rules: > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > + - if: $SELECTED_JOBS_ONLY > + when: never > - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - qubes-hw2 > -- > git-series 0.9.1 >
On Thu, Feb 13, 2025 at 05:36:47PM -0800, Stefano Stabellini wrote: > On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote: > > Debugging sometimes involves running specific jobs on different > > versions. It's useful to easily avoid running all of the not interesting > > ones (for given case) to save both time and CI resources. Doing so used > > to require changing the yaml files, usually in several places. > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. > > Note that one needs to satisfy job dependencies on their own (for > > example if a test job needs a build job, that specific build job > > needs to be included too). > > > > The variable can be specified via Gitlab web UI when scheduling a > > pipeline, but it can be also set when doing git push directly: > > > > git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" > > > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html > > > > The variable needs to include regex for selecting jobs, including > > enclosing slashes. > > A coma/space separated list of jobs to select would be friendlier UX, > > but unfortunately that is not supported: > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed > > workaround doesn't work for job-level CI_JOB_NAME). > > On the other hand, the regex is more flexible (one can select for > > example all arm32 jobs). > > I was trying to find workarounds so that we could also support the > simple list of comma-separated jobs you mentioned, but I couldn't find > an easy way to do that. > > However, one thing we can do is to support writing SELECTED_JOBS_ONLY in > .gitlab-ci.yml as a commit in xen.git, for people that don't know or > don't remember how to set ci.variable using the git command line. You can always do it, in `variables` setting AFAIR. > Given that this is for testing, I think it would be no problem to adding > a special commit on top of your tree. We are just trying to make it > easier compared to having to manually delete the list of jobs we don't > need. Yes, manually delete was awful. In practice I usually added always-false rules, but still. > But even with the special commit, I couldn't think of an easy way to > achieve the nicer comma-separated list of jobs... > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > --- > > This probably wants documenting beyond this commit message. I don't > > think we have any CI-related docs anywhere, do we? Some new file in > > docs/misc? > > Yes please :-) > > It would be also worth documenting how to create a simple pipeline by > removing everything that you don't need for a single test You mean how to find what jobs you need? > > And also, it's possible to extend web ui for starting pipelines to > > include pre-defined variables. I use it in qubes here if you want to > > see: > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new > > I don't have access to this Oh, sorry. Screenshot attached. And its definition looks like this: https://gitlab.com/QubesOS/qubes-continuous-integration/-/blob/main/.gitlab-ci.yml?ref_type=heads#L15-26 > > Does it make sense to include SELECTED_JOBS_ONLY this way too? > > Personally, I'll probably use it via cmdline push only anyway, but I > > don't know what workflows other people have. > > --- > > automation/gitlab-ci/build.yaml | 6 ++++++ > > automation/gitlab-ci/test.yaml | 14 ++++++++++++++ > > 2 files changed, 20 insertions(+) > > > > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > > index 35e224366f62..f12de00a164a 100644 > > --- a/automation/gitlab-ci/build.yaml > > +++ b/automation/gitlab-ci/build.yaml > > @@ -12,6 +12,12 @@ > > - '*/*.log' > > when: always > > needs: [] > > + rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + when: always > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > + - when: on_success > > > > .gcc-tmpl: > > variables: &gcc > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > index c21a37933881..93632f1f9204 100644 > > --- a/automation/gitlab-ci/test.yaml > > +++ b/automation/gitlab-ci/test.yaml > > @@ -1,6 +1,11 @@ > > .test-jobs-common: > > stage: test > > image: ${XEN_REGISTRY}/${CONTAINER} > > + rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > + - when: on_success > > > > .arm64-test-needs: &arm64-test-needs > > - alpine-3.18-arm64-rootfs-export > > @@ -99,6 +104,9 @@ > > - '*.dtb' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - xilinx > > @@ -117,6 +125,9 @@ > > - '*.log' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - xilinx > > @@ -137,6 +148,9 @@ > > - '*.log' > > when: always > > rules: > > + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY > > + - if: $SELECTED_JOBS_ONLY > > + when: never > > - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > > tags: > > - qubes-hw2 > > -- > > git-series 0.9.1 > >
On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote: > On Thu, Feb 13, 2025 at 05:36:47PM -0800, Stefano Stabellini wrote: > > On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote: > > > Debugging sometimes involves running specific jobs on different > > > versions. It's useful to easily avoid running all of the not interesting > > > ones (for given case) to save both time and CI resources. Doing so used > > > to require changing the yaml files, usually in several places. > > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. > > > Note that one needs to satisfy job dependencies on their own (for > > > example if a test job needs a build job, that specific build job > > > needs to be included too). > > > > > > The variable can be specified via Gitlab web UI when scheduling a > > > pipeline, but it can be also set when doing git push directly: > > > > > > git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" > > > > > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html > > > > > > The variable needs to include regex for selecting jobs, including > > > enclosing slashes. > > > A coma/space separated list of jobs to select would be friendlier UX, > > > but unfortunately that is not supported: > > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed > > > workaround doesn't work for job-level CI_JOB_NAME). > > > On the other hand, the regex is more flexible (one can select for > > > example all arm32 jobs). > > > > I was trying to find workarounds so that we could also support the > > simple list of comma-separated jobs you mentioned, but I couldn't find > > an easy way to do that. > > > > However, one thing we can do is to support writing SELECTED_JOBS_ONLY in > > .gitlab-ci.yml as a commit in xen.git, for people that don't know or > > don't remember how to set ci.variable using the git command line. > > You can always do it, in `variables` setting AFAIR. > > > Given that this is for testing, I think it would be no problem to adding > > a special commit on top of your tree. We are just trying to make it > > easier compared to having to manually delete the list of jobs we don't > > need. > > Yes, manually delete was awful. In practice I usually added always-false > rules, but still. > > > But even with the special commit, I couldn't think of an easy way to > > achieve the nicer comma-separated list of jobs... > > > > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > > --- > > > This probably wants documenting beyond this commit message. I don't > > > think we have any CI-related docs anywhere, do we? Some new file in > > > docs/misc? > > > > Yes please :-) > > > > It would be also worth documenting how to create a simple pipeline by > > removing everything that you don't need for a single test > > You mean how to find what jobs you need? > > > > And also, it's possible to extend web ui for starting pipelines to > > > include pre-defined variables. I use it in qubes here if you want to > > > see: > > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new > > > > I don't have access to this > > Oh, sorry. Screenshot attached. > > And its definition looks like this: > https://gitlab.com/QubesOS/qubes-continuous-integration/-/blob/main/.gitlab-ci.yml?ref_type=heads#L15-26 Actually this gave me an idea. What if we flip the default? Normally we want to run all the jobs. When doing development, we typically want to run one specific job attached to the work we are currently doing. This patch introduces a blacklist, but it looks like we want a whitelist instead? Wouldn't it be easier to say: "delete everything except for adl-smoke-x86-64-dom0pvh-hvm-gcc-debug" Then we can arrange it so adl-smoke-x86-64-dom0pvh-hvm-gcc-debug and its dependencies are left enabled. Everything else is disabled?
On Fri, 14 Feb 2025, Stefano Stabellini wrote: > On Fri, 14 Feb 2025, Marek Marczykowski-Górecki wrote: > > On Thu, Feb 13, 2025 at 05:36:47PM -0800, Stefano Stabellini wrote: > > > On Thu, 13 Feb 2025, Marek Marczykowski-Górecki wrote: > > > > Debugging sometimes involves running specific jobs on different > > > > versions. It's useful to easily avoid running all of the not interesting > > > > ones (for given case) to save both time and CI resources. Doing so used > > > > to require changing the yaml files, usually in several places. > > > > Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. > > > > Note that one needs to satisfy job dependencies on their own (for > > > > example if a test job needs a build job, that specific build job > > > > needs to be included too). > > > > > > > > The variable can be specified via Gitlab web UI when scheduling a > > > > pipeline, but it can be also set when doing git push directly: > > > > > > > > git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" > > > > > > > > More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html > > > > > > > > The variable needs to include regex for selecting jobs, including > > > > enclosing slashes. > > > > A coma/space separated list of jobs to select would be friendlier UX, > > > > but unfortunately that is not supported: > > > > https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed > > > > workaround doesn't work for job-level CI_JOB_NAME). > > > > On the other hand, the regex is more flexible (one can select for > > > > example all arm32 jobs). > > > > > > I was trying to find workarounds so that we could also support the > > > simple list of comma-separated jobs you mentioned, but I couldn't find > > > an easy way to do that. > > > > > > However, one thing we can do is to support writing SELECTED_JOBS_ONLY in > > > .gitlab-ci.yml as a commit in xen.git, for people that don't know or > > > don't remember how to set ci.variable using the git command line. > > > > You can always do it, in `variables` setting AFAIR. > > > > > Given that this is for testing, I think it would be no problem to adding > > > a special commit on top of your tree. We are just trying to make it > > > easier compared to having to manually delete the list of jobs we don't > > > need. > > > > Yes, manually delete was awful. In practice I usually added always-false > > rules, but still. > > > > > But even with the special commit, I couldn't think of an easy way to > > > achieve the nicer comma-separated list of jobs... > > > > > > > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > > > --- > > > > This probably wants documenting beyond this commit message. I don't > > > > think we have any CI-related docs anywhere, do we? Some new file in > > > > docs/misc? > > > > > > Yes please :-) > > > > > > It would be also worth documenting how to create a simple pipeline by > > > removing everything that you don't need for a single test > > > > You mean how to find what jobs you need? > > > > > > And also, it's possible to extend web ui for starting pipelines to > > > > include pre-defined variables. I use it in qubes here if you want to > > > > see: > > > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new > > > > > > I don't have access to this > > > > Oh, sorry. Screenshot attached. > > > > And its definition looks like this: > > https://gitlab.com/QubesOS/qubes-continuous-integration/-/blob/main/.gitlab-ci.yml?ref_type=heads#L15-26 > > Actually this gave me an idea. What if we flip the default? Normally we > want to run all the jobs. > > When doing development, we typically want to run one specific job > attached to the work we are currently doing. This patch introduces a > blacklist, but it looks like we want a whitelist instead? > > Wouldn't it be easier to say: "delete everything except for > adl-smoke-x86-64-dom0pvh-hvm-gcc-debug" > > Then we can arrange it so adl-smoke-x86-64-dom0pvh-hvm-gcc-debug and its > dependencies are left enabled. Everything else is disabled? Sorry I sent it too fast without thinking. This patch is already implementing the whitelist. The only thing we are really missing is the automatic dependency enablement, but I don't know how that could be implemented.
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index 35e224366f62..f12de00a164a 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -12,6 +12,12 @@ - '*/*.log' when: always needs: [] + rules: + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY + when: always + - if: $SELECTED_JOBS_ONLY + when: never + - when: on_success .gcc-tmpl: variables: &gcc diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index c21a37933881..93632f1f9204 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -1,6 +1,11 @@ .test-jobs-common: stage: test image: ${XEN_REGISTRY}/${CONTAINER} + rules: + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY + - if: $SELECTED_JOBS_ONLY + when: never + - when: on_success .arm64-test-needs: &arm64-test-needs - alpine-3.18-arm64-rootfs-export @@ -99,6 +104,9 @@ - '*.dtb' when: always rules: + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY + - if: $SELECTED_JOBS_ONLY + when: never - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -117,6 +125,9 @@ - '*.log' when: always rules: + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY + - if: $SELECTED_JOBS_ONLY + when: never - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -137,6 +148,9 @@ - '*.log' when: always rules: + - if: $SELECTED_JOBS_ONLY && $CI_JOB_NAME =~ $SELECTED_JOBS_ONLY + - if: $SELECTED_JOBS_ONLY + when: never - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - qubes-hw2
Debugging sometimes involves running specific jobs on different versions. It's useful to easily avoid running all of the not interesting ones (for given case) to save both time and CI resources. Doing so used to require changing the yaml files, usually in several places. Ease this step by adding SELECTED_JOBS_ONLY variable that takes a regex. Note that one needs to satisfy job dependencies on their own (for example if a test job needs a build job, that specific build job needs to be included too). The variable can be specified via Gitlab web UI when scheduling a pipeline, but it can be also set when doing git push directly: git push -o ci.variable=SELECTED_JOBS_ONLY="/job1|job2/" More details at https://docs.gitlab.co.jp/ee/user/project/push_options.html The variable needs to include regex for selecting jobs, including enclosing slashes. A coma/space separated list of jobs to select would be friendlier UX, but unfortunately that is not supported: https://gitlab.com/gitlab-org/gitlab/-/issues/209904 (note the proposed workaround doesn't work for job-level CI_JOB_NAME). On the other hand, the regex is more flexible (one can select for example all arm32 jobs). Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> --- This probably wants documenting beyond this commit message. I don't think we have any CI-related docs anywhere, do we? Some new file in docs/misc? And also, it's possible to extend web ui for starting pipelines to include pre-defined variables. I use it in qubes here if you want to see: https://gitlab.com/QubesOS/qubes-continuous-integration/-/pipelines/new Does it make sense to include SELECTED_JOBS_ONLY this way too? Personally, I'll probably use it via cmdline push only anyway, but I don't know what workflows other people have. --- automation/gitlab-ci/build.yaml | 6 ++++++ automation/gitlab-ci/test.yaml | 14 ++++++++++++++ 2 files changed, 20 insertions(+)