| Message ID | 20240228225527.1052240-1-helen.koike@collabora.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing | expand |
Hi Helen, I appreciate the amount of work you've put into this, to improve testing of the kernel as a whole. I'll need more time to answer, but please see below for a quick comment already. On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html . > > > Motivation for this Work > ======================== > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the ci/ > folder. > > > Future Work > =========== > > **Expanding Static Checks:** > We have the opportunity to integrate a variety of static analysis tools, > including: > - dtbs_checks > - sparse > - yamllint > - dt-doc-validate > - coccicheck > > **Adding Userspace Tests on VMs:** > To further our testing, we can implement userspace tests that run on virtual > machines (VMs), such as: > - kselftests > - kunit tests > - Subsystem-specific tests, customizable in the scenarios. > > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB > > **Lightweight Implementation for All Developers:** > We aim to design these tests to be lightweight, ensuring developers with limited > computing resources can still run essential tests. Resource-intensive tests can > be set to trigger manually, rather than automatically, to accommodate diverse > development environments. > > > Chat Discussions > ================ > > For those interested in further discussions: > > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > and KernelCI community for their valuable feedback and support in this proposal. > > > I eagerly await your thoughts and suggestions on this initiative. > > Also, if you want to see this initiave move faster, we are happy to discuss > funding options. > > Best regards, > Helen Koike > > Helen Koike (3): > kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing > kci-gitlab: Add documentation > kci-gitlab: docs: Add images > > .gitlab-ci.yml | 2 + > Documentation/ci/gitlab-ci/gitlab-ci.rst | 404 ++++++++++++++++++ > .../ci/gitlab-ci/images/job-matrix.png | Bin 0 -> 159752 bytes > .../gitlab-ci/images/new-project-runner.png | Bin 0 -> 607737 bytes > .../ci/gitlab-ci/images/pipelines-on-push.png | Bin 0 -> 532143 bytes > .../ci/gitlab-ci/images/the-pipeline.png | Bin 0 -> 91675 bytes > .../ci/gitlab-ci/images/variables.png | Bin 0 -> 277518 bytes > Documentation/index.rst | 7 + > MAINTAINERS | 9 + > ci/gitlab-ci/bootstrap-gitlab-runner.sh | 55 +++ > ci/gitlab-ci/ci-scripts/build-docs.sh | 35 ++ > ci/gitlab-ci/ci-scripts/build-kernel.sh | 35 ++ > ci/gitlab-ci/ci-scripts/ici-functions.sh | 104 +++++ > ci/gitlab-ci/ci-scripts/install-smatch.sh | 13 + > .../ci-scripts/parse_commit_message.sh | 27 ++ > ci/gitlab-ci/ci-scripts/run-checkpatch.sh | 19 + > ci/gitlab-ci/ci-scripts/run-smatch.sh | 45 ++ > ci/gitlab-ci/docker-compose.yaml | 18 + > ci/gitlab-ci/linux.code-workspace | 11 + > ci/gitlab-ci/yml/build.yml | 43 ++ > ci/gitlab-ci/yml/cache.yml | 26 ++ > ci/gitlab-ci/yml/container.yml | 36 ++ > ci/gitlab-ci/yml/gitlab-ci.yml | 71 +++ > ci/gitlab-ci/yml/kernel-combinations.yml | 18 + > ci/gitlab-ci/yml/scenarios.yml | 12 + > ci/gitlab-ci/yml/scenarios/file-systems.yml | 21 + > ci/gitlab-ci/yml/scenarios/media.yml | 21 + > ci/gitlab-ci/yml/scenarios/network.yml | 21 + > ci/gitlab-ci/yml/static-checks.yml | 21 + > 29 files changed, 1074 insertions(+) > create mode 100644 .gitlab-ci.yml > create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst > create mode 100644 Documentation/ci/gitlab-ci/images/job-matrix.png > create mode 100644 Documentation/ci/gitlab-ci/images/new-project-runner.png > create mode 100644 Documentation/ci/gitlab-ci/images/pipelines-on-push.png > create mode 100644 Documentation/ci/gitlab-ci/images/the-pipeline.png > create mode 100644 Documentation/ci/gitlab-ci/images/variables.png > create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh > create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh > create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh > create mode 100644 ci/gitlab-ci/docker-compose.yaml > create mode 100644 ci/gitlab-ci/linux.code-workspace > create mode 100644 ci/gitlab-ci/yml/build.yml > create mode 100644 ci/gitlab-ci/yml/cache.yml > create mode 100644 ci/gitlab-ci/yml/container.yml > create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml > create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml > create mode 100644 ci/gitlab-ci/yml/scenarios.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml > create mode 100644 ci/gitlab-ci/yml/static-checks.yml
On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > Chat Discussions > > ================ > > > > For those interested in further discussions: > > > > **Join Our Slack Channel:** > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > Feel free to join and contribute to the conversation. The KernelCI team has > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. Yeah, and that list looks super restrictive and arbitrary. Like, microsoft is there but kernel.org isn't? I'm sure there's a reason, but we should at the very least open it to everyone. Maxime
On 2/29/24 01:07, Laurent Pinchart wrote: > On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: >> **Join Our Slack Channel:** >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >> Feel free to join and contribute to the conversation. The KernelCI team has >> weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. Yes, it's not ideal that we use closed-source software for communication, but FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with a Google account, I'd be able to see and approve your attempt too. Otherwise, our video meetings are generally open to everyone and run in Jitsi. Nick
On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: > On 2/29/24 01:07, Laurent Pinchart wrote: > > On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > >> **Join Our Slack Channel:** > >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > >> Feel free to join and contribute to the conversation. The KernelCI team has > >> weekly calls where we also discuss the GitLab-CI pipeline. > > > > Could we communicate using free software please ? Furthermore, it's not > > possible to create an account on that slack instance unless you have an > > e-mail address affiliated with a small number of companies > > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > > for me. > > Yes, it's not ideal that we use closed-source software for communication, but > FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with > a Google account, I'd be able to see and approve your attempt too. I don't use Google accounts to authenticate to third-party services, sorry. And in any case, that won't make slack free software. > Otherwise, our video meetings are generally open to everyone and run in Jitsi.
Hi Laurent, Helen, On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > **Join Our Slack Channel:** > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > Feel free to join and contribute to the conversation. The KernelCI team has > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. I agree. There is no shortage of good alternatives either: we have IRC, Matrix and XMPP for instance. Just pick one of them.
On Thu, Feb 29, 2024 at 09:39:01AM +0000, Sakari Ailus wrote: > On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > > Feel free to join and contribute to the conversation. The KernelCI team has > > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > > possible to create an account on that slack instance unless you have an > > e-mail address affiliated with a small number of companies > > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > > for me. > I agree. There is no shortage of good alternatives either: we have IRC, > Matrix and XMPP for instance. Just pick one of them. And indeed KernelCI does actually already have #kernelci on libera.chat.
On 2/29/24 11:34 AM, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: >> On 2/29/24 01:07, Laurent Pinchart wrote: >>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: >>>> **Join Our Slack Channel:** >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >>>> Feel free to join and contribute to the conversation. The KernelCI team has >>>> weekly calls where we also discuss the GitLab-CI pipeline. >>> >>> Could we communicate using free software please ? Furthermore, it's not >>> possible to create an account on that slack instance unless you have an >>> e-mail address affiliated with a small number of companies >>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go >>> for me. >> >> Yes, it's not ideal that we use closed-source software for communication, but >> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with >> a Google account, I'd be able to see and approve your attempt too. > > I don't use Google accounts to authenticate to third-party services, > sorry. And in any case, that won't make slack free software. Of course. You're also welcome to join the #kernelci channel on libera.chat. It's much quieter there, though. Nick
On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > On 2/29/24 11:34 AM, Laurent Pinchart wrote: > > On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: > >> On 2/29/24 01:07, Laurent Pinchart wrote: > >>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > >>>> **Join Our Slack Channel:** > >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > >>>> Feel free to join and contribute to the conversation. The KernelCI team has > >>>> weekly calls where we also discuss the GitLab-CI pipeline. > >>> > >>> Could we communicate using free software please ? Furthermore, it's not > >>> possible to create an account on that slack instance unless you have an > >>> e-mail address affiliated with a small number of companies > >>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > >>> for me. > >> > >> Yes, it's not ideal that we use closed-source software for communication, but > >> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with > >> a Google account, I'd be able to see and approve your attempt too. > > > > I don't use Google accounts to authenticate to third-party services, > > sorry. And in any case, that won't make slack free software. > > Of course. You're also welcome to join the #kernelci channel on libera.chat. Isn't that a bit pointless if it's no the main IM channel ? > It's much quieter there, though.
On 2/29/24 1:19 PM, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: >> On 2/29/24 11:34 AM, Laurent Pinchart wrote: >>> On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: >>>> On 2/29/24 01:07, Laurent Pinchart wrote: >>>>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: >>>>>> **Join Our Slack Channel:** >>>>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >>>>>> Feel free to join and contribute to the conversation. The KernelCI team has >>>>>> weekly calls where we also discuss the GitLab-CI pipeline. >>>>> >>>>> Could we communicate using free software please ? Furthermore, it's not >>>>> possible to create an account on that slack instance unless you have an >>>>> e-mail address affiliated with a small number of companies >>>>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go >>>>> for me. >>>> >>>> Yes, it's not ideal that we use closed-source software for communication, but >>>> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with >>>> a Google account, I'd be able to see and approve your attempt too. >>> >>> I don't use Google accounts to authenticate to third-party services, >>> sorry. And in any case, that won't make slack free software. >> >> Of course. You're also welcome to join the #kernelci channel on libera.chat. > > Isn't that a bit pointless if it's no the main IM channel ? Yes, it's not ideal, but if more people come there, more discussions will happen there too. Nick
On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > Of course. You're also welcome to join the #kernelci channel on libera.chat. > Isn't that a bit pointless if it's no the main IM channel ? It *was* the original channel and still gets some usage (mostly started by me admittedly since I've never joined slack for a bunch of reasons that make it hassle), IIRC the Slack was started because there were some interns who had trouble figuring out IRC and intermittent connectivity but people seem to have migrated.
On 29/02/2024 12:41, Mark Brown wrote: > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > >>> Of course. You're also welcome to join the #kernelci channel on libera.chat. > >> Isn't that a bit pointless if it's no the main IM channel ? > > It *was* the original channel and still gets some usage (mostly started > by me admittedly since I've never joined slack for a bunch of reasons > that make it hassle), IIRC the Slack was started because there were some > interns who had trouble figuring out IRC and intermittent connectivity > but people seem to have migrated. In fact it was initially created for the members of the Linux Foundation project only, which is why registration is moderated for emails that don't have a domain linked to a member (BTW not any Google account will just work e.g. @gmail.com is moderated, only @google.com for Google employees isn't). And yes IRC is the "least common denominator" chat platform. Maybe having a bridge between the main Slack channel and IRC would help. Guillaume
On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: > On 29/02/2024 12:41, Mark Brown wrote: > > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > > >>> Of course. You're also welcome to join the #kernelci channel on libera.chat. > > > >> Isn't that a bit pointless if it's no the main IM channel ? > > > > It *was* the original channel and still gets some usage (mostly started > > by me admittedly since I've never joined slack for a bunch of reasons > > that make it hassle), IIRC the Slack was started because there were some > > interns who had trouble figuring out IRC and intermittent connectivity > > but people seem to have migrated. > > In fact it was initially created for the members of the Linux > Foundation project only, which is why registration is moderated > for emails that don't have a domain linked to a member (BTW not > any Google account will just work e.g. @gmail.com is moderated, > only @google.com for Google employees isn't). > > And yes IRC is the "least common denominator" chat platform. > Maybe having a bridge between the main Slack channel and IRC > would help. If the gitlab CI pipeline proposal wants to be considered for inclusion in the kernel, I think it needs to switch to a free software solution for its *main* communication channels.
Hello, On 28/02/2024 23:55, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. This sounds like a nice starting point to me as an additional way to run tests upstream. I have one particular question as I see a pattern through the rest of the email, please see below. [...] > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. [...] > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB [...] > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. > > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > and KernelCI community for their valuable feedback and support in this proposal. Where does this fit on the KernelCI roadmap? I see it mentioned a few times but it's not entirely clear whether this initiative is an independent one or in some way linked to KernelCI. Say, are you planning to use the kci tool, new API, compiler toolchains, user-space and Docker images etc? Or, are KernelCI plans evolving to follow this move? Thanks, Guillaume
On Thu, Feb 29, 2024 at 02:20:41PM +0200, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: > > On 29/02/2024 12:41, Mark Brown wrote: > > > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > > >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > > > > >>> Of course. You're also welcome to join the #kernelci channel on libera.chat. > > > > > >> Isn't that a bit pointless if it's no the main IM channel ? > > > > > > It *was* the original channel and still gets some usage (mostly started > > > by me admittedly since I've never joined slack for a bunch of reasons > > > that make it hassle), IIRC the Slack was started because there were some > > > interns who had trouble figuring out IRC and intermittent connectivity > > > but people seem to have migrated. > > > > In fact it was initially created for the members of the Linux > > Foundation project only, which is why registration is moderated > > for emails that don't have a domain linked to a member (BTW not > > any Google account will just work e.g. @gmail.com is moderated, > > only @google.com for Google employees isn't). > > > > And yes IRC is the "least common denominator" chat platform. > > Maybe having a bridge between the main Slack channel and IRC > > would help. > > If the gitlab CI pipeline proposal wants to be considered for inclusion > in the kernel, I think it needs to switch to a free software solution > for its *main* communication channels. And to clarify, I didn't meant the kernel CI project, but only the gitlab CI pipeline for the Linux kernel project. I don't know how tightly integrated the two projects are though.
On 2/29/24 2:25 PM, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 02:20:41PM +0200, Laurent Pinchart wrote: >> On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: >>> On 29/02/2024 12:41, Mark Brown wrote: >>>> On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: >>>>> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: >>>> >>>>>> Of course. You're also welcome to join the #kernelci channel on libera.chat. >>>> >>>>> Isn't that a bit pointless if it's no the main IM channel ? >>>> >>>> It *was* the original channel and still gets some usage (mostly started >>>> by me admittedly since I've never joined slack for a bunch of reasons >>>> that make it hassle), IIRC the Slack was started because there were some >>>> interns who had trouble figuring out IRC and intermittent connectivity >>>> but people seem to have migrated. >>> >>> In fact it was initially created for the members of the Linux >>> Foundation project only, which is why registration is moderated >>> for emails that don't have a domain linked to a member (BTW not >>> any Google account will just work e.g. @gmail.com is moderated, >>> only @google.com for Google employees isn't). >>> >>> And yes IRC is the "least common denominator" chat platform. >>> Maybe having a bridge between the main Slack channel and IRC >>> would help. >> >> If the gitlab CI pipeline proposal wants to be considered for inclusion >> in the kernel, I think it needs to switch to a free software solution >> for its *main* communication channels. > > And to clarify, I didn't meant the kernel CI project, but only the > gitlab CI pipeline for the Linux kernel project. I don't know how > tightly integrated the two projects are though. They're not tightly integrated so far. However, we're exploring the idea of letting GitLab CI submit jobs to KernelCI and get results as a part of the pipeline. Helen already joined the #kernelci channel, and we will maintain a presence there for the GitLab CI project. Nick
On 2/29/24 2:20 PM, Guillaume Tucker wrote: > Hello, > > On 28/02/2024 23:55, Helen Koike wrote: >> Dear Kernel Community, >> >> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a >> basic test pipeline triggered by code pushes to a GitLab-CI instance. This >> initial version includes static checks (checkpatch and smatch for now) and build >> tests across various architectures and configurations. It leverages an >> integrated cache for efficient build times and introduces a flexible 'scenarios' >> mechanism for subsystem-specific extensions. > > This sounds like a nice starting point to me as an additional way > to run tests upstream. I have one particular question as I see a > pattern through the rest of the email, please see below. > > [...] > >> 4. **Collaborative Testing Environment:** The kernel community is already >> engaged in numerous testing efforts, including various GitLab-CI pipelines such >> as DRM-CI, which I maintain, along with other solutions like KernelCI and >> BPF-CI. This proposal is designed to further stimulate contributions to the >> evolving testing landscape. Our goal is to establish a comprehensive suite of >> common tools and files. > > [...] > >> **Leveraging External Test Labs:** >> We can extend our testing to external labs, similar to what DRM-CI currently >> does. This includes: >> - Lava labs >> - Bare metal labs >> - Using KernelCI-provided labs >> >> **Other integrations** >> - Submit results to KCIDB > > [...] > >> **Join Our Slack Channel:** >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >> Feel free to join and contribute to the conversation. The KernelCI team has >> weekly calls where we also discuss the GitLab-CI pipeline. >> >> **Acknowledgments:** >> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - >> and KernelCI community for their valuable feedback and support in this proposal. > > Where does this fit on the KernelCI roadmap? > > I see it mentioned a few times but it's not entirely clear > whether this initiative is an independent one or in some way > linked to KernelCI. Say, are you planning to use the kci tool, > new API, compiler toolchains, user-space and Docker images etc? > Or, are KernelCI plans evolving to follow this move? I would say this is an important part of KernelCI the project, considering its aim to improve testing and CI in the kernel. It's not a part of KernelCI the service as it is right now, although I would say it would be good to have ability to submit KernelCI jobs from GitLab CI and pull results in the same pipeline, as we discussed earlier. Nick
Hi, Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : > On 2/29/24 2:20 PM, Guillaume Tucker wrote: > > Hello, > > > > On 28/02/2024 23:55, Helen Koike wrote: > > > Dear Kernel Community, > > > > > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > > > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > > > initial version includes static checks (checkpatch and smatch for now) and build > > > tests across various architectures and configurations. It leverages an > > > integrated cache for efficient build times and introduces a flexible 'scenarios' > > > mechanism for subsystem-specific extensions. > > > > This sounds like a nice starting point to me as an additional way > > to run tests upstream. I have one particular question as I see a > > pattern through the rest of the email, please see below. > > > > [...] > > > > > 4. **Collaborative Testing Environment:** The kernel community is already > > > engaged in numerous testing efforts, including various GitLab-CI pipelines such > > > as DRM-CI, which I maintain, along with other solutions like KernelCI and > > > BPF-CI. This proposal is designed to further stimulate contributions to the > > > evolving testing landscape. Our goal is to establish a comprehensive suite of > > > common tools and files. > > > > [...] > > > > > **Leveraging External Test Labs:** > > > We can extend our testing to external labs, similar to what DRM-CI currently > > > does. This includes: > > > - Lava labs > > > - Bare metal labs > > > - Using KernelCI-provided labs > > > > > > **Other integrations** > > > - Submit results to KCIDB > > > > [...] > > > > > **Join Our Slack Channel:** > > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > > Feel free to join and contribute to the conversation. The KernelCI team has > > > weekly calls where we also discuss the GitLab-CI pipeline. > > > > > > **Acknowledgments:** > > > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > > > and KernelCI community for their valuable feedback and support in this proposal. > > > > Where does this fit on the KernelCI roadmap? > > > > I see it mentioned a few times but it's not entirely clear > > whether this initiative is an independent one or in some way > > linked to KernelCI. Say, are you planning to use the kci tool, > > new API, compiler toolchains, user-space and Docker images etc? > > Or, are KernelCI plans evolving to follow this move? > > I would say this is an important part of KernelCI the project, considering its > aim to improve testing and CI in the kernel. It's not a part of KernelCI the > service as it is right now, although I would say it would be good to have > ability to submit KernelCI jobs from GitLab CI and pull results in the same > pipeline, as we discussed earlier. I'd like to add that both CI have a different purpose in the Linux project. This CI work is a pre-merge verification. Everyone needs to run checkpatch and smatch, this is automating it (and will catch those that forgot or ran it incorrectly). But it can go further by effectively testing specific patches on real hardware (with pretty narrow filters). It will help catch submission issues earlier, and reduce kernelCI regression rate. As a side effect, kernelCI infra will endup catching the "integration" issues, which are the issue as a result of simultenous changes in different trees. They are also often more complex and benefit from the bisection capabilities. kernelCI tests are also a lot more intensive, they usually covers everything, but they bundle multiple changes per run. The pre-merge tests will be reduced to what seems meaningful for the changes. Its important to understand that pre- merge CI have a time cost, and we need to make sure CI time does not exceed the merge window period. Nicolas
On 29/02/2024 17:28, Nicolas Dufresne wrote: > Hi, > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: >>> Hello, >>> >>> On 28/02/2024 23:55, Helen Koike wrote: >>>> Dear Kernel Community, >>>> >>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a >>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This >>>> initial version includes static checks (checkpatch and smatch for now) and build >>>> tests across various architectures and configurations. It leverages an >>>> integrated cache for efficient build times and introduces a flexible 'scenarios' >>>> mechanism for subsystem-specific extensions. >>> >>> This sounds like a nice starting point to me as an additional way >>> to run tests upstream. I have one particular question as I see a >>> pattern through the rest of the email, please see below. >>> >>> [...] >>> >>>> 4. **Collaborative Testing Environment:** The kernel community is already >>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such >>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and >>>> BPF-CI. This proposal is designed to further stimulate contributions to the >>>> evolving testing landscape. Our goal is to establish a comprehensive suite of >>>> common tools and files. >>> >>> [...] >>> >>>> **Leveraging External Test Labs:** >>>> We can extend our testing to external labs, similar to what DRM-CI currently >>>> does. This includes: >>>> - Lava labs >>>> - Bare metal labs >>>> - Using KernelCI-provided labs >>>> >>>> **Other integrations** >>>> - Submit results to KCIDB >>> >>> [...] >>> >>>> **Join Our Slack Channel:** >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >>>> Feel free to join and contribute to the conversation. The KernelCI team has >>>> weekly calls where we also discuss the GitLab-CI pipeline. >>>> >>>> **Acknowledgments:** >>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - >>>> and KernelCI community for their valuable feedback and support in this proposal. >>> >>> Where does this fit on the KernelCI roadmap? >>> >>> I see it mentioned a few times but it's not entirely clear >>> whether this initiative is an independent one or in some way >>> linked to KernelCI. Say, are you planning to use the kci tool, >>> new API, compiler toolchains, user-space and Docker images etc? >>> Or, are KernelCI plans evolving to follow this move? >> >> I would say this is an important part of KernelCI the project, considering its >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the >> service as it is right now, although I would say it would be good to have >> ability to submit KernelCI jobs from GitLab CI and pull results in the same >> pipeline, as we discussed earlier. Right, I think this needs a bit of disambiguation. The legacy KernelCI system from the Linaro days several years ago is really a service on its own like the many other CIs out there. However, the new KernelCI API and related tooling (kci command line, new web dashboard, modular runtime design etc.) is not that. It's about addressing all the community requirements and that includes being able to run a same test manually in a shell, or in a VM, or automatically from GitLab CI or using a main generic pipeline hosted by KernelCI itself. With this approach, there's no distinction between "the project" and "the service", and as we discussed before there shouldn't even be a distinction with KCIDB. Just KernelCI. However I don't really see this happening, unless I'm missing a part of the story or some upcoming announcement with an updated roadmap. For some reason the old and established paradigm seems unshakeable. The new KernelCI implementation is starting to look just like a refresh of the old one with newer components - which is a huge missed opportunity to really change things IMHO. This may sound like a bit of a tangent, facilitating GitLab CI for the upstream kernel is of course significant progress in any case - no question about that. My comment is more about why it's being driven hand-in-hand with KernelCI in what seems like a diverging direction from KernelCI's announced plans. Why push for a GitLab-centered orchestration when there's a more universal solution being proposed by the project? I would find it easier to understand - and I sense I'm not the only one here reading the thread - if KernelCI wasn't mentioned that many times in the cover letter and if the scripts didn't have KCI_* in so many places, basically if this was clearly an independent initiative such as KUnit, 0-day or regzbot. > I'd like to add that both CI have a different purpose in the Linux project. This > CI work is a pre-merge verification. Everyone needs to run checkpatch and > smatch, this is automating it (and will catch those that forgot or ran it > incorrectly). But it can go further by effectively testing specific patches on > real hardware (with pretty narrow filters). It will help catch submission issues > earlier, and reduce kernelCI regression rate. As a side effect, kernelCI infra > will endup catching the "integration" issues, which are the issue as a result of > simultenous changes in different trees. They are also often more complex and > benefit from the bisection capabilities. > > kernelCI tests are also a lot more intensive, they usually covers everything, > but they bundle multiple changes per run. The pre-merge tests will be reduced to > what seems meaningful for the changes. Its important to understand that pre- > merge CI have a time cost, and we need to make sure CI time does not exceed the > merge window period. You're referring to the legacy KernelCI, to illustrate the point I made earlier. The plan with the new implementation was to be able to do pre-merge testing as well as many other things, basically to provide a platform able to cope with the diversity of workflows across the kernel subsystems and the complexity of the "system under test" itself. Well, let's see how this goes and it does look quite promising. Evolution is always a chaotic process, especially in a complex project like this. I'm not expecting to get all the answers to the questions I have but it seemed important to raise this point and seek a bit more clarity around KernelCI. Guillaume
On Friday, March 01, 2024 18:56 -03, Guillaume Tucker <gtucker@gtucker.io> wrote: > On 29/02/2024 17:28, Nicolas Dufresne wrote: > > Hi, > > > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : > >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: > >>> Hello, > >>> > >>> On 28/02/2024 23:55, Helen Koike wrote: > >>>> Dear Kernel Community, > >>>> > >>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > >>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This > >>>> initial version includes static checks (checkpatch and smatch for now) and build > >>>> tests across various architectures and configurations. It leverages an > >>>> integrated cache for efficient build times and introduces a flexible 'scenarios' > >>>> mechanism for subsystem-specific extensions. > >>> > >>> This sounds like a nice starting point to me as an additional way > >>> to run tests upstream. I have one particular question as I see a > >>> pattern through the rest of the email, please see below. > >>> > >>> [...] > >>> > >>>> 4. **Collaborative Testing Environment:** The kernel community is already > >>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such > >>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and > >>>> BPF-CI. This proposal is designed to further stimulate contributions to the > >>>> evolving testing landscape. Our goal is to establish a comprehensive suite of > >>>> common tools and files. > >>> > >>> [...] > >>> > >>>> **Leveraging External Test Labs:** > >>>> We can extend our testing to external labs, similar to what DRM-CI currently > >>>> does. This includes: > >>>> - Lava labs > >>>> - Bare metal labs > >>>> - Using KernelCI-provided labs > >>>> > >>>> **Other integrations** > >>>> - Submit results to KCIDB > >>> > >>> [...] > >>> > >>>> **Join Our Slack Channel:** > >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > >>>> Feel free to join and contribute to the conversation. The KernelCI team has > >>>> weekly calls where we also discuss the GitLab-CI pipeline. > >>>> > >>>> **Acknowledgments:** > >>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > >>>> and KernelCI community for their valuable feedback and support in this proposal. > >>> > >>> Where does this fit on the KernelCI roadmap? > >>> > >>> I see it mentioned a few times but it's not entirely clear > >>> whether this initiative is an independent one or in some way > >>> linked to KernelCI. Say, are you planning to use the kci tool, > >>> new API, compiler toolchains, user-space and Docker images etc? > >>> Or, are KernelCI plans evolving to follow this move? > >> > >> I would say this is an important part of KernelCI the project, considering its > >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the > >> service as it is right now, although I would say it would be good to have > >> ability to submit KernelCI jobs from GitLab CI and pull results in the same > >> pipeline, as we discussed earlier. > > Right, I think this needs a bit of disambiguation. The legacy > KernelCI system from the Linaro days several years ago is really > a service on its own like the many other CIs out there. However, > the new KernelCI API and related tooling (kci command line, new > web dashboard, modular runtime design etc.) is not that. It's > about addressing all the community requirements and that includes > being able to run a same test manually in a shell, or in a VM, or > automatically from GitLab CI or using a main generic pipeline > hosted by KernelCI itself. With this approach, there's no > distinction between "the project" and "the service", and as we > discussed before there shouldn't even be a distinction with > KCIDB. Just KernelCI. > > However I don't really see this happening, unless I'm missing a > part of the story or some upcoming announcement with an updated > roadmap. For some reason the old and established paradigm seems > unshakeable. The new KernelCI implementation is starting to look > just like a refresh of the old one with newer components - which > is a huge missed opportunity to really change things IMHO. Calling that a missed opportunity is a subjective perspective about the latest developments in KernelCI. The system implementation is one level less important than the actual kernel community engagement the project can generate. If one asks people around, the lack of community engagement with KernelCI is evident. However, after the recent leadership change in the project there is a growing effort to bring the kernel community closer to the KernelCI project with a renewed focus on high quality test results, clean regression reporting, among other things. Then, with an increased number of community members involved, we will have the necessary feedback (and funding!) to evolve the KernelCI infrastructure and technology to new levels. Otherwise, envisioning something that can solve *all* community testing needs would never be anything more than a fantasy in people's heads. - Gus
On 02/03/2024 10:48 pm, Gustavo Padovan wrote: > On Friday, March 01, 2024 18:56 -03, Guillaume Tucker <gtucker@gtucker.io> wrote: > >> On 29/02/2024 17:28, Nicolas Dufresne wrote: >>> Hi, >>> >>> Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : >>>> On 2/29/24 2:20 PM, Guillaume Tucker wrote: >>>>> Hello, >>>>> >>>>> On 28/02/2024 23:55, Helen Koike wrote: >>>>>> Dear Kernel Community, >>>>>> >>>>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a >>>>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This >>>>>> initial version includes static checks (checkpatch and smatch for now) and build >>>>>> tests across various architectures and configurations. It leverages an >>>>>> integrated cache for efficient build times and introduces a flexible 'scenarios' >>>>>> mechanism for subsystem-specific extensions. >>>>> >>>>> This sounds like a nice starting point to me as an additional way >>>>> to run tests upstream. I have one particular question as I see a >>>>> pattern through the rest of the email, please see below. >>>>> >>>>> [...] >>>>> >>>>>> 4. **Collaborative Testing Environment:** The kernel community is already >>>>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such >>>>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and >>>>>> BPF-CI. This proposal is designed to further stimulate contributions to the >>>>>> evolving testing landscape. Our goal is to establish a comprehensive suite of >>>>>> common tools and files. >>>>> >>>>> [...] >>>>> >>>>>> **Leveraging External Test Labs:** >>>>>> We can extend our testing to external labs, similar to what DRM-CI currently >>>>>> does. This includes: >>>>>> - Lava labs >>>>>> - Bare metal labs >>>>>> - Using KernelCI-provided labs >>>>>> >>>>>> **Other integrations** >>>>>> - Submit results to KCIDB >>>>> >>>>> [...] >>>>> >>>>>> **Join Our Slack Channel:** >>>>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >>>>>> Feel free to join and contribute to the conversation. The KernelCI team has >>>>>> weekly calls where we also discuss the GitLab-CI pipeline. >>>>>> >>>>>> **Acknowledgments:** >>>>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - >>>>>> and KernelCI community for their valuable feedback and support in this proposal. >>>>> >>>>> Where does this fit on the KernelCI roadmap? >>>>> >>>>> I see it mentioned a few times but it's not entirely clear >>>>> whether this initiative is an independent one or in some way >>>>> linked to KernelCI. Say, are you planning to use the kci tool, >>>>> new API, compiler toolchains, user-space and Docker images etc? >>>>> Or, are KernelCI plans evolving to follow this move? >>>> >>>> I would say this is an important part of KernelCI the project, considering its >>>> aim to improve testing and CI in the kernel. It's not a part of KernelCI the >>>> service as it is right now, although I would say it would be good to have >>>> ability to submit KernelCI jobs from GitLab CI and pull results in the same >>>> pipeline, as we discussed earlier. >> >> Right, I think this needs a bit of disambiguation. The legacy >> KernelCI system from the Linaro days several years ago is really >> a service on its own like the many other CIs out there. However, >> the new KernelCI API and related tooling (kci command line, new >> web dashboard, modular runtime design etc.) is not that. It's >> about addressing all the community requirements and that includes >> being able to run a same test manually in a shell, or in a VM, or >> automatically from GitLab CI or using a main generic pipeline >> hosted by KernelCI itself. With this approach, there's no >> distinction between "the project" and "the service", and as we >> discussed before there shouldn't even be a distinction with >> KCIDB. Just KernelCI. >> >> However I don't really see this happening, unless I'm missing a >> part of the story or some upcoming announcement with an updated >> roadmap. For some reason the old and established paradigm seems >> unshakeable. The new KernelCI implementation is starting to look >> just like a refresh of the old one with newer components - which >> is a huge missed opportunity to really change things IMHO. > > Calling that a missed opportunity is a subjective perspective about > the latest developments in KernelCI. The system implementation is > one level less important than the actual kernel community engagement > the project can generate. If one asks people around, the lack of > community engagement with KernelCI is evident. Well I would argue that community engagement and technical development work side-by-side, not as a hierarchy. You can't run Android phones or data centers with community engagement, and you can't write the code without the people. I was enquiring about this in particluar because I'm preparing a LF webinar, so I've started another thread to avoid spamming this one as it's really a side topic: https://lore.kernel.org/all/71f59a56-aef3-4bae-867b-769a0cdd1c1b@gtucker.io/T/#u > However, after the recent leadership change in the project there is a > growing effort to bring the kernel community closer to the KernelCI > project with a renewed focus on high quality test results, clean regression > reporting, among other things. Then, with an increased number of community > members involved, we will have the necessary feedback (and funding!) to > evolve the KernelCI infrastructure and technology to new levels. In a nutshell, KernelCI started small and then joined the LF. The scope changed to encompass the whole kernel community, and as a result a number of things were done: community survey, lots of conf talks and email discussions with kernel devs etc. Then some plan was put in place with the new API and web dashboard designs, but other priorities slowed things down on this front which is why it's still not quite there two years later. That's also why community engagement has been low. But that's OK, plans are just plans and things are catching up again now I think. And once again, I think this GitLab CI move is great :) Guillaume