| Message ID | 20240724095505.33544-1-berrange@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | gitlab: expose installed package info in build logs | expand |
Daniel P. Berrangé <berrange@redhat.com> writes: > Many times we see a build job start failing, we wonder if the installed > packages have changed since the last passing build. We can rarely > diagnose this, however, since we only have the new container image, not > the old one. > > The lcitool generated containers create a /packages.txt file whose > content is intended to be output in the build phase, so record the > packages associated with the build. > > This adds packages.txt to the manually written containers, and modifies > the build jobs to display this content. This will improve our future > debuggability of CI problems. Queued to maintainer/for-9.1, thanks.
On 24/7/24 11:55, Daniel P. Berrangé wrote: > Daniel P. Berrangé (2): > gitlab: record installed packages in /packages.txt in containers > gitlab: display /packages.txt in build jobs Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Hello Daniel, On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote: >Many times we see a build job start failing, we wonder if the installed >packages have changed since the last passing build. We can rarely >diagnose this, however, since we only have the new container image, not >the old one. > APT allows you to specify to pin package versions when installing; wouldn't that help ensure our tests are deterministic? Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few months and inform of any updates so that we can manually bump them. Manos >The lcitool generated containers create a /packages.txt file whose >content is intended to be output in the build phase, so record the >packages associated with the build. > >This adds packages.txt to the manually written containers, and modifies >the build jobs to display this content. This will improve our future >debuggability of CI problems. > >Daniel P. Berrangé (2): > gitlab: record installed packages in /packages.txt in containers > gitlab: display /packages.txt in build jobs > > .gitlab-ci.d/buildtest-template.yml | 1 + > .gitlab-ci.d/crossbuild-template.yml | 2 ++ > tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- > tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- > tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++- > tests/docker/dockerfiles/debian-loongarch-cross.docker | 3 ++- > tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- > tests/docker/dockerfiles/debian-xtensa-cross.docker | 3 ++- > 8 files changed, 15 insertions(+), 6 deletions(-) > >-- >2.45.2 > >
On 25/07/2024 11.42, Manos Pitsidianakis wrote: > Hello Daniel, > > On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote: >> Many times we see a build job start failing, we wonder if the installed >> packages have changed since the last passing build. We can rarely >> diagnose this, however, since we only have the new container image, not >> the old one. >> > > APT allows you to specify to pin package versions when installing; wouldn't > that help ensure our tests are deterministic? Even if tests would be more deterministic that way, I also don't think that's what we want: We would test with backlevel packages that users are not using (since they'll normally use the latest version of a package from a certain version of a distro). So in our CI, we would not be able to reproduce the issues that the users are seeing on their systems. Thomas
On Thu, Jul 25, 2024 at 12:42:18PM +0300, Manos Pitsidianakis wrote: > Hello Daniel, > > On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote: > > Many times we see a build job start failing, we wonder if the installed > > packages have changed since the last passing build. We can rarely > > diagnose this, however, since we only have the new container image, not > > the old one. > > > > APT allows you to specify to pin package versions when installing; wouldn't > that help ensure our tests are deterministic? We want to be testing against latest packages because that reflects what our users are likely to see on their own systems. IOW, having failures after newly updated packages is a good thing - provided we can easily identify what caused the breakage, to enable us to fix it promptly - by promptly I mean same-day. > Furthermore, a gitlab cron job pipeline can be set up to run every e.g. few > months and inform of any updates so that we can manually bump them. Only seeing breakage from new packages as long as a few months after it happens is a bad thing. We benefit from fast detection and fast fixing. In cases where the new package is broken, rather than QEMU, it is also better to be able to report that back to the distro within days of them rolling out the problem as it'll be fresh in their minds to fix. With regards, Daniel
Manos Pitsidianakis <manos.pitsidianakis@linaro.org> writes: > Hello Daniel, > > On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote: >>Many times we see a build job start failing, we wonder if the installed >>packages have changed since the last passing build. We can rarely >>diagnose this, however, since we only have the new container image, not >>the old one. >> > > APT allows you to specify to pin package versions when installing; > wouldn't that help ensure our tests are deterministic? Generally we want track the latest LTS version and usually the distros are pretty good about being cautious about updates. I think the recent case was because BSD's have a different policy about updating python. IOW this is a nice to have for debugging but I don't think we want to freeze the acceptable package set for any given install. > Furthermore, a gitlab cron job pipeline can be set up to run every > e.g. few months and inform of any updates so that we can manually bump > them. > > Manos > > >>The lcitool generated containers create a /packages.txt file whose >>content is intended to be output in the build phase, so record the >>packages associated with the build. >> >>This adds packages.txt to the manually written containers, and modifies >>the build jobs to display this content. This will improve our future >>debuggability of CI problems. >> >>Daniel P. Berrangé (2): >> gitlab: record installed packages in /packages.txt in containers >> gitlab: display /packages.txt in build jobs >> >> .gitlab-ci.d/buildtest-template.yml | 1 + >> .gitlab-ci.d/crossbuild-template.yml | 2 ++ >> tests/docker/dockerfiles/debian-all-test-cross.docker | 3 ++- >> tests/docker/dockerfiles/debian-hexagon-cross.docker | 3 ++- >> tests/docker/dockerfiles/debian-legacy-test-cross.docker | 3 ++- >> tests/docker/dockerfiles/debian-loongarch-cross.docker | 3 ++- >> tests/docker/dockerfiles/debian-tricore-cross.docker | 3 ++- >> tests/docker/dockerfiles/debian-xtensa-cross.docker | 3 ++- >> 8 files changed, 15 insertions(+), 6 deletions(-) >> >> -- 2.45.2 >> >>
On Thu, 25 Jul 2024 12:56, Thomas Huth <thuth@redhat.com> wrote: >On 25/07/2024 11.42, Manos Pitsidianakis wrote: >> Hello Daniel, >> >> On Wed, 24 Jul 2024 12:55, "Daniel P. Berrangé" <berrange@redhat.com> wrote: >>> Many times we see a build job start failing, we wonder if the installed >>> packages have changed since the last passing build. We can rarely >>> diagnose this, however, since we only have the new container image, not >>> the old one. >>> >> >> APT allows you to specify to pin package versions when installing; wouldn't >> that help ensure our tests are deterministic? > >Even if tests would be more deterministic that way, I also don't think >that's what we want: We would test with backlevel packages that users are >not using (since they'll normally use the latest version of a package from a >certain version of a distro). So in our CI, we would not be able to >reproduce the issues that the users are seeing on their systems. > Definitely, but the current testing makes it harder to discern the lines between QEMU regression and QEMU not working with latest updates. If we had e.g. a pinned job and a "latest" job disagreeing we'd know the cause. Of course that means extra CI minutes, maintainer upkeep etc etc, I know, but this was my reasoning, I don't dismiss testing for what users experience. :) Manos