| Message ID | 20241008155023.2571517-1-andrew.cooper3@citrix.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | CI: Stop building QEMU in general | expand |
On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index c668736bdc2f..c83e0bdbe119 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: > extends: .gcc-x86-64-build-debug > variables: > CONTAINER: alpine:3.18 > + QEMU: y Could you use a different name for the variable? This is exposed as an environment variable, it could easily be used in a build system already, like used to store a path to a QEMU to use. We don't really have a name space for CI variable, but maybe BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other usages. Otherwise patch looks fine. Cheers,
On 08/10/2024 5:27 pm, Anthony PERARD wrote: > On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml >> index c668736bdc2f..c83e0bdbe119 100644 >> --- a/automation/gitlab-ci/build.yaml >> +++ b/automation/gitlab-ci/build.yaml >> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: >> extends: .gcc-x86-64-build-debug >> variables: >> CONTAINER: alpine:3.18 >> + QEMU: y > Could you use a different name for the variable? This is exposed as an > environment variable, it could easily be used in a build system already, > like used to store a path to a QEMU to use. > > We don't really have a name space for CI variable, but maybe > BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other > usages. > > Otherwise patch looks fine. A can switch to BUILD_QEMU, but I don't plan to repost just for that. ~Andrew
On Tue, 8 Oct 2024, Andrew Cooper wrote: > On 08/10/2024 5:27 pm, Anthony PERARD wrote: > > On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: > >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > >> index c668736bdc2f..c83e0bdbe119 100644 > >> --- a/automation/gitlab-ci/build.yaml > >> +++ b/automation/gitlab-ci/build.yaml > >> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: > >> extends: .gcc-x86-64-build-debug > >> variables: > >> CONTAINER: alpine:3.18 > >> + QEMU: y > > Could you use a different name for the variable? This is exposed as an > > environment variable, it could easily be used in a build system already, > > like used to store a path to a QEMU to use. > > > > We don't really have a name space for CI variable, but maybe > > BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other > > usages. > > > > Otherwise patch looks fine. > > A can switch to BUILD_QEMU, but I don't plan to repost just for that. The patch is fine. BUILD_QEMU is a bit better. BUILD_QEMU_XEN or BUILD_QEMU_DM would be even better to clarify we are building a QEMU version to serve as device model. It is a pity that typically we call qemu-dm the ancient QEMU fork because "qemu device model" is more meaningful than "qemu xen". Anyway, pick your preference and feel free to change the variable name on commit. Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
On 08/10/2024 7:54 pm, Stefano Stabellini wrote: > On Tue, 8 Oct 2024, Andrew Cooper wrote: >> On 08/10/2024 5:27 pm, Anthony PERARD wrote: >>> On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: >>>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml >>>> index c668736bdc2f..c83e0bdbe119 100644 >>>> --- a/automation/gitlab-ci/build.yaml >>>> +++ b/automation/gitlab-ci/build.yaml >>>> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: >>>> extends: .gcc-x86-64-build-debug >>>> variables: >>>> CONTAINER: alpine:3.18 >>>> + QEMU: y >>> Could you use a different name for the variable? This is exposed as an >>> environment variable, it could easily be used in a build system already, >>> like used to store a path to a QEMU to use. >>> >>> We don't really have a name space for CI variable, but maybe >>> BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other >>> usages. >>> >>> Otherwise patch looks fine. >> A can switch to BUILD_QEMU, but I don't plan to repost just for that. > The patch is fine. > > BUILD_QEMU is a bit better. BUILD_QEMU_XEN or BUILD_QEMU_DM would be > even better to clarify we are building a QEMU version to serve as device > model. It is a pity that typically we call qemu-dm the ancient QEMU > fork because "qemu device model" is more meaningful than "qemu xen". > > Anyway, pick your preference and feel free to change the variable name > on commit. > > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> Thanks. I think I'll go with BUILD_QEMU_XEN because that what the rest of Xen's build system calls the thing being discussed here. ~Andrew
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index c668736bdc2f..c83e0bdbe119 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: extends: .gcc-x86-64-build-debug variables: CONTAINER: alpine:3.18 + QEMU: y debian-bookworm-gcc-debug: extends: .gcc-x86-64-build-debug diff --git a/automation/scripts/build b/automation/scripts/build index 34416297a4b7..926cf5b46d17 100755 --- a/automation/scripts/build +++ b/automation/scripts/build @@ -91,11 +91,8 @@ else cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"") fi - # Qemu requires Python 3.8 or later, and ninja - # and Clang 10 or later - if ! type python3 || ! python3 automation/scripts/qemu-deps-check.py \ - || [[ "$cc_is_clang" == y && "$cc_ver" -lt 0x0a0000 ]] \ - || ! type ninja; then + # QEMU is only for those who ask + if [[ "$QEMU" != "y" ]]; then cfgargs+=("--with-system-qemu=/bin/false") fi diff --git a/automation/scripts/qemu-deps-check.py b/automation/scripts/qemu-deps-check.py deleted file mode 100755 index f6188afb3e2a..000000000000 --- a/automation/scripts/qemu-deps-check.py +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env python3 -# -*- coding: utf-8 -*- -import sys - -if sys.version_info < (3, 8): - print("Python %d.%d.%d too old" % - (sys.version_info.major, - sys.version_info.minor, - sys.version_info.micro)) - exit(1) - -try: - import tomllib -except ImportError: - try: - import tomli - except ImportError: - print("No tomli") - exit(1)
We spend an awful lot of CI time building QEMU, even though most changes don't touch the subset of tools/libs/ used by QEMU. Some numbers taken at a time when CI was otherwise quiet: With Without Alpine: 13m38s 6m04s Debian 12: 10m05s 8m10s OpenSUSE Tumbleweed: 11m40s 7m54s Ubuntu 24.04: 14m56s 8m06s which is a >50% improvement in wallclock time in some cases. The only build we have that needs QEMU is alpine-3.18-gcc-debug. This is the build deployed and used by the QubesOS ADL-* and Zen3p-* jobs. Xilinx-x86_64 deploys it too, but is PVH-only and doesn't use QEMU. QEMU is also built by CirrusCI for FreeBSD (fully Clang/LLVM toolchain). This should help quite a lot with Gitlab CI capacity. Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> --- CC: Anthony PERARD <anthony.perard@vates.tech> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Michal Orzel <michal.orzel@amd.com> CC: Doug Goldstein <cardoe@cardoe.com> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> I did consider putting in a build of QEMU on arm64 for good measure, but I can't find any build job which currently does; none of the arm64 containers have ninja by the looks of things. Run with this patch in place: https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/1486822209 --- automation/gitlab-ci/build.yaml | 1 + automation/scripts/build | 7 ++----- automation/scripts/qemu-deps-check.py | 19 ------------------- 3 files changed, 3 insertions(+), 24 deletions(-) delete mode 100755 automation/scripts/qemu-deps-check.py