diff mbox series

[v2,13/12] CI: Refresh OpenSUSE Tumbleweed container

Message ID 20240712111426.3244115-1-andrew.cooper3@citrix.com (mailing list archive)
State New
Headers show
Series None | expand

Commit Message

Andrew Cooper July 12, 2024, 11:14 a.m. UTC
Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially.  Testing docs isn't very interesting and
saves a lot of space.  Other savings come from removing a huge pile of
optional QEMU dependencies (QEMU just needs to build the Xen parts to be
useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

This includes both of Olaf's recent changes, python3-packaging (now required
for a rebuilt container), and findutils (not required yet; still gets pulled
in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine.
staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
hence it's safe to drop `which`.

v2:
 * git -> git-core
 * Use 'pkgconfig()' references
---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile

Comments

Anthony PERARD July 12, 2024, 12:33 p.m. UTC | #1
On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
> while removed are:
> 
>   bc                   # ?

I don't know why, but this one is also installed by osstest. The commit
adding it doesn't say why it was needed.

Also, looks like many QEMU containers are installing `bc`, but I can't
find a single use. Maybe historic.

>   libnl3-devel         # Libxl COLO(?), option

Yes.

>   libnuma-devel        # ?

There's some mention of it in QEMU.

>   libpng16-devel       # Qemu ?

It's added to a few QEMU containers, but I can't find any use.

>   libssh2-devel        # ?

It seems like it was used by QEMU at some point, but they switch to
'libssh'. Probably optional, like many QEMU deps.

>   libtasn1-devel       # ?

QEMU, and it seems to be only used in tests.

>   nasm                 # ?

'nasm' is used to build OVMF.


If we don't want to build OVMF in our GitLab CI, which I don't think
would be useful to do anyway (no dependency on xen.git), then:

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,
Andrew Cooper July 12, 2024, 12:48 p.m. UTC | #2
On 12/07/2024 1:33 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
>> while removed are:
>>
>>   bc                   # ?
> I don't know why, but this one is also installed by osstest. The commit
> adding it doesn't say why it was needed.
>
> Also, looks like many QEMU containers are installing `bc`, but I can't
> find a single use. Maybe historic.

Part of the reason why our containers are so massive is because
dependencies only ever accumulate...

But yes - this one is especially odd.
>>   nasm                 # ?
> 'nasm' is used to build OVMF.
>
>
> If we don't want to build OVMF in our GitLab CI, which I don't think
> would be useful to do anyway (no dependency on xen.git)

Oh, interesting.

I'd assumed (by virtue of the jobs succeeding) that we were building
OVMF by default, but we're clearly not.

./configure says:

  --enable-ovmf           Enable OVMF (default is DISABLED)

so it's really just a simple opt-in, which our build script does not.


In the longer term, we will want OVMF in Gitlab CI, but we'd want to
pull it in as a separate artefact and use
--with-system-ovmf=/path/to/artefact

So I agree we don't need to build OVMF inside this container.

> , then:
>
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

~Andrew
diff mbox series

Patch

diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
new file mode 100644
index 000000000000..61e840fc67da
--- /dev/null
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -0,0 +1,76 @@ 
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
deleted file mode 100644
index 077fed22260c..000000000000
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ /dev/null
@@ -1,74 +0,0 @@ 
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/tumbleweed
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper dup -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        libzstd-devel \
-        lzo-devel \
-        make \
-        meson \
-        nasm \
-        ncurses-devel \
-        ninja \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index b4139414bc8e..fd6ca1270e4e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -668,25 +668,25 @@  opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 4d5669c5ad01..c4f013702122 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -43,7 +43,7 @@  case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used