diff mbox series

[XEN,2/2] automation: Check if ninja is available before building QEMU

Message ID 20210707164001.894805-3-anthony.perard@citrix.com (mailing list archive)
State New
Headers show
Series automation: Fix build with new qemu-xen version | expand

Commit Message

Anthony PERARD July 7, 2021, 4:40 p.m. UTC
ninja is now required to build the latest version of QEMU, some
container still don't have ninja and attempting to add it breaks the
build for different reasons, so QEMU will be skip on those containers.

Failures:
- ubuntu/xenial:
    fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32)
    /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
- suse/opensuse-tumbleweed:
    failed to build iPXE
- debian/unstable:
    update of the container fails with:
        The following packages have unmet dependencies:
         clang-8 : Depends: libstdc++-8-dev but it is not installable
                   Depends: libgcc-8-dev but it is not installable
                   Depends: libobjc-8-dev but it is not installable
                   Recommends: llvm-8-dev but it is not going to be installed
                   Recommends: libomp-8-dev but it is not going to be installed

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/scripts/build | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comments

Andrew Cooper July 7, 2021, 5:46 p.m. UTC | #1
On 07/07/2021 17:40, Anthony PERARD wrote:
> ninja is now required to build the latest version of QEMU, some
> container still don't have ninja and attempting to add it breaks the
> build for different reasons, so QEMU will be skip on those containers.
>
> Failures:
> - ubuntu/xenial:
>     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
> - debian/unstable-i386: (debian-unstable-32)
>     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
> - suse/opensuse-tumbleweed:
>     failed to build iPXE
> - debian/unstable:
>     update of the container fails with:
>         The following packages have unmet dependencies:
>          clang-8 : Depends: libstdc++-8-dev but it is not installable
>                    Depends: libgcc-8-dev but it is not installable
>                    Depends: libobjc-8-dev but it is not installable
>                    Recommends: llvm-8-dev but it is not going to be installed
>                    Recommends: libomp-8-dev but it is not going to be installed

Only the first failure seems to be related to ninja.

The second is a known bug which still needs addressing.  Our build
system's logic to cope with `checkpolicy` being absent doesn't work.

Tumbleweed is generally broken and fails at ./configure due to missing
compression libraries.  It is quite possible that the iPXE failure is
related.

The final one is probably because unstable has dropped clang-8.  The
container hasn't been rebuilt in a while because we still haven't got
auto-rebuild sorted.

~Andrew
Olaf Hering July 7, 2021, 7:52 p.m. UTC | #2
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> iPXE failure

it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Olaf
Olaf Hering July 7, 2021, 8:35 p.m. UTC | #3
Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> Tumbleweed is generally broken and fails at ./configure due to missing
> compression libraries.

Something requests zlib-devel to be installed.

I suggest to provide all config.logs, not just the one from the top directory.

Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

Olaf
Andrew Cooper July 7, 2021, 10:44 p.m. UTC | #4
On 07/07/2021 20:52, Olaf Hering wrote:
> Am Wed, 7 Jul 2021 18:46:03 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> iPXE failure
> it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for
fixing build with GCC11" patch which has been sitting neglected on the list?

~Andrew
Andrew Cooper July 7, 2021, 10:47 p.m. UTC | #5
On 07/07/2021 21:35, Olaf Hering wrote:
> Am Wed, 7 Jul 2021 18:46:03 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> Tumbleweed is generally broken and fails at ./configure due to missing
>> compression libraries.
> Something requests zlib-devel to be installed.

Yeah - it shouldn't be hard to fix.

> I suggest to provide all config.logs, not just the one from the top directory.
>
> Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

We ought to yes, although I think at the moment we probably don't want
to realise how stale all the containers are.

~Andrew
Olaf Hering July 8, 2021, 12:29 a.m. UTC | #6
Am Wed, 7 Jul 2021 23:44:33 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> On 07/07/2021 20:52, Olaf Hering wrote:
> > Am Wed, 7 Jul 2021 18:46:03 +0100
> > schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
> >> iPXE failure  
> > it just needs to be updated to ipxe.git#master to make it compatible with gcc11.  
> Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for
> fixing build with GCC11" patch which has been sitting neglected on the list?

Yes, but the build logs indicate gcc10 is used in this container. So it might be something else.

Olaf
Olaf Hering July 8, 2021, 12:32 a.m. UTC | #7
Am Wed, 7 Jul 2021 23:47:02 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> We ought to yes, although I think at the moment we probably don't want
> to realise how stale all the containers are.

At least for Tumbleweed it is useful information.

opensuse-tumbleweed.dockerfile requests zlib-devel already.

tools/config.log is required to see what the failure actually is.
The toplevel config.log is not helpful in this particular case.

Olaf
Anthony PERARD July 8, 2021, 10:52 a.m. UTC | #8
On Wed, Jul 07, 2021 at 06:46:03PM +0100, Andrew Cooper wrote:
> On 07/07/2021 17:40, Anthony PERARD wrote:
> > ninja is now required to build the latest version of QEMU, some
> > container still don't have ninja and attempting to add it breaks the
> > build for different reasons, so QEMU will be skip on those containers.
> >
> > Failures:
> > - ubuntu/xenial:
> >     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
> > - debian/unstable-i386: (debian-unstable-32)
> >     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
> > - suse/opensuse-tumbleweed:
> >     failed to build iPXE
> > - debian/unstable:
> >     update of the container fails with:
> >         The following packages have unmet dependencies:
> >          clang-8 : Depends: libstdc++-8-dev but it is not installable
> >                    Depends: libgcc-8-dev but it is not installable
> >                    Depends: libobjc-8-dev but it is not installable
> >                    Recommends: llvm-8-dev but it is not going to be installed
> >                    Recommends: libomp-8-dev but it is not going to be installed
> 
> Only the first failure seems to be related to ninja.
> 
> The second is a known bug which still needs addressing.  Our build
> system's logic to cope with `checkpolicy` being absent doesn't work.
> 
> Tumbleweed is generally broken and fails at ./configure due to missing
> compression libraries.  It is quite possible that the iPXE failure is
> related.
> 
> The final one is probably because unstable has dropped clang-8.  The
> container hasn't been rebuilt in a while because we still haven't got
> auto-rebuild sorted.


So those containers are ones that I've noticed needed ninja (as found
out by looking at a single pipeline), but one which trying to install
ninja and thus rebuilding the container mean that the build was broken.
I just didn't want to spend time trying to fix issues that should have
been found out by a regular rebuild of the containers. So I've just
listed the reason there and moved on.
Andrew Cooper July 8, 2021, 11:23 a.m. UTC | #9
On 08/07/2021 11:52, Anthony PERARD wrote:
> On Wed, Jul 07, 2021 at 06:46:03PM +0100, Andrew Cooper wrote:
>> On 07/07/2021 17:40, Anthony PERARD wrote:
>>> ninja is now required to build the latest version of QEMU, some
>>> container still don't have ninja and attempting to add it breaks the
>>> build for different reasons, so QEMU will be skip on those containers.
>>>
>>> Failures:
>>> - ubuntu/xenial:
>>>     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
>>> - debian/unstable-i386: (debian-unstable-32)
>>>     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
>>> - suse/opensuse-tumbleweed:
>>>     failed to build iPXE
>>> - debian/unstable:
>>>     update of the container fails with:
>>>         The following packages have unmet dependencies:
>>>          clang-8 : Depends: libstdc++-8-dev but it is not installable
>>>                    Depends: libgcc-8-dev but it is not installable
>>>                    Depends: libobjc-8-dev but it is not installable
>>>                    Recommends: llvm-8-dev but it is not going to be installed
>>>                    Recommends: libomp-8-dev but it is not going to be installed
>> Only the first failure seems to be related to ninja.
>>
>> The second is a known bug which still needs addressing.  Our build
>> system's logic to cope with `checkpolicy` being absent doesn't work.
>>
>> Tumbleweed is generally broken and fails at ./configure due to missing
>> compression libraries.  It is quite possible that the iPXE failure is
>> related.
>>
>> The final one is probably because unstable has dropped clang-8.  The
>> container hasn't been rebuilt in a while because we still haven't got
>> auto-rebuild sorted.
>
> So those containers are ones that I've noticed needed ninja (as found
> out by looking at a single pipeline), but one which trying to install
> ninja and thus rebuilding the container mean that the build was broken.
> I just didn't want to spend time trying to fix issues that should have
> been found out by a regular rebuild of the containers.

That's fine, but as these containers weren't rebuilt, the details of any
knock-on error aren't relevant here.

I'd go with simply:

"ninja is now required to build the latest version of QEMU, and not all
distros have a suitable version.  Skip the QEMU build when ninja is not
available".

Can fix on commit if you're happy with this.

~Andrew
Anthony PERARD July 8, 2021, 11:30 a.m. UTC | #10
On Thu, Jul 08, 2021 at 12:23:38PM +0100, Andrew Cooper wrote:
> I'd go with simply:
> 
> "ninja is now required to build the latest version of QEMU, and not all
> distros have a suitable version.  Skip the QEMU build when ninja is not
> available".
> 
> Can fix on commit if you're happy with this.

Sounds good, thanks.
Olaf Hering July 8, 2021, 12:23 p.m. UTC | #11
Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <olaf@aepfle.de>:

> At least for Tumbleweed it is useful information.

CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"

Two years old for a rolling release.

Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Olaf
Andrew Cooper July 8, 2021, 12:51 p.m. UTC | #12
On 08/07/2021 13:23, Olaf Hering wrote:
> Am Thu, 8 Jul 2021 02:32:24 +0200
> schrieb Olaf Hering <olaf@aepfle.de>:
>
>> At least for Tumbleweed it is useful information.
> CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"
>
> Two years old for a rolling release.
>
> Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Automatically rebuilding the containers is still a task on the CI
working group's todo list.

That said, the tumbleweed containers were rebuilt far more recently than
that, so something is wonky...


In a xen tree, see automation/build/Makefile for details, but tl;dr to
rebuild:

$ make -C automation/build/ tumbleweed

and to push:

$ docker login registry.gitlab.com/xen-project/xen
$ make -C automation/build/ tumbleweed PUSH=1

Bear in mind that the final line will update the live CI system.  What I
normally do to test new containers is to give it a test suffix, push to
the live system, and push a branch to xen.git with the container names
edited in automation/gitlab-ci/*.yaml.

~Andrew
Olaf Hering July 8, 2021, 2:20 p.m. UTC | #13
Am Thu, 8 Jul 2021 13:51:49 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> That said, the tumbleweed containers were rebuilt far more recently than
> that, so something is wonky...

After playing with 'automation/scripts/containerize', docker image ls shows:
registry.gitlab.com/xen-project/xen/suse   opensuse-tumbleweed   f9b84e2346f0   7 weeks ago         2.27GB

No idea how that created a container with broken content.
zlib-devel.rpm exists, but libz.so is a broken symlink.

I will update the README to list the required steps for updating.

Olaf
diff mbox series

Patch

diff --git a/automation/scripts/build b/automation/scripts/build
index 46b6903d2922..d8c77e244cda 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -46,7 +46,9 @@  if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
 fi
 
 # Qemu requires Python 3.5 or later
-if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))"; then
+# QEMU requires ninja
+if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
+        || ! type ninja; then
     cfgargs+=("--with-system-qemu=/bin/false")
 fi