diff mbox series

[RFC,8/8] tests/docker: Selective line reading by python script

Message ID 20220727163632.59806-9-lucas.araujo@eldorado.org.br (mailing list archive)
State New, archived
Headers show
Series Patch series to set up a ppc64le CI | expand

Commit Message

Lucas Mateus Martins Araujo e Castro July 27, 2022, 4:36 p.m. UTC
Building some images failed on ppc64le because the dockerfile tried to
install some packages that are only available in x86 and arm64, to solve
this while still having those packages be available in those architectures
a comment was put before the installation command to instruct the python
script into ignoring those lines for some architectures (in this case
ppc64le)

Overall I'm not a big fan of the way I solved this problem, so I'd like
to know if anyone has a better way to make these dockerfilse work in
PPC64LE.

For context the base images used here are available in PPC64LE but some
of the packages installed are not (in alpine's case it's XEN, which is
only available to x86 and ARM), so this patch create a ignore_list which
is set on a per-architecture basis, and any packages in a dockerfile in
this ignore_list will not be copied to the temporary dockerfile used in
the docker command.

Signed-off-by: Lucas Mateus Castro(alqotel) <lucas.araujo@eldorado.org.br>
---
 tests/docker/docker.py                 | 15 ++++++++++++---
 tests/docker/dockerfiles/alpine.docker |  2 ++
 2 files changed, 14 insertions(+), 3 deletions(-)

Comments

Philippe Mathieu-Daudé July 27, 2022, 9:20 p.m. UTC | #1
+Erik/Daniel

On Wed, Jul 27, 2022 at 6:37 PM Lucas Mateus Castro(alqotel)
<lucas.araujo@eldorado.org.br> wrote:
>
> Building some images failed on ppc64le because the dockerfile tried to
> install some packages that are only available in x86 and arm64, to solve
> this while still having those packages be available in those architectures
> a comment was put before the installation command to instruct the python
> script into ignoring those lines for some architectures (in this case
> ppc64le)
>
> Overall I'm not a big fan of the way I solved this problem, so I'd like
> to know if anyone has a better way to make these dockerfilse work in
> PPC64LE.
>
> For context the base images used here are available in PPC64LE but some
> of the packages installed are not (in alpine's case it's XEN, which is
> only available to x86 and ARM), so this patch create a ignore_list which
> is set on a per-architecture basis, and any packages in a dockerfile in
> this ignore_list will not be copied to the temporary dockerfile used in
> the docker command.

Shouldn't this be done on lcitool side?
(https://gitlab.com/libvirt/libvirt-ci/-/tree/master/lcitool)

> Signed-off-by: Lucas Mateus Castro(alqotel) <lucas.araujo@eldorado.org.br>
> ---
>  tests/docker/docker.py                 | 15 ++++++++++++---
>  tests/docker/dockerfiles/alpine.docker |  2 ++
>  2 files changed, 14 insertions(+), 3 deletions(-)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index d0af2861b8..9b962d1c78 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -14,6 +14,7 @@
>  import os
>  import sys
>  import subprocess
> +import platform
>  import json
>  import hashlib
>  import atexit
> @@ -207,8 +208,15 @@ def _read_qemu_dockerfile(img_name):
>
>  def _dockerfile_preprocess(df):
>      out = ""
> +    ignore_list = []
>      for l in df.splitlines():
> -        if len(l.strip()) == 0 or l.startswith("#"):
> +        if len(l.strip()) == 0:
> +            continue
> +        if l.startswith("#"):
> +            if len(l.split()) >= 3:
> +                if l.split()[1] == "ignore":
> +                    if platform.processor() in l.split()[2].split(','):
> +                        ignore_list += l.split()[3].split(',')
>              continue
>          from_pref = "FROM qemu/"
>          if l.startswith(from_pref):
> @@ -219,7 +227,8 @@ def _dockerfile_preprocess(df):
>              inlining = _read_qemu_dockerfile(l[len(from_pref):])
>              out += _dockerfile_preprocess(inlining)
>              continue
> -        out += l + "\n"
> +        if not any(x in l.split() for x in ignore_list):
> +            out += l + "\n"
>      return out
>
>
> @@ -330,7 +339,7 @@ def build_image(self, tag, docker_dir, dockerfile,
>          tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
>                                               encoding='utf-8',
>                                               dir=docker_dir, suffix=".docker")
> -        tmp_df.write(dockerfile)
> +        tmp_df.write(_dockerfile_preprocess(dockerfile))
>
>          if user:
>              uid = os.getuid()
> diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
> index 2943a99730..5cec46d8f2 100644
> --- a/tests/docker/dockerfiles/alpine.docker
> +++ b/tests/docker/dockerfiles/alpine.docker
> @@ -6,6 +6,8 @@
>
>  FROM docker.io/library/alpine:edge
>
> +# Lines to by ignored when this file is read by the python script
> +# ignore ppc64le,ppc64 xen-dev
>  RUN apk update && \
>      apk upgrade && \
>      apk add \
> --
> 2.25.1
>
Daniel P. Berrangé July 28, 2022, 8:43 a.m. UTC | #2
On Wed, Jul 27, 2022 at 11:20:56PM +0200, Philippe Mathieu-Daudé wrote:
> +Erik/Daniel
> 
> On Wed, Jul 27, 2022 at 6:37 PM Lucas Mateus Castro(alqotel)
> <lucas.araujo@eldorado.org.br> wrote:
> >
> > Building some images failed on ppc64le because the dockerfile tried to
> > install some packages that are only available in x86 and arm64, to solve
> > this while still having those packages be available in those architectures
> > a comment was put before the installation command to instruct the python
> > script into ignoring those lines for some architectures (in this case
> > ppc64le)
> >
> > Overall I'm not a big fan of the way I solved this problem, so I'd like
> > to know if anyone has a better way to make these dockerfilse work in
> > PPC64LE.
> >
> > For context the base images used here are available in PPC64LE but some
> > of the packages installed are not (in alpine's case it's XEN, which is
> > only available to x86 and ARM), so this patch create a ignore_list which
> > is set on a per-architecture basis, and any packages in a dockerfile in
> > this ignore_list will not be copied to the temporary dockerfile used in
> > the docker command.
> 
> Shouldn't this be done on lcitool side?
> (https://gitlab.com/libvirt/libvirt-ci/-/tree/master/lcitool)

Yes, to fix this properly we'll need to identify which packages are
architecture specific, and create separate 'RUN' commands that skip
those packages on arches in question. Doable, but not entirely easy.

> 
> > Signed-off-by: Lucas Mateus Castro(alqotel) <lucas.araujo@eldorado.org.br>
> > ---
> >  tests/docker/docker.py                 | 15 ++++++++++++---
> >  tests/docker/dockerfiles/alpine.docker |  2 ++
> >  2 files changed, 14 insertions(+), 3 deletions(-)
> >
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index d0af2861b8..9b962d1c78 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -14,6 +14,7 @@
> >  import os
> >  import sys
> >  import subprocess
> > +import platform
> >  import json
> >  import hashlib
> >  import atexit
> > @@ -207,8 +208,15 @@ def _read_qemu_dockerfile(img_name):
> >
> >  def _dockerfile_preprocess(df):
> >      out = ""
> > +    ignore_list = []
> >      for l in df.splitlines():
> > -        if len(l.strip()) == 0 or l.startswith("#"):
> > +        if len(l.strip()) == 0:
> > +            continue
> > +        if l.startswith("#"):
> > +            if len(l.split()) >= 3:
> > +                if l.split()[1] == "ignore":
> > +                    if platform.processor() in l.split()[2].split(','):
> > +                        ignore_list += l.split()[3].split(',')
> >              continue
> >          from_pref = "FROM qemu/"
> >          if l.startswith(from_pref):
> > @@ -219,7 +227,8 @@ def _dockerfile_preprocess(df):
> >              inlining = _read_qemu_dockerfile(l[len(from_pref):])
> >              out += _dockerfile_preprocess(inlining)
> >              continue
> > -        out += l + "\n"
> > +        if not any(x in l.split() for x in ignore_list):
> > +            out += l + "\n"
> >      return out
> >
> >
> > @@ -330,7 +339,7 @@ def build_image(self, tag, docker_dir, dockerfile,
> >          tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
> >                                               encoding='utf-8',
> >                                               dir=docker_dir, suffix=".docker")
> > -        tmp_df.write(dockerfile)
> > +        tmp_df.write(_dockerfile_preprocess(dockerfile))
> >
> >          if user:
> >              uid = os.getuid()
> > diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
> > index 2943a99730..5cec46d8f2 100644
> > --- a/tests/docker/dockerfiles/alpine.docker
> > +++ b/tests/docker/dockerfiles/alpine.docker
> > @@ -6,6 +6,8 @@
> >
> >  FROM docker.io/library/alpine:edge
> >
> > +# Lines to by ignored when this file is read by the python script
> > +# ignore ppc64le,ppc64 xen-dev
> >  RUN apk update && \
> >      apk upgrade && \
> >      apk add \
> > --
> > 2.25.1
> >
> 

With regards,
Daniel
diff mbox series

Patch

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index d0af2861b8..9b962d1c78 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -14,6 +14,7 @@ 
 import os
 import sys
 import subprocess
+import platform
 import json
 import hashlib
 import atexit
@@ -207,8 +208,15 @@  def _read_qemu_dockerfile(img_name):
 
 def _dockerfile_preprocess(df):
     out = ""
+    ignore_list = []
     for l in df.splitlines():
-        if len(l.strip()) == 0 or l.startswith("#"):
+        if len(l.strip()) == 0:
+            continue
+        if l.startswith("#"):
+            if len(l.split()) >= 3:
+                if l.split()[1] == "ignore":
+                    if platform.processor() in l.split()[2].split(','):
+                        ignore_list += l.split()[3].split(',')
             continue
         from_pref = "FROM qemu/"
         if l.startswith(from_pref):
@@ -219,7 +227,8 @@  def _dockerfile_preprocess(df):
             inlining = _read_qemu_dockerfile(l[len(from_pref):])
             out += _dockerfile_preprocess(inlining)
             continue
-        out += l + "\n"
+        if not any(x in l.split() for x in ignore_list):
+            out += l + "\n"
     return out
 
 
@@ -330,7 +339,7 @@  def build_image(self, tag, docker_dir, dockerfile,
         tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
                                              encoding='utf-8',
                                              dir=docker_dir, suffix=".docker")
-        tmp_df.write(dockerfile)
+        tmp_df.write(_dockerfile_preprocess(dockerfile))
 
         if user:
             uid = os.getuid()
diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
index 2943a99730..5cec46d8f2 100644
--- a/tests/docker/dockerfiles/alpine.docker
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -6,6 +6,8 @@ 
 
 FROM docker.io/library/alpine:edge
 
+# Lines to by ignored when this file is read by the python script
+# ignore ppc64le,ppc64 xen-dev
 RUN apk update && \
     apk upgrade && \
     apk add \