| Message ID | 20210222230106.7030-2-dbuono@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | gitlab-ci.yml: Add jobs to test CFI | expand |
On 23/02/21 00:01, Daniele Buono wrote: > Currently, make parallelism at build time is defined as #cpus+1. Some > build jobs may need (or benefit from) a different number. An example is > builds with LTO where, because of the huge demand of memory at link > time, gitlab runners fails if two linkers are run concurrently This > patch retains the default value of #cpus+1 but allows setting the "JOBS" > variable to a different number when applying the template As I just found out, you can add -Dbackend_max_links=1 to the meson command line instead if LTO is enabled. Paolo
On 2/22/21 8:01 PM, Daniele Buono wrote: > Currently, make parallelism at build time is defined as #cpus+1. > Some build jobs may need (or benefit from) a different number. > > An example is builds with LTO where, because of the huge demand > of memory at link time, gitlab runners fails if two linkers are > run concurrently > > This patch retains the default value of #cpus+1 but allows setting > the "JOBS" variable to a different number when applying the template > > Signed-off-by: Daniele Buono <dbuono@linux.vnet.ibm.com> > --- > .gitlab-ci.yml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 8b6d495288..5c198f05d4 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -17,7 +17,7 @@ include: > stage: build > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest > before_script: > - - JOBS=$(expr $(nproc) + 1) > + - JOBS=${JOBS:-$(expr $(nproc) + 1)} > script: > - mkdir build > - cd build
This works, but setting this value to 1 for everybody seems a bit too restrictive. While the gitlab ci runners don't have enough memory for this, that's not necessarily true for every build platform, and linking multiple targets in parallel with LTO can result in a big save in time, so I'd prefer a customizable way. How about adding a flag `--max-ld-procs` to configure to manually set backend_max_links? This would also allow setting it up to any specific number above 1, which looking at the Makefile seems to not be possible now: because of how the -j flag is passed from make to ninja, a compilation is either sequential or parallel based on #cpus On 2/23/2021 3:12 AM, Paolo Bonzini wrote: > On 23/02/21 00:01, Daniele Buono wrote: >> Currently, make parallelism at build time is defined as #cpus+1. Some >> build jobs may need (or benefit from) a different number. An example >> is builds with LTO where, because of the huge demand of memory at link >> time, gitlab runners fails if two linkers are run concurrently This >> patch retains the default value of #cpus+1 but allows setting the >> "JOBS" variable to a different number when applying the template > > As I just found out, you can add -Dbackend_max_links=1 to the meson > command line instead if LTO is enabled. > > Paolo >
On 23/02/21 20:34, Daniele Buono wrote: > This works, but setting this value to 1 for everybody seems a bit too > restrictive. While the gitlab ci runners don't have enough memory for > this, that's not necessarily true for every build platform, and linking > multiple targets in parallel with LTO can result in a big save in time, > so I'd prefer a customizable way. > > How about adding a flag `--max-ld-procs` to configure to manually set > backend_max_links? Another possibility is to invoke "meson configure build -Dbackend_max_links=1" after configure. Paolo > This would also allow setting it up to any specific number above 1, > which looking at the Makefile seems to not be possible now: because of > how the -j flag is passed from make to ninja, a compilation is either > sequential or parallel based on #cpus
On 2/24/2021 2:44 AM, Paolo Bonzini wrote: > On 23/02/21 20:34, Daniele Buono wrote: >> This works, but setting this value to 1 for everybody seems a bit too >> restrictive. While the gitlab ci runners don't have enough memory for >> this, that's not necessarily true for every build platform, and linking >> multiple targets in parallel with LTO can result in a big save in time, >> so I'd prefer a customizable way. >> >> How about adding a flag `--max-ld-procs` to configure to manually set >> backend_max_links? > > Another possibility is to invoke "meson configure build > -Dbackend_max_links=1" after configure. I like this, I'll send a v2 soon where I replace this patch with one just for linking. Daniele
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 8b6d495288..5c198f05d4 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -17,7 +17,7 @@ include: stage: build image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest before_script: - - JOBS=$(expr $(nproc) + 1) + - JOBS=${JOBS:-$(expr $(nproc) + 1)} script: - mkdir build - cd build
Currently, make parallelism at build time is defined as #cpus+1. Some build jobs may need (or benefit from) a different number. An example is builds with LTO where, because of the huge demand of memory at link time, gitlab runners fails if two linkers are run concurrently This patch retains the default value of #cpus+1 but allows setting the "JOBS" variable to a different number when applying the template Signed-off-by: Daniele Buono <dbuono@linux.vnet.ibm.com> --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)