| Message ID | 20240804033309.890181-1-masahiroy@kernel.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | kbuild: modinst: remove the multithread option from zstd compression | expand |
On Sun, Aug 4, 2024 at 5:33 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > Parallel execution is supported by GNU Make: > > $ make -j<N> modules_install > > It is questionable to enable multithreading within each zstd process > by default. > > If you still want to do it, you can use the environment variable: > > $ ZSTD_NBTHREADS=<N> make modules_install > Hi Masahiro, I have some understanding problems. [ start-build.txt ] dileks 24225 24217 0 17:55 tty2 00:00:00 /usr/bin/perf stat make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/ PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@gmail.com KBUILD_BUILD_TIMESTAMP=2024-08-03 KDEB_PKGVERSION=6.10.3-1~trixie+dileks1 LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg ^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make modules_install ... is used? As far as I understood, each kernel-module file is taken - in the Debian build-process - sequentially file for file - ZSTD compressed and afterwards deleted. Is there a benefit when 'make -j<N>' is used? [ EXAMPLE - with my patch from [1] ] zstd -T0 -19 --rm -f -q debian/linux-image-6.10.3-1-amd64-clang18-kcfi/lib/modules/6.10.3-1-amd64-clang18-kcfi/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko ^^ How should ZSTD_NBTHREADS=<N> replace 'zstd -T0'? When I pass '-T0 -L19' to ZSTD two cores are used on my CPU and a better compress-level produces smaller kernel-module files. I would like to benefit from both. Thanks. Best regards, -Sedat- Link: https://lore.kernel.org/all/CA+icZUUQadYjAXiCNx7PmKDV20WctvnzkXC3R2F7FM4Gzcm39Q@mail.gmail.com/ [1] > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > scripts/Makefile.modinst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index 0afd75472679..04f5229efa6b 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP $@ > quiet_cmd_xz = XZ $@ > cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > quiet_cmd_zstd = ZSTD $@ > - cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > + cmd_zstd = $(ZSTD) --rm -f -q $< > > $(dst)/%.ko.gz: $(dst)/%.ko FORCE > $(call cmd,gzip) > -- > 2.43.0 > >
On Sun, Aug 4, 2024 at 3:36 PM Sedat Dilek <sedat.dilek@gmail.com> wrote: > > On Sun, Aug 4, 2024 at 5:33 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > Parallel execution is supported by GNU Make: > > > > $ make -j<N> modules_install > > > > It is questionable to enable multithreading within each zstd process > > by default. > > > > If you still want to do it, you can use the environment variable: > > > > $ ZSTD_NBTHREADS=<N> make modules_install > > > > Hi Masahiro, > > I have some understanding problems. > > [ start-build.txt ] > dileks 24225 24217 0 17:55 tty2 00:00:00 /usr/bin/perf stat > make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/ > PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza > KBUILD_BUILD_USER=sedat.dilek@gmail.com > KBUILD_BUILD_TIMESTAMP=2024-08-03 > KDEB_PKGVERSION=6.10.3-1~trixie+dileks1 > LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg > > ^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make > modules_install ... is used? Option 1 $ export ZSTD_NBTHREADS=<N> $ make V=1 -k [snip] olddefconfig bindeb-pkg Option 2 $ ZSTD_NBTHREADS=<N> make V=1 -k [snip] olddefconfig bindeb-pkg > > As far as I understood, each kernel-module file is taken - in the > Debian build-process - sequentially file for file - ZSTD compressed > and afterwards deleted. > Is there a benefit when 'make -j<N>' is used? zstd --rm -f -q foo.ko zstd --rm -f -q bar.ko zstd --rm -f -q baz.ko will run in parallel with Make's -j<N> option.
On Sun, Aug 04, 2024 at 12:33:07PM +0900, Masahiro Yamada wrote: > Parallel execution is supported by GNU Make: > > $ make -j<N> modules_install > > It is questionable to enable multithreading within each zstd process > by default. > > If you still want to do it, you can use the environment variable: > > $ ZSTD_NBTHREADS=<N> make modules_install > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > --- > > scripts/Makefile.modinst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index 0afd75472679..04f5229efa6b 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP $@ > quiet_cmd_xz = XZ $@ > cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > quiet_cmd_zstd = ZSTD $@ > - cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > + cmd_zstd = $(ZSTD) --rm -f -q $< > > $(dst)/%.ko.gz: $(dst)/%.ko FORCE > $(call cmd,gzip) > -- > 2.43.0 > Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
On Tue, Aug 6, 2024 at 1:02 PM Nicolas Schier <nicolas@fjasle.eu> wrote: > > On Sun, Aug 04, 2024 at 12:33:07PM +0900, Masahiro Yamada wrote: > > Parallel execution is supported by GNU Make: > > > > $ make -j<N> modules_install > > > > It is questionable to enable multithreading within each zstd process > > by default. > > > > If you still want to do it, you can use the environment variable: > > > > $ ZSTD_NBTHREADS=<N> make modules_install > > > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> > > --- > > > > scripts/Makefile.modinst | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > > index 0afd75472679..04f5229efa6b 100644 > > --- a/scripts/Makefile.modinst > > +++ b/scripts/Makefile.modinst > > @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP $@ > > quiet_cmd_xz = XZ $@ > > cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > > quiet_cmd_zstd = ZSTD $@ > > - cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > > + cmd_zstd = $(ZSTD) --rm -f -q $< > > > > $(dst)/%.ko.gz: $(dst)/%.ko FORCE > > $(call cmd,gzip) > > -- > > 2.43.0 > > > > Reviewed-by: Nicolas Schier <nicolas@fjasle.eu> > Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # Linux v6.11-rc4+ on x86-64 -Sedat-
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 0afd75472679..04f5229efa6b 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP $@ quiet_cmd_xz = XZ $@ cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< quiet_cmd_zstd = ZSTD $@ - cmd_zstd = $(ZSTD) -T0 --rm -f -q $< + cmd_zstd = $(ZSTD) --rm -f -q $< $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(call cmd,gzip)
Parallel execution is supported by GNU Make: $ make -j<N> modules_install It is questionable to enable multithreading within each zstd process by default. If you still want to do it, you can use the environment variable: $ ZSTD_NBTHREADS=<N> make modules_install Signed-off-by: Masahiro Yamada <masahiroy@kernel.org> --- scripts/Makefile.modinst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)