Message ID | 20210407160927.222092-1-lucjan.lucjanov@gmail.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | [v3] kbuild: add support for zstd compressed modules | expand |
On Thu, Apr 8, 2021 at 1:09 AM Piotr Gorski <lucjan.lucjanov@gmail.com> wrote: > > kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > > V2 -> V3 > > * Fix a typo > > V1 -> V2 > > * Rebuild against linux-kbuild tree > > Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> Applied to linux-kbuild. Thanks! I slightly changed to fix the log alignment. diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index f9fa2a3808b2..ff9b09e4cfca 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -96,7 +96,7 @@ quiet_cmd_gzip = GZIP $@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< -quiet_cmd_zstd = ZSTD $@ +quiet_cmd_zstd = ZSTD $@ cmd_zstd = $(ZSTD) -T0 --rm -f -q $< $(dst)/%.ko.gz: $(dst)/%.ko FORCE > --- > init/Kconfig | 8 +++++++- > scripts/Makefile.modinst | 6 ++++++ > 2 files changed, 13 insertions(+), 1 deletion(-) > > diff --git a/init/Kconfig b/init/Kconfig > index 510f6fcd9b7f..b5744d32c4df 100644 > --- a/init/Kconfig > +++ b/init/Kconfig > @@ -2242,7 +2242,7 @@ choice > > Please note that the tool used to load modules needs to support the > corresponding algorithm. module-init-tools MAY support gzip, and kmod > - MAY support gzip and xz. > + MAY support gzip, xz and zstd. > > Your build system needs to provide the appropriate compression tool > to compress the modules. > @@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ > Compress modules with XZ. The installed modules are suffixed > with .ko.xz. > > +config MODULE_COMPRESS_ZSTD > + bool "ZSTD" > + help > + Compress modules with ZSTD. The installed modules are suffixed > + with .ko.zst. > + > endchoice > > config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index 191408f7a91a..f9fa2a3808b2 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -21,6 +21,7 @@ endif > suffix-y := > suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz > suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz > +suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst > > modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules)) > > @@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP $@ > cmd_gzip = $(KGZIP) -n -f $< > quiet_cmd_xz = XZ $@ > cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< > +quiet_cmd_zstd = ZSTD $@ > + cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > > $(dst)/%.ko.gz: $(dst)/%.ko FORCE > $(call cmd,gzip) > @@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE > $(dst)/%.ko.xz: $(dst)/%.ko FORCE > $(call cmd,xz) > > +$(dst)/%.ko.zst: $(dst)/%.ko FORCE > + $(call cmd,zstd) > + > PHONY += FORCE > FORCE: > > -- > 2.31.0.97.g1424303384 >
No, the --rm option is essential. xz and gzip have the --rm option built in as opposed to zstd, which is why I used it. I've been using zstd module compression since last december (although I set a different compression level on mine) and everything works fine. Oleksandr also tested it at his place and didn't report any objections.
On Thu, Apr 8, 2021 at 11:05 PM Piotr Gorski <lucjan.lucjanov@gmail.com> wrote: > > No, the --rm option is essential. xz and gzip have the --rm option built in as opposed to zstd, which is why I used it. I've been using zstd module compression since last december (although I set a different compression level on mine) and everything works fine. Oleksandr also tested it at his place and didn't report any objections. [ CC me I am not subscribed to linux-kbuild or linux-kernel ] [ CC Nick ] Unfortunately, I do not find my initial posting which has all information. I add the link to the thread on linux-kbuild ML. So, I gave you as much information as I have (linux-config, make-line etc.) and you write "everything works fine"? What do you mean by "everything" - different compressors and none? Is that working "fine"? What build environment do you use? Here: Debian/testing AMD64. Did you try with... CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_DWARF5=y CONFIG_MODULE_COMPRESS_ZSTD=y ...Kconfigs enabled? As said I use builddeb from scripts directory to generate my Debian packages. Any chance you can test with builddeb? I have enabled Clang-LTO Kconfig. Tried with Clang-LTO Kconfig? This worked *before* and *after*... kbuild: add support for zstd compressed modules kbuild: remove CONFIG_MODULE_COMPRESS (CC Nick as he is listed as a reviewer here) ... not within my build-environment. For me this is a *regression*. - Sedat - [1] https://marc.info/?t=161790914600002&r=1&w=2
I originally posted the patch in a different form [1] even before Masahiro's changes. I've been testing this solution since December last year and posted it in March this year, after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2] Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did. My current logs: lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_DEBUG_INFO_COMPRESSED is not set # CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF4=y CONFIG_DEBUG_INFO_BTF=y CONFIG_DEBUG_INFO_BTF_MODULES=y lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz CONFIG_MODULE_COMPRESS_ZSTD=y CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use. The only difference - I don't use clang. Maybe those who use will comment on this. I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression. [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2 [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2 [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2
On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski <lucjan.lucjanov@gmail.com> wrote: > > I originally posted the patch in a different form [1] even before Masahiro's changes. > I've been testing this solution since December last year and posted it in March this year, > after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2] > > Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did. > > My current logs: > > lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz > CONFIG_DEBUG_INFO=y > # CONFIG_DEBUG_INFO_REDUCED is not set > # CONFIG_DEBUG_INFO_COMPRESSED is not set > # CONFIG_DEBUG_INFO_SPLIT is not set > CONFIG_DEBUG_INFO_DWARF4=y > CONFIG_DEBUG_INFO_BTF=y > CONFIG_DEBUG_INFO_BTF_MODULES=y > lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz > CONFIG_MODULE_COMPRESS_ZSTD=y > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 > > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use. > > The only difference - I don't use clang. Maybe those who use will comment on this. > I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression. > > [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2 > > [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2 > > [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2 I am a big fan of ZSTD and have it as default in all available Linux Kconfigs and Debian's initramfs-tools. So, I am highly interested in getting this fixed. Unfortunately, I have thrown away my yesterday's Clang-LTO build and switched to Clang-CFI with builddeb - should do handle the same way. I see three iwlwifi.ko (as an example): $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07 drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with debug_info, not stripped That iwlwifi.ko with debug-info is optimized when moving to debian/linux-image-dbg directory: $ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18 debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with debug_info, not stripped And think it's shrunk down and included debian/linux-image directory: $ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko -rw-r--r-- 1 dileks dileks 694K Apr 9 11:18 debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko $ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed. Fact is the one with debug-info is done correctly. Might be builddeb script needs a special treatment. - Sedat -
On Fri, Apr 9, 2021 at 1:31 PM Sedat Dilek <sedat.dilek@gmail.com> wrote: > > On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski <lucjan.lucjanov@gmail.com> wrote: > > > > I originally posted the patch in a different form [1] even before Masahiro's changes. > > I've been testing this solution since December last year and posted it in March this year, > > after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2] > > > > Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did. > > > > My current logs: > > > > lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz > > CONFIG_DEBUG_INFO=y > > # CONFIG_DEBUG_INFO_REDUCED is not set > > # CONFIG_DEBUG_INFO_COMPRESSED is not set > > # CONFIG_DEBUG_INFO_SPLIT is not set > > CONFIG_DEBUG_INFO_DWARF4=y > > CONFIG_DEBUG_INFO_BTF=y > > CONFIG_DEBUG_INFO_BTF_MODULES=y > > lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz > > CONFIG_MODULE_COMPRESS_ZSTD=y > > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19 > > > > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use. > > > > The only difference - I don't use clang. Maybe those who use will comment on this. > > I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression. > > > > [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2 > > > > [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2 > > > > [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2 > > I am a big fan of ZSTD and have it as default in all available Linux > Kconfigs and Debian's initramfs-tools. > So, I am highly interested in getting this fixed. > > Unfortunately, I have thrown away my yesterday's Clang-LTO build and > switched to Clang-CFI with builddeb - should do handle the same way. > > I see three iwlwifi.ko (as an example): > > $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07 > drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB > relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with > debug_info, not stripped > > That iwlwifi.ko with debug-info is optimized when moving to > debian/linux-image-dbg directory: > > $ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18 > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: > ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with > debug_info, not stripped > > And think it's shrunk down and included debian/linux-image directory: > > $ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > -rw-r--r-- 1 dileks dileks 694K Apr 9 11:18 > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > > $ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko > debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: > ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), > BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped > > I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed. > Fact is the one with debug-info is done correctly. > Might be builddeb script needs a special treatment. > OK, I see (sorry Gmail truncates my paste of snippet). We need to add in this block a CONFIG_MODULE_COMPRESS_XXX handling: [ scripts/package/builddeb ] #159: if is_enabled CONFIG_MODULES; then ... #184: fi Maybe other scripts in scripts/package/ directory, too. What do you say Masahiro? I have to admit I never used any compression for kernel-modules before and after recent changes in <kbuild.git#kbuild>. - Sedat - [1] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/tree/scripts/package/builddeb?h=kbuild#n159
diff --git a/init/Kconfig b/init/Kconfig index 510f6fcd9b7f..b5744d32c4df 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2242,7 +2242,7 @@ choice Please note that the tool used to load modules needs to support the corresponding algorithm. module-init-tools MAY support gzip, and kmod - MAY support gzip and xz. + MAY support gzip, xz and zstd. Your build system needs to provide the appropriate compression tool to compress the modules. @@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ Compress modules with XZ. The installed modules are suffixed with .ko.xz. +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + help + Compress modules with ZSTD. The installed modules are suffixed + with .ko.zst. + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 191408f7a91a..f9fa2a3808b2 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -21,6 +21,7 @@ endif suffix-y := suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz +suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules)) @@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP $@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< +quiet_cmd_zstd = ZSTD $@ + cmd_zstd = $(ZSTD) -T0 --rm -f -q $< $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(call cmd,gzip) @@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE $(dst)/%.ko.xz: $(dst)/%.ko FORCE $(call cmd,xz) +$(dst)/%.ko.zst: $(dst)/%.ko FORCE + $(call cmd,zstd) + PHONY += FORCE FORCE:
kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. V2 -> V3 * Fix a typo V1 -> V2 * Rebuild against linux-kbuild tree Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> --- init/Kconfig | 8 +++++++- scripts/Makefile.modinst | 6 ++++++ 2 files changed, 13 insertions(+), 1 deletion(-)