| Message ID | 20200730190841.2071656-5-nickrterrell@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Add support for ZSTD-compressed kernel and initramfs | expand |
Hi Nick, On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote: > From: Nick Terrell <terrelln@fb.com> > > * Add support for a zstd compressed initramfs. > * Add compression for compressing built-in initramfs with zstd. > > I have tested this patch by boot testing with buildroot and QEMU. > Specifically, I booted the kernel with both a zstd and gzip compressed > initramfs, both built into the kernel and separate. I ensured that the > correct compression algorithm was used. I tested on arm, aarch64, i386, > and x86_64. > > This patch has been tested in production on aarch64 and x86_64 devices. > > Additionally, I have performance measurements from internal use in > production. On an aarch64 device we saw 19 second boot time improvement > from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64 > device we saw a 9 second boot time reduction from switching from xz to > zstd. > > Reviewed-by: Kees Cook <keescook@chromium.org> > Tested-by: Sedat Dilek <sedat.dilek@gmail.com> > Signed-off-by: Nick Terrell <terrelln@fb.com> Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add support for zstd compressed initramfs"). > --- a/usr/Kconfig > +++ b/usr/Kconfig > @@ -100,6 +100,15 @@ config RD_LZ4 > Support loading of a LZ4 encoded initial ramdisk or cpio buffer > If unsure, say N. > > +config RD_ZSTD > + bool "Support initial ramdisk/ramfs compressed using ZSTD" > + default y > + depends on BLK_DEV_INITRD > + select DECOMPRESS_ZSTD > + help > + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. > + If unsure, say N. I'm aware you copied this from the other entries, but IMHO "default y", and "If unsure, say N" are not a good combination. Gr{oetje,eeting}s, Geert
On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > Hi Nick, > > On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote: > > From: Nick Terrell <terrelln@fb.com> > > > > * Add support for a zstd compressed initramfs. > > * Add compression for compressing built-in initramfs with zstd. > > > > I have tested this patch by boot testing with buildroot and QEMU. > > Specifically, I booted the kernel with both a zstd and gzip compressed > > initramfs, both built into the kernel and separate. I ensured that the > > correct compression algorithm was used. I tested on arm, aarch64, i386, > > and x86_64. > > > > This patch has been tested in production on aarch64 and x86_64 devices. > > > > Additionally, I have performance measurements from internal use in > > production. On an aarch64 device we saw 19 second boot time improvement > > from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64 > > device we saw a 9 second boot time reduction from switching from xz to > > zstd. > > > > Reviewed-by: Kees Cook <keescook@chromium.org> > > Tested-by: Sedat Dilek <sedat.dilek@gmail.com> > > Signed-off-by: Nick Terrell <terrelln@fb.com> > > Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add > support for zstd compressed initramfs"). > > > --- a/usr/Kconfig > > +++ b/usr/Kconfig > > @@ -100,6 +100,15 @@ config RD_LZ4 > > Support loading of a LZ4 encoded initial ramdisk or cpio buffer > > If unsure, say N. > > > > +config RD_ZSTD > > + bool "Support initial ramdisk/ramfs compressed using ZSTD" > > + default y > > + depends on BLK_DEV_INITRD > > + select DECOMPRESS_ZSTD > > + help > > + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. > > + If unsure, say N. > > I'm aware you copied this from the other entries, but IMHO "default y", > and "If unsure, say N" are not a good combination. > Hi Geert, you are right - for new stuff it should be "default n". What I am missing - still - is a note - that your user-space should have the correct bits to support zstd-initramfs. Unsure where to place such an information. If you send a patch for above feel free to add my: Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com> Thanks. Regards, - Sedat - [1] https://bugs.debian.org/955469 > Gr{oetje,eeting}s, > > Geert > > -- > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > In personal conversations with technical people, I call myself a hacker. But > when I'm talking to journalists I just say "programmer" or something like that. > -- Linus Torvalds
On Tue, Aug 04, 2020 at 09:25:23AM +0200, Sedat Dilek wrote: > On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote: > > > From: Nick Terrell <terrelln@fb.com> > > > * Add support for a zstd compressed initramfs. > > > * Add compression for compressing built-in initramfs with zstd. > > > --- a/usr/Kconfig > > > +++ b/usr/Kconfig > > > @@ -100,6 +100,15 @@ config RD_LZ4 > > > Support loading of a LZ4 encoded initial ramdisk or cpio buffer > > > If unsure, say N. > > > > > > +config RD_ZSTD > > > + bool "Support initial ramdisk/ramfs compressed using ZSTD" > > > + default y > > > + depends on BLK_DEV_INITRD > > > + select DECOMPRESS_ZSTD > > > + help > > > + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. > > > + If unsure, say N. > > > > I'm aware you copied this from the other entries, but IMHO "default y", > > and "If unsure, say N" are not a good combination. > you are right - for new stuff it should be "default n". It got already applied to Linus' tree with "y", and I think it'd be nice to have it as a default. Let's disable other compressors instead. On the other hand, having an unsupported rd compressor results in a boot failure that's not immediately obvious, so that's a reason for keeping the setting as "y". On the third hand, distributions default to either gz or xz, thus I'd say: * let's have gz xz zstd default to y, all others to n * drop bzip2 lzma1 completely * distros can't switch the mkinitramfs default yet, but if RD_ZSTD=y now, they'll be able to once they drop support for old kernels in a few years > What I am missing - still - is a note - that your user-space should > have the correct bits to support zstd-initramfs. > Unsure where to place such an information. Looks like INITRAMFS_COMPRESSION_* have lengthy prose but are not shown in menuconfig, while RD_*, with no such prose, are shown. The prose itself is grossly obsolete, too. I have some updates in: https://github.com/kilobyte/linux/commits/nobz2-v3 but that patchset needs rebasing and refreshing. Meow!
diff --git a/usr/Kconfig b/usr/Kconfig index 96afb03b65f9..2599bc21c1b2 100644 --- a/usr/Kconfig +++ b/usr/Kconfig @@ -100,6 +100,15 @@ config RD_LZ4 Support loading of a LZ4 encoded initial ramdisk or cpio buffer If unsure, say N. +config RD_ZSTD + bool "Support initial ramdisk/ramfs compressed using ZSTD" + default y + depends on BLK_DEV_INITRD + select DECOMPRESS_ZSTD + help + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. + If unsure, say N. + choice prompt "Built-in initramfs compression mode" depends on INITRAMFS_SOURCE != "" @@ -196,6 +205,17 @@ config INITRAMFS_COMPRESSION_LZ4 If you choose this, keep in mind that most distros don't provide lz4 by default which could cause a build failure. +config INITRAMFS_COMPRESSION_ZSTD + bool "ZSTD" + depends on RD_ZSTD + help + ZSTD is a compression algorithm targeting intermediate compression + with fast decompression speed. It will compress better than GZIP and + decompress around the same speed as LZO, but slower than LZ4. + + If you choose this, keep in mind that you may need to install the zstd + tool to be able to compress the initram. + config INITRAMFS_COMPRESSION_NONE bool "None" help diff --git a/usr/Makefile b/usr/Makefile index c12e6b15ce72..b1a81a40eab1 100644 --- a/usr/Makefile +++ b/usr/Makefile @@ -15,6 +15,7 @@ compress-$(CONFIG_INITRAMFS_COMPRESSION_LZMA) := lzma compress-$(CONFIG_INITRAMFS_COMPRESSION_XZ) := xzmisc compress-$(CONFIG_INITRAMFS_COMPRESSION_LZO) := lzo compress-$(CONFIG_INITRAMFS_COMPRESSION_LZ4) := lz4 +compress-$(CONFIG_INITRAMFS_COMPRESSION_ZSTD) := zstd obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o