[v10,4/8] usr: add support for zstd compressed initramfs
diff mbox series

Message ID 20200730190841.2071656-5-nickrterrell@gmail.com
State New
Headers show
Series
  • Add support for ZSTD-compressed kernel and initramfs
Related show

Commit Message

Nick Terrell July 30, 2020, 7:08 p.m. UTC
From: Nick Terrell <terrelln@fb.com>

* Add support for a zstd compressed initramfs.
* Add compression for compressing built-in initramfs with zstd.

I have tested this patch by boot testing with buildroot and QEMU.
Specifically, I booted the kernel with both a zstd and gzip compressed
initramfs, both built into the kernel and separate. I ensured that the
correct compression algorithm was used. I tested on arm, aarch64, i386,
and x86_64.

This patch has been tested in production on aarch64 and x86_64 devices.

Additionally, I have performance measurements from internal use in
production. On an aarch64 device we saw 19 second boot time improvement
from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64
device we saw a 9 second boot time reduction from switching from xz to
zstd.

Reviewed-by: Kees Cook <keescook@chromium.org>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
 usr/Kconfig  | 20 ++++++++++++++++++++
 usr/Makefile |  1 +
 2 files changed, 21 insertions(+)

Comments

Geert Uytterhoeven Aug. 4, 2020, 6:52 a.m. UTC | #1
Hi Nick,

On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote:
> From: Nick Terrell <terrelln@fb.com>
>
> * Add support for a zstd compressed initramfs.
> * Add compression for compressing built-in initramfs with zstd.
>
> I have tested this patch by boot testing with buildroot and QEMU.
> Specifically, I booted the kernel with both a zstd and gzip compressed
> initramfs, both built into the kernel and separate. I ensured that the
> correct compression algorithm was used. I tested on arm, aarch64, i386,
> and x86_64.
>
> This patch has been tested in production on aarch64 and x86_64 devices.
>
> Additionally, I have performance measurements from internal use in
> production. On an aarch64 device we saw 19 second boot time improvement
> from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64
> device we saw a 9 second boot time reduction from switching from xz to
> zstd.
>
> Reviewed-by: Kees Cook <keescook@chromium.org>
> Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
> Signed-off-by: Nick Terrell <terrelln@fb.com>

Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add
support for zstd compressed initramfs").

> --- a/usr/Kconfig
> +++ b/usr/Kconfig
> @@ -100,6 +100,15 @@ config RD_LZ4
>           Support loading of a LZ4 encoded initial ramdisk or cpio buffer
>           If unsure, say N.
>
> +config RD_ZSTD
> +       bool "Support initial ramdisk/ramfs compressed using ZSTD"
> +       default y
> +       depends on BLK_DEV_INITRD
> +       select DECOMPRESS_ZSTD
> +       help
> +         Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
> +         If unsure, say N.

I'm aware you copied this from the other entries, but IMHO "default y",
and "If unsure, say N" are not a good combination.

Gr{oetje,eeting}s,

                        Geert
Sedat Dilek Aug. 4, 2020, 7:25 a.m. UTC | #2
On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
> Hi Nick,
>
> On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote:
> > From: Nick Terrell <terrelln@fb.com>
> >
> > * Add support for a zstd compressed initramfs.
> > * Add compression for compressing built-in initramfs with zstd.
> >
> > I have tested this patch by boot testing with buildroot and QEMU.
> > Specifically, I booted the kernel with both a zstd and gzip compressed
> > initramfs, both built into the kernel and separate. I ensured that the
> > correct compression algorithm was used. I tested on arm, aarch64, i386,
> > and x86_64.
> >
> > This patch has been tested in production on aarch64 and x86_64 devices.
> >
> > Additionally, I have performance measurements from internal use in
> > production. On an aarch64 device we saw 19 second boot time improvement
> > from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64
> > device we saw a 9 second boot time reduction from switching from xz to
> > zstd.
> >
> > Reviewed-by: Kees Cook <keescook@chromium.org>
> > Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
> > Signed-off-by: Nick Terrell <terrelln@fb.com>
>
> Thanks for your patch, which is now commit a30d8a39f0571425 ("usr: Add
> support for zstd compressed initramfs").
>
> > --- a/usr/Kconfig
> > +++ b/usr/Kconfig
> > @@ -100,6 +100,15 @@ config RD_LZ4
> >           Support loading of a LZ4 encoded initial ramdisk or cpio buffer
> >           If unsure, say N.
> >
> > +config RD_ZSTD
> > +       bool "Support initial ramdisk/ramfs compressed using ZSTD"
> > +       default y
> > +       depends on BLK_DEV_INITRD
> > +       select DECOMPRESS_ZSTD
> > +       help
> > +         Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
> > +         If unsure, say N.
>
> I'm aware you copied this from the other entries, but IMHO "default y",
> and "If unsure, say N" are not a good combination.
>

Hi Geert,

you are right - for new stuff it should be "default n".

What I am missing - still - is a note - that your user-space should
have the correct bits to support zstd-initramfs.
Unsure where to place such an information.

If you send a patch for above feel free to add my:

Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>

Thanks.

Regards,
- Sedat -

[1] https://bugs.debian.org/955469

> Gr{oetje,eeting}s,
>
>                         Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds
Adam Borowski Aug. 4, 2020, 9:19 a.m. UTC | #3
On Tue, Aug 04, 2020 at 09:25:23AM +0200, Sedat Dilek wrote:
> On Tue, Aug 4, 2020 at 8:52 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > On Thu, Jul 30, 2020 at 9:13 PM Nick Terrell <nickrterrell@gmail.com> wrote:
> > > From: Nick Terrell <terrelln@fb.com>
> > > * Add support for a zstd compressed initramfs.
> > > * Add compression for compressing built-in initramfs with zstd.

> > > --- a/usr/Kconfig
> > > +++ b/usr/Kconfig
> > > @@ -100,6 +100,15 @@ config RD_LZ4
> > >           Support loading of a LZ4 encoded initial ramdisk or cpio buffer
> > >           If unsure, say N.
> > >
> > > +config RD_ZSTD
> > > +       bool "Support initial ramdisk/ramfs compressed using ZSTD"
> > > +       default y
> > > +       depends on BLK_DEV_INITRD
> > > +       select DECOMPRESS_ZSTD
> > > +       help
> > > +         Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
> > > +         If unsure, say N.
> >
> > I'm aware you copied this from the other entries, but IMHO "default y",
> > and "If unsure, say N" are not a good combination.

> you are right - for new stuff it should be "default n".

It got already applied to Linus' tree with "y", and I think it'd be nice
to have it as a default.  Let's disable other compressors instead.

On the other hand, having an unsupported rd compressor results in a boot
failure that's not immediately obvious, so that's a reason for keeping
the setting as "y".

On the third hand, distributions default to either gz or xz, thus I'd say:
* let's have gz xz zstd default to y, all others to n
* drop bzip2 lzma1 completely
* distros can't switch the mkinitramfs default yet, but if RD_ZSTD=y now,
  they'll be able to once they drop support for old kernels in a few years

> What I am missing - still - is a note - that your user-space should
> have the correct bits to support zstd-initramfs.
> Unsure where to place such an information.

Looks like INITRAMFS_COMPRESSION_* have lengthy prose but are not shown in
menuconfig, while RD_*, with no such prose, are shown.

The prose itself is grossly obsolete, too.  I have some updates in:
    https://github.com/kilobyte/linux/commits/nobz2-v3
but that patchset needs rebasing and refreshing.


Meow!

Patch
diff mbox series

diff --git a/usr/Kconfig b/usr/Kconfig
index 96afb03b65f9..2599bc21c1b2 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -100,6 +100,15 @@  config RD_LZ4
 	  Support loading of a LZ4 encoded initial ramdisk or cpio buffer
 	  If unsure, say N.
 
+config RD_ZSTD
+	bool "Support initial ramdisk/ramfs compressed using ZSTD"
+	default y
+	depends on BLK_DEV_INITRD
+	select DECOMPRESS_ZSTD
+	help
+	  Support loading of a ZSTD encoded initial ramdisk or cpio buffer.
+	  If unsure, say N.
+
 choice
 	prompt "Built-in initramfs compression mode"
 	depends on INITRAMFS_SOURCE != ""
@@ -196,6 +205,17 @@  config INITRAMFS_COMPRESSION_LZ4
 	  If you choose this, keep in mind that most distros don't provide lz4
 	  by default which could cause a build failure.
 
+config INITRAMFS_COMPRESSION_ZSTD
+	bool "ZSTD"
+	depends on RD_ZSTD
+	help
+	  ZSTD is a compression algorithm targeting intermediate compression
+	  with fast decompression speed. It will compress better than GZIP and
+	  decompress around the same speed as LZO, but slower than LZ4.
+
+	  If you choose this, keep in mind that you may need to install the zstd
+	  tool to be able to compress the initram.
+
 config INITRAMFS_COMPRESSION_NONE
 	bool "None"
 	help
diff --git a/usr/Makefile b/usr/Makefile
index c12e6b15ce72..b1a81a40eab1 100644
--- a/usr/Makefile
+++ b/usr/Makefile
@@ -15,6 +15,7 @@  compress-$(CONFIG_INITRAMFS_COMPRESSION_LZMA)	:= lzma
 compress-$(CONFIG_INITRAMFS_COMPRESSION_XZ)	:= xzmisc
 compress-$(CONFIG_INITRAMFS_COMPRESSION_LZO)	:= lzo
 compress-$(CONFIG_INITRAMFS_COMPRESSION_LZ4)	:= lz4
+compress-$(CONFIG_INITRAMFS_COMPRESSION_ZSTD)	:= zstd
 
 obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o