mbox series

[GIT,PULL,v6,0/8] Add support for ZSTD-compressed kernel and initramfs

Message ID 20200707034604.1539157-1-nickrterrell@gmail.com
Headers show
Series Add support for ZSTD-compressed kernel and initramfs | expand

Message

Nick Terrell July 7, 2020, 3:45 a.m. UTC
From: Nick Terrell <terrelln@fb.com>

Please pull from

  git@github.com:terrelln/linux.git tags/v6-zstd

to get these changes. Alternatively the patchset is included.

Hi all,

This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
are supported on all architectures. The ZSTD-compressed kernel is only
hooked up to x86 in this patch set.

Zstandard requires slightly more memory during the kernel decompression
on x86 (192 KB vs 64 KB), and the memory usage is independent of the
window size.

Zstandard requires memory proprortional to the window size used during
compression for decompressing the ramdisk image, since streaming mode is
used. Newer versions of zstd (1.3.2+) list the window size of a file
with `zstd -lv <file>'. The absolute maximum amount of memory required
is just over 8 MB, but it can be controlled at compression time.

This patch set has been boot tested with buildroot and QEMU based off
of linux-5.8-rc4.

On i386 and x86_64 I have tested the following configurations:
* zstd compressed kernel and a separate zstd compressed initramfs
* zstd compressed kernel and a built-in zstd compressed initramfs
* gzip compressed kernel and a separate gzip compressed initramfs
* gzip compressed kernel and a built-in gzip compressed initramfs

On arm and aarch64 I tested the same configurations, except that the kernel is
always gzip compressed.

Facebook has been using v1 of these patches on x86_64 devices for more than 6
months. When we switched from a xz compressed initramfs to a zstd compressed
initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
switched from a xz compressed kernel to a zstd compressed kernel we saved 2
seconds of boot time.

Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
When we switched from an lzma compressed initramfs to a zstd compressed initramfs
decompression time shrunk from 27 seconds to 8 seconds.

The zstd compressed kernel is smaller than the gzip compressed kernel but larger
than the xz or lzma compressed kernels, and it decompresses faster than
everything except lz4. See the table below for the measurement of an x86_64
kernel ordered by compressed size:

algo	size
xz  	 6,509,792
lzma	 6,856,576
zstd	 7,399,157
gzip	 8,522,527
bzip	 8,629,603
lzo 	 9,808,035
lz4 	10,705,570
none	32,565,672

Alex Xu ran benchmarks in https://lkml.org/lkml/2020/7/1/722.

v1 -> v2:
- Rebase
  - usr/Makefile and init/Kconfig were changed so the patches were updated
- No functional changes except to rebase
- Split the patches up into smaller chunks

v2 -> v3:
- Add *.zst to the .gitignore in patch 8
- Style nits in patch 3
- Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
  1 through 3

v3 -> v4:
- Increase the ZSTD_IOBUF_SIZE from 4KB to 128KB to improve performance.
  With this change I switch from malloc() to large_malloc() for the
  buffers.
- Increase the maximum allowed window size from 8 MB to 128 MB, which is
  the max that zstd in the kernel supports.

v4 -> v5:
- Update commit message for patch 6 in response to comments
- Rebase onto next-20200408

v5 -> v6:
- Rebase onto v5.8-rc4

Best,
Nick Terrell

Adam Borowski (1):
  .gitignore: add ZSTD-compressed files

Nick Terrell (7):
  lib: prepare zstd for preboot environment
  lib: prepare xxhash for preboot environment
  lib: add zstd support to decompress
  init: add support for zstd compressed kernel
  usr: add support for zstd compressed initramfs
  x86: bump ZO_z_extra_bytes margin for zstd
  x86: Add support for ZSTD compressed kernel

 .gitignore                        |   1 +
 Documentation/x86/boot.rst        |   6 +-
 arch/x86/Kconfig                  |   1 +
 arch/x86/boot/compressed/Makefile |   5 +-
 arch/x86/boot/compressed/misc.c   |   4 +
 arch/x86/boot/header.S            |   8 +-
 arch/x86/include/asm/boot.h       |   6 +-
 include/linux/decompress/unzstd.h |  11 +
 init/Kconfig                      |  15 +-
 lib/Kconfig                       |   4 +
 lib/Makefile                      |   1 +
 lib/decompress.c                  |   5 +
 lib/decompress_unzstd.c           | 342 ++++++++++++++++++++++++++++++
 lib/xxhash.c                      |  21 +-
 lib/zstd/decompress.c             |   2 +
 lib/zstd/fse_decompress.c         |   9 +-
 lib/zstd/zstd_internal.h          |  14 +-
 scripts/Makefile.lib              |  15 ++
 usr/Kconfig                       |  20 ++
 usr/Makefile                      |   1 +
 20 files changed, 464 insertions(+), 27 deletions(-)
 create mode 100644 include/linux/decompress/unzstd.h
 create mode 100644 lib/decompress_unzstd.c

Comments

Sedat Dilek July 7, 2020, 4:03 a.m. UTC | #1
On Tue, Jul 7, 2020 at 5:50 AM Nick Terrell <nickrterrell@gmail.com> wrote:
>
> From: Nick Terrell <terrelln@fb.com>
>
> Please pull from
>
>   git@github.com:terrelln/linux.git tags/v6-zstd
>
> to get these changes. Alternatively the patchset is included.
>

Hi Nick,

cool, I just pulled from their.
Yesterday, I switched over from Linux v5.7.y to v5.8-rc4.
What a good timing :-).

I will report later.

Regards,
- Sedat -

> Hi all,
>
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.
>
> Zstandard requires slightly more memory during the kernel decompression
> on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> window size.
>
> Zstandard requires memory proprortional to the window size used during
> compression for decompressing the ramdisk image, since streaming mode is
> used. Newer versions of zstd (1.3.2+) list the window size of a file
> with `zstd -lv <file>'. The absolute maximum amount of memory required
> is just over 8 MB, but it can be controlled at compression time.
>
> This patch set has been boot tested with buildroot and QEMU based off
> of linux-5.8-rc4.
>
> On i386 and x86_64 I have tested the following configurations:
> * zstd compressed kernel and a separate zstd compressed initramfs
> * zstd compressed kernel and a built-in zstd compressed initramfs
> * gzip compressed kernel and a separate gzip compressed initramfs
> * gzip compressed kernel and a built-in gzip compressed initramfs
>
> On arm and aarch64 I tested the same configurations, except that the kernel is
> always gzip compressed.
>
> Facebook has been using v1 of these patches on x86_64 devices for more than 6
> months. When we switched from a xz compressed initramfs to a zstd compressed
> initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> seconds of boot time.
>
> Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> decompression time shrunk from 27 seconds to 8 seconds.
>
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
>
> algo    size
> xz       6,509,792
> lzma     6,856,576
> zstd     7,399,157
> gzip     8,522,527
> bzip     8,629,603
> lzo      9,808,035
> lz4     10,705,570
> none    32,565,672
>
> Alex Xu ran benchmarks in https://lkml.org/lkml/2020/7/1/722.
>
> v1 -> v2:
> - Rebase
>   - usr/Makefile and init/Kconfig were changed so the patches were updated
> - No functional changes except to rebase
> - Split the patches up into smaller chunks
>
> v2 -> v3:
> - Add *.zst to the .gitignore in patch 8
> - Style nits in patch 3
> - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
>   1 through 3
>
> v3 -> v4:
> - Increase the ZSTD_IOBUF_SIZE from 4KB to 128KB to improve performance.
>   With this change I switch from malloc() to large_malloc() for the
>   buffers.
> - Increase the maximum allowed window size from 8 MB to 128 MB, which is
>   the max that zstd in the kernel supports.
>
> v4 -> v5:
> - Update commit message for patch 6 in response to comments
> - Rebase onto next-20200408
>
> v5 -> v6:
> - Rebase onto v5.8-rc4
>
> Best,
> Nick Terrell
>
> Adam Borowski (1):
>   .gitignore: add ZSTD-compressed files
>
> Nick Terrell (7):
>   lib: prepare zstd for preboot environment
>   lib: prepare xxhash for preboot environment
>   lib: add zstd support to decompress
>   init: add support for zstd compressed kernel
>   usr: add support for zstd compressed initramfs
>   x86: bump ZO_z_extra_bytes margin for zstd
>   x86: Add support for ZSTD compressed kernel
>
>  .gitignore                        |   1 +
>  Documentation/x86/boot.rst        |   6 +-
>  arch/x86/Kconfig                  |   1 +
>  arch/x86/boot/compressed/Makefile |   5 +-
>  arch/x86/boot/compressed/misc.c   |   4 +
>  arch/x86/boot/header.S            |   8 +-
>  arch/x86/include/asm/boot.h       |   6 +-
>  include/linux/decompress/unzstd.h |  11 +
>  init/Kconfig                      |  15 +-
>  lib/Kconfig                       |   4 +
>  lib/Makefile                      |   1 +
>  lib/decompress.c                  |   5 +
>  lib/decompress_unzstd.c           | 342 ++++++++++++++++++++++++++++++
>  lib/xxhash.c                      |  21 +-
>  lib/zstd/decompress.c             |   2 +
>  lib/zstd/fse_decompress.c         |   9 +-
>  lib/zstd/zstd_internal.h          |  14 +-
>  scripts/Makefile.lib              |  15 ++
>  usr/Kconfig                       |  20 ++
>  usr/Makefile                      |   1 +
>  20 files changed, 464 insertions(+), 27 deletions(-)
>  create mode 100644 include/linux/decompress/unzstd.h
>  create mode 100644 lib/decompress_unzstd.c
>
> --
> 2.27.0
>
Sedat Dilek July 7, 2020, 9:01 a.m. UTC | #2
On Tue, Jul 7, 2020 at 6:03 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Tue, Jul 7, 2020 at 5:50 AM Nick Terrell <nickrterrell@gmail.com> wrote:
> >
> > From: Nick Terrell <terrelln@fb.com>
> >
> > Please pull from
> >
> >   git@github.com:terrelln/linux.git tags/v6-zstd
> >
> > to get these changes. Alternatively the patchset is included.
> >
>
> Hi Nick,
>
> cool, I just pulled from their.
> Yesterday, I switched over from Linux v5.7.y to v5.8-rc4.
> What a good timing :-).
>
> I will report later.
>

Looks good.

$ scripts/diffconfig /boot/config-5.8.0-rc4-3-amd64-llvm11-ias
/boot/config-5.8.0-rc4-4-amd64-llvm11-ias
 BUILD_SALT "5.8.0-rc4-3-amd64-llvm11-ias" -> "5.8.0-rc4-4-amd64-llvm11-ias"
 KERNEL_XZ y -> n
+DECOMPRESS_ZSTD y
+HAVE_KERNEL_ZSTD y
+KERNEL_ZSTD y
+RD_ZSTD y

# du -s -k initrd.img-5.8.0-rc4-3-amd64-llvm11-ias
initrd.img-5.8.0-rc4-4-amd64-llvm11-ias
31208   initrd.img-5.8.0-rc4-3-amd64-llvm11-ias <--- GZIP
24768   initrd.img-5.8.0-rc4-4-amd64-llvm11-ias <--- ZSTD

- Sedat -

> Regards,
> - Sedat -
>
> > Hi all,
> >
> > This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> > initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> > are supported on all architectures. The ZSTD-compressed kernel is only
> > hooked up to x86 in this patch set.
> >
> > Zstandard requires slightly more memory during the kernel decompression
> > on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> > window size.
> >
> > Zstandard requires memory proprortional to the window size used during
> > compression for decompressing the ramdisk image, since streaming mode is
> > used. Newer versions of zstd (1.3.2+) list the window size of a file
> > with `zstd -lv <file>'. The absolute maximum amount of memory required
> > is just over 8 MB, but it can be controlled at compression time.
> >
> > This patch set has been boot tested with buildroot and QEMU based off
> > of linux-5.8-rc4.
> >
> > On i386 and x86_64 I have tested the following configurations:
> > * zstd compressed kernel and a separate zstd compressed initramfs
> > * zstd compressed kernel and a built-in zstd compressed initramfs
> > * gzip compressed kernel and a separate gzip compressed initramfs
> > * gzip compressed kernel and a built-in gzip compressed initramfs
> >
> > On arm and aarch64 I tested the same configurations, except that the kernel is
> > always gzip compressed.
> >
> > Facebook has been using v1 of these patches on x86_64 devices for more than 6
> > months. When we switched from a xz compressed initramfs to a zstd compressed
> > initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> > switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> > seconds of boot time.
> >
> > Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> > When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> > decompression time shrunk from 27 seconds to 8 seconds.
> >
> > The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> > than the xz or lzma compressed kernels, and it decompresses faster than
> > everything except lz4. See the table below for the measurement of an x86_64
> > kernel ordered by compressed size:
> >
> > algo    size
> > xz       6,509,792
> > lzma     6,856,576
> > zstd     7,399,157
> > gzip     8,522,527
> > bzip     8,629,603
> > lzo      9,808,035
> > lz4     10,705,570
> > none    32,565,672
> >
> > Alex Xu ran benchmarks in https://lkml.org/lkml/2020/7/1/722.
> >
> > v1 -> v2:
> > - Rebase
> >   - usr/Makefile and init/Kconfig were changed so the patches were updated
> > - No functional changes except to rebase
> > - Split the patches up into smaller chunks
> >
> > v2 -> v3:
> > - Add *.zst to the .gitignore in patch 8
> > - Style nits in patch 3
> > - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
> >   1 through 3
> >
> > v3 -> v4:
> > - Increase the ZSTD_IOBUF_SIZE from 4KB to 128KB to improve performance.
> >   With this change I switch from malloc() to large_malloc() for the
> >   buffers.
> > - Increase the maximum allowed window size from 8 MB to 128 MB, which is
> >   the max that zstd in the kernel supports.
> >
> > v4 -> v5:
> > - Update commit message for patch 6 in response to comments
> > - Rebase onto next-20200408
> >
> > v5 -> v6:
> > - Rebase onto v5.8-rc4
> >
> > Best,
> > Nick Terrell
> >
> > Adam Borowski (1):
> >   .gitignore: add ZSTD-compressed files
> >
> > Nick Terrell (7):
> >   lib: prepare zstd for preboot environment
> >   lib: prepare xxhash for preboot environment
> >   lib: add zstd support to decompress
> >   init: add support for zstd compressed kernel
> >   usr: add support for zstd compressed initramfs
> >   x86: bump ZO_z_extra_bytes margin for zstd
> >   x86: Add support for ZSTD compressed kernel
> >
> >  .gitignore                        |   1 +
> >  Documentation/x86/boot.rst        |   6 +-
> >  arch/x86/Kconfig                  |   1 +
> >  arch/x86/boot/compressed/Makefile |   5 +-
> >  arch/x86/boot/compressed/misc.c   |   4 +
> >  arch/x86/boot/header.S            |   8 +-
> >  arch/x86/include/asm/boot.h       |   6 +-
> >  include/linux/decompress/unzstd.h |  11 +
> >  init/Kconfig                      |  15 +-
> >  lib/Kconfig                       |   4 +
> >  lib/Makefile                      |   1 +
> >  lib/decompress.c                  |   5 +
> >  lib/decompress_unzstd.c           | 342 ++++++++++++++++++++++++++++++
> >  lib/xxhash.c                      |  21 +-
> >  lib/zstd/decompress.c             |   2 +
> >  lib/zstd/fse_decompress.c         |   9 +-
> >  lib/zstd/zstd_internal.h          |  14 +-
> >  scripts/Makefile.lib              |  15 ++
> >  usr/Kconfig                       |  20 ++
> >  usr/Makefile                      |   1 +
> >  20 files changed, 464 insertions(+), 27 deletions(-)
> >  create mode 100644 include/linux/decompress/unzstd.h
> >  create mode 100644 lib/decompress_unzstd.c
> >
> > --
> > 2.27.0
> >
Kees Cook July 7, 2020, 9:32 p.m. UTC | #3
On Mon, Jul 06, 2020 at 08:45:56PM -0700, Nick Terrell wrote:
> From: Nick Terrell <terrelln@fb.com>
> 
> Please pull from
> 
>   git@github.com:terrelln/linux.git tags/v6-zstd
> 
> to get these changes. Alternatively the patchset is included.
> 
> Hi all,
> 
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.

Hello x86 maintainers!

I think this series is ready to go. Notes below...

> [...]
>   x86: bump ZO_z_extra_bytes margin for zstd

The above patch is really the only thing that has any external visibility
to kernels that have ZSTD disabled. Given the ratios of memory sizes
involved (an extra 64K when we're dealing with 2MB windows) seems
reasonable to me. If that isn't acceptable, it should be trivial to make
it CONFIG-selectable (like we already do with BOOT_HEAP_SIZE).

What do you think? If the non-x86 parts should land first in -mm, I
guess that would be okay, but I think it makes sense for all of this to
go via -tip.

-Kees