From patchwork Thu Jul 30 19:08:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693555 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77968138A for ; Thu, 30 Jul 2020 19:11:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A15C206F5 for ; Thu, 30 Jul 2020 19:11:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LNxjwrKZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730411AbgG3TLT (ORCPT ); Thu, 30 Jul 2020 15:11:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730413AbgG3TLT (ORCPT ); Thu, 30 Jul 2020 15:11:19 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61746C061574; Thu, 30 Jul 2020 12:11:19 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id t15so5427440pjq.5; Thu, 30 Jul 2020 12:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M/PjmAninS+4waGJS0k7TVpi9mX9COho3YSX6SxTl+Y=; b=LNxjwrKZpykbDtTMF6elkiLywrLlNwej5RqnOIPN9chduUY1rXrii9ppSPSGNy52sc UaDhUnw2sk2Bc5Vf+FR8lmHPvYASUzSYyDKjLqN07fF3YoDjNslS2OORe7gEuXkorzKo s+dbpIY1gNtoYsB7e6NyqlCpeYKOBvIxwE/X21v7eXTttPPYYjiez5kI/Q4XNZdw89uX c2cTznWG3JvnNhPkXw0jc1lWRQML8E1XDWjbSO+pT5UfWlcTQOzg578qZKqRznc042JJ 4v3kt75lTT4RFTa9oh3Af2pUNhrKgVOEC/A8KohZDvUwVvUy7KPKJ8fykKvzIa0tYoBl HBkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M/PjmAninS+4waGJS0k7TVpi9mX9COho3YSX6SxTl+Y=; b=RfX/H5tRIzf17jfKVDZecL2oFIBpZfWNIaSJTC8iZWYQS5AvRRmQ/kL1gda7oI8gqo Jbuzv7IJ7EZAyn5RZL2jDoba1l6yvS/WDOYxrdXu89vkbIa/UhiMIbBD7LjriU0dlQSO 8y0Qab7miBOF5vshVVvJ7JCY6amaF3lOITWs2CMzpk9aD+Ph7e1BiE3Ec/k8D/YcjK95 P+uANfuK2b9WtkH00S1MD6Tj5fK9b2qbVakl8KVTa5G8infA5SR1OZjTdssTaGaIuP8w cARqi0bu6wFEiiI6SHgT+CSb8ScyrqwwAJjtj26h+I/Fzjc9pZBP+dkusmVuhPlq5Di9 H2yA== X-Gm-Message-State: AOAM531yOx69ZVrIRMOf1nPZx8YfzsbU3eq/SoWPzfX9IeNYRKvddVUx 9v2LBnF5ELVod5tu2/RqwPU= X-Google-Smtp-Source: ABdhPJyGZsKMnMMxhKnxh2QvxabyupjVY6yf71zX4yUtkpn/zeRewwmsW+jD0DfIr8O05892+CBrwg== X-Received: by 2002:a62:84d5:: with SMTP id k204mr420518pfd.66.1596136278844; Thu, 30 Jul 2020 12:11:18 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:18 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 1/8] lib: prepare zstd for preboot environment Date: Thu, 30 Jul 2020 12:08:34 -0700 Message-Id: <20200730190841.2071656-2-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell * Remove a double definition of the CHECK_F macro when the zstd library is amalgamated. * Switch ZSTD_copy8() to __builtin_memcpy(), because in the preboot environment on x86 gcc can't inline `memcpy()` otherwise. * Limit the gcc hack in ZSTD_wildcopy() to the broken gcc version. See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388. These changes are necessary to get the build to work in the preboot environment, and to get reasonable performance. ZSTD_copy8() and ZSTD_wildcopy() are in the core of the zstd hot loop. So outlining these calls to memcpy(), and having an extra branch are very detrimental to performance. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- lib/zstd/fse_decompress.c | 9 +-------- lib/zstd/zstd_internal.h | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/lib/zstd/fse_decompress.c b/lib/zstd/fse_decompress.c index a84300e5a013..0b353530fb3f 100644 --- a/lib/zstd/fse_decompress.c +++ b/lib/zstd/fse_decompress.c @@ -47,6 +47,7 @@ ****************************************************************/ #include "bitstream.h" #include "fse.h" +#include "zstd_internal.h" #include #include #include /* memcpy, memset */ @@ -60,14 +61,6 @@ enum { FSE_static_assert = 1 / (int)(!!(c)) }; \ } /* use only *after* variable declarations */ -/* check and forward error code */ -#define CHECK_F(f) \ - { \ - size_t const e = f; \ - if (FSE_isError(e)) \ - return e; \ - } - /* ************************************************************** * Templates ****************************************************************/ diff --git a/lib/zstd/zstd_internal.h b/lib/zstd/zstd_internal.h index 1a79fab9e13a..dac753397f86 100644 --- a/lib/zstd/zstd_internal.h +++ b/lib/zstd/zstd_internal.h @@ -127,7 +127,14 @@ static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG; * Shared functions to include for inlining *********************************************/ ZSTD_STATIC void ZSTD_copy8(void *dst, const void *src) { - memcpy(dst, src, 8); + /* + * zstd relies heavily on gcc being able to analyze and inline this + * memcpy() call, since it is called in a tight loop. Preboot mode + * is compiled in freestanding mode, which stops gcc from analyzing + * memcpy(). Use __builtin_memcpy() to tell gcc to analyze this as a + * regular memcpy(). + */ + __builtin_memcpy(dst, src, 8); } /*! ZSTD_wildcopy() : * custom version of memcpy(), can copy up to 7 bytes too many (8 bytes if length==0) */ @@ -137,13 +144,16 @@ ZSTD_STATIC void ZSTD_wildcopy(void *dst, const void *src, ptrdiff_t length) const BYTE* ip = (const BYTE*)src; BYTE* op = (BYTE*)dst; BYTE* const oend = op + length; - /* Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388. +#if defined(GCC_VERSION) && GCC_VERSION >= 70000 && GCC_VERSION < 70200 + /* + * Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388. * Avoid the bad case where the loop only runs once by handling the * special case separately. This doesn't trigger the bug because it * doesn't involve pointer/integer overflow. */ if (length <= 8) return ZSTD_copy8(dst, src); +#endif do { ZSTD_copy8(op, ip); op += 8; From patchwork Thu Jul 30 19:08:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693557 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D945912 for ; Thu, 30 Jul 2020 19:11:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6C07121883 for ; Thu, 30 Jul 2020 19:11:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ii1liHGj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730454AbgG3TLY (ORCPT ); Thu, 30 Jul 2020 15:11:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730413AbgG3TLX (ORCPT ); Thu, 30 Jul 2020 15:11:23 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AF8FC061574; Thu, 30 Jul 2020 12:11:23 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id q17so15020086pls.9; Thu, 30 Jul 2020 12:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Kqo3YaFETXJxqVXUPh2pgTsm/1m3Qevrm7fbVjDuWew=; b=ii1liHGjVmtc7z5vtU9iT3jF//xnURDEHQQcNzIlTKy38USjqvOjUIBurlYVBi9bGY bb9deU4KnI8/sDZ3SmIG5RpCOdPGW28YMIOGSFCfH20i+ElYY5o97fSH24paEy6Yt5WC uMQ5XRhATwxfWWRG9Ll/h0JK3vQNSbDA7EB0kB88xRoW4eFNFroCVOIU8tBi9lU5sA// j1fw8D1RbM48RsHdXXrL2LuJz+fuFJWqE4s76F/gpA2O/rFYygzyCg63vAyW+d97eDyJ Raoc8GZAY0eNxhRT+z0Tx0pG7Y0CEhmyQplR0bHVheRKkp2/UdmLbqqYWw3S3/zq1ZdO Xt+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Kqo3YaFETXJxqVXUPh2pgTsm/1m3Qevrm7fbVjDuWew=; b=kd0N6B2e1LsMIMxzFZGxSFvI0ldoKZhDqTu3B62UfV99ig5lcYqU6RSlCZy+vJLXD3 n0+K7GGcPb+zvP1rvk4T/hlESBd9xkB46sErLqo7w1kXK34iD2fGZ89ztrlTYNpKvmqY qAbndtsASbxJ6+xS+tBPpTDtIWdIz8H6a3iw+Yt+rnm3zvmaTOhiy9tjkwrp9GCFCcek 2Mo0XM0Mjs02e0Wu3hNRzLhUJ8//k+oqgSixppXiE2AFgK/gn4/kbycIfJbY+m3dDGMa yNW16BfXJsZqZJp9H0Zcj+TlX1q8b2fzGblkV/sMY1XlnICYLpDindvFZkNf8dznsEWK W7BQ== X-Gm-Message-State: AOAM532faZi6Ac2qtJPQlW4GPWyWhneYpSlDzutb1YZuOMhT0Cvqd5GD 4g8spT3SOOi0VlJnmyms0Xc= X-Google-Smtp-Source: ABdhPJyX/NKlTjT34YyHfx6cTzqpwFdRO/9iBowpwHE1Tt4yCXUWYgzdg9fja3g5OdyvSpXP5mVCww== X-Received: by 2002:a17:902:6b45:: with SMTP id g5mr634460plt.42.1596136282347; Thu, 30 Jul 2020 12:11:22 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:21 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 2/8] lib: add zstd support to decompress Date: Thu, 30 Jul 2020 12:08:35 -0700 Message-Id: <20200730190841.2071656-3-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell * Add unzstd() and the zstd decompress interface. * Add zstd support to decompress_method(). The decompress_method() and unzstd() functions are used to decompress the initramfs and the initrd. The __decompress() function is used in the preboot environment to decompress a zstd compressed kernel. The zstd decompression function allows the input and output buffers to overlap because that is used by x86 kernel decompression. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- include/linux/decompress/unzstd.h | 11 + lib/Kconfig | 4 + lib/Makefile | 1 + lib/decompress.c | 5 + lib/decompress_unzstd.c | 345 ++++++++++++++++++++++++++++++ 5 files changed, 366 insertions(+) create mode 100644 include/linux/decompress/unzstd.h create mode 100644 lib/decompress_unzstd.c diff --git a/include/linux/decompress/unzstd.h b/include/linux/decompress/unzstd.h new file mode 100644 index 000000000000..56d539ae880f --- /dev/null +++ b/include/linux/decompress/unzstd.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef LINUX_DECOMPRESS_UNZSTD_H +#define LINUX_DECOMPRESS_UNZSTD_H + +int unzstd(unsigned char *inbuf, long len, + long (*fill)(void*, unsigned long), + long (*flush)(void*, unsigned long), + unsigned char *output, + long *pos, + void (*error_fn)(char *x)); +#endif diff --git a/lib/Kconfig b/lib/Kconfig index df3f3da95990..a5d6f23c4cab 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -342,6 +342,10 @@ config DECOMPRESS_LZ4 select LZ4_DECOMPRESS tristate +config DECOMPRESS_ZSTD + select ZSTD_DECOMPRESS + tristate + # # Generic allocator support is selected if needed # diff --git a/lib/Makefile b/lib/Makefile index b1c42c10073b..2ba9642a3a87 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -170,6 +170,7 @@ lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o lib-$(CONFIG_DECOMPRESS_XZ) += decompress_unxz.o lib-$(CONFIG_DECOMPRESS_LZO) += decompress_unlzo.o lib-$(CONFIG_DECOMPRESS_LZ4) += decompress_unlz4.o +lib-$(CONFIG_DECOMPRESS_ZSTD) += decompress_unzstd.o obj-$(CONFIG_TEXTSEARCH) += textsearch.o obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o diff --git a/lib/decompress.c b/lib/decompress.c index 857ab1af1ef3..ab3fc90ffc64 100644 --- a/lib/decompress.c +++ b/lib/decompress.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include @@ -37,6 +38,9 @@ #ifndef CONFIG_DECOMPRESS_LZ4 # define unlz4 NULL #endif +#ifndef CONFIG_DECOMPRESS_ZSTD +# define unzstd NULL +#endif struct compress_format { unsigned char magic[2]; @@ -52,6 +56,7 @@ static const struct compress_format compressed_formats[] __initconst = { { {0xfd, 0x37}, "xz", unxz }, { {0x89, 0x4c}, "lzo", unlzo }, { {0x02, 0x21}, "lz4", unlz4 }, + { {0x28, 0xb5}, "zstd", unzstd }, { {0, 0}, NULL, NULL } }; diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c new file mode 100644 index 000000000000..0ad2c15479ed --- /dev/null +++ b/lib/decompress_unzstd.c @@ -0,0 +1,345 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Important notes about in-place decompression + * + * At least on x86, the kernel is decompressed in place: the compressed data + * is placed to the end of the output buffer, and the decompressor overwrites + * most of the compressed data. There must be enough safety margin to + * guarantee that the write position is always behind the read position. + * + * The safety margin for ZSTD with a 128 KB block size is calculated below. + * Note that the margin with ZSTD is bigger than with GZIP or XZ! + * + * The worst case for in-place decompression is that the beginning of + * the file is compressed extremely well, and the rest of the file is + * uncompressible. Thus, we must look for worst-case expansion when the + * compressor is encoding uncompressible data. + * + * The structure of the .zst file in case of a compresed kernel is as follows. + * Maximum sizes (as bytes) of the fields are in parenthesis. + * + * Frame Header: (18) + * Blocks: (N) + * Checksum: (4) + * + * The frame header and checksum overhead is at most 22 bytes. + * + * ZSTD stores the data in blocks. Each block has a header whose size is + * a 3 bytes. After the block header, there is up to 128 KB of payload. + * The maximum uncompressed size of the payload is 128 KB. The minimum + * uncompressed size of the payload is never less than the payload size + * (excluding the block header). + * + * The assumption, that the uncompressed size of the payload is never + * smaller than the payload itself, is valid only when talking about + * the payload as a whole. It is possible that the payload has parts where + * the decompressor consumes more input than it produces output. Calculating + * the worst case for this would be tricky. Instead of trying to do that, + * let's simply make sure that the decompressor never overwrites any bytes + * of the payload which it is currently reading. + * + * Now we have enough information to calculate the safety margin. We need + * - 22 bytes for the .zst file format headers; + * - 3 bytes per every 128 KiB of uncompressed size (one block header per + * block); and + * - 128 KiB (biggest possible zstd block size) to make sure that the + * decompressor never overwrites anything from the block it is currently + * reading. + * + * We get the following formula: + * + * safety_margin = 22 + uncompressed_size * 3 / 131072 + 131072 + * <= 22 + (uncompressed_size >> 15) + 131072 + */ + +/* + * Preboot environments #include "path/to/decompress_unzstd.c". + * All of the source files we depend on must be #included. + * zstd's only source dependeny is xxhash, which has no source + * dependencies. + * + * When UNZSTD_PREBOOT is defined we declare __decompress(), which is + * used for kernel decompression, instead of unzstd(). + * + * Define __DISABLE_EXPORTS in preboot environments to prevent symbols + * from xxhash and zstd from being exported by the EXPORT_SYMBOL macro. + */ +#ifdef STATIC +# define UNZSTD_PREBOOT +# include "xxhash.c" +# include "zstd/entropy_common.c" +# include "zstd/fse_decompress.c" +# include "zstd/huf_decompress.c" +# include "zstd/zstd_common.c" +# include "zstd/decompress.c" +#endif + +#include +#include +#include + +/* 128MB is the maximum window size supported by zstd. */ +#define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX) +/* + * Size of the input and output buffers in multi-call mode. + * Pick a larger size because it isn't used during kernel decompression, + * since that is single pass, and we have to allocate a large buffer for + * zstd's window anyway. The larger size speeds up initramfs decompression. + */ +#define ZSTD_IOBUF_SIZE (1 << 17) + +static int INIT handle_zstd_error(size_t ret, void (*error)(char *x)) +{ + const int err = ZSTD_getErrorCode(ret); + + if (!ZSTD_isError(ret)) + return 0; + + switch (err) { + case ZSTD_error_memory_allocation: + error("ZSTD decompressor ran out of memory"); + break; + case ZSTD_error_prefix_unknown: + error("Input is not in the ZSTD format (wrong magic bytes)"); + break; + case ZSTD_error_dstSize_tooSmall: + case ZSTD_error_corruption_detected: + case ZSTD_error_checksum_wrong: + error("ZSTD-compressed data is corrupt"); + break; + default: + error("ZSTD-compressed data is probably corrupt"); + break; + } + return -1; +} + +/* + * Handle the case where we have the entire input and output in one segment. + * We can allocate less memory (no circular buffer for the sliding window), + * and avoid some memcpy() calls. + */ +static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf, + long out_len, long *in_pos, + void (*error)(char *x)) +{ + const size_t wksp_size = ZSTD_DCtxWorkspaceBound(); + void *wksp = large_malloc(wksp_size); + ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size); + int err; + size_t ret; + + if (dctx == NULL) { + error("Out of memory while allocating ZSTD_DCtx"); + err = -1; + goto out; + } + /* + * Find out how large the frame actually is, there may be junk at + * the end of the frame that ZSTD_decompressDCtx() can't handle. + */ + ret = ZSTD_findFrameCompressedSize(in_buf, in_len); + err = handle_zstd_error(ret, error); + if (err) + goto out; + in_len = (long)ret; + + ret = ZSTD_decompressDCtx(dctx, out_buf, out_len, in_buf, in_len); + err = handle_zstd_error(ret, error); + if (err) + goto out; + + if (in_pos != NULL) + *in_pos = in_len; + + err = 0; +out: + if (wksp != NULL) + large_free(wksp); + return err; +} + +static int INIT __unzstd(unsigned char *in_buf, long in_len, + long (*fill)(void*, unsigned long), + long (*flush)(void*, unsigned long), + unsigned char *out_buf, long out_len, + long *in_pos, + void (*error)(char *x)) +{ + ZSTD_inBuffer in; + ZSTD_outBuffer out; + ZSTD_frameParams params; + void *in_allocated = NULL; + void *out_allocated = NULL; + void *wksp = NULL; + size_t wksp_size; + ZSTD_DStream *dstream; + int err; + size_t ret; + + if (out_len == 0) + out_len = LONG_MAX; /* no limit */ + + if (fill == NULL && flush == NULL) + /* + * We can decompress faster and with less memory when we have a + * single chunk. + */ + return decompress_single(in_buf, in_len, out_buf, out_len, + in_pos, error); + + /* + * If in_buf is not provided, we must be using fill(), so allocate + * a large enough buffer. If it is provided, it must be at least + * ZSTD_IOBUF_SIZE large. + */ + if (in_buf == NULL) { + in_allocated = large_malloc(ZSTD_IOBUF_SIZE); + if (in_allocated == NULL) { + error("Out of memory while allocating input buffer"); + err = -1; + goto out; + } + in_buf = in_allocated; + in_len = 0; + } + /* Read the first chunk, since we need to decode the frame header. */ + if (fill != NULL) + in_len = fill(in_buf, ZSTD_IOBUF_SIZE); + if (in_len < 0) { + error("ZSTD-compressed data is truncated"); + err = -1; + goto out; + } + /* Set the first non-empty input buffer. */ + in.src = in_buf; + in.pos = 0; + in.size = in_len; + /* Allocate the output buffer if we are using flush(). */ + if (flush != NULL) { + out_allocated = large_malloc(ZSTD_IOBUF_SIZE); + if (out_allocated == NULL) { + error("Out of memory while allocating output buffer"); + err = -1; + goto out; + } + out_buf = out_allocated; + out_len = ZSTD_IOBUF_SIZE; + } + /* Set the output buffer. */ + out.dst = out_buf; + out.pos = 0; + out.size = out_len; + + /* + * We need to know the window size to allocate the ZSTD_DStream. + * Since we are streaming, we need to allocate a buffer for the sliding + * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX + * (8 MB), so it is important to use the actual value so as not to + * waste memory when it is smaller. + */ + ret = ZSTD_getFrameParams(¶ms, in.src, in.size); + err = handle_zstd_error(ret, error); + if (err) + goto out; + if (ret != 0) { + error("ZSTD-compressed data has an incomplete frame header"); + err = -1; + goto out; + } + if (params.windowSize > ZSTD_WINDOWSIZE_MAX) { + error("ZSTD-compressed data has too large a window size"); + err = -1; + goto out; + } + + /* + * Allocate the ZSTD_DStream now that we know how much memory is + * required. + */ + wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize); + wksp = large_malloc(wksp_size); + dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size); + if (dstream == NULL) { + error("Out of memory while allocating ZSTD_DStream"); + err = -1; + goto out; + } + + /* + * Decompression loop: + * Read more data if necessary (error if no more data can be read). + * Call the decompression function, which returns 0 when finished. + * Flush any data produced if using flush(). + */ + if (in_pos != NULL) + *in_pos = 0; + do { + /* + * If we need to reload data, either we have fill() and can + * try to get more data, or we don't and the input is truncated. + */ + if (in.pos == in.size) { + if (in_pos != NULL) + *in_pos += in.pos; + in_len = fill ? fill(in_buf, ZSTD_IOBUF_SIZE) : -1; + if (in_len < 0) { + error("ZSTD-compressed data is truncated"); + err = -1; + goto out; + } + in.pos = 0; + in.size = in_len; + } + /* Returns zero when the frame is complete. */ + ret = ZSTD_decompressStream(dstream, &out, &in); + err = handle_zstd_error(ret, error); + if (err) + goto out; + /* Flush all of the data produced if using flush(). */ + if (flush != NULL && out.pos > 0) { + if (out.pos != flush(out.dst, out.pos)) { + error("Failed to flush()"); + err = -1; + goto out; + } + out.pos = 0; + } + } while (ret != 0); + + if (in_pos != NULL) + *in_pos += in.pos; + + err = 0; +out: + if (in_allocated != NULL) + large_free(in_allocated); + if (out_allocated != NULL) + large_free(out_allocated); + if (wksp != NULL) + large_free(wksp); + return err; +} + +#ifndef UNZSTD_PREBOOT +STATIC int INIT unzstd(unsigned char *buf, long len, + long (*fill)(void*, unsigned long), + long (*flush)(void*, unsigned long), + unsigned char *out_buf, + long *pos, + void (*error)(char *x)) +{ + return __unzstd(buf, len, fill, flush, out_buf, 0, pos, error); +} +#else +STATIC int INIT __decompress(unsigned char *buf, long len, + long (*fill)(void*, unsigned long), + long (*flush)(void*, unsigned long), + unsigned char *out_buf, long out_len, + long *pos, + void (*error)(char *x)) +{ + return __unzstd(buf, len, fill, flush, out_buf, out_len, pos, error); +} +#endif From patchwork Thu Jul 30 19:08:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693559 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4927B912 for ; Thu, 30 Jul 2020 19:11:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CB9A2084D for ; Thu, 30 Jul 2020 19:11:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Egm1hQ44" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730466AbgG3TL1 (ORCPT ); Thu, 30 Jul 2020 15:11:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730465AbgG3TL0 (ORCPT ); Thu, 30 Jul 2020 15:11:26 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8841EC061574; Thu, 30 Jul 2020 12:11:26 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id p1so15034772pls.4; Thu, 30 Jul 2020 12:11:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uzi0WeuypATsmeWiS85QI65j2KXCUQ4AAy0p8hgi3lg=; b=Egm1hQ444fzjJK7rCeONLMT5TbJjgu/ukR64wqV0nx/AxbavjSIl/NlawU2GU7MnoI zosm43ntOwVMDTN9E1CGXeuMk+OsACupxiqBooY3XwazByX7PJ9rcU80JRL+/g4mJ6xu 5v/NZdN6Y7a+p2BwvIXcjkoOyVuERET5sdpw37Y2vx4JvjwbJLizfH4FjVHb46NELmeS w/cjUuGzvYLowdFAAhEL1QyOMh0QXQskuOVGU1yE0Wyq9K8HJzdh9Or9CivHbvGm/Gik 69e+hKUvXKJF1VleTCeLTPv08YMjKwI+xY/NliXunYXSIcI5gdur9zfHSak8kZ/JCx9E /pZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uzi0WeuypATsmeWiS85QI65j2KXCUQ4AAy0p8hgi3lg=; b=E2kjAxhiD556gbhCQzgTh2XqyjXr/qsHyqrPB4w80Cj1Dlppm6O8T8dD3k2LwHExEl YPDBXVvy6pTGzH5A7kxURN8/22UJmLSx44VtV0L6jEAV/87jevX8H/HQzpRy8LOyS+CG 11SA+8EWJyDXcLLe4zXbZO+jLTzvQlaMgzHhWi/b5d5VczTt0bDk4wCQcWuzOCGhNIgF 7JJqUOqeYcHeqwowJXhM3vw2ACzYkZyVEw5Ng+oQDwRGXofW+N/ZIwq/C3+S+IaTEvfk FZScotvG4KTPA+aGb19EKfDRJTVRdmgh4RBRy5pelq3JuTKfpmjqMasXuwenw2VpgQnH /v/Q== X-Gm-Message-State: AOAM5330/wt+w6FIbI1iJAUDguHp0IRZf+IWf5e3zHNWibXFZZjPr01m dIcVFTVRS2qII7IFs7jkQos= X-Google-Smtp-Source: ABdhPJyocbRoEDo9xHBCd6nsKyERitAM37A0W4anwAeGC3H4yAogMCDBANiSc9iW3AWUaPfKHs1zcA== X-Received: by 2002:a17:90a:110:: with SMTP id b16mr511953pjb.235.1596136286039; Thu, 30 Jul 2020 12:11:26 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:25 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 3/8] init: add support for zstd compressed kernel Date: Thu, 30 Jul 2020 12:08:36 -0700 Message-Id: <20200730190841.2071656-4-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell * Adds the zstd and zstd22 cmds to scripts/Makefile.lib * Adds the HAVE_KERNEL_ZSTD and KERNEL_ZSTD options Architecture specific support is still needed for decompression. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- Makefile | 3 ++- init/Kconfig | 15 ++++++++++++++- scripts/Makefile.lib | 22 ++++++++++++++++++++++ 3 files changed, 38 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 229e67f2ff75..565084f347bd 100644 --- a/Makefile +++ b/Makefile @@ -464,6 +464,7 @@ KLZOP = lzop LZMA = lzma LZ4 = lz4c XZ = xz +ZSTD = zstd CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF) @@ -512,7 +513,7 @@ CLANG_FLAGS := export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE LD CC export CPP AR NM STRIP OBJCOPY OBJDUMP OBJSIZE READELF PAHOLE LEX YACC AWK INSTALLKERNEL export PERL PYTHON PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX -export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ +export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ ZSTD export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS LDFLAGS_MODULE export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS KBUILD_LDFLAGS diff --git a/init/Kconfig b/init/Kconfig index 0498af567f70..2b6409fec53f 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -191,13 +191,16 @@ config HAVE_KERNEL_LZO config HAVE_KERNEL_LZ4 bool +config HAVE_KERNEL_ZSTD + bool + config HAVE_KERNEL_UNCOMPRESSED bool choice prompt "Kernel compression mode" default KERNEL_GZIP - depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 || HAVE_KERNEL_UNCOMPRESSED + depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4 || HAVE_KERNEL_ZSTD || HAVE_KERNEL_UNCOMPRESSED help The linux kernel is a kind of self-extracting executable. Several compression algorithms are available, which differ @@ -276,6 +279,16 @@ config KERNEL_LZ4 is about 8% bigger than LZO. But the decompression speed is faster than LZO. +config KERNEL_ZSTD + bool "ZSTD" + depends on HAVE_KERNEL_ZSTD + help + ZSTD is a compression algorithm targeting intermediate compression + with fast decompression speed. It will compress better than GZIP and + decompress around the same speed as LZO, but slower than LZ4. You + will need at least 192 KB RAM or more for booting. The zstd command + line tool is required for compression. + config KERNEL_UNCOMPRESSED bool "None" depends on HAVE_KERNEL_UNCOMPRESSED diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 916b2f7f7098..54f7b7eb580b 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -413,6 +413,28 @@ quiet_cmd_xzkern = XZKERN $@ quiet_cmd_xzmisc = XZMISC $@ cmd_xzmisc = cat $(real-prereqs) | $(XZ) --check=crc32 --lzma2=dict=1MiB > $@ +# ZSTD +# --------------------------------------------------------------------------- +# Appends the uncompressed size of the data using size_append. The .zst +# format has the size information available at the beginning of the file too, +# but it's in a more complex format and it's good to avoid changing the part +# of the boot code that reads the uncompressed size. +# +# Note that the bytes added by size_append will make the zstd tool think that +# the file is corrupt. This is expected. +# +# zstd uses a maximum window size of 8 MB. zstd22 uses a maximum window size of +# 128 MB. zstd22 is used for kernel compression because it is decompressed in a +# single pass, so zstd doesn't need to allocate a window buffer. When streaming +# decompression is used, like initramfs decompression, zstd22 should likely not +# be used because it would require zstd to allocate a 128 MB buffer. + +quiet_cmd_zstd = ZSTD $@ + cmd_zstd = { cat $(real-prereqs) | $(ZSTD) -19; $(size_append); } > $@ + +quiet_cmd_zstd22 = ZSTD22 $@ + cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@ + # ASM offsets # --------------------------------------------------------------------------- From patchwork Thu Jul 30 19:08:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693561 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E80C4912 for ; Thu, 30 Jul 2020 19:11:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D04812084D for ; Thu, 30 Jul 2020 19:11:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OYeysaA5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730465AbgG3TLb (ORCPT ); Thu, 30 Jul 2020 15:11:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730444AbgG3TLa (ORCPT ); Thu, 30 Jul 2020 15:11:30 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABE63C061574; Thu, 30 Jul 2020 12:11:30 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id ha11so5437714pjb.1; Thu, 30 Jul 2020 12:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fmIcvsol2hVEYD4hslBcgfIPlSJK4NAnFqyLqCM1gdk=; b=OYeysaA5vFkzYebKJCg+7lQgoWOFfnVI8qfBmxjzSgbrPdZlPs+z2ksxmwX26xsYnc zSWS1H+spEaGmRGX2HNqeZF3iornchIkd+7WkCoP9XL7Q6F+ji+Qq7LFIhhaiuwp/MoG 2nXLpEHp8m69FULHvHS2JOEN8shBNYIaMHm+6A/4oAv89gdaeI3Y6HVtNvr1fwxHfKiR upEP9svLS3lV4uDlHBXrY8vQH6LWrp0m4v3XXvNT4Kg0viM8+zeDb7cgvKIwfJC1GvmM lQuRpxMnnQKj73J1S4cUmEbjZDmRcxwDPMqo9YzdY/iDZH2jjV7eg5SGAX17bnx9M45s CGXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fmIcvsol2hVEYD4hslBcgfIPlSJK4NAnFqyLqCM1gdk=; b=F++8V7NVT39sPq+QctanlAsmQXlJyACnQijGdJSZSEOg0hsD9nl1QyGnStds+Os8LB rSK9aRCoWf0IB4LwCWpTylQodaIo0LV0KkcFeGvAnr7TWpozDqayiB/jmSlKXW93No5q QBXWSzwxUMOTaObfgtcJ3wh8X5k0m0iH95qw17ODKTE4p2a2edSFnBBbElhAWxev4Y8n kjG31723ynttVrgM14hWG1yrMgvm76kQ6tClQNonn/hP9JIZGPE3tfYQ/fbBs8rs/l/Y Tryomtm900CE09fFGodXzx7pqc8wprXXY9nsM0h7CAhySeTSVRTv/3/hbT7f6Msa/HCl OgDQ== X-Gm-Message-State: AOAM532rwaRm4OWb7cT1LuaJIPeBGut5Po/sYH9U0gEHz6JisHVzTzV1 /9ERpeSP0cnYUujtB/ozW+o= X-Google-Smtp-Source: ABdhPJxRJeItJ5swbNPdsSfqu+8axOrjQ8VxVtnh0EXmjqeV+W+ckMRslnabITdfqF0FeK0Eku9EsQ== X-Received: by 2002:a17:90a:8543:: with SMTP id a3mr569088pjw.31.1596136289012; Thu, 30 Jul 2020 12:11:29 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:28 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 4/8] usr: add support for zstd compressed initramfs Date: Thu, 30 Jul 2020 12:08:37 -0700 Message-Id: <20200730190841.2071656-5-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell * Add support for a zstd compressed initramfs. * Add compression for compressing built-in initramfs with zstd. I have tested this patch by boot testing with buildroot and QEMU. Specifically, I booted the kernel with both a zstd and gzip compressed initramfs, both built into the kernel and separate. I ensured that the correct compression algorithm was used. I tested on arm, aarch64, i386, and x86_64. This patch has been tested in production on aarch64 and x86_64 devices. Additionally, I have performance measurements from internal use in production. On an aarch64 device we saw 19 second boot time improvement from switching from lzma to zstd (27 seconds to 8 seconds). On an x86_64 device we saw a 9 second boot time reduction from switching from xz to zstd. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell Reviewed-by: Sedat Dilek --- usr/Kconfig | 20 ++++++++++++++++++++ usr/Makefile | 1 + 2 files changed, 21 insertions(+) diff --git a/usr/Kconfig b/usr/Kconfig index 96afb03b65f9..2599bc21c1b2 100644 --- a/usr/Kconfig +++ b/usr/Kconfig @@ -100,6 +100,15 @@ config RD_LZ4 Support loading of a LZ4 encoded initial ramdisk or cpio buffer If unsure, say N. +config RD_ZSTD + bool "Support initial ramdisk/ramfs compressed using ZSTD" + default y + depends on BLK_DEV_INITRD + select DECOMPRESS_ZSTD + help + Support loading of a ZSTD encoded initial ramdisk or cpio buffer. + If unsure, say N. + choice prompt "Built-in initramfs compression mode" depends on INITRAMFS_SOURCE != "" @@ -196,6 +205,17 @@ config INITRAMFS_COMPRESSION_LZ4 If you choose this, keep in mind that most distros don't provide lz4 by default which could cause a build failure. +config INITRAMFS_COMPRESSION_ZSTD + bool "ZSTD" + depends on RD_ZSTD + help + ZSTD is a compression algorithm targeting intermediate compression + with fast decompression speed. It will compress better than GZIP and + decompress around the same speed as LZO, but slower than LZ4. + + If you choose this, keep in mind that you may need to install the zstd + tool to be able to compress the initram. + config INITRAMFS_COMPRESSION_NONE bool "None" help diff --git a/usr/Makefile b/usr/Makefile index c12e6b15ce72..b1a81a40eab1 100644 --- a/usr/Makefile +++ b/usr/Makefile @@ -15,6 +15,7 @@ compress-$(CONFIG_INITRAMFS_COMPRESSION_LZMA) := lzma compress-$(CONFIG_INITRAMFS_COMPRESSION_XZ) := xzmisc compress-$(CONFIG_INITRAMFS_COMPRESSION_LZO) := lzo compress-$(CONFIG_INITRAMFS_COMPRESSION_LZ4) := lz4 +compress-$(CONFIG_INITRAMFS_COMPRESSION_ZSTD) := zstd obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o From patchwork Thu Jul 30 19:08:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693563 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAEF6912 for ; Thu, 30 Jul 2020 19:11:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9351F22BF5 for ; Thu, 30 Jul 2020 19:11:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BaU8IpH/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730486AbgG3TLf (ORCPT ); Thu, 30 Jul 2020 15:11:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730444AbgG3TLe (ORCPT ); Thu, 30 Jul 2020 15:11:34 -0400 Received: from mail-pj1-x1044.google.com (mail-pj1-x1044.google.com [IPv6:2607:f8b0:4864:20::1044]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95BD3C061574; Thu, 30 Jul 2020 12:11:32 -0700 (PDT) Received: by mail-pj1-x1044.google.com with SMTP id f9so5429163pju.4; Thu, 30 Jul 2020 12:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n6QCGzWBzwdRunC88ZR2ssAXgvTx6O91V0TzBT8uQq0=; b=BaU8IpH/hEJ0Bt6I9r0CnbjI95drWEkwS26FyN9SQ1C0SLb+l/GctZKrp+6iAniMJ/ V8IEULsf6CXvq+y2jGJ40+COD9TBdDmqEpVcCPrrq8syEAsFnIadi+22m87xWSRDyKB1 uiZTPD3w7Nu0x3me1WmwN3q3qadMKKRtCBd7dFCzAmRcuVsdhaTg3ML5fc2M306y2Grx WQMiuvGYA61ahT0KfWdswFR/IC13gZbg1FAmwOFP/610ojzXQKvffSQqP21jbW0jd24u T/Khm1jATVxJv3+6lsuzrV/S3X/veq5yo285bZq+RTV4Y7SP67+G942fjSVvlvxwOH1U m/wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n6QCGzWBzwdRunC88ZR2ssAXgvTx6O91V0TzBT8uQq0=; b=hLiZTMdpxTZ2Ics7lUqaq6EADBg/mUvqJHKpRtyxY5QjeNefUgtaDRbxzplO5QfhZz FgGVPFEaJOooQBe5BjxTHTqvRPQuQfTyEmP5OJ5lekMeHj93Lv32kmRWCWdfD33te0Vd BDNT74GqB2Ry/HHcuL55FkBqIJJOg7aV7OKFO1FYjpLP3MmJGK8bpsJGzM4Nu0zUZ/SL IB7PIZnsojm1uqesHe/5GtGP7W5XJ/vOvqaxLqIL6MV+VjyySMGAMXqx4ajYXR+PDb2L 2NCWoYghBxvEt293LeD20MaJKIgjt2+2AyrIfU/cL0E1O4BsVD3MDIwiGKv1BcH9wcmx Nckw== X-Gm-Message-State: AOAM5305Vs5yhNghjvwVrDVfMpz6h/GAXZmChKgMiH288/rQhuhtNMRl 82yLmdCGP16xRLJENQvupOE= X-Google-Smtp-Source: ABdhPJwm/oxQy2a4+WOCoFsWEMnAGdqfL644K97nSedNa+5kBmRN+lfDPnrOEYaqPuy6ibdgdZHvpA== X-Received: by 2002:a65:6212:: with SMTP id d18mr315986pgv.402.1596136292141; Thu, 30 Jul 2020 12:11:32 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:31 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 5/8] x86: bump ZO_z_extra_bytes margin for zstd Date: Thu, 30 Jul 2020 12:08:38 -0700 Message-Id: <20200730190841.2071656-6-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell Bump the ZO_z_extra_bytes margin for zstd. Zstd needs 3 bytes per 128 KB, and has a 22 byte fixed overhead. Zstd needs to maintain 128 KB of space at all times, since that is the maximum block size. See the comments regarding in-place decompression added in lib/decompress_unzstd.c for details. The existing code is written so that all the compression algorithms use the same ZO_z_extra_bytes. It is taken to be the maximum of the growth rate plus the maximum fixed overhead. The comments just above this diff state that: Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- arch/x86/boot/header.S | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S index 735ad7f21ab0..6dbd7e9f74c9 100644 --- a/arch/x86/boot/header.S +++ b/arch/x86/boot/header.S @@ -539,8 +539,14 @@ pref_address: .quad LOAD_PHYSICAL_ADDR # preferred load addr # the size-dependent part now grows so fast. # # extra_bytes = (uncompressed_size >> 8) + 65536 +# +# ZSTD compressed data grows by at most 3 bytes per 128K, and only has a 22 +# byte fixed overhead but has a maximum block size of 128K, so it needs a +# larger margin. +# +# extra_bytes = (uncompressed_size >> 8) + 131072 -#define ZO_z_extra_bytes ((ZO_z_output_len >> 8) + 65536) +#define ZO_z_extra_bytes ((ZO_z_output_len >> 8) + 131072) #if ZO_z_output_len > ZO_z_input_len # define ZO_z_extract_offset (ZO_z_output_len + ZO_z_extra_bytes - \ ZO_z_input_len) From patchwork Thu Jul 30 19:08:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693567 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E6BE138C for ; Thu, 30 Jul 2020 19:11:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 12B802084D for ; Thu, 30 Jul 2020 19:11:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RngkvXG5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730497AbgG3TLj (ORCPT ); Thu, 30 Jul 2020 15:11:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730488AbgG3TLf (ORCPT ); Thu, 30 Jul 2020 15:11:35 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1E74C061574; Thu, 30 Jul 2020 12:11:35 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id b9so15033192plx.6; Thu, 30 Jul 2020 12:11:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZESbWSWufIqTsPaugjsg63j9f9ZzfVJX0mg3KbrzmvI=; b=RngkvXG5oTuNOO3i9nOOG65xKCkH3/E886NUiPg9ZgUA7ZLhZmGWxr5bUxipPWkWKc KGs7/Pg/F0ktpXYjGTX2WCOarqdVdt11rClXlHSRLf9ZCAWy1RWFLUkxwOKE2eQXUOuO XTdHE0RQuUrWQ98nA2LUezBja483fIqWJAg4UDhI+UhLBV3AbZobS/fn15xKiluvjxZ/ DFyMEUIYqiUJiec9ZD+B3JNggid4SBCPa1XaYwKuAW/hQBF9HxCIF+SPiBJ7eQtUhgtZ virKPNuXEE65Q0gOVY4sNmX5lKNo5luTpm8aAH1CCLcC/hYnnUOUMT8l9UCXHnW6sZjw ObHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZESbWSWufIqTsPaugjsg63j9f9ZzfVJX0mg3KbrzmvI=; b=e7OPpJG/aJLZJd3F15gj81vVpdzH/qCHCM7e8lx4nCO5sNg+VJi60cFthVq8dRIwRu UE661VSN9aIlDVNB1R1GcA6YX1ydZjz0xAPEoa7JzOeQ0b+2jwm3qJSDMPZTp1X1IkPV OmRJnGxxViHgfK2OlzumZOGZnCi1kOMie+QYLyJnCNKSWeQrkLt3EvWNAfvfj4zIDoxf pkF9vmB6r65bReZN5oArdTMjs9J9xIoUt94nlDzoRBtfLrHrBBNh90ZGzrs6fDBjfHRY AOGmBLi9K0WkByRoXdAEEtBeBL3X5JJapvLqo/OrFZTLp5vDzqGwTrpeujqFr6NleyET jUcw== X-Gm-Message-State: AOAM533IP/rDGw9Lg6AUYinAe20bpT2ckCvHwNieQXk274DLOtQ+JzcN 04GVoR3sd72olYH1vgXaGWE= X-Google-Smtp-Source: ABdhPJy+CA7y3GmHqbiWgUrRgUS+uVHqMDcMc6oXxZXEymdV4XrH3A8io87j/+2S8NsNI/fzYkEaUg== X-Received: by 2002:a63:5a17:: with SMTP id o23mr352413pgb.218.1596136295129; Thu, 30 Jul 2020 12:11:35 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:34 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 6/8] x86: Add support for ZSTD compressed kernel Date: Thu, 30 Jul 2020 12:08:39 -0700 Message-Id: <20200730190841.2071656-7-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell * Add support for zstd compressed kernel * Define __DISABLE_EXPORTS in Makefile * Remove __DISABLE_EXPORTS definition from kaslr.c * Bump the heap size for zstd. * Update the documentation. Integrates the ZSTD decompression code to the x86 pre-boot code. Zstandard requires slightly more memory during the kernel decompression on x86 (192 KB vs 64 KB), and the memory usage is independent of the window size. __DISABLE_EXPORTS is now defined in the Makefile, which covers both the existing use in kaslr.c, and the use needed by the zstd decompressor in misc.c. This patch has been boot tested with both a zstd and gzip compressed kernel on i386 and x86_64 using buildroot and QEMU. Additionally, this has been tested in production on x86_64 devices. We saw a 2 second boot time reduction by switching kernel compression from xz to zstd. Reviewed-by: Kees Cook Tested-by: Sedat Dilek Signed-off-by: Nick Terrell --- Documentation/x86/boot.rst | 6 +++--- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 6 +++++- arch/x86/boot/compressed/kaslr.c | 7 ------- arch/x86/boot/compressed/misc.c | 4 ++++ arch/x86/include/asm/boot.h | 11 +++++++++-- 6 files changed, 22 insertions(+), 13 deletions(-) diff --git a/Documentation/x86/boot.rst b/Documentation/x86/boot.rst index 5325c71ca877..7fafc7ac00d7 100644 --- a/Documentation/x86/boot.rst +++ b/Documentation/x86/boot.rst @@ -782,9 +782,9 @@ Protocol: 2.08+ uncompressed data should be determined using the standard magic numbers. The currently supported compression formats are gzip (magic numbers 1F 8B or 1F 9E), bzip2 (magic number 42 5A), LZMA - (magic number 5D 00), XZ (magic number FD 37), and LZ4 (magic number - 02 21). The uncompressed payload is currently always ELF (magic - number 7F 45 4C 46). + (magic number 5D 00), XZ (magic number FD 37), LZ4 (magic number + 02 21) and ZSTD (magic number 28 B5). The uncompressed payload is + currently always ELF (magic number 7F 45 4C 46). ============ ============== Field name: payload_length diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 883da0abf779..4a64395bc35d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -188,6 +188,7 @@ config X86 select HAVE_KERNEL_LZMA select HAVE_KERNEL_LZO select HAVE_KERNEL_XZ + select HAVE_KERNEL_ZSTD select HAVE_KPROBES select HAVE_KPROBES_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 5a828fde7a42..c08714ae76ec 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -26,7 +26,7 @@ OBJECT_FILES_NON_STANDARD := y KCOV_INSTRUMENT := n targets := vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma \ - vmlinux.bin.xz vmlinux.bin.lzo vmlinux.bin.lz4 + vmlinux.bin.xz vmlinux.bin.lzo vmlinux.bin.lz4 vmlinux.bin.zst KBUILD_CFLAGS := -m$(BITS) -O2 KBUILD_CFLAGS += -fno-strict-aliasing $(call cc-option, -fPIE, -fPIC) @@ -42,6 +42,7 @@ KBUILD_CFLAGS += $(call cc-disable-warning, gnu) KBUILD_CFLAGS += -Wno-pointer-sign KBUILD_CFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables +KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ GCOV_PROFILE := n @@ -145,6 +146,8 @@ $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE $(call if_changed,lzo) $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE $(call if_changed,lz4) +$(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE + $(call if_changed,zstd22) suffix-$(CONFIG_KERNEL_GZIP) := gz suffix-$(CONFIG_KERNEL_BZIP2) := bz2 @@ -152,6 +155,7 @@ suffix-$(CONFIG_KERNEL_LZMA) := lzma suffix-$(CONFIG_KERNEL_XZ) := xz suffix-$(CONFIG_KERNEL_LZO) := lzo suffix-$(CONFIG_KERNEL_LZ4) := lz4 +suffix-$(CONFIG_KERNEL_ZSTD) := zst quiet_cmd_mkpiggy = MKPIGGY $@ cmd_mkpiggy = $(obj)/mkpiggy $< > $@ diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index d7408af55738..0048269180d5 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -19,13 +19,6 @@ */ #define BOOT_CTYPE_H -/* - * _ctype[] in lib/ctype.c is needed by isspace() of linux/ctype.h. - * While both lib/ctype.c and lib/cmdline.c will bring EXPORT_SYMBOL - * which is meaningless and will cause compiling error in some cases. - */ -#define __DISABLE_EXPORTS - #include "misc.h" #include "error.h" #include "../string.h" diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index 9652d5c2afda..39e592d0e0b4 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -77,6 +77,10 @@ static int lines, cols; #ifdef CONFIG_KERNEL_LZ4 #include "../../../../lib/decompress_unlz4.c" #endif + +#ifdef CONFIG_KERNEL_ZSTD +#include "../../../../lib/decompress_unzstd.c" +#endif /* * NOTE: When adding a new decompressor, please update the analysis in * ../header.S. diff --git a/arch/x86/include/asm/boot.h b/arch/x86/include/asm/boot.h index 680c320363db..9191280d9ea3 100644 --- a/arch/x86/include/asm/boot.h +++ b/arch/x86/include/asm/boot.h @@ -24,9 +24,16 @@ # error "Invalid value for CONFIG_PHYSICAL_ALIGN" #endif -#ifdef CONFIG_KERNEL_BZIP2 +#if defined(CONFIG_KERNEL_BZIP2) # define BOOT_HEAP_SIZE 0x400000 -#else /* !CONFIG_KERNEL_BZIP2 */ +#elif defined(CONFIG_KERNEL_ZSTD) +/* + * Zstd needs to allocate the ZSTD_DCtx in order to decompress the kernel. + * The ZSTD_DCtx is ~160KB, so set the heap size to 192KB because it is a + * round number and to allow some slack. + */ +# define BOOT_HEAP_SIZE 0x30000 +#else # define BOOT_HEAP_SIZE 0x10000 #endif From patchwork Thu Jul 30 19:08:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693569 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 70E4C912 for ; Thu, 30 Jul 2020 19:11:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 583B222B40 for ; Thu, 30 Jul 2020 19:11:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VOJH1I3/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730517AbgG3TLr (ORCPT ); Thu, 30 Jul 2020 15:11:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730444AbgG3TLj (ORCPT ); Thu, 30 Jul 2020 15:11:39 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10CBEC061574; Thu, 30 Jul 2020 12:11:39 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id g19so2463370plq.0; Thu, 30 Jul 2020 12:11:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=033/koxi8tBQP0JLIdsncSOzrcGFA+WfwhEA16y7708=; b=VOJH1I3/2EUXK2+wuA1Qck7sGAXLG+sr+IPGIjUgVGyQ8D8d3YnpK09z6N6KE7fG2Y cu1nLPOjj4U3vJ1cNHVd1qhL17ZzhCjjMEXqwgiNg/hTEZvxwMR7uXhbXO7+YcFnDR+X jgfOWzGjbtrV+R41GOlTRCKS/cOUwYDrBwVLuMjtiKY5OFa/4xSELAsl9dET4AA/X/UR lPraacU5i7lhcurzr/uZs1ax0zi7JPXwEn+vGnfvXgoHWd0NPxnys6vyAPkm5YCeD1lH kBoRnAvnjbjQeC9Ucqy04omLCjYOZZ/shrpY+PLLPIil4Xz0CZN4vpvlW0hqFja2Muyl D5JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=033/koxi8tBQP0JLIdsncSOzrcGFA+WfwhEA16y7708=; b=N8mkWXX3K7eOymuHAPkfrK7ibmb81qGZsDMHubEpGMmdbsYbcsUE41w0kiqg+0uEcv l4pzUGVshty3HTox/gBAIcnC+OsHsjeqRMRXzP2qVsnNi66QPNfQHKnUJh3hPzxNPT2i +dRvUQa/wrJ70AZszUu+tTB9307rFAvxiUkeWhc9K/VW9fd5pEND4tZP2AuQYFOQaGT2 IccWZo61f8LsKo7F9lxw2w3lBBvqDtIG02V0PVes4+7sKL1SccTO+xgoYJ0RUSndtcpH wk7FugNeqyyd4QRanlLKuNDNvsA9q+asITuPDdPvr/ocjHKb2jiuXKlku6tB/zBoD1p5 JDfQ== X-Gm-Message-State: AOAM53208XR29xsrkHw3YVZvhBFU7+KOhqfsZe1Z2AGgI5YZ5q6mL8AG 3F2tNfobZ1rMmlqqOispgBI= X-Google-Smtp-Source: ABdhPJzhH1KCDavhn1SyTstAYmZDkp0/vMIpOz8PMPIheimgqMBK62Ia4E0QQnxqNbKZdMMQAwxB3g== X-Received: by 2002:a17:902:b7c2:: with SMTP id v2mr623822plz.34.1596136298588; Thu, 30 Jul 2020 12:11:38 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:37 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 7/8] .gitignore: add ZSTD-compressed files Date: Thu, 30 Jul 2020 12:08:40 -0700 Message-Id: <20200730190841.2071656-8-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Adam Borowski For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more will come, thus let's be consistent with all other compressors. Tested-by: Sedat Dilek Reviewed-by: Kees Cook Signed-off-by: Nick Terrell Signed-off-by: Adam Borowski --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index d5f4804ed07c..162bd2b67bdf 100644 --- a/.gitignore +++ b/.gitignore @@ -44,6 +44,7 @@ *.tab.[ch] *.tar *.xz +*.zst Module.symvers modules.builtin modules.order From patchwork Thu Jul 30 19:08:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Terrell X-Patchwork-Id: 11693565 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE093138A for ; Thu, 30 Jul 2020 19:11:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D467322CA1 for ; Thu, 30 Jul 2020 19:11:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="se+kmvyq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730507AbgG3TLm (ORCPT ); Thu, 30 Jul 2020 15:11:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730488AbgG3TLm (ORCPT ); Thu, 30 Jul 2020 15:11:42 -0400 Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07325C061574; Thu, 30 Jul 2020 12:11:42 -0700 (PDT) Received: by mail-pj1-x1041.google.com with SMTP id k1so5044254pjt.5; Thu, 30 Jul 2020 12:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2qrM1HvX0q0CDfLlkMQtnEdV4MfzTfdneBe5DtTAa1M=; b=se+kmvyqv/0yBAoSxgdaaFmZtUjRhQ+tp1A3dml97ntARklAYnFwYx9gM7AMfj5wVW 9yJLnRnfCtftCWUbFWLgnx8VNH+HC2BCDisouyEirG1FclBiozhZMN3JllkHhCnKTiob somDNAKbzeWyVlNpwdY3OvOdKDfCNW6sWuW785/tPoarljl3QZZJXnEGKVOSnYY02MFH 91fg3CNSFvFMfeiv8RZxhOJZMlz+7TWt0NZnVuiqwi7+pAJ+r390AWvBnkuZuLXydDno J14m+CyG0JqBBNSA0EVIxrXDX4Ujg5fgl0PG+WzPFIs49/R/6raWEjoGU024/HY8nrZX krAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2qrM1HvX0q0CDfLlkMQtnEdV4MfzTfdneBe5DtTAa1M=; b=b37CuJlEPjxghfVtvConHIl7Z8seEkKKdcIpJb+1VsfZxrMT8V0oYz5eiyz9doR796 wMWb2lh31MNQtmhNRfl0Vj9r2dthdCaV1V1zxjKgtZzBo4u8zbcOzHuG60L8xwx63m0U IJCnvq3V0v9viFpf8zObC7QXlryOFyNlBycwn7fkvG2klYSwgFumDQjnxSCF+Tz5DkaU fc/jZwg0hINQH+D1Cr59TXp3wctBvChfXxWKfsEmWaslg6FiyS52XDMAde1c+v1GM2IH Y3FdEzl0tl4Ll8wYeEWSozyuM8WA8ELNkqFltAO7cLyKG3Lb8E3+vShfFQ6WqIFUJXR/ q2kA== X-Gm-Message-State: AOAM530YIxRrPKjRXtGp4cIDEPgRRtSyGTzFYznboPyDHZDYADXvHawY 29KPa8HAaS6NU2t0yfT00tk= X-Google-Smtp-Source: ABdhPJyw/p/oqLAHCYYpuecDmhxocUWLlJApao4mjZexeaT+ADb3kgXNASVpphJLSL9aEks8cZbr6g== X-Received: by 2002:a63:4b1d:: with SMTP id y29mr362889pga.264.1596136301573; Thu, 30 Jul 2020 12:11:41 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id o22sm8644976pfu.206.2020.07.30.12.11.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2020 12:11:40 -0700 (PDT) From: Nick Terrell To: Borislav Petkov , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, Chris Mason , linux-kbuild@vger.kernel.org, x86@kernel.org, gregkh@linuxfoundation.org, Petr Malat , Kees Cook , Kernel Team , Nick Terrell , Adam Borowski , Patrick Williams , rmikey@fb.com, mingo@kernel.org, Patrick Williams , Sedat Dilek , Norbert Lange , Andrew Morton , Alex Xu , Masahiro Yamada , Nick Terrell Subject: [PATCH v10 8/8] Documentation: dontdiff: Add zstd compressed files Date: Thu, 30 Jul 2020 12:08:41 -0700 Message-Id: <20200730190841.2071656-9-nickrterrell@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200730190841.2071656-1-nickrterrell@gmail.com> References: <20200730190841.2071656-1-nickrterrell@gmail.com> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nick Terrell For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more will come, thus let's be consistent with all other compressors. Signed-off-by: Nick Terrell --- Documentation/dontdiff | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/dontdiff b/Documentation/dontdiff index ef9519c32c55..e361fc95ca29 100644 --- a/Documentation/dontdiff +++ b/Documentation/dontdiff @@ -55,6 +55,7 @@ *.ver *.xml *.xz +*.zst *_MODULES *_vga16.c *~