[PATCHv2,00/17] zram: convert to custom comp API and allow algorithms configuration

Message

Sergey Senozhatsky May 6, 2024, 7:58 a.m. UTC

This patch set moves zram from crypto API to a custom compression
API which allows us to tune and configure compression algorithms,
something that crypto API, unfortunately, doesn't support. Basically,
this seroes brings back the bits of comp "backend" code that we had
many years ago. This means that if we want zram to support new
compression algorithms we need to implement corresponding backends.

        Currently, zram supports a pretty decent number of comp backends:
lzo, lzorle, lz4, lz4hc, 842, deflate, zstd

        At this point we handle 2 parameters: a compression level and
a pre-trained compression dictionary. Which seems like a good enough
start. The list will be extended in the future.

Examples:

- changes default compression level
        echo "algo=zstd level=11" > /sys/block/zram0/comp_algorithm

- passes path to a pre-trained dictionary
        echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/comp_algorithm

TEST
====

using default zstd
/sys/block/zram0/mm_stat
1750315008 504602831 514256896        0 514256896        1        0    34204    34204

using zstd level=7 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750310912 432540606 441712640        0 441712640        1        0    34187    34187

v2:
-- forcibly select LZO when nono backeneds were selected (Masahiro Yamada)
-- add 842 compression backend
-- added comp config preprocessing and caching
-- reworked zstd dict handling

Sergey Senozhatsky (17):
  zram: move from crypto API to custom comp backends API
  zram: add lzo and lzorle compression backends support
  zram: add lz4 compression backend support
  zram: add lz4hc compression backend support
  zram: add zstd compression backend support
  zram: pass estimated src size hint to zstd
  zram: add zlib compression backend support
  zram: add 842 compression backend support
  zram: check that backends array has at least one backend
  zram: introduce zcomp_config structure
  zram: extend comp_algorithm attr write handling
  zram: support compression level comp config
  zram: add support for dict comp config
  zram: add dictionary support to zstd backend
  zram: add config init/release backend callbacks
  zram: share dictionaries between per-CPU contexts
  Documentation/zram: add documentation for algorithm parameters

 Documentation/admin-guide/blockdev/zram.rst |  38 +++-
 drivers/block/zram/Kconfig                  |  76 +++++--
 drivers/block/zram/Makefile                 |   8 +
 drivers/block/zram/backend_842.c            |  79 +++++++
 drivers/block/zram/backend_842.h            |  10 +
 drivers/block/zram/backend_deflate.c        | 143 +++++++++++++
 drivers/block/zram/backend_deflate.h        |  10 +
 drivers/block/zram/backend_lz4.c            |  58 +++++
 drivers/block/zram/backend_lz4.h            |  10 +
 drivers/block/zram/backend_lz4hc.c          |  84 ++++++++
 drivers/block/zram/backend_lz4hc.h          |  10 +
 drivers/block/zram/backend_lzo.c            |  55 +++++
 drivers/block/zram/backend_lzo.h            |  10 +
 drivers/block/zram/backend_lzorle.c         |  55 +++++
 drivers/block/zram/backend_lzorle.h         |  10 +
 drivers/block/zram/backend_zstd.c           | 221 ++++++++++++++++++++
 drivers/block/zram/backend_zstd.h           |  10 +
 drivers/block/zram/zcomp.c                  | 155 +++++++-------
 drivers/block/zram/zcomp.h                  |  42 +++-
 drivers/block/zram/zram_drv.c               | 120 ++++++++++-
 drivers/block/zram/zram_drv.h               |   1 +
 21 files changed, 1094 insertions(+), 111 deletions(-)
 create mode 100644 drivers/block/zram/backend_842.c
 create mode 100644 drivers/block/zram/backend_842.h
 create mode 100644 drivers/block/zram/backend_deflate.c
 create mode 100644 drivers/block/zram/backend_deflate.h
 create mode 100644 drivers/block/zram/backend_lz4.c
 create mode 100644 drivers/block/zram/backend_lz4.h
 create mode 100644 drivers/block/zram/backend_lz4hc.c
 create mode 100644 drivers/block/zram/backend_lz4hc.h
 create mode 100644 drivers/block/zram/backend_lzo.c
 create mode 100644 drivers/block/zram/backend_lzo.h
 create mode 100644 drivers/block/zram/backend_lzorle.c
 create mode 100644 drivers/block/zram/backend_lzorle.h
 create mode 100644 drivers/block/zram/backend_zstd.c
 create mode 100644 drivers/block/zram/backend_zstd.h

Comments

Sergey Senozhatsky May 8, 2024, 7:41 a.m. UTC | #1

On (24/05/06 16:58), Sergey Senozhatsky wrote:
> 	This patch set moves zram from crypto API to a custom compression
> API which allows us to tune and configure compression algorithms,
> something that crypto API, unfortunately, doesn't support. Basically,
> this seroes brings back the bits of comp "backend" code that we had
> many years ago. This means that if we want zram to support new
> compression algorithms we need to implement corresponding backends.
> 
>         Currently, zram supports a pretty decent number of comp backends:
> lzo, lzorle, lz4, lz4hc, 842, deflate, zstd
> 
>         At this point we handle 2 parameters: a compression level and
> a pre-trained compression dictionary. Which seems like a good enough
> start. The list will be extended in the future.
> 
> Examples:
> 
> - changes default compression level
>         echo "algo=zstd level=11" > /sys/block/zram0/comp_algorithm
> 
> - passes path to a pre-trained dictionary
>         echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/comp_algorithm

I'll send v3 shortly, which adds pre-trained dictionary support to
lz4 and lz4hc compression backends. Apparently lz4/lz4hc can use
dictionaries pre-trained with zstd --init, just like zstd.