[v3,00/19] crypto: scatterlist handling improvements

Message

Eric Biggers Feb. 19, 2025, 6:23 p.m. UTC

This series can also be retrieved from:

    git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crypto-scatterlist-v3

This series cleans up and optimizes the code that translates between
scatterlists (the input to the API) and virtual addresses (what software
implementations operate on) for skcipher and aead algorithms.

This takes the form of cleanups and optimizations to the skcipher_walk
functions and a rework of the underlying scatter_walk functions.

The unnecessary use of scatterlists still remains a huge pain point of
many of the crypto APIs, with the exception of lib/crypto/, shash, and
scomp which do it properly.  But this series at least reduces (but not
eliminates) the impact on performance that the scatterlists have.

An an example, this patchset improves IPsec throughput by about 5%, as
measured using iperf3 bidirectional TCP between two c3d-standard-4 (AMD
Genoa) instances in Google Compute Engine using transport mode IPsec
with AES-256-GCM.

This series is organized as follows:

- Patch 1-5 improve scatter_walk, introducing easier-to-use functions
  and optimizing performance in some cases.
- Patch 6-17 convert users to use the new functions.
- Patch 18 removes functions that are no longer needed.
- Patch 19 optimizes the walker on !HIGHMEM platforms to start returning
  data segments that can cross a page boundary.  This can significantly
  improve performance in cases where messages can cross pages, such as
  IPsec.  Previously there was a large overhead caused by packets being
  unnecessarily divided into multiple parts by the walker, including
  hitting skcipher_next_slow() which uses a single-block bounce buffer.

Changed in v3:
- Dropped patches that were upstreamed.
- Added a Reviewed-by and Tested-by.

Changed in v2:
- Added comment to scatterwalk_done_dst().
- Added scatterwalk_get_sglist() and use it in net/tls/.
- Dropped the keywrap patch, as keywrap is being removed by
  https://lore.kernel.org/r/[e-mail]

Eric Biggers (19):
  crypto: scatterwalk - move to next sg entry just in time
  crypto: scatterwalk - add new functions for skipping data
  crypto: scatterwalk - add new functions for iterating through data
  crypto: scatterwalk - add new functions for copying data
  crypto: scatterwalk - add scatterwalk_get_sglist()
  crypto: skcipher - use scatterwalk_start_at_pos()
  crypto: aegis - use the new scatterwalk functions
  crypto: arm/ghash - use the new scatterwalk functions
  crypto: arm64 - use the new scatterwalk functions
  crypto: nx - use the new scatterwalk functions
  crypto: s390/aes-gcm - use the new scatterwalk functions
  crypto: s5p-sss - use the new scatterwalk functions
  crypto: stm32 - use the new scatterwalk functions
  crypto: x86/aes-gcm - use the new scatterwalk functions
  crypto: x86/aegis - use the new scatterwalk functions
  net/tls: use the new scatterwalk functions
  crypto: skcipher - use the new scatterwalk functions
  crypto: scatterwalk - remove obsolete functions
  crypto: scatterwalk - don't split at page boundaries when !HIGHMEM

 arch/arm/crypto/ghash-ce-glue.c       |  15 +-
 arch/arm64/crypto/aes-ce-ccm-glue.c   |  17 +--
 arch/arm64/crypto/ghash-ce-glue.c     |  16 +-
 arch/arm64/crypto/sm4-ce-ccm-glue.c   |  27 ++--
 arch/arm64/crypto/sm4-ce-gcm-glue.c   |  31 ++--
 arch/s390/crypto/aes_s390.c           |  33 ++---
 arch/x86/crypto/aegis128-aesni-glue.c |  10 +-
 arch/x86/crypto/aesni-intel_glue.c    |  28 ++--
 crypto/aegis128-core.c                |  10 +-
 crypto/scatterwalk.c                  |  91 +++++++-----
 crypto/skcipher.c                     |  65 +++------
 drivers/crypto/nx/nx-aes-ccm.c        |  16 +-
 drivers/crypto/nx/nx-aes-gcm.c        |  17 +--
 drivers/crypto/nx/nx.c                |  31 +---
 drivers/crypto/nx/nx.h                |   3 -
 drivers/crypto/s5p-sss.c              |  38 ++---
 drivers/crypto/stm32/stm32-cryp.c     |  34 ++---
 include/crypto/scatterwalk.h          | 203 +++++++++++++++++++++-----
 net/tls/tls_device_fallback.c         |  31 +---
 19 files changed, 363 insertions(+), 353 deletions(-)


base-commit: c346fef6fef53fa57ff323b701e7bad82290d0e7