[v2,rdma-next,0/8] Introduce mlx5 Memory Scheme ODP

Message

Michael Guralnik Sept. 9, 2024, 10:04 a.m. UTC

This series introduces a new ODP scheme in mlx5 where the FW takes the
responsibility of parsing and providing page fault data to the driver
to handle the fault.
As opposed to the current ODP transport scheme where the driver is
responsible for reading and parsing work queues and querying mkeys to
acquire needed info to handle the page fault.

The new scheme allows driver to support ODP over Devx QPs where driver
is not able to access the QP buffers, owned by the user application,
to read the work queue requests.
Furthermore, the new scheme allows support for ODP with new indirect
MKEY types as the driver doesn't need to query or parse indirect mkeys
in this scheme.

The driver will enable the new scheme on devices that have the relevant
capabilities. Otherwise, transport scheme ODP will be the default.

The move to memory scheme ODP is transparent to existing ODP
applications and no change is needed.
New application that want to take advantage of the new functionality
should query which scheme is active and it's capabilities using Devx.

v1->v2:
- switch mlx5_umem_find_best_pgsz to a function and rename

Michael Guralnik (8):
  net/mlx5: Expand mkey page size to support 6 bits
  net/mlx5: Expose HW bits for Memory scheme ODP
  RDMA/mlx5: Add new ODP memory scheme eqe format
  RDMA/mlx5: Enforce umem boundaries for explicit ODP page faults
  RDMA/mlx5: Split ODP mkey search logic
  RDMA/mlx5: Add handling for memory scheme page fault events
  RDMA/mlx5: Add implicit MR handling to ODP memory scheme
  net/mlx5: Handle memory scheme ODP capabilities

 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  30 +-
 drivers/infiniband/hw/mlx5/mr.c               |  10 +-
 drivers/infiniband/hw/mlx5/odp.c              | 400 ++++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/main.c    |  54 ++-
 include/linux/mlx5/device.h                   |  30 +-
 include/linux/mlx5/mlx5_ifc.h                 |  64 ++-
 6 files changed, 457 insertions(+), 131 deletions(-)

Comments

Leon Romanovsky Sept. 11, 2024, 12:07 p.m. UTC | #1

On Mon, 09 Sep 2024 13:04:56 +0300, Michael Guralnik wrote:
> This series introduces a new ODP scheme in mlx5 where the FW takes the
> responsibility of parsing and providing page fault data to the driver
> to handle the fault.
> As opposed to the current ODP transport scheme where the driver is
> responsible for reading and parsing work queues and querying mkeys to
> acquire needed info to handle the page fault.
> 
> [...]

Applied, thanks!

[1/8] net/mlx5: Expand mkey page size to support 6 bits
      https://git.kernel.org/rdma/rdma/c/cef7dde8836ab0
[2/8] net/mlx5: Expose HW bits for Memory scheme ODP
      https://git.kernel.org/rdma/rdma/c/6cd9171d04cff7
[3/8] RDMA/mlx5: Add new ODP memory scheme eqe format
      https://git.kernel.org/rdma/rdma/c/64c68385a39bb6
[4/8] RDMA/mlx5: Enforce umem boundaries for explicit ODP page faults
      https://git.kernel.org/rdma/rdma/c/8c6d097d830f77
[5/8] RDMA/mlx5: Split ODP mkey search logic
      https://git.kernel.org/rdma/rdma/c/7f91510af938b4
[6/8] RDMA/mlx5: Add handling for memory scheme page fault events
      https://git.kernel.org/rdma/rdma/c/e4fda2320f8e6b
[7/8] RDMA/mlx5: Add implicit MR handling to ODP memory scheme
      https://git.kernel.org/rdma/rdma/c/6f2487bfafce5e
[8/8] net/mlx5: Handle memory scheme ODP capabilities
      https://git.kernel.org/rdma/rdma/c/5416992c011080

Best regards,