[mlx5-next,0/7] Add mlx5 live migration driver

Message

Leon Romanovsky Sept. 22, 2021, 10:38 a.m. UTC

From: Leon Romanovsky <[e-mail]>

From Yishai:

This series adds mlx5 live migration driver for VFs that are migrated
capable.

It uses vfio_pci_core to register to the VFIO subsystem and then
implements the mlx5 specific logic in the migration area.

The migration implementation follows the definition from uapi/vfio.h and
uses the mlx5 VF->PF command channel to achieve it.

The series adds an API in the vfio core layer to check migration state
transition validity as part of a migration flow. This ensures that all
migration implementations follow a consistent migration state machine.

As part of the migration process the VF doesn't ride on mlx5_core, the
device is driving *two* different PCI devices, the PF owned by mlx5_core
and the VF owned by the mlx5 vfio driver.

The mlx5_core of the PF is accessed only during the narrow window of the
VF's ioctl that requires its services.

The series also exposes from the PCI sub system an API named
pci_iov_vf_id() to get the index of the VF. The PCI core uses this index
internally, often called the vf_id, during the setup of the VF, eg
pci_iov_add_virtfn().

The returned VF index is needed by the mlx5 vfio driver for its internal
operations to configure/control its VFs as part of the migration
process.

With the above functionality in place the driver implements the
suspend/resume flows to work over QEMU.

Thanks

-----------------------------------------------------
Alex,

This series touches our ethernet and RDMA drivers, so we will need to
route the patches through separate shared branch (mlx5-next) in order
to eliminate the chances of merge conflicts between different subsystems.

Thanks

Jason Gunthorpe (1):
  PCI/IOV: Provide internal VF index

Yishai Hadas (6):
  vfio: Add an API to check migration state transition validity
  vfio/pci_core: Make the region->release() function optional
  net/mlx5: Introduce migration bits and structures
  net/mlx5: Expose APIs to get/put the mlx5 core device
  mlx5_vfio_pci: Expose migration commands over mlx5 device
  mlx5_vfio_pci: Implement vfio_pci driver for mlx5 devices

 .../net/ethernet/mellanox/mlx5/core/main.c    |  43 +
 drivers/pci/iov.c                             |  14 +
 drivers/vfio/pci/Kconfig                      |  11 +
 drivers/vfio/pci/Makefile                     |   3 +
 drivers/vfio/pci/mlx5_vfio_pci.c              | 736 ++++++++++++++++++
 drivers/vfio/pci/mlx5_vfio_pci_cmd.c          | 358 +++++++++
 drivers/vfio/pci/mlx5_vfio_pci_cmd.h          |  43 +
 drivers/vfio/pci/vfio_pci_core.c              |   3 +-
 drivers/vfio/vfio.c                           |  41 +
 include/linux/mlx5/driver.h                   |   3 +
 include/linux/mlx5/mlx5_ifc.h                 | 145 +++-
 include/linux/pci.h                           |   7 +-
 include/linux/vfio.h                          |   1 +
 13 files changed, 1405 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c
 create mode 100644 drivers/vfio/pci/mlx5_vfio_pci_cmd.c
 create mode 100644 drivers/vfio/pci/mlx5_vfio_pci_cmd.h