[RFC,0/5] crypto: qat - enable SRIOV VF live migration

Message

Xin Zeng June 30, 2023, 1:12 p.m. UTC

This set enables live migration for Intel QAT GEN4 SRIOV Virtual
Functions (VFs). It is composed of 5 patches.
The first is a pre-requisite. It adds logic to the QAT PF driver that
allows to save and restore the state of a bank (a virtual function is a
wrapper around a bank) and drain a ring pair. The second patch adds to
the QAT PF driver a set of interfaces to allow to save and restore the
state of a VF that will be called by the modules qat_vfio_pci which will
be introduced in the last patch. The third adds HZ_PER_GHZ which will be
required by the fourth one. The fourth one implements the defined
interfaces. The last one adds a vfio pci extension specific for QAT
which intercepts the vfio device operations for a QAT VF to allow live
migration.

Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko 
2. Assign the VFs to the virtual machine and enable device live
migration 
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib) 
4. Migrate the VM from the source node to a destination node

For P2P states support and AER support, we are going to implement these
in the final version. Any feedback is appreciated!

Andy Shevchenko (1):
  units: Add HZ_PER_GHZ

Siming Wan (1):
  crypto: qat - add bank save/restore and RP drain

Xin Zeng (3):
  crypto: qat - add interface for live migration
  crypto: qat - implement interface for live migration
  vfio/qat: Add vfio_pci driver for Intel QAT VF devices

 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |   9 +-
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.h     |   3 +-
 .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c   |   2 +-
 .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c     |   2 +-
 .../intel/qat/qat_c62x/adf_c62x_hw_data.c     |   2 +-
 .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c |   2 +-
 drivers/crypto/intel/qat/qat_common/Makefile  |   4 +-
 .../intel/qat/qat_common/adf_accel_devices.h  |  78 ++-
 .../intel/qat/qat_common/adf_gen2_hw_data.c   |  17 +-
 .../intel/qat/qat_common/adf_gen2_hw_data.h   |  10 +-
 .../intel/qat/qat_common/adf_gen4_hw_data.c   | 415 +++++++++++-
 .../intel/qat/qat_common/adf_gen4_hw_data.h   | 152 ++++-
 .../intel/qat/qat_common/adf_gen4_pfvf.c      |   7 +-
 .../intel/qat/qat_common/adf_gen4_pfvf.h      |   7 +
 .../intel/qat/qat_common/adf_gen4_vf_mig.c    | 609 ++++++++++++++++++
 .../intel/qat/qat_common/adf_mstate_mgr.c     | 267 ++++++++
 .../intel/qat/qat_common/adf_mstate_mgr.h     |  99 +++
 .../intel/qat/qat_common/adf_transport.c      |  11 +-
 .../crypto/intel/qat/qat_common/adf_vf_isr.c  |   2 +-
 .../crypto/intel/qat/qat_common/qat_vf_mig.c  | 106 +++
 .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c   |   2 +-
 .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c   |   2 +-
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   1 +
 drivers/vfio/pci/qat/Kconfig                  |  13 +
 drivers/vfio/pci/qat/Makefile                 |   4 +
 drivers/vfio/pci/qat/qat_vfio_pci_main.c      | 518 +++++++++++++++
 include/linux/qat/qat_vf_mig.h                |  15 +
 include/linux/units.h                         |   6 +-
 29 files changed, 2332 insertions(+), 35 deletions(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/qat_vf_mig.c
 create mode 100644 drivers/vfio/pci/qat/Kconfig
 create mode 100644 drivers/vfio/pci/qat/Makefile
 create mode 100644 drivers/vfio/pci/qat/qat_vfio_pci_main.c
 create mode 100644 include/linux/qat/qat_vf_mig.h