mbox series

[RFC,v2,vfio,0/7] pds vfio driver

Message ID 20221214232136.64220-1-brett.creeley@amd.com (mailing list archive)
Headers show
Series pds vfio driver | expand

Message

Brett Creeley Dec. 14, 2022, 11:21 p.m. UTC 
This is a first draft patchset for a new vendor specific VFIO driver for
use with the AMD/Pensando Distributed Services Card (DSC). This driver
(pds_vfio) is a client of the newly introduced pds_core driver.

Reference to the pds_core patchset:
https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@amd.com/

AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the
Distributed Services Card (DSC). This patchset adds the new pds_vfio
driver in order to support NVMe VF live migration.

This driver will use the pds_core device and auxiliary_bus as the VFIO
control path to the DSC. The pds_core device creates auxiliary_bus devices
for each live migratable VF. The devices are named by their feature plus
the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find
its related VF PCI driver instance. Once this auxiliary bus connection
is configured, the pds_vfio driver can send admin queue commands to the
device and receive events from pds_core.

An ASCII diagram of a VFIO instance looks something like this and can
be used with the VFIO subsystem to provide devices VFIO and live
migration support.

                               .------.  .--------------------------.
                               | QEMU |--|  VM     .-------------.  |
                               '......'  |         | nvme driver |  |
                                  |      |         .-------------.  |
                                  |      |         |  SR-IOV VF  |  |
                                  |      |         '-------------'  |
                                  |      '---------------||---------'
                               .--------------.          ||
                               |/dev/<vfio_fd>|          ||
                               '--------------'          ||
Host Userspace                         |                 ||
===================================================      ||
Host Kernel                            |                 ||
                                       |                 ||
           pds_core.LM.2305 <--+   .--------.            ||
                   |           |   |vfio-pci|            ||
                   |           |   '--------'            ||
                   |           |       |                 ||
         .------------.       .-------------.            ||
         |  pds_core  |       |   pds_vfio  |            ||
         '------------'       '-------------'            ||
               ||                   ||                   ||
             09:00.0              09:00.1                ||
== PCI ==================================================||=====
               ||                   ||                   ||
          .----------.         .----------.              ||
    ,-----|    PF    |---------|    VF    |-------------------,
    |     '----------'         '----------'  |      nvme      |
    |                     DSC                |  data/control  |
    |                                        |      path      |
    -----------------------------------------------------------


The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds.
It makes use of and introduces new files in the common include/linux/pds
include directory.

Changes:
v2:
- Implement state transitions for VFIO_MIGRATION_P2P flag
- Improve auxiliary driver probe by returning EPROBE_DEFER
  when the PCI driver is not set up correctly
- Add pointer to docs in
  Documentation/networking/device_drivers/ethernet/index.rst

Brett Creeley (7):
  pds_vfio: Initial support for pds_vfio VFIO driver
  pds_vfio: Add support to register as PDS client
  pds_vfio: Add VFIO live migration support
  vfio: Commonize combine_ranges for use in other VFIO drivers
  pds_vfio: Add support for dirty page tracking
  pds_vfio: Add support for firmware recovery
  pds_vfio: Add documentation files

 .../ethernet/pensando/pds_vfio.rst            |  88 +++
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/mlx5/cmd.c                   |  48 +-
 drivers/vfio/pci/pds/Kconfig                  |  10 +
 drivers/vfio/pci/pds/Makefile                 |  12 +
 drivers/vfio/pci/pds/aux_drv.c                | 216 +++++++
 drivers/vfio/pci/pds/aux_drv.h                |  30 +
 drivers/vfio/pci/pds/cmds.c                   | 486 ++++++++++++++++
 drivers/vfio/pci/pds/cmds.h                   |  44 ++
 drivers/vfio/pci/pds/dirty.c                  | 541 ++++++++++++++++++
 drivers/vfio/pci/pds/dirty.h                  |  49 ++
 drivers/vfio/pci/pds/lm.c                     | 484 ++++++++++++++++
 drivers/vfio/pci/pds/lm.h                     |  53 ++
 drivers/vfio/pci/pds/pci_drv.c                | 134 +++++
 drivers/vfio/pci/pds/pci_drv.h                |   9 +
 drivers/vfio/pci/pds/vfio_dev.c               | 238 ++++++++
 drivers/vfio/pci/pds/vfio_dev.h               |  42 ++
 drivers/vfio/vfio_main.c                      |  48 ++
 include/linux/pds/pds_core_if.h               |   1 +
 include/linux/pds/pds_lm.h                    | 356 ++++++++++++
 include/linux/vfio.h                          |   3 +
 21 files changed, 2847 insertions(+), 47 deletions(-)
 create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst
 create mode 100644 drivers/vfio/pci/pds/Kconfig
 create mode 100644 drivers/vfio/pci/pds/Makefile
 create mode 100644 drivers/vfio/pci/pds/aux_drv.c
 create mode 100644 drivers/vfio/pci/pds/aux_drv.h
 create mode 100644 drivers/vfio/pci/pds/cmds.c
 create mode 100644 drivers/vfio/pci/pds/cmds.h
 create mode 100644 drivers/vfio/pci/pds/dirty.c
 create mode 100644 drivers/vfio/pci/pds/dirty.h
 create mode 100644 drivers/vfio/pci/pds/lm.c
 create mode 100644 drivers/vfio/pci/pds/lm.h
 create mode 100644 drivers/vfio/pci/pds/pci_drv.c
 create mode 100644 drivers/vfio/pci/pds/pci_drv.h
 create mode 100644 drivers/vfio/pci/pds/vfio_dev.c
 create mode 100644 drivers/vfio/pci/pds/vfio_dev.h
 create mode 100644 include/linux/pds/pds_lm.h

--
2.17.1

Comments

Christoph Hellwig Dec. 15, 2022, 6:55 a.m. UTC | #1 
On Wed, Dec 14, 2022 at 03:21:29PM -0800, Brett Creeley wrote:
> This is a first draft patchset for a new vendor specific VFIO driver for
> use with the AMD/Pensando Distributed Services Card (DSC). This driver
> (pds_vfio) is a client of the newly introduced pds_core driver.

Hi Brett,

just as said last time and completely ignored by you:  any nvme live
migration needs to be standardized in the NVMe technical working group
before it can be merged in Linux.
Brett Creeley Dec. 15, 2022, 10:22 p.m. UTC | #2 
On 12/14/2022 10:55 PM, Christoph Hellwig wrote:
> just as said last time and completely ignored by you:  any nvme live
> migration needs to be standardized in the NVMe technical working group
> before it can be merged in Linux.

Hey Christoph,

I'm not trying to ignore you and that's why this series is still marked 
RFC. I appreciate the feedback, regardless of the end result.

I am paying attention to the other thread on the KVM mailing list and do 
understand your stance on the topic.

Thanks,

Brett