| Message ID | 20221207010705.35128-1-brett.creeley@amd.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | pds vfio driver | expand |
On Tue, Dec 06, 2022 at 05:06:58PM -0800, Brett Creeley wrote: > AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the > Distributed Services Card (DSC). This patchset adds the new pds_vfio > driver in order to support NVMe VF live migration. If you want NVMe live migration, please work with the nvme technical working group to standardize it. We will not add support for a gazillion incompatible and probably broken concepts of this.
On 12/7/2022 3:06 AM, Brett Creeley wrote: > This is a first draft patchset for a new vendor specific VFIO driver for > use with the AMD/Pensando Distributed Services Card (DSC). This driver > (pds_vfio) is a client of the newly introduced pds_core driver. > > Reference to the pds_core patchset: > https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@amd.com/ > > AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the > Distributed Services Card (DSC). This patchset adds the new pds_vfio > driver in order to support NVMe VF live migration. > > This driver will use the pds_core device and auxiliary_bus as the VFIO > control path to the DSC. The pds_core device creates auxiliary_bus devices > for each live migratable VF. The devices are named by their feature plus > the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find > its related VF PCI driver instance. Once this auxiliary bus connection > is configured, the pds_vfio driver can send admin queue commands to the > device and receive events from pds_core. > > An ASCII diagram of a VFIO instance looks something like this and can > be used with the VFIO subsystem to provide devices VFIO and live > migration support. > > .------. .--------------------------. > | QEMU |--| VM .-------------. | > '......' | | nvme driver | | > | | .-------------. | > | | | SR-IOV VF | | > | | '-------------' | > | '---------------||---------' > .--------------. || > |/dev/<vfio_fd>| || > '--------------' || > Host Userspace | || > =================================================== || > Host Kernel | || > | || > pds_core.LM.2305 <--+ .--------. || > | | |vfio-pci| || > | | '--------' || > | | | || > .------------. .-------------. || > | pds_core | | pds_vfio | || > '------------' '-------------' || > || || || > 09:00.0 09:00.1 || > == PCI ==================================================||===== > || || || > .----------. .----------. || > ,-----| PF |---------| VF |-------------------, > | '----------' '----------' | nvme | > | DSC | data/control | > | | path | > ----------------------------------------------------------- Hi Brett, what is the class code of the pds_core device ? I see that pds_vfio class_code is PCI_CLASS_STORAGE_EXPRESS. > > > The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. > It makes use of and introduces new files in the common include/linux/pds > include directory. > > Brett Creeley (7): > pds_vfio: Initial support for pds_vfio VFIO driver > pds_vfio: Add support to register as PDS client > pds_vfio: Add VFIO live migration support > vfio: Commonize combine_ranges for use in other VFIO drivers > pds_vfio: Add support for dirty page tracking > pds_vfio: Add support for firmware recovery > pds_vfio: Add documentation files > > .../ethernet/pensando/pds_vfio.rst | 88 +++ > drivers/vfio/pci/Kconfig | 2 + > drivers/vfio/pci/mlx5/cmd.c | 48 +- > drivers/vfio/pci/pds/Kconfig | 10 + > drivers/vfio/pci/pds/Makefile | 12 + > drivers/vfio/pci/pds/aux_drv.c | 216 +++++++ > drivers/vfio/pci/pds/aux_drv.h | 30 + > drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++ > drivers/vfio/pci/pds/cmds.h | 44 ++ > drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ > drivers/vfio/pci/pds/dirty.h | 49 ++ > drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++ > drivers/vfio/pci/pds/lm.h | 53 ++ > drivers/vfio/pci/pds/pci_drv.c | 134 +++++ > drivers/vfio/pci/pds/pci_drv.h | 9 + > drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++ > drivers/vfio/pci/pds/vfio_dev.h | 42 ++ > drivers/vfio/vfio_main.c | 48 ++ > include/linux/pds/pds_core_if.h | 1 + > include/linux/pds/pds_lm.h | 356 ++++++++++++ > include/linux/vfio.h | 3 + > 21 files changed, 2847 insertions(+), 47 deletions(-) > create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst > create mode 100644 drivers/vfio/pci/pds/Kconfig > create mode 100644 drivers/vfio/pci/pds/Makefile > create mode 100644 drivers/vfio/pci/pds/aux_drv.c > create mode 100644 drivers/vfio/pci/pds/aux_drv.h > create mode 100644 drivers/vfio/pci/pds/cmds.c > create mode 100644 drivers/vfio/pci/pds/cmds.h > create mode 100644 drivers/vfio/pci/pds/dirty.c > create mode 100644 drivers/vfio/pci/pds/dirty.h > create mode 100644 drivers/vfio/pci/pds/lm.c > create mode 100644 drivers/vfio/pci/pds/lm.h > create mode 100644 drivers/vfio/pci/pds/pci_drv.c > create mode 100644 drivers/vfio/pci/pds/pci_drv.h > create mode 100644 drivers/vfio/pci/pds/vfio_dev.c > create mode 100644 drivers/vfio/pci/pds/vfio_dev.h > create mode 100644 include/linux/pds/pds_lm.h > > -- > 2.17.1 >
On 12/11/2022 4:54 AM, Max Gurtovoy wrote: > Caution: This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > On 12/7/2022 3:06 AM, Brett Creeley wrote: >> This is a first draft patchset for a new vendor specific VFIO driver for >> use with the AMD/Pensando Distributed Services Card (DSC). This driver >> (pds_vfio) is a client of the newly introduced pds_core driver. >> >> Reference to the pds_core patchset: >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fnetdev%2F20221207004443.33779-1-shannon.nelson%40amd.com%2F&data=05%7C01%7Cbrett.creeley%40amd.com%7C0591fe11a7c24bf8789908dadb76db84%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638063600829691750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3VMvNtUto4HwCap65NvWrIADbLzGk4Ef9ZnA9fAB458%3D&reserved=0 >> >> AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the >> Distributed Services Card (DSC). This patchset adds the new pds_vfio >> driver in order to support NVMe VF live migration. >> >> This driver will use the pds_core device and auxiliary_bus as the VFIO >> control path to the DSC. The pds_core device creates auxiliary_bus >> devices >> for each live migratable VF. The devices are named by their feature plus >> the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can >> find >> its related VF PCI driver instance. Once this auxiliary bus connection >> is configured, the pds_vfio driver can send admin queue commands to the >> device and receive events from pds_core. >> >> An ASCII diagram of a VFIO instance looks something like this and can >> be used with the VFIO subsystem to provide devices VFIO and live >> migration support. >> >> .------. .--------------------------. >> | QEMU |--| VM .-------------. | >> '......' | | nvme driver | | >> | | .-------------. | >> | | | SR-IOV VF | | >> | | '-------------' | >> | '---------------||---------' >> .--------------. || >> |/dev/<vfio_fd>| || >> '--------------' || >> Host Userspace | || >> =================================================== || >> Host Kernel | || >> | || >> pds_core.LM.2305 <--+ .--------. || >> | | |vfio-pci| || >> | | '--------' || >> | | | || >> .------------. .-------------. || >> | pds_core | | pds_vfio | || >> '------------' '-------------' || >> || || || >> 09:00.0 09:00.1 || >> == PCI ==================================================||===== >> || || || >> .----------. .----------. || >> ,-----| PF |---------| VF |-------------------, >> | '----------' '----------' | nvme | >> | DSC | data/control | >> | | path | >> ----------------------------------------------------------- > > Hi Brett, > > what is the class code of the pds_core device ? > > I see that pds_vfio class_code is PCI_CLASS_STORAGE_EXPRESS. The pds_core driver has the following as its only pci_device_id entry: PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) > >> >> >> The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. >> It makes use of and introduces new files in the common include/linux/pds >> include directory. >> >> Brett Creeley (7): >> pds_vfio: Initial support for pds_vfio VFIO driver >> pds_vfio: Add support to register as PDS client >> pds_vfio: Add VFIO live migration support >> vfio: Commonize combine_ranges for use in other VFIO drivers >> pds_vfio: Add support for dirty page tracking >> pds_vfio: Add support for firmware recovery >> pds_vfio: Add documentation files >> >> .../ethernet/pensando/pds_vfio.rst | 88 +++ >> drivers/vfio/pci/Kconfig | 2 + >> drivers/vfio/pci/mlx5/cmd.c | 48 +- >> drivers/vfio/pci/pds/Kconfig | 10 + >> drivers/vfio/pci/pds/Makefile | 12 + >> drivers/vfio/pci/pds/aux_drv.c | 216 +++++++ >> drivers/vfio/pci/pds/aux_drv.h | 30 + >> drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++ >> drivers/vfio/pci/pds/cmds.h | 44 ++ >> drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ >> drivers/vfio/pci/pds/dirty.h | 49 ++ >> drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++ >> drivers/vfio/pci/pds/lm.h | 53 ++ >> drivers/vfio/pci/pds/pci_drv.c | 134 +++++ >> drivers/vfio/pci/pds/pci_drv.h | 9 + >> drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++ >> drivers/vfio/pci/pds/vfio_dev.h | 42 ++ >> drivers/vfio/vfio_main.c | 48 ++ >> include/linux/pds/pds_core_if.h | 1 + >> include/linux/pds/pds_lm.h | 356 ++++++++++++ >> include/linux/vfio.h | 3 + >> 21 files changed, 2847 insertions(+), 47 deletions(-) >> create mode 100644 >> Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst >> create mode 100644 drivers/vfio/pci/pds/Kconfig >> create mode 100644 drivers/vfio/pci/pds/Makefile >> create mode 100644 drivers/vfio/pci/pds/aux_drv.c >> create mode 100644 drivers/vfio/pci/pds/aux_drv.h >> create mode 100644 drivers/vfio/pci/pds/cmds.c >> create mode 100644 drivers/vfio/pci/pds/cmds.h >> create mode 100644 drivers/vfio/pci/pds/dirty.c >> create mode 100644 drivers/vfio/pci/pds/dirty.h >> create mode 100644 drivers/vfio/pci/pds/lm.c >> create mode 100644 drivers/vfio/pci/pds/lm.h >> create mode 100644 drivers/vfio/pci/pds/pci_drv.c >> create mode 100644 drivers/vfio/pci/pds/pci_drv.h >> create mode 100644 drivers/vfio/pci/pds/vfio_dev.c >> create mode 100644 drivers/vfio/pci/pds/vfio_dev.h >> create mode 100644 include/linux/pds/pds_lm.h >> >> -- >> 2.17.1 >>
On 12/11/2022 5:16 PM, Brett Creeley wrote: > > On 12/11/2022 4:54 AM, Max Gurtovoy wrote: >> Caution: This message originated from an External Source. Use proper >> caution when opening attachments, clicking links, or responding. >> >> >> On 12/7/2022 3:06 AM, Brett Creeley wrote: >>> This is a first draft patchset for a new vendor specific VFIO driver for >>> use with the AMD/Pensando Distributed Services Card (DSC). This driver >>> (pds_vfio) is a client of the newly introduced pds_core driver. >>> >>> Reference to the pds_core patchset: >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fnetdev%2F20221207004443.33779-1-shannon.nelson%40amd.com%2F&data=05%7C01%7Cbrett.creeley%40amd.com%7C0591fe11a7c24bf8789908dadb76db84%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638063600829691750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3VMvNtUto4HwCap65NvWrIADbLzGk4Ef9ZnA9fAB458%3D&reserved=0 >>> >>> AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the >>> Distributed Services Card (DSC). This patchset adds the new pds_vfio >>> driver in order to support NVMe VF live migration. >>> >>> This driver will use the pds_core device and auxiliary_bus as the VFIO >>> control path to the DSC. The pds_core device creates auxiliary_bus >>> devices >>> for each live migratable VF. The devices are named by their feature plus >>> the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio >>> can find >>> its related VF PCI driver instance. Once this auxiliary bus connection >>> is configured, the pds_vfio driver can send admin queue commands to the >>> device and receive events from pds_core. >>> >>> An ASCII diagram of a VFIO instance looks something like this and can >>> be used with the VFIO subsystem to provide devices VFIO and live >>> migration support. >>> >>> .------. .--------------------------. >>> | QEMU |--| VM .-------------. | >>> '......' | | nvme driver | | >>> | | .-------------. | >>> | | | SR-IOV VF | | >>> | | '-------------' | >>> | '---------------||---------' >>> .--------------. || >>> |/dev/<vfio_fd>| || >>> '--------------' || >>> Host Userspace | || >>> =================================================== || >>> Host Kernel | || >>> | || >>> pds_core.LM.2305 <--+ .--------. || >>> | | |vfio-pci| || >>> | | '--------' || >>> | | | || >>> .------------. .-------------. || >>> | pds_core | | pds_vfio | || >>> '------------' '-------------' || >>> || || || >>> 09:00.0 09:00.1 || >>> == PCI ==================================================||===== >>> || || || >>> .----------. .----------. || >>> ,-----| PF |---------| VF |-------------------, >>> | '----------' '----------' | nvme | >>> | DSC | data/control | >>> | | path | >>> ----------------------------------------------------------- >> >> Hi Brett, >> >> what is the class code of the pds_core device ? >> >> I see that pds_vfio class_code is PCI_CLASS_STORAGE_EXPRESS. > > The pds_core driver has the following as its only pci_device_id > entry: > > PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) The PCI class code for this device is 0x12 (Processing accelerator). Thanks, Brett > >> >>> >>> >>> The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. >>> It makes use of and introduces new files in the common include/linux/pds >>> include directory. >>> >>> Brett Creeley (7): >>> pds_vfio: Initial support for pds_vfio VFIO driver >>> pds_vfio: Add support to register as PDS client >>> pds_vfio: Add VFIO live migration support >>> vfio: Commonize combine_ranges for use in other VFIO drivers >>> pds_vfio: Add support for dirty page tracking >>> pds_vfio: Add support for firmware recovery >>> pds_vfio: Add documentation files >>> >>> .../ethernet/pensando/pds_vfio.rst | 88 +++ >>> drivers/vfio/pci/Kconfig | 2 + >>> drivers/vfio/pci/mlx5/cmd.c | 48 +- >>> drivers/vfio/pci/pds/Kconfig | 10 + >>> drivers/vfio/pci/pds/Makefile | 12 + >>> drivers/vfio/pci/pds/aux_drv.c | 216 +++++++ >>> drivers/vfio/pci/pds/aux_drv.h | 30 + >>> drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++ >>> drivers/vfio/pci/pds/cmds.h | 44 ++ >>> drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ >>> drivers/vfio/pci/pds/dirty.h | 49 ++ >>> drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++ >>> drivers/vfio/pci/pds/lm.h | 53 ++ >>> drivers/vfio/pci/pds/pci_drv.c | 134 +++++ >>> drivers/vfio/pci/pds/pci_drv.h | 9 + >>> drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++ >>> drivers/vfio/pci/pds/vfio_dev.h | 42 ++ >>> drivers/vfio/vfio_main.c | 48 ++ >>> include/linux/pds/pds_core_if.h | 1 + >>> include/linux/pds/pds_lm.h | 356 ++++++++++++ >>> include/linux/vfio.h | 3 + >>> 21 files changed, 2847 insertions(+), 47 deletions(-) >>> create mode 100644 >>> Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst >>> create mode 100644 drivers/vfio/pci/pds/Kconfig >>> create mode 100644 drivers/vfio/pci/pds/Makefile >>> create mode 100644 drivers/vfio/pci/pds/aux_drv.c >>> create mode 100644 drivers/vfio/pci/pds/aux_drv.h >>> create mode 100644 drivers/vfio/pci/pds/cmds.c >>> create mode 100644 drivers/vfio/pci/pds/cmds.h >>> create mode 100644 drivers/vfio/pci/pds/dirty.c >>> create mode 100644 drivers/vfio/pci/pds/dirty.h >>> create mode 100644 drivers/vfio/pci/pds/lm.c >>> create mode 100644 drivers/vfio/pci/pds/lm.h >>> create mode 100644 drivers/vfio/pci/pds/pci_drv.c >>> create mode 100644 drivers/vfio/pci/pds/pci_drv.h >>> create mode 100644 drivers/vfio/pci/pds/vfio_dev.c >>> create mode 100644 drivers/vfio/pci/pds/vfio_dev.h >>> create mode 100644 include/linux/pds/pds_lm.h >>> >>> -- >>> 2.17.1 >>>
This is a first draft patchset for a new vendor specific VFIO driver for use with the AMD/Pensando Distributed Services Card (DSC). This driver (pds_vfio) is a client of the newly introduced pds_core driver. Reference to the pds_core patchset: https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@amd.com/ AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the Distributed Services Card (DSC). This patchset adds the new pds_vfio driver in order to support NVMe VF live migration. This driver will use the pds_core device and auxiliary_bus as the VFIO control path to the DSC. The pds_core device creates auxiliary_bus devices for each live migratable VF. The devices are named by their feature plus the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find its related VF PCI driver instance. Once this auxiliary bus connection is configured, the pds_vfio driver can send admin queue commands to the device and receive events from pds_core. An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide devices VFIO and live migration support. .------. .--------------------------. | QEMU |--| VM .-------------. | '......' | | nvme driver | | | | .-------------. | | | | SR-IOV VF | | | | '-------------' | | '---------------||---------' .--------------. || |/dev/<vfio_fd>| || '--------------' || Host Userspace | || =================================================== || Host Kernel | || | || pds_core.LM.2305 <--+ .--------. || | | |vfio-pci| || | | '--------' || | | | || .------------. .-------------. || | pds_core | | pds_vfio | || '------------' '-------------' || || || || 09:00.0 09:00.1 || == PCI ==================================================||===== || || || .----------. .----------. || ,-----| PF |---------| VF |-------------------, | '----------' '----------' | nvme | | DSC | data/control | | | path | ----------------------------------------------------------- The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory. Brett Creeley (7): pds_vfio: Initial support for pds_vfio VFIO driver pds_vfio: Add support to register as PDS client pds_vfio: Add VFIO live migration support vfio: Commonize combine_ranges for use in other VFIO drivers pds_vfio: Add support for dirty page tracking pds_vfio: Add support for firmware recovery pds_vfio: Add documentation files .../ethernet/pensando/pds_vfio.rst | 88 +++ drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/mlx5/cmd.c | 48 +- drivers/vfio/pci/pds/Kconfig | 10 + drivers/vfio/pci/pds/Makefile | 12 + drivers/vfio/pci/pds/aux_drv.c | 216 +++++++ drivers/vfio/pci/pds/aux_drv.h | 30 + drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 44 ++ drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 49 ++ drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++ drivers/vfio/pci/pds/lm.h | 53 ++ drivers/vfio/pci/pds/pci_drv.c | 134 +++++ drivers/vfio/pci/pds/pci_drv.h | 9 + drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++ drivers/vfio/pci/pds/vfio_dev.h | 42 ++ drivers/vfio/vfio_main.c | 48 ++ include/linux/pds/pds_core_if.h | 1 + include/linux/pds/pds_lm.h | 356 ++++++++++++ include/linux/vfio.h | 3 + 21 files changed, 2847 insertions(+), 47 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst create mode 100644 drivers/vfio/pci/pds/Kconfig create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/aux_drv.c create mode 100644 drivers/vfio/pci/pds/aux_drv.h create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h create mode 100644 include/linux/pds/pds_lm.h -- 2.17.1