| Message ID | 20181220082350.58113-1-aik@ozlabs.ru (mailing list archive) |
|---|---|
| Headers | show |
| Series | powerpc/powernv/npu, vfio: NVIDIA V100 + P9 passthrough | expand |
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > My bad, I was not cc-ing everyone but now with v7 I am, sorry about that. I've already applied v6, I'll assume this is unchanged from that unless you tell me otherwise. cheers > This is for passing through NVIDIA V100 GPUs on POWER9 systems. > 20/20 has the details of hardware setup. > > This implements support for NVIDIA V100 GPU with coherent memory and > NPU/ATS support available in the POWER9 CPU. The aim is to support > unmodified vendor driver in the guest. > > This is pushed to (both guest and host kernels): > https://github.com/aik/linux/tree/nv2 > > Matching qemu is pushed to github: > https://github.com/aik/qemu/tree/nv2 > > Skiboot bits are here: > https://github.com/aik/skiboot/tree/nv2 > > The individual patches have changelogs. v7 fixes compile warning > and updates a VFIO capability comment in 20/20. > > Please comment. Thanks. > > > > Alexey Kardashevskiy (20): > powerpc/ioda/npu: Call skiboot's hot reset hook when disabling NPU2 > powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a > region > powerpc/vfio/iommu/kvm: Do not pin device memory > powerpc/powernv: Move npu struct from pnv_phb to pci_controller > powerpc/powernv/npu: Move OPAL calls away from context manipulation > powerpc/pseries/iommu: Use memory@ nodes in max RAM address > calculation > powerpc/pseries/npu: Enable platform support > powerpc/pseries: Remove IOMMU API support for non-LPAR systems > powerpc/powernv/pseries: Rework device adding to IOMMU groups > powerpc/iommu_api: Move IOMMU groups setup to a single place > powerpc/powernv: Reference iommu_table while it is linked to a group > powerpc/powernv/npu: Move single TVE handling to NPU PE > powerpc/powernv/npu: Convert NPU IOMMU helpers to > iommu_table_group_ops > powerpc/powernv/npu: Add compound IOMMU groups > powerpc/powernv/npu: Add release_ownership hook > powerpc/powernv/npu: Check mmio_atsd array bounds when populating > powerpc/powernv/npu: Fault user page into the hypervisor's pagetable > vfio_pci: Allow mapping extra regions > vfio_pci: Allow regions to add own capabilities > vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver > > drivers/vfio/pci/Makefile | 1 + > arch/powerpc/include/asm/iommu.h | 17 +- > arch/powerpc/include/asm/mmu_context.h | 15 +- > arch/powerpc/include/asm/pci-bridge.h | 1 + > arch/powerpc/include/asm/pci.h | 4 + > arch/powerpc/platforms/powernv/pci.h | 30 +- > drivers/vfio/pci/trace.h | 102 ++++ > drivers/vfio/pci/vfio_pci_private.h | 20 + > include/uapi/linux/vfio.h | 37 ++ > arch/powerpc/kernel/iommu.c | 69 +-- > arch/powerpc/kvm/book3s_64_vio.c | 18 +- > arch/powerpc/mm/mmu_context_iommu.c | 110 +++- > arch/powerpc/platforms/powernv/npu-dma.c | 549 +++++++++++++++--- > arch/powerpc/platforms/powernv/pci-ioda-tce.c | 3 +- > arch/powerpc/platforms/powernv/pci-ioda.c | 237 ++++---- > arch/powerpc/platforms/powernv/pci.c | 43 +- > arch/powerpc/platforms/pseries/iommu.c | 88 ++- > arch/powerpc/platforms/pseries/pci.c | 22 + > drivers/vfio/pci/vfio_pci.c | 42 +- > drivers/vfio/pci/vfio_pci_nvlink2.c | 482 +++++++++++++++ > drivers/vfio/vfio_iommu_spapr_tce.c | 64 +- > drivers/vfio/pci/Kconfig | 6 + > 22 files changed, 1569 insertions(+), 391 deletions(-) > create mode 100644 drivers/vfio/pci/trace.h > create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c > > -- > 2.17.1
On 20/12/2018 20:38, Michael Ellerman wrote: > Alexey Kardashevskiy <aik@ozlabs.ru> writes: > >> My bad, I was not cc-ing everyone but now with v7 I am, sorry about that. > > I've already applied v6, I'll assume this is unchanged from that unless > you tell me otherwise. 14/20 has fixed warning about uninitialized npdev, 20/20 has fixed comment about one capability: [fstn1-p1 kernel]$ git diff 7e04f09 9128bd1 diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c index ed81426..12b8421 100644 --- a/arch/powerpc/platforms/powernv/npu-dma.c +++ b/arch/powerpc/platforms/powernv/npu-dma.c @@ -540,7 +540,7 @@ struct iommu_table_group *pnv_try_setup_npu_table_group(struct pnv_ioda_pe *pe) struct npu_comp *npucomp; struct pci_dev *gpdev = NULL; struct pci_controller *hose; - struct pci_dev *npdev; + struct pci_dev *npdev = NULL; list_for_each_entry(gpdev, &pe->pbus->devices, bus_list) { npdev = pnv_pci_get_npu_dev(gpdev, 0); diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 22b825c..5562587 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -390,8 +390,7 @@ struct vfio_region_info_cap_nvlink2_ssatgt { }; /* - * Capability with compressed real address (aka SSA - small system address), - * used to match the NVLink bridge with a GPU. Also contains a link speed. + * Capability with an NVLink link speed. */ #define VFIO_REGION_INFO_CAP_NVLINK2_LNKSPD 5 > > cheers > >> This is for passing through NVIDIA V100 GPUs on POWER9 systems. >> 20/20 has the details of hardware setup. >> >> This implements support for NVIDIA V100 GPU with coherent memory and >> NPU/ATS support available in the POWER9 CPU. The aim is to support >> unmodified vendor driver in the guest. >> >> This is pushed to (both guest and host kernels): >> https://github.com/aik/linux/tree/nv2 >> >> Matching qemu is pushed to github: >> https://github.com/aik/qemu/tree/nv2 >> >> Skiboot bits are here: >> https://github.com/aik/skiboot/tree/nv2 >> >> The individual patches have changelogs. v7 fixes compile warning >> and updates a VFIO capability comment in 20/20. >> >> Please comment. Thanks. >> >> >> >> Alexey Kardashevskiy (20): >> powerpc/ioda/npu: Call skiboot's hot reset hook when disabling NPU2 >> powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a >> region >> powerpc/vfio/iommu/kvm: Do not pin device memory >> powerpc/powernv: Move npu struct from pnv_phb to pci_controller >> powerpc/powernv/npu: Move OPAL calls away from context manipulation >> powerpc/pseries/iommu: Use memory@ nodes in max RAM address >> calculation >> powerpc/pseries/npu: Enable platform support >> powerpc/pseries: Remove IOMMU API support for non-LPAR systems >> powerpc/powernv/pseries: Rework device adding to IOMMU groups >> powerpc/iommu_api: Move IOMMU groups setup to a single place >> powerpc/powernv: Reference iommu_table while it is linked to a group >> powerpc/powernv/npu: Move single TVE handling to NPU PE >> powerpc/powernv/npu: Convert NPU IOMMU helpers to >> iommu_table_group_ops >> powerpc/powernv/npu: Add compound IOMMU groups >> powerpc/powernv/npu: Add release_ownership hook >> powerpc/powernv/npu: Check mmio_atsd array bounds when populating >> powerpc/powernv/npu: Fault user page into the hypervisor's pagetable >> vfio_pci: Allow mapping extra regions >> vfio_pci: Allow regions to add own capabilities >> vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver >> >> drivers/vfio/pci/Makefile | 1 + >> arch/powerpc/include/asm/iommu.h | 17 +- >> arch/powerpc/include/asm/mmu_context.h | 15 +- >> arch/powerpc/include/asm/pci-bridge.h | 1 + >> arch/powerpc/include/asm/pci.h | 4 + >> arch/powerpc/platforms/powernv/pci.h | 30 +- >> drivers/vfio/pci/trace.h | 102 ++++ >> drivers/vfio/pci/vfio_pci_private.h | 20 + >> include/uapi/linux/vfio.h | 37 ++ >> arch/powerpc/kernel/iommu.c | 69 +-- >> arch/powerpc/kvm/book3s_64_vio.c | 18 +- >> arch/powerpc/mm/mmu_context_iommu.c | 110 +++- >> arch/powerpc/platforms/powernv/npu-dma.c | 549 +++++++++++++++--- >> arch/powerpc/platforms/powernv/pci-ioda-tce.c | 3 +- >> arch/powerpc/platforms/powernv/pci-ioda.c | 237 ++++---- >> arch/powerpc/platforms/powernv/pci.c | 43 +- >> arch/powerpc/platforms/pseries/iommu.c | 88 ++- >> arch/powerpc/platforms/pseries/pci.c | 22 + >> drivers/vfio/pci/vfio_pci.c | 42 +- >> drivers/vfio/pci/vfio_pci_nvlink2.c | 482 +++++++++++++++ >> drivers/vfio/vfio_iommu_spapr_tce.c | 64 +- >> drivers/vfio/pci/Kconfig | 6 + >> 22 files changed, 1569 insertions(+), 391 deletions(-) >> create mode 100644 drivers/vfio/pci/trace.h >> create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c >> >> -- >> 2.17.1
My bad, I was not cc-ing everyone but now with v7 I am, sorry about that. This is for passing through NVIDIA V100 GPUs on POWER9 systems. 20/20 has the details of hardware setup. This implements support for NVIDIA V100 GPU with coherent memory and NPU/ATS support available in the POWER9 CPU. The aim is to support unmodified vendor driver in the guest. This is pushed to (both guest and host kernels): https://github.com/aik/linux/tree/nv2 Matching qemu is pushed to github: https://github.com/aik/qemu/tree/nv2 Skiboot bits are here: https://github.com/aik/skiboot/tree/nv2 The individual patches have changelogs. v7 fixes compile warning and updates a VFIO capability comment in 20/20. Please comment. Thanks. Alexey Kardashevskiy (20): powerpc/ioda/npu: Call skiboot's hot reset hook when disabling NPU2 powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a region powerpc/vfio/iommu/kvm: Do not pin device memory powerpc/powernv: Move npu struct from pnv_phb to pci_controller powerpc/powernv/npu: Move OPAL calls away from context manipulation powerpc/pseries/iommu: Use memory@ nodes in max RAM address calculation powerpc/pseries/npu: Enable platform support powerpc/pseries: Remove IOMMU API support for non-LPAR systems powerpc/powernv/pseries: Rework device adding to IOMMU groups powerpc/iommu_api: Move IOMMU groups setup to a single place powerpc/powernv: Reference iommu_table while it is linked to a group powerpc/powernv/npu: Move single TVE handling to NPU PE powerpc/powernv/npu: Convert NPU IOMMU helpers to iommu_table_group_ops powerpc/powernv/npu: Add compound IOMMU groups powerpc/powernv/npu: Add release_ownership hook powerpc/powernv/npu: Check mmio_atsd array bounds when populating powerpc/powernv/npu: Fault user page into the hypervisor's pagetable vfio_pci: Allow mapping extra regions vfio_pci: Allow regions to add own capabilities vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver drivers/vfio/pci/Makefile | 1 + arch/powerpc/include/asm/iommu.h | 17 +- arch/powerpc/include/asm/mmu_context.h | 15 +- arch/powerpc/include/asm/pci-bridge.h | 1 + arch/powerpc/include/asm/pci.h | 4 + arch/powerpc/platforms/powernv/pci.h | 30 +- drivers/vfio/pci/trace.h | 102 ++++ drivers/vfio/pci/vfio_pci_private.h | 20 + include/uapi/linux/vfio.h | 37 ++ arch/powerpc/kernel/iommu.c | 69 +-- arch/powerpc/kvm/book3s_64_vio.c | 18 +- arch/powerpc/mm/mmu_context_iommu.c | 110 +++- arch/powerpc/platforms/powernv/npu-dma.c | 549 +++++++++++++++--- arch/powerpc/platforms/powernv/pci-ioda-tce.c | 3 +- arch/powerpc/platforms/powernv/pci-ioda.c | 237 ++++---- arch/powerpc/platforms/powernv/pci.c | 43 +- arch/powerpc/platforms/pseries/iommu.c | 88 ++- arch/powerpc/platforms/pseries/pci.c | 22 + drivers/vfio/pci/vfio_pci.c | 42 +- drivers/vfio/pci/vfio_pci_nvlink2.c | 482 +++++++++++++++ drivers/vfio/vfio_iommu_spapr_tce.c | 64 +- drivers/vfio/pci/Kconfig | 6 + 22 files changed, 1569 insertions(+), 391 deletions(-) create mode 100644 drivers/vfio/pci/trace.h create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c