| Message ID | 0-v3-50561e12d92b+313-vfio_iommufd_jgg@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Connect VFIO to IOMMUFD | expand |
Hi Jason, On 2022/11/17 05:05, Jason Gunthorpe wrote: > This series provides an alternative container layer for VFIO implemented > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will > not be compiled in. > > At this point iommufd can be injected by passing in a iommfd FD to > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd > to obtain the compat IOAS and then connect up all the VFIO drivers as > appropriate. > > This is temporary stopping point, a following series will provide a way to > directly open a VFIO device FD and directly connect it to IOMMUFD using > native ioctls that can expose the IOMMUFD features like hwpt, future > vPASID and dynamic attachment. > > This series, in compat mode, has passed all the qemu tests we have > available, including the test suites for the Intel GVT mdev. Aside from > the temporary limitation with P2P memory this is belived to be fully > compatible with VFIO. > > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd > > It requires the iommufd series: > > https://lore.kernel.org/r/0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com gvtg test encountered broken display with below commit in your for-next branch. https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/commit/?h=for-next&id=57f62422b6f0477afaddd2fc77a4bb9b94275f42 I noticed there are diffs in drivers/vfio/ and drivers/iommu/iommufd/ between this commit and the last tested commit (37c9e6e44d77a). Seems to have regression due to the diffs. > v3: > - Fix iommufd_attached to be only used in the vfio_iommufd_physical_* > funcs > - Always check for iommufd before invoking a iommufd function > - Fix mismatch between vfio_pin_pages and iommufd_access when the IOVA > is not aligned. Resolves problems on S390 > v2: https://lore.kernel.org/r/0-v2-65016290f146+33e-vfio_iommufd_jgg@nvidia.com > - Rebase to v6.1-rc3, v4 iommufd series > - Fixup comments and commit messages from list remarks > - Fix leaking of the iommufd for mdevs > - New patch to fix vfio modaliases when vfio container is disabled > - Add a dmesg once when the iommufd provided /dev/vfio/vfio is opened > to signal that iommufd is providing this > v1: https://lore.kernel.org/r/0-v1-4991695894d8+211-vfio_iommufd_jgg@nvidia.com > > Jason Gunthorpe (11): > vfio: Move vfio_device driver open/close code to a function > vfio: Move vfio_device_assign_container() into > vfio_device_first_open() > vfio: Rename vfio_device_assign/unassign_container() > vfio: Move storage of allow_unsafe_interrupts to vfio_main.c > vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for > vfio_file_enforced_coherent() > vfio-iommufd: Allow iommufd to be used in place of a container fd > vfio-iommufd: Support iommufd for physical VFIO devices > vfio-iommufd: Support iommufd for emulated VFIO devices > vfio: Move container related MODULE_ALIAS statements into container.c > vfio: Make vfio_container optionally compiled > iommufd: Allow iommufd to supply /dev/vfio/vfio > > drivers/gpu/drm/i915/gvt/kvmgt.c | 3 + > drivers/iommu/iommufd/Kconfig | 12 + > drivers/iommu/iommufd/main.c | 36 ++ > drivers/s390/cio/vfio_ccw_ops.c | 3 + > drivers/s390/crypto/vfio_ap_ops.c | 3 + > drivers/vfio/Kconfig | 36 +- > drivers/vfio/Makefile | 5 +- > drivers/vfio/container.c | 141 ++----- > drivers/vfio/fsl-mc/vfio_fsl_mc.c | 3 + > drivers/vfio/iommufd.c | 161 ++++++++ > .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 6 + > drivers/vfio/pci/mlx5/main.c | 3 + > drivers/vfio/pci/vfio_pci.c | 3 + > drivers/vfio/platform/vfio_amba.c | 3 + > drivers/vfio/platform/vfio_platform.c | 3 + > drivers/vfio/vfio.h | 100 ++++- > drivers/vfio/vfio_iommu_type1.c | 5 +- > drivers/vfio/vfio_main.c | 348 ++++++++++++++---- > include/linux/vfio.h | 39 ++ > 19 files changed, 714 insertions(+), 199 deletions(-) > create mode 100644 drivers/vfio/iommufd.c > > > base-commit: 9d367dc905dd278614aaf601afb28e511b82fb3b
On Wed, Nov 23, 2022 at 10:44:12AM +0800, Yi Liu wrote: > Hi Jason, > > On 2022/11/17 05:05, Jason Gunthorpe wrote: > > This series provides an alternative container layer for VFIO implemented > > using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will > > not be compiled in. > > > > At this point iommufd can be injected by passing in a iommfd FD to > > VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd > > to obtain the compat IOAS and then connect up all the VFIO drivers as > > appropriate. > > > > This is temporary stopping point, a following series will provide a way to > > directly open a VFIO device FD and directly connect it to IOMMUFD using > > native ioctls that can expose the IOMMUFD features like hwpt, future > > vPASID and dynamic attachment. > > > > This series, in compat mode, has passed all the qemu tests we have > > available, including the test suites for the Intel GVT mdev. Aside from > > the temporary limitation with P2P memory this is belived to be fully > > compatible with VFIO. > > > > This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd > > > > It requires the iommufd series: > > > > https://lore.kernel.org/r/0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com > > gvtg test encountered broken display with below commit in your for-next > branch. > > https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/commit/?h=for-next&id=57f62422b6f0477afaddd2fc77a4bb9b94275f42 > > I noticed there are diffs in drivers/vfio/ and drivers/iommu/iommufd/ > between this commit and the last tested commit (37c9e6e44d77a). Seems > to have regression due to the diffs. Do you have something more to go on? I am checking the diff and not getting any idea. The above also merges v6.1-rc5 into the tree, is there a chance rc5 is the gvt problem? Jason
On 2022/11/23 20:59, Jason Gunthorpe wrote: > On Wed, Nov 23, 2022 at 10:44:12AM +0800, Yi Liu wrote: >> Hi Jason, >> >> On 2022/11/17 05:05, Jason Gunthorpe wrote: >>> This series provides an alternative container layer for VFIO implemented >>> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will >>> not be compiled in. >>> >>> At this point iommufd can be injected by passing in a iommfd FD to >>> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd >>> to obtain the compat IOAS and then connect up all the VFIO drivers as >>> appropriate. >>> >>> This is temporary stopping point, a following series will provide a way to >>> directly open a VFIO device FD and directly connect it to IOMMUFD using >>> native ioctls that can expose the IOMMUFD features like hwpt, future >>> vPASID and dynamic attachment. >>> >>> This series, in compat mode, has passed all the qemu tests we have >>> available, including the test suites for the Intel GVT mdev. Aside from >>> the temporary limitation with P2P memory this is belived to be fully >>> compatible with VFIO. >>> >>> This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd >>> >>> It requires the iommufd series: >>> >>> https://lore.kernel.org/r/0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com >> >> gvtg test encountered broken display with below commit in your for-next >> branch. >> >> https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/commit/?h=for-next&id=57f62422b6f0477afaddd2fc77a4bb9b94275f42 >> >> I noticed there are diffs in drivers/vfio/ and drivers/iommu/iommufd/ >> between this commit and the last tested commit (37c9e6e44d77a). Seems >> to have regression due to the diffs. > > Do you have something more to go on? I am checking the diff and not > getting any idea. The above also merges v6.1-rc5 into the tree, is > there a chance rc5 is the gvt problem? that is possible, I'll let my colleague revert it and try.
On 2022/11/23 21:04, Yi Liu wrote: > On 2022/11/23 20:59, Jason Gunthorpe wrote: >> On Wed, Nov 23, 2022 at 10:44:12AM +0800, Yi Liu wrote: >>> Hi Jason, >>> >>> On 2022/11/17 05:05, Jason Gunthorpe wrote: >>>> This series provides an alternative container layer for VFIO implemented >>>> using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will >>>> not be compiled in. >>>> >>>> At this point iommufd can be injected by passing in a iommfd FD to >>>> VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd >>>> to obtain the compat IOAS and then connect up all the VFIO drivers as >>>> appropriate. >>>> >>>> This is temporary stopping point, a following series will provide a way to >>>> directly open a VFIO device FD and directly connect it to IOMMUFD using >>>> native ioctls that can expose the IOMMUFD features like hwpt, future >>>> vPASID and dynamic attachment. >>>> >>>> This series, in compat mode, has passed all the qemu tests we have >>>> available, including the test suites for the Intel GVT mdev. Aside from >>>> the temporary limitation with P2P memory this is belived to be fully >>>> compatible with VFIO. >>>> >>>> This is on github: >>>> https://github.com/jgunthorpe/linux/commits/vfio_iommufd >>>> >>>> It requires the iommufd series: >>>> >>>> https://lore.kernel.org/r/0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com >>> >>> gvtg test encountered broken display with below commit in your for-next >>> branch. >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/commit/?h=for-next&id=57f62422b6f0477afaddd2fc77a4bb9b94275f42 >>> >>> >>> I noticed there are diffs in drivers/vfio/ and drivers/iommu/iommufd/ >>> between this commit and the last tested commit (37c9e6e44d77a). Seems >>> to have regression due to the diffs. >> >> Do you have something more to go on? I am checking the diff and not >> getting any idea. The above also merges v6.1-rc5 into the tree, is >> there a chance rc5 is the gvt problem? > > that is possible, I'll let my colleague revert it and try. after reverting the v6.1-rc5 merge commit, the test is passed. also tried your latest branch which has v6.1-rc6 merge commit, still observed blank display. but this time, this issue can be recovered by restarting gdm. It looks like the regression is not due to iommufd. May due to diff in gvt and i915 itself. So reported a regression as below link. https://lists.freedesktop.org/archives/intel-gvt-dev/2022-November/011488.html
This series provides an alternative container layer for VFIO implemented using iommufd. This is optional, if CONFIG_IOMMUFD is not set then it will not be compiled in. At this point iommufd can be injected by passing in a iommfd FD to VFIO_GROUP_SET_CONTAINER which will use the VFIO compat layer in iommufd to obtain the compat IOAS and then connect up all the VFIO drivers as appropriate. This is temporary stopping point, a following series will provide a way to directly open a VFIO device FD and directly connect it to IOMMUFD using native ioctls that can expose the IOMMUFD features like hwpt, future vPASID and dynamic attachment. This series, in compat mode, has passed all the qemu tests we have available, including the test suites for the Intel GVT mdev. Aside from the temporary limitation with P2P memory this is belived to be fully compatible with VFIO. This is on github: https://github.com/jgunthorpe/linux/commits/vfio_iommufd It requires the iommufd series: https://lore.kernel.org/r/0-v5-4001c2997bd0+30c-iommufd_jgg@nvidia.com v3: - Fix iommufd_attached to be only used in the vfio_iommufd_physical_* funcs - Always check for iommufd before invoking a iommufd function - Fix mismatch between vfio_pin_pages and iommufd_access when the IOVA is not aligned. Resolves problems on S390 v2: https://lore.kernel.org/r/0-v2-65016290f146+33e-vfio_iommufd_jgg@nvidia.com - Rebase to v6.1-rc3, v4 iommufd series - Fixup comments and commit messages from list remarks - Fix leaking of the iommufd for mdevs - New patch to fix vfio modaliases when vfio container is disabled - Add a dmesg once when the iommufd provided /dev/vfio/vfio is opened to signal that iommufd is providing this v1: https://lore.kernel.org/r/0-v1-4991695894d8+211-vfio_iommufd_jgg@nvidia.com Jason Gunthorpe (11): vfio: Move vfio_device driver open/close code to a function vfio: Move vfio_device_assign_container() into vfio_device_first_open() vfio: Rename vfio_device_assign/unassign_container() vfio: Move storage of allow_unsafe_interrupts to vfio_main.c vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent() vfio-iommufd: Allow iommufd to be used in place of a container fd vfio-iommufd: Support iommufd for physical VFIO devices vfio-iommufd: Support iommufd for emulated VFIO devices vfio: Move container related MODULE_ALIAS statements into container.c vfio: Make vfio_container optionally compiled iommufd: Allow iommufd to supply /dev/vfio/vfio drivers/gpu/drm/i915/gvt/kvmgt.c | 3 + drivers/iommu/iommufd/Kconfig | 12 + drivers/iommu/iommufd/main.c | 36 ++ drivers/s390/cio/vfio_ccw_ops.c | 3 + drivers/s390/crypto/vfio_ap_ops.c | 3 + drivers/vfio/Kconfig | 36 +- drivers/vfio/Makefile | 5 +- drivers/vfio/container.c | 141 ++----- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 3 + drivers/vfio/iommufd.c | 161 ++++++++ .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 6 + drivers/vfio/pci/mlx5/main.c | 3 + drivers/vfio/pci/vfio_pci.c | 3 + drivers/vfio/platform/vfio_amba.c | 3 + drivers/vfio/platform/vfio_platform.c | 3 + drivers/vfio/vfio.h | 100 ++++- drivers/vfio/vfio_iommu_type1.c | 5 +- drivers/vfio/vfio_main.c | 348 ++++++++++++++---- include/linux/vfio.h | 39 ++ 19 files changed, 714 insertions(+), 199 deletions(-) create mode 100644 drivers/vfio/iommufd.c base-commit: 9d367dc905dd278614aaf601afb28e511b82fb3b