mbox series

[v4,00/17] IOMMUFD Generic interface

Message ID 0-v4-0de2f6c78ed0+9d1-iommufd_jgg@nvidia.com (mailing list archive)
Headers show
Series IOMMUFD Generic interface | expand

Message

Jason Gunthorpe Nov. 8, 2022, 12:48 a.m. UTC
[
This has been in linux-next for a little while now, and we've completed
the syzkaller run. 1300 hours of CPU time have been invested since the
last report with no improvement in coverage or new detections. syzkaller
coverage reached 69%(75%), and review of the misses show substantial
amounts are WARN_ON's and other debugging which are not expected to be
covered.
]

iommufd is the user API to control the IOMMU subsystem as it relates to
managing IO page tables that point at user space memory.

It takes over from drivers/vfio/vfio_iommu_type1.c (aka the VFIO
container) which is the VFIO specific interface for a similar idea.

We see a broad need for extended features, some being highly IOMMU device
specific:
 - Binding iommu_domain's to PASID/SSID
 - Userspace IO page tables, for ARM, x86 and S390
 - Kernel bypassed invalidation of user page tables
 - Re-use of the KVM page table in the IOMMU
 - Dirty page tracking in the IOMMU
 - Runtime Increase/Decrease of IOPTE size
 - PRI support with faults resolved in userspace

Many of these HW features exist to support VM use cases - for instance the
combination of PASID, PRI and Userspace IO Page Tables allows an
implementation of DMA Shared Virtual Addressing (vSVA) within a
guest. Dirty tracking enables VM live migration with SRIOV devices and
PASID support allow creating "scalable IOV" devices, among other things.

As these features are fundamental to a VM platform they need to be
uniformly exposed to all the driver families that do DMA into VMs, which
is currently VFIO and VDPA.

The pre-v1 series proposed re-using the VFIO type 1 data structure,
however it was suggested that if we are doing this big update then we
should also come with an improved data structure that solves the
limitations that VFIO type1 has. Notably this addresses:

 - Multiple IOAS/'containers' and multiple domains inside a single FD

 - Single-pin operation no matter how many domains and containers use
   a page

 - A fine grained locking scheme supporting user managed concurrency for
   multi-threaded map/unmap

 - A pre-registration mechanism to optimize vIOMMU use cases by
   pre-pinning pages

 - Extended ioctl API that can manage these new objects and exposes
   domains directly to user space

 - domains are sharable between subsystems, eg VFIO and VDPA

The bulk of this code is a new data structure design to track how the
IOVAs are mapped to PFNs.

iommufd intends to be general and consumable by any driver that wants to
DMA to userspace. From a driver perspective it can largely be dropped in
in-place of iommu_attach_device() and provides a uniform full feature set
to all consumers.

As this is a larger project this series is the first step. This series
provides the iommfd "generic interface" which is designed to be suitable
for applications like DPDK and VMM flows that are not optimized to
specific HW scenarios. It is close to being a drop in replacement for the
existing VFIO type 1 and supports existing qemu based VM flows.

Several follow-on series are being prepared:

- Patches integrating with qemu in native mode:
  https://github.com/yiliu1765/qemu/commits/qemu-iommufd-6.0-rc2

- A completed integration with VFIO now exists that covers "emulated" mdev
  use cases now, and can pass testing with qemu/etc in compatability mode:
  https://github.com/jgunthorpe/linux/commits/vfio_iommufd

- A draft providing system iommu dirty tracking on top of iommufd,
  including iommu driver implementations:
  https://github.com/jpemartins/linux/commits/x86-iommufd

  This pairs with patches for providing a similar API to support VFIO-device
  tracking to give a complete vfio solution:
  https://lore.kernel.org/kvm/20220901093853.60194-1-yishaih@nvidia.com/

- Userspace page tables aka 'nested translation' for ARM and Intel iommu
  drivers:
  https://github.com/nicolinc/iommufd/commits/iommufd_nesting

- "device centric" vfio series to expose the vfio_device FD directly as a
  normal cdev, and provide an extended API allowing dynamically changing
  the IOAS binding:
  https://github.com/yiliu1765/iommufd/commits/iommufd-v6.0-rc2-nesting-0901

- Drafts for PASID and PRI interfaces are included above as well

Overall enough work is done now to show the merit of the new API design
and at least draft solutions to many of the main problems.

Several people have contributed directly to this work: Eric Auger, Joao
Martins, Kevin Tian, Lu Baolu, Nicolin Chen, Yi L Liu. Many more have
participated in the discussions that lead here, and provided ideas. Thanks
to all!

The v1/v2 iommufd series has been used to guide a large amount of preparatory
work that has now been merged. The general theme is to organize things in
a way that makes injecting iommufd natural:

 - VFIO live migration support with mlx5 and hisi_acc drivers.
   These series need a dirty tracking solution to be really usable.
   https://lore.kernel.org/kvm/20220224142024.147653-1-yishaih@nvidia.com/
   https://lore.kernel.org/kvm/20220308184902.2242-1-shameerali.kolothum.thodi@huawei.com/

 - Significantly rework the VFIO gvt mdev and remove struct
   mdev_parent_ops
   https://lore.kernel.org/lkml/20220411141403.86980-1-hch@lst.de/

 - Rework how PCIe no-snoop blocking works
   https://lore.kernel.org/kvm/0-v3-2cf356649677+a32-intel_no_snoop_jgg@nvidia.com/

 - Consolidate dma ownership into the iommu core code
   https://lore.kernel.org/linux-iommu/20220418005000.897664-1-baolu.lu@linux.intel.com/

 - Make all vfio driver interfaces use struct vfio_device consistently
   https://lore.kernel.org/kvm/0-v4-8045e76bf00b+13d-vfio_mdev_no_group_jgg@nvidia.com/

 - Remove the vfio_group from the kvm/vfio interface
   https://lore.kernel.org/kvm/0-v3-f7729924a7ea+25e33-vfio_kvm_no_group_jgg@nvidia.com/

 - Simplify locking in vfio
   https://lore.kernel.org/kvm/0-v2-d035a1842d81+1bf-vfio_group_locking_jgg@nvidia.com/

 - Remove the vfio notifiter scheme that faces drivers
   https://lore.kernel.org/kvm/0-v4-681e038e30fd+78-vfio_unmap_notif_jgg@nvidia.com/

 - Improve the driver facing API for vfio pin/unpin pages to make the
   presence of struct page clear
   https://lore.kernel.org/kvm/20220723020256.30081-1-nicolinc@nvidia.com/

 - Clean up in the Intel IOMMU driver
   https://lore.kernel.org/linux-iommu/20220301020159.633356-1-baolu.lu@linux.intel.com/
   https://lore.kernel.org/linux-iommu/20220510023407.2759143-1-baolu.lu@linux.intel.com/
   https://lore.kernel.org/linux-iommu/20220514014322.2927339-1-baolu.lu@linux.intel.com/
   https://lore.kernel.org/linux-iommu/20220706025524.2904370-1-baolu.lu@linux.intel.com/
   https://lore.kernel.org/linux-iommu/20220702015610.2849494-1-baolu.lu@linux.intel.com/

 - Rework s390 vfio drivers
   https://lore.kernel.org/kvm/20220707135737.720765-1-farman@linux.ibm.com/

 - Normalize vfio ioctl handling
   https://lore.kernel.org/kvm/0-v2-0f9e632d54fb+d6-vfio_ioctl_split_jgg@nvidia.com/

 - VFIO API for dirty tracking (aka dma logging) managed inside a PCI
   device, with mlx5 implementation
   https://lore.kernel.org/kvm/20220901093853.60194-1-yishaih@nvidia.com

 - Introduce a struct device sysfs presence for struct vfio_device
   https://lore.kernel.org/kvm/20220901143747.32858-1-kevin.tian@intel.com/

 - Complete restructuring the vfio mdev model
   https://lore.kernel.org/kvm/20220822062208.152745-1-hch@lst.de/

 - Isolate VFIO container code in preperation for iommufd to provide an
   alternative implementation of it all
   https://lore.kernel.org/kvm/0-v1-a805b607f1fb+17b-vfio_container_split_jgg@nvidia.com

 - Simplify and consolidate iommu_domain/device compatability checking
   https://lore.kernel.org/linux-iommu/cover.1666042872.git.nicolinc@nvidia.com/

 - Align iommu SVA support with the domain-centric model
   https://lore.kernel.org/all/20221031005917.45690-1-baolu.lu@linux.intel.com/

This is about 233 patches applied since March, thank you to everyone
involved in all this work!

Currently there are a number of supporting series still in progress:

 - DMABUF exporter support for VFIO to allow PCI P2P with VFIO
   https://lore.kernel.org/r/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com

 - Start to provide iommu_domain ops for POWER
   https://lore.kernel.org/all/20220714081822.3717693-1-aik@ozlabs.ru/

However, these are not necessary for this series to advance.

This is on github: https://github.com/jgunthorpe/linux/commits/iommufd

v4:
 - Rebase to v6.1-rc3, include the iommu branch with the needed EINVAL
   patch series and also the SVA rework
 - All bug fixes and comments with no API or behavioral changes
 - gvt tests are passing again
 - Syzkaller is no longer finding issues and achieved high coverage of
   69%(75%)
 - Coverity has been run by two people
 - new "nth failure" test that systematically sweeps all error unwind paths
   looking for splats
 - All fixes noted in the mailing list
   If you sent an email and I didn't reply please ping it, I have lost it.
 - The selftest patch has been broken into three to make the additional
   modification to the main code clearer
 - The interdiff is 1.8k lines for the main code, with another 3k of
   test suite changes
v3: https://lore.kernel.org/r/0-v3-402a7d6459de+24b-iommufd_jgg@nvidia.com
 - Rebase to v6.1-rc1
 - Improve documentation
 - Use EXPORT_SYMBOL_NS
 - Fix W1, checkpatch stuff
 - Revise pages.c to resolve the FIXMEs. Create a
   interval_tree_double_span_iter which allows a simple expression of the
   previously problematic algorithms
 - Consistently use the word 'access' instead of user to refer to an
   access from an in-kernel user (eg vfio mdev)
 - Support two forms of rlimit accounting and make the vfio compatible one
   the default in compatability mode (following series)
 - Support old VFIO type1 by disabling huge pages and implementing a
   simple algorithm to split a struct iopt_area
 - Full implementation of access support, test coverage and optimizations
 - Complete COPY to be able to copy across contiguous areas. Improve
   all the algorithms around contiguous areas with a dedicated iterator
 - Functional ENFORCED_COHERENT support
 - Support multi-device groups
 - Lots of smaller changes (the interdiff is 5k lines)
v2: https://lore.kernel.org/r/0-v2-f9436d0bde78+4bb-iommufd_jgg@nvidia.com
 - Rebase to v6.0-rc3
 - Improve comments
 - Change to an iterative destruction approach to avoid cycles
 - Near rewrite of the vfio facing implementation, supported by a complete
   implementation on the vfio side
 - New IOMMU_IOAS_ALLOW_IOVAS API as discussed. Allows userspace to
   assert that ranges of IOVA must always be mappable. To be used by a VMM
   that has promised a guest a certain availability of IOVA. May help
   guide PPC's multi-window implementation.
 - Rework how unmap_iova works, user can unmap the whole ioas now
 - The no-snoop / wbinvd support is implemented
 - Bug fixes
 - Test suite improvements
 - Lots of smaller changes (the interdiff is 3k lines)
v1: https://lore.kernel.org/r/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com

Jason Gunthorpe (15):
  iommu: Add IOMMU_CAP_ENFORCE_CACHE_COHERENCY
  interval-tree: Add a utility to iterate over spans in an interval tree
  iommufd: File descriptor, context, kconfig and makefiles
  kernel/user: Allow user::locked_vm to be usable for iommufd
  iommufd: PFN handling for iopt_pages
  iommufd: Algorithms for PFN storage
  iommufd: Data structure to provide IOVA to PFN mapping
  iommufd: IOCTLs for the io_pagetable
  iommufd: Add a HW pagetable object
  iommufd: Add kAPI toward external drivers for physical devices
  iommufd: Add kAPI toward external drivers for kernel access
  iommufd: vfio container FD ioctl compatibility
  iommufd: Add a selftest
  iommufd: Add some fault injection points
  iommufd: Add additional invariant assertions

Kevin Tian (1):
  iommufd: Document overview of iommufd

Lu Baolu (1):
  iommu: Add device-centric DMA ownership interfaces

 .clang-format                                 |    3 +
 Documentation/userspace-api/index.rst         |    1 +
 .../userspace-api/ioctl/ioctl-number.rst      |    1 +
 Documentation/userspace-api/iommufd.rst       |  222 ++
 MAINTAINERS                                   |   12 +
 drivers/iommu/Kconfig                         |    1 +
 drivers/iommu/Makefile                        |    2 +-
 drivers/iommu/amd/iommu.c                     |    2 +
 drivers/iommu/intel/iommu.c                   |   16 +-
 drivers/iommu/iommu.c                         |  124 +-
 drivers/iommu/iommufd/Kconfig                 |   23 +
 drivers/iommu/iommufd/Makefile                |   13 +
 drivers/iommu/iommufd/device.c                |  748 +++++++
 drivers/iommu/iommufd/double_span.h           |   98 +
 drivers/iommu/iommufd/hw_pagetable.c          |   57 +
 drivers/iommu/iommufd/io_pagetable.c          | 1214 +++++++++++
 drivers/iommu/iommufd/io_pagetable.h          |  241 +++
 drivers/iommu/iommufd/ioas.c                  |  390 ++++
 drivers/iommu/iommufd/iommufd_private.h       |  307 +++
 drivers/iommu/iommufd/iommufd_test.h          |   93 +
 drivers/iommu/iommufd/main.c                  |  419 ++++
 drivers/iommu/iommufd/pages.c                 | 1884 +++++++++++++++++
 drivers/iommu/iommufd/selftest.c              |  853 ++++++++
 drivers/iommu/iommufd/vfio_compat.c           |  452 ++++
 include/linux/interval_tree.h                 |   58 +
 include/linux/iommu.h                         |   17 +
 include/linux/iommufd.h                       |  102 +
 include/linux/sched/user.h                    |    2 +-
 include/uapi/linux/iommufd.h                  |  332 +++
 kernel/user.c                                 |    1 +
 lib/Kconfig                                   |    4 +
 lib/interval_tree.c                           |  132 ++
 tools/testing/selftests/Makefile              |    1 +
 tools/testing/selftests/iommu/.gitignore      |    3 +
 tools/testing/selftests/iommu/Makefile        |   12 +
 tools/testing/selftests/iommu/config          |    2 +
 tools/testing/selftests/iommu/iommufd.c       | 1627 ++++++++++++++
 .../selftests/iommu/iommufd_fail_nth.c        |  580 +++++
 tools/testing/selftests/iommu/iommufd_utils.h |  278 +++
 39 files changed, 10294 insertions(+), 33 deletions(-)
 create mode 100644 Documentation/userspace-api/iommufd.rst
 create mode 100644 drivers/iommu/iommufd/Kconfig
 create mode 100644 drivers/iommu/iommufd/Makefile
 create mode 100644 drivers/iommu/iommufd/device.c
 create mode 100644 drivers/iommu/iommufd/double_span.h
 create mode 100644 drivers/iommu/iommufd/hw_pagetable.c
 create mode 100644 drivers/iommu/iommufd/io_pagetable.c
 create mode 100644 drivers/iommu/iommufd/io_pagetable.h
 create mode 100644 drivers/iommu/iommufd/ioas.c
 create mode 100644 drivers/iommu/iommufd/iommufd_private.h
 create mode 100644 drivers/iommu/iommufd/iommufd_test.h
 create mode 100644 drivers/iommu/iommufd/main.c
 create mode 100644 drivers/iommu/iommufd/pages.c
 create mode 100644 drivers/iommu/iommufd/selftest.c
 create mode 100644 drivers/iommu/iommufd/vfio_compat.c
 create mode 100644 include/linux/iommufd.h
 create mode 100644 include/uapi/linux/iommufd.h
 create mode 100644 tools/testing/selftests/iommu/.gitignore
 create mode 100644 tools/testing/selftests/iommu/Makefile
 create mode 100644 tools/testing/selftests/iommu/config
 create mode 100644 tools/testing/selftests/iommu/iommufd.c
 create mode 100644 tools/testing/selftests/iommu/iommufd_fail_nth.c
 create mode 100644 tools/testing/selftests/iommu/iommufd_utils.h


base-commit: 69e61edebea030f177de7a23b8d5d9b8c4a90bda

Comments

Jason Gunthorpe Nov. 8, 2022, 1:01 a.m. UTC | #1
On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote:
> Cover the essential functionality of the iommufd with a directed
> test. This aims to achieve reasonable functional coverage using the
> in-kernel self test framework.
> 
> It provides a mock kernel module for the iommu_domain that allows it to
> run without any HW and the mocking provides a way to directly validate
> that the PFNs loaded into the iommu_domain are correct.
> 
> The mock also simulates the rare case of PAGE_SIZE > iommu page size as
> the mock will operate at a 2K iommu page size. This allows exercising all
> of the calculations to support this mismatch.
> 
> This allows achieving high coverage of the corner cases in the iopt_pages.
> 
> However, it is an unusually invasive config option to enable all of
> this. The config option should not be enabled in a production kernel.

This patch crossed 100k so it was bounced from the vger lists.

If anyone didn't get it and would like to see it lore has it:

https://lore.kernel.org/linux-iommu/15-v4-0de2f6c78ed0+9d1-iommufd_jgg@nvidia.com/

For the eventual v5 I will split the nth test into its own patch

Jason
Nicolin Chen Nov. 8, 2022, 5:48 a.m. UTC | #2
On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote:

> diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selftests/iommu/iommufd.c

> +TEST_F(iommufd, cmd_length)
> +{
> +#define TEST_LENGTH(_struct, _ioctl)                                     \
> +	{                                                                \
> +		struct {                                                 \
> +			struct _struct cmd;                              \
> +			uint8_t extra;                                   \
> +		} cmd = { .cmd = { .size = sizeof(struct _struct) - 1 }, \
> +			  .extra = UINT8_MAX };                          \
> +		int old_errno;                                           \
> +		int rc;                                                  \
> +									 \
> +		EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, _ioctl, &cmd)); \

I guess it should be EINVAL corresponding to updated kernel code?

> +TEST_F(iommufd, cmd_ex_fail)
> +{
> +	struct {
> +		struct iommu_destroy cmd;
> +		__u64 future;
> +	} cmd = { .cmd = { .size = sizeof(cmd), .id = 0 } };
> +
> +	/* object id is invalid and command is longer */
> +	EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> +	/* future area is non-zero */
> +	cmd.future = 1;
> +	EXPECT_ERRNO(E2BIG, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> +	/* Original command "works" */
> +	cmd.cmd.size = sizeof(cmd.cmd);
> +	EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> +	/* Short command fails */
> +	cmd.cmd.size = sizeof(cmd.cmd) - 1;
> +	EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, IOMMU_DESTROY, &cmd));

Ditto

> +TEST_HARNESS_MAIN
> diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c

> +static void fail_nth_first(struct __test_metadata *_metadata,
> +			   struct fail_nth_state *nth_state)
> +{
> +	char buf[300];
> +
> +	snprintf(buf, sizeof(buf), "/proc/self/task/%u/fail-nth", gettid());

Not sure what's missing, I have a build error at gettid. Copying
a solution from tools/perf/jvmti/jvmti_agent.c file, can fix with:
------------------------------
diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c
index 99eaa9f32e0b..7704b3a754d3 100644
--- a/tools/testing/selftests/iommu/iommufd_fail_nth.c
+++ b/tools/testing/selftests/iommu/iommufd_fail_nth.c
@@ -19,6 +19,7 @@
 
 #define __EXPORTED_HEADERS__
 #include <linux/vfio.h>
+#include <syscall.h> /* for gettid() */
 
 #include "iommufd_utils.h"
 
@@ -84,6 +85,13 @@ struct fail_nth_state {
        unsigned int iteration;
 };
 
+#ifndef HAVE_GETTID
+static inline pid_t gettid(void)
+{
+       return (pid_t)syscall(__NR_gettid);
+}
+#endif
+
 static void fail_nth_first(struct __test_metadata *_metadata,
                           struct fail_nth_state *nth_state)
 {
------------------------------
Jason Gunthorpe Nov. 8, 2022, 1:27 p.m. UTC | #3
On Mon, Nov 07, 2022 at 09:48:32PM -0800, Nicolin Chen wrote:
> On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote:
> 
> > diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selftests/iommu/iommufd.c
> 
> > +TEST_F(iommufd, cmd_length)
> > +{
> > +#define TEST_LENGTH(_struct, _ioctl)                                     \
> > +	{                                                                \
> > +		struct {                                                 \
> > +			struct _struct cmd;                              \
> > +			uint8_t extra;                                   \
> > +		} cmd = { .cmd = { .size = sizeof(struct _struct) - 1 }, \
> > +			  .extra = UINT8_MAX };                          \
> > +		int old_errno;                                           \
> > +		int rc;                                                  \
> > +									 \
> > +		EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, _ioctl, &cmd)); \
> 
> I guess it should be EINVAL corresponding to updated kernel code?
> 
> > +TEST_F(iommufd, cmd_ex_fail)
> > +{
> > +	struct {
> > +		struct iommu_destroy cmd;
> > +		__u64 future;
> > +	} cmd = { .cmd = { .size = sizeof(cmd), .id = 0 } };
> > +
> > +	/* object id is invalid and command is longer */
> > +	EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> > +	/* future area is non-zero */
> > +	cmd.future = 1;
> > +	EXPECT_ERRNO(E2BIG, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> > +	/* Original command "works" */
> > +	cmd.cmd.size = sizeof(cmd.cmd);
> > +	EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> > +	/* Short command fails */
> > +	cmd.cmd.size = sizeof(cmd.cmd) - 1;
> > +	EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, IOMMU_DESTROY, &cmd));
> 
> Ditto

Oops, yes, I fixed these

> 
> > +TEST_HARNESS_MAIN
> > diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c
> 
> > +static void fail_nth_first(struct __test_metadata *_metadata,
> > +			   struct fail_nth_state *nth_state)
> > +{
> > +	char buf[300];
> > +
> > +	snprintf(buf, sizeof(buf), "/proc/self/task/%u/fail-nth", gettid());
> 
> Not sure what's missing, I have a build error at gettid. Copying
> a solution from tools/perf/jvmti/jvmti_agent.c file, can fix with:

I think your glibc is probably old

> ------------------------------
> diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c
> index 99eaa9f32e0b..7704b3a754d3 100644
> --- a/tools/testing/selftests/iommu/iommufd_fail_nth.c
> +++ b/tools/testing/selftests/iommu/iommufd_fail_nth.c
> @@ -19,6 +19,7 @@
>  
>  #define __EXPORTED_HEADERS__
>  #include <linux/vfio.h>
> +#include <syscall.h> /* for gettid() */
>  
>  #include "iommufd_utils.h"
>  
> @@ -84,6 +85,13 @@ struct fail_nth_state {
>         unsigned int iteration;
>  };
>  
> +#ifndef HAVE_GETTID
> +static inline pid_t gettid(void)
> +{
> +       return (pid_t)syscall(__NR_gettid);
> +}
> +#endif

Ah, there is a lot of complicated makefile stuff to make this work,
and it only works for perf/bpf not selftests

It looks like there is no reason for this to need gettid, we don't use
threads. So this can just be getpid and that is portable.

Thanks,
Jason
Shameerali Kolothum Thodi Nov. 11, 2022, 3:51 p.m. UTC | #4
> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> Sent: 08 November 2022 00:49
> To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David
> Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg Roedel
> <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>;
> linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org;
> llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick
> Desaulniers <ndesaulniers@google.com>; Miguel Ojeda <ojeda@kernel.org>;
> Robin Murphy <robin.murphy@arm.com>; Shuah Khan <shuah@kernel.org>;
> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix
> <trix@redhat.com>; Will Deacon <will@kernel.org>
> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu
> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan
> <daniel.m.jordan@oracle.com>; David Gibson
> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric
> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>;
> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins
> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato
> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin
> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>;
> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi
> Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com>
> Subject: [PATCH v4 00/17] IOMMUFD Generic interface
[...]
> 
> - Userspace page tables aka 'nested translation' for ARM and Intel iommu
>   drivers:
>   https://github.com/nicolinc/iommufd/commits/iommufd_nesting

Hi Eric/Yi/Nicolin,

Could you please provide a latest Kernel/Qemu branch for the ARM nesting support?
The above link points to Yi's git, but not sure which one is latest/stable to 
have a play.

Thanks,
Shameer
Yi Liu Nov. 12, 2022, 12:44 p.m. UTC | #5
Hi Shameer,

On 2022/11/11 23:51, Shameerali Kolothum Thodi wrote:
> 
> 
>> -----Original Message-----
>> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
>> Sent: 08 November 2022 00:49
>> To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David
>> Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg Roedel
>> <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>;
>> linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org;
>> llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick
>> Desaulniers <ndesaulniers@google.com>; Miguel Ojeda <ojeda@kernel.org>;
>> Robin Murphy <robin.murphy@arm.com>; Shuah Khan <shuah@kernel.org>;
>> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix
>> <trix@redhat.com>; Will Deacon <will@kernel.org>
>> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu
>> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
>> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan
>> <daniel.m.jordan@oracle.com>; David Gibson
>> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric
>> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>;
>> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins
>> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato
>> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin
>> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>;
>> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi
>> Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com>
>> Subject: [PATCH v4 00/17] IOMMUFD Generic interface
> [...]
>>
>> - Userspace page tables aka 'nested translation' for ARM and Intel iommu
>>    drivers:
>>    https://github.com/nicolinc/iommufd/commits/iommufd_nesting
> 
> Hi Eric/Yi/Nicolin,
> 
> Could you please provide a latest Kernel/Qemu branch for the ARM nesting support?
> The above link points to Yi's git, but not sure which one is latest/stable to
> have a play.

Nicolin and I are working on the new version for nesting support. Below 
kernl branch is our latest progress so far. As the naming, it's still
wip. We also need to workout a Qemu version, so still need some time
before sharing with you.

https://github.com/yiliu1765/iommufd/tree/wip/iommufd-v6.1-rc3-nesting
Shameerali Kolothum Thodi Jan. 10, 2023, 11:35 a.m. UTC | #6
> -----Original Message-----
> From: Yi Liu [mailto:yi.l.liu@intel.com]
> Sent: 12 November 2022 12:45
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> Jason Gunthorpe <jgg@nvidia.com>; bpf@vger.kernel.org; Jonathan Corbet
> <corbet@lwn.net>; David Woodhouse <dwmw2@infradead.org>;
> iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; Kevin Tian
> <kevin.tian@intel.com>; linux-doc@vger.kernel.org;
> linux-kselftest@vger.kernel.org; llvm@lists.linux.dev; Nathan Chancellor
> <nathan@kernel.org>; Nick Desaulniers <ndesaulniers@google.com>; Miguel
> Ojeda <ojeda@kernel.org>; Robin Murphy <robin.murphy@arm.com>; Shuah
> Khan <shuah@kernel.org>; Suravee Suthikulpanit
> <suravee.suthikulpanit@amd.com>; Tom Rix <trix@redhat.com>; Will
> Deacon <will@kernel.org>
> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu
> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>;
> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan
> <daniel.m.jordan@oracle.com>; David Gibson
> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric
> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>;
> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins
> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato
> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin
> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>;
> zhukeqian <zhukeqian1@huawei.com>
> Subject: Re: [PATCH v4 00/17] IOMMUFD Generic interface
> 
> Hi Shameer,
> 
> On 2022/11/11 23:51, Shameerali Kolothum Thodi wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> >> Sent: 08 November 2022 00:49
> >> To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David
> >> Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg
> Roedel
> >> <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>;
> >> linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org;
> >> llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick
> >> Desaulniers <ndesaulniers@google.com>; Miguel Ojeda
> <ojeda@kernel.org>;
> >> Robin Murphy <robin.murphy@arm.com>; Shuah Khan
> <shuah@kernel.org>;
> >> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix
> >> <trix@redhat.com>; Will Deacon <will@kernel.org>
> >> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu
> >> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni
> <chaitanyak@nvidia.com>;
> >> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan
> >> <daniel.m.jordan@oracle.com>; David Gibson
> >> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>;
> Eric
> >> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>;
> >> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins
> >> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato
> >> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>;
> Nicolin
> >> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>;
> >> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>;
> Yi
> >> Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com>
> >> Subject: [PATCH v4 00/17] IOMMUFD Generic interface
> > [...]
> >>
> >> - Userspace page tables aka 'nested translation' for ARM and Intel iommu
> >>    drivers:
> >>    https://github.com/nicolinc/iommufd/commits/iommufd_nesting
> >
> > Hi Eric/Yi/Nicolin,
> >
> > Could you please provide a latest Kernel/Qemu branch for the ARM nesting
> support?
> > The above link points to Yi's git, but not sure which one is latest/stable to
> > have a play.
> 
> Nicolin and I are working on the new version for nesting support. Below
> kernl branch is our latest progress so far. As the naming, it's still
> wip. We also need to workout a Qemu version, so still need some time
> before sharing with you.
> 
> https://github.com/yiliu1765/iommufd/tree/wip/iommufd-v6.1-rc3-nesting

Hi Yi,

Thanks for that. I attempted ARM vSVA support based on your above branch
and related Qemu branch. With few hacks and additional patches the prototype
code works well on HiSilicon ARM platform. 

Please find the corresponding branches ere,
https://github.com/hisilicon/kernel-dev/tree/iommufd-v6.1-rc3-nesting-arm-vSVA
https://github.com/hisilicon/qemu/tree/qemu-iommufd-6.1-rc3-arm-vSVA

Please let me know if there are any recent branches for ARM support.

Thanks,
Shameer
 
> 
> --
> Regards,
> Yi Liu
Jason Gunthorpe Jan. 10, 2023, 1:49 p.m. UTC | #7
On Tue, Jan 10, 2023 at 11:35:59AM +0000, Shameerali Kolothum Thodi wrote:

> Thanks for that. I attempted ARM vSVA support based on your above branch
> and related Qemu branch. With few hacks and additional patches the prototype
> code works well on HiSilicon ARM platform. 

Nice!
 
> Please find the corresponding branches ere,
> https://github.com/hisilicon/kernel-dev/tree/iommufd-v6.1-rc3-nesting-arm-vSVA

We need to have a big think about how the PASID/PRI caps should be
working in VFIO..

The whole PRI thing needs to be its own series and need a careful look

Can you also look at the dirty tracking stuff? I'd really like to see
that done for the huawei vfio live migration driver

Thanks,
Jason
Joao Martins Jan. 10, 2023, 3:16 p.m. UTC | #8
On 10/01/2023 13:49, Jason Gunthorpe wrote:
> Can you also look at the dirty tracking stuff? I'd really like to see
> that done for the huawei vfio live migration driver
> 

He did look and provides comments based on his testing of the initial series
(IIUC from what we spoke at LPC). v2 should be simpler, though I am still
working it out the merging of unit tests into iommufd.

My plan once I post the v2 was to handover the SMMUv3 part back to Shameerali
given the fact he has hardware that has the feature and can iterate more
meaningfully than me.

	Joao
Jason Gunthorpe Jan. 10, 2023, 3:18 p.m. UTC | #9
On Tue, Jan 10, 2023 at 03:16:23PM +0000, Joao Martins wrote:
> On 10/01/2023 13:49, Jason Gunthorpe wrote:
> > Can you also look at the dirty tracking stuff? I'd really like to see
> > that done for the huawei vfio live migration driver
> > 
> 
> He did look and provides comments based on his testing of the initial series
> (IIUC from what we spoke at LPC). v2 should be simpler, though I am still
> working it out the merging of unit tests into iommufd.
> 
> My plan once I post the v2 was to handover the SMMUv3 part back to Shameerali
> given the fact he has hardware that has the feature and can iterate more
> meaningfully than me.

Sounds good

Thanks,
Jason
Shameerali Kolothum Thodi Jan. 10, 2023, 3:30 p.m. UTC | #10
> -----Original Message-----
> From: Joao Martins [mailto:joao.m.martins@oracle.com]
> Sent: 10 January 2023 15:16
> To: Jason Gunthorpe <jgg@nvidia.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> Cc: Yi Liu <yi.l.liu@intel.com>; bpf@vger.kernel.org; Jonathan Corbet
> <corbet@lwn.net>; David Woodhouse <dwmw2@infradead.org>;
> iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; Kevin Tian
> <kevin.tian@intel.com>; linux-doc@vger.kernel.org;
> linux-kselftest@vger.kernel.org; llvm@lists.linux.dev; Nathan Chancellor
> <nathan@kernel.org>; Nick Desaulniers <ndesaulniers@google.com>; Miguel
> Ojeda <ojeda@kernel.org>; Robin Murphy <robin.murphy@arm.com>; Shuah
> Khan <shuah@kernel.org>; Suravee Suthikulpanit
> <suravee.suthikulpanit@amd.com>; Tom Rix <trix@redhat.com>; Will
> Deacon <will@kernel.org>; Alex Williamson <alex.williamson@redhat.com>;
> Lu Baolu <baolu.lu@linux.intel.com>; Chaitanya Kulkarni
> <chaitanyak@nvidia.com>; Cornelia Huck <cohuck@redhat.com>; Daniel
> Jordan <daniel.m.jordan@oracle.com>; David Gibson
> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric
> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>;
> Jean-Philippe Brucker <jean-philippe@linaro.org>; kvm@vger.kernel.org;
> Matthew Rosato <mjrosato@linux.ibm.com>; Michael S. Tsirkin
> <mst@redhat.com>; Nicolin Chen <nicolinc@nvidia.com>; Niklas Schnelle
> <schnelle@linux.ibm.com>; zhukeqian <zhukeqian1@huawei.com>
> Subject: Re: [PATCH v4 00/17] IOMMUFD Generic interface
> 
> On 10/01/2023 13:49, Jason Gunthorpe wrote:
> > Can you also look at the dirty tracking stuff? I'd really like to see
> > that done for the huawei vfio live migration driver
> >
> 
> He did look and provides comments based on his testing of the initial series
> (IIUC from what we spoke at LPC). v2 should be simpler, though I am still
> working it out the merging of unit tests into iommufd.
> 
> My plan once I post the v2 was to handover the SMMUv3 part back to
> Shameerali
> given the fact he has hardware that has the feature and can iterate more
> meaningfully than me.

No problem. Happy to help.

Thanks,
Shameer