| Message ID | 20231117093512.1999666-1-zhenzhong.duan@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | docs/devel: Add VFIO iommufd backend documentation | expand |
On 11/17/23 10:35, Zhenzhong Duan wrote: > Suggested-by: Cédric Le Goater <clg@redhat.com> > Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com> The content looks good but it lacks formatting. Please try to generate the docs. Thanks, C. > --- > MAINTAINERS | 1 + > docs/devel/index-internals.rst | 1 + > docs/devel/vfio-iommufd.rst | 115 +++++++++++++++++++++++++++++++++ > 3 files changed, 117 insertions(+) > create mode 100644 docs/devel/vfio-iommufd.rst > > diff --git a/MAINTAINERS b/MAINTAINERS > index d86ba56a49..07990456ed 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -2175,6 +2175,7 @@ F: backends/iommufd.c > F: include/sysemu/iommufd.h > F: include/qemu/chardev_open.h > F: util/chardev_open.c > +F: docs/devel/vfio-iommufd.rst > > vhost > M: Michael S. Tsirkin <mst@redhat.com> > diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst > index 6f81df92bc..3def4a138b 100644 > --- a/docs/devel/index-internals.rst > +++ b/docs/devel/index-internals.rst > @@ -18,5 +18,6 @@ Details about QEMU's various subsystems including how to add features to them. > s390-dasd-ipl > tracing > vfio-migration > + vfio-iommufd > writing-monitor-commands > virtio-backends > diff --git a/docs/devel/vfio-iommufd.rst b/docs/devel/vfio-iommufd.rst > new file mode 100644 > index 0000000000..59804a7f26 > --- /dev/null > +++ b/docs/devel/vfio-iommufd.rst > @@ -0,0 +1,115 @@ > +=============================== > +IOMMUFD BACKEND usage with VFIO > +=============================== > + > +(Same meaning for backend/container/BE) > + > +With the introduction of iommufd, the Linux kernel provides a generic > +interface for user space drivers to propagate their DMA mappings to kernel > +for assigned devices. While the legacy kernel interface is group-centric, > +the new iommufd interface is device-centric, relying on device fd and iommufd. > + > +To support both interfaces in the QEMU VFIO device, introduce a base container > +to abstract the common part of VFIO legacy and iommufd container. So that the > +generic VFIO code can use either container. > + > +The base container implements generic functions such as memory_listener and > +address space management whereas the derived container implements callbacks > +specific to either legacy or iommufd. Each container has its own way to setup > +secure context and dma management interface. The below diagram shows how it > +looks like with both containers. > + > + VFIO AddressSpace/Memory > + +-------+ +----------+ +-----+ +-----+ > + | pci | | platform | | ap | | ccw | > + +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ > + | | | | | AddressSpace | > + | | | | +------------+---------+ > + +---V-----------V-----------V--------V----+ / > + | VFIOAddressSpace | <------------+ > + | | | MemoryListener > + | VFIOContainerBase list | > + +-------+----------------------------+----+ > + | | > + | | > + +-------V------+ +--------V----------+ > + | iommufd | | vfio legacy | > + | container | | container | > + +-------+------+ +--------+----------+ > + | | > + | /dev/iommu | /dev/vfio/vfio > + | /dev/vfio/devices/vfioX | /dev/vfio/$group_id > +Userspace | | > +============+============================+=========================== > +Kernel | device fd | > + +---------------+ | group/container fd > + | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU) > + | ATTACH_IOAS) | | device fd > + | | | > + | +-------V------------V-----------------+ > + iommufd | | vfio | > +(map/unmap | +---------+--------------------+-------+ > +ioas_copy) | | | map/unmap > + | | | > + +------V------+ +-----V------+ +------V--------+ > + | iommfd core | | device | | vfio iommu | > + +-------------+ +------------+ +---------------+ > + > +[Secure Context setup] > +- iommufd BE: uses device fd and iommufd to setup secure context > + (bind_iommufd, attach_ioas) > +- vfio legacy BE: uses group fd and container fd to setup secure context > + (set_container, set_iommu) > + > +[Device access] > +- iommufd BE: device fd is opened through /dev/vfio/devices/vfioX > +- vfio legacy BE: device fd is retrieved from group fd ioctl > + > +[DMA Mapping flow] > +1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener > +2. VFIO populates DMA map/unmap via the container BEs > + *) iommufd BE: uses iommufd > + *) vfio legacy BE: uses container fd > + > + > +Example configuration > +===================== > + > +Step 1: configure the host device > +--------------------------------- > + > +It's exactly same as the VFIO device with legacy VFIO container. > + > +Step 2: configure QEMU > +---------------------- > + > +Interactions with the /dev/iommu are abstracted by a new > +iommufd object (compiled in with the CONFIG_IOMMUFD option). > + > +Any QEMU device (e.g. VFIO device) wishing to use /dev/iommu must be > +linked with an iommufd object. It gets a new optional property named > +iommufd which allows to pass an iommufd object. Take vfio-pci device > +for example: > + > + -object iommufd,id=iommufd0 > + -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 > + > +Note the /dev/iommu and VFIO cdev can be externally opened by a > +management layer. In such a case the fd is passed, the fd supports > +a string naming the fd or a number, for example: > + > + -object iommufd,id=iommufd0,fd=22 > + -device vfio-pci,iommufd=iommufd0,fd=23 > + > +If the fd property is not passed, the fd is opened by QEMU. > + > +If no iommufd property is passed to the vfio-pci device, iommufd is > +not used and the user gets the behavior based on the legacy VFIO > +container: > + > + -device vfio-pci,host=0000:02:00.0 > + > +Supported platform > +================== > + > +Supports X86, ARM and S390X currently.
On 11/17/23 13:58, Cédric Le Goater wrote: > On 11/17/23 10:35, Zhenzhong Duan wrote: >> Suggested-by: Cédric Le Goater <clg@redhat.com> >> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com> > > The content looks good but it lacks formatting. Please try to generate > the docs. Please check my vfio-8.2 branch. Thanks, C.
>-----Original Message----- >From: Cédric Le Goater <clg@redhat.com> >Sent: Friday, November 17, 2023 10:28 PM >To: Duan, Zhenzhong <zhenzhong.duan@intel.com>; qemu-devel@nongnu.org >Cc: alex.williamson@redhat.com; jgg@nvidia.com; nicolinc@nvidia.com; >joao.m.martins@oracle.com; eric.auger@redhat.com; peterx@redhat.com; >jasowang@redhat.com; Tian, Kevin <kevin.tian@intel.com>; Liu, Yi L ><yi.l.liu@intel.com>; Sun, Yi Y <yi.y.sun@intel.com>; Peng, Chao P ><chao.p.peng@intel.com> >Subject: Re: [PATCH] docs/devel: Add VFIO iommufd backend documentation > >On 11/17/23 13:58, Cédric Le Goater wrote: >> On 11/17/23 10:35, Zhenzhong Duan wrote: >>> Suggested-by: Cédric Le Goater <clg@redhat.com> >>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com> >> >> The content looks good but it lacks formatting. Please try to generate >> the docs. > >Please check my vfio-8.2 branch. Got it, thanks for help. BRs. Zhenzhong
diff --git a/MAINTAINERS b/MAINTAINERS index d86ba56a49..07990456ed 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2175,6 +2175,7 @@ F: backends/iommufd.c F: include/sysemu/iommufd.h F: include/qemu/chardev_open.h F: util/chardev_open.c +F: docs/devel/vfio-iommufd.rst vhost M: Michael S. Tsirkin <mst@redhat.com> diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst index 6f81df92bc..3def4a138b 100644 --- a/docs/devel/index-internals.rst +++ b/docs/devel/index-internals.rst @@ -18,5 +18,6 @@ Details about QEMU's various subsystems including how to add features to them. s390-dasd-ipl tracing vfio-migration + vfio-iommufd writing-monitor-commands virtio-backends diff --git a/docs/devel/vfio-iommufd.rst b/docs/devel/vfio-iommufd.rst new file mode 100644 index 0000000000..59804a7f26 --- /dev/null +++ b/docs/devel/vfio-iommufd.rst @@ -0,0 +1,115 @@ +=============================== +IOMMUFD BACKEND usage with VFIO +=============================== + +(Same meaning for backend/container/BE) + +With the introduction of iommufd, the Linux kernel provides a generic +interface for user space drivers to propagate their DMA mappings to kernel +for assigned devices. While the legacy kernel interface is group-centric, +the new iommufd interface is device-centric, relying on device fd and iommufd. + +To support both interfaces in the QEMU VFIO device, introduce a base container +to abstract the common part of VFIO legacy and iommufd container. So that the +generic VFIO code can use either container. + +The base container implements generic functions such as memory_listener and +address space management whereas the derived container implements callbacks +specific to either legacy or iommufd. Each container has its own way to setup +secure context and dma management interface. The below diagram shows how it +looks like with both containers. + + VFIO AddressSpace/Memory + +-------+ +----------+ +-----+ +-----+ + | pci | | platform | | ap | | ccw | + +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ + | | | | | AddressSpace | + | | | | +------------+---------+ + +---V-----------V-----------V--------V----+ / + | VFIOAddressSpace | <------------+ + | | | MemoryListener + | VFIOContainerBase list | + +-------+----------------------------+----+ + | | + | | + +-------V------+ +--------V----------+ + | iommufd | | vfio legacy | + | container | | container | + +-------+------+ +--------+----------+ + | | + | /dev/iommu | /dev/vfio/vfio + | /dev/vfio/devices/vfioX | /dev/vfio/$group_id +Userspace | | +============+============================+=========================== +Kernel | device fd | + +---------------+ | group/container fd + | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU) + | ATTACH_IOAS) | | device fd + | | | + | +-------V------------V-----------------+ + iommufd | | vfio | +(map/unmap | +---------+--------------------+-------+ +ioas_copy) | | | map/unmap + | | | + +------V------+ +-----V------+ +------V--------+ + | iommfd core | | device | | vfio iommu | + +-------------+ +------------+ +---------------+ + +[Secure Context setup] +- iommufd BE: uses device fd and iommufd to setup secure context + (bind_iommufd, attach_ioas) +- vfio legacy BE: uses group fd and container fd to setup secure context + (set_container, set_iommu) + +[Device access] +- iommufd BE: device fd is opened through /dev/vfio/devices/vfioX +- vfio legacy BE: device fd is retrieved from group fd ioctl + +[DMA Mapping flow] +1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener +2. VFIO populates DMA map/unmap via the container BEs + *) iommufd BE: uses iommufd + *) vfio legacy BE: uses container fd + + +Example configuration +===================== + +Step 1: configure the host device +--------------------------------- + +It's exactly same as the VFIO device with legacy VFIO container. + +Step 2: configure QEMU +---------------------- + +Interactions with the /dev/iommu are abstracted by a new +iommufd object (compiled in with the CONFIG_IOMMUFD option). + +Any QEMU device (e.g. VFIO device) wishing to use /dev/iommu must be +linked with an iommufd object. It gets a new optional property named +iommufd which allows to pass an iommufd object. Take vfio-pci device +for example: + + -object iommufd,id=iommufd0 + -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 + +Note the /dev/iommu and VFIO cdev can be externally opened by a +management layer. In such a case the fd is passed, the fd supports +a string naming the fd or a number, for example: + + -object iommufd,id=iommufd0,fd=22 + -device vfio-pci,iommufd=iommufd0,fd=23 + +If the fd property is not passed, the fd is opened by QEMU. + +If no iommufd property is passed to the vfio-pci device, iommufd is +not used and the user gets the behavior based on the legacy VFIO +container: + + -device vfio-pci,host=0000:02:00.0 + +Supported platform +================== + +Supports X86, ARM and S390X currently.
Suggested-by: Cédric Le Goater <clg@redhat.com> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com> --- MAINTAINERS | 1 + docs/devel/index-internals.rst | 1 + docs/devel/vfio-iommufd.rst | 115 +++++++++++++++++++++++++++++++++ 3 files changed, 117 insertions(+) create mode 100644 docs/devel/vfio-iommufd.rst