[v2,0/5] drm/xe/sriov: Don't migrate dmabuf BO to System RAM while running in VM

Message

Kasireddy, Vivek Oct. 21, 2024, 5:21 a.m. UTC

While testing [1] and [2] with a SRIOV enabled dGPU, it was noticed
that migrating a BO to System RAM before exporting it as a dmabuf
results in considerable performance degradation while running in a
Guest VM. For example, running a simple 3D app such as weston-simple-egl
would yield ~50 FPS instead of ~59 FPS, assuming a mode of 1920x1080@60.

One fix to this problem is to not migrate the BO and keep it in LMEM
during export. However, given that the GPU running in PF mode on the
Host cannot effectively access the PCI BAR addresses backing the
imported dmabuf BO, they need to be translated into LMEM addresses
(DPA) to enable this use-case to work properly.

With this patch series applied, it would become possible to display
(via Qemu GTK UI) Guest VM compositor's framebuffer (created in its LMEM)
on the Host without having to make any copies of it or a roundtrip
to System RAM. And, weston-simple-egl can now achieve ~59 FPS while
running with Gnome Wayland in the Guest VM.

Changelog:

v1 -> v2:
- Use a dma_addr array instead of SG table to store translated DMA
  addresses (Matt)
- Use a cursor to iterate over the entries in the dma_addr array
  instead of relying on SG iterator (Matt)
- Rebased and tested this series on top of [3] that introduces
  drm_pagemap_dma_addr and xe_res_first_dma/__xe_res_dma_next
  that this version relies on

[1] https://patchwork.freedesktop.org/series/131746/
[2] https://patchwork.freedesktop.org/series/135273/
[3] https://patchwork.freedesktop.org/series/137870/

Patchset overview:

Patch 1: PCI driver patch to unblock P2P DMA between VF and PF
Patch 2: Prevent BO migration to System RAM while running in VM
Patch 3: Helper function to get VF's backing object in LMEM
Patch 4-5: Create a new dma_addr array for LMEM based dmabuf BOs
	   to store translated addresses (DPAs)

This series is tested using the following method:
- Run Qemu with the following relevant options:
  qemu-system-x86_64 -m 4096m ....
  -device vfio-pci,host=0000:03:00.1
  -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080
  -display gtk,gl=on
  -object memory-backend-memfd,id=mem1,size=4096M
  -machine memory-backend=mem1 ...
- Run Gnome Wayland with the following options in the Guest VM:
  # cat /usr/lib/udev/rules.d/61-mutter-primary-gpu.rules
  ENV{DEVNAME}=="/dev/dri/card1", TAG+="mutter-device-preferred-primary", TAG+="mutter-device-disable-kms-modifiers"
  # XDG_SESSION_TYPE=wayland dbus-run-session -- /usr/bin/gnome-shell --wayland --no-x11 &

Cc: Michal Wajdeczko <[e-mail]>
Cc: Michał Winiarski <[e-mail]>
Cc: Simona Vetter <[e-mail]>
Cc: Matthew Auld <[e-mail]>
Cc: Matthew Brost <[e-mail]>
Cc: Thomas Hellström <[e-mail]>
Cc: Dongwon Kim <[e-mail]>

Vivek Kasireddy (5):
  PCI/P2PDMA: Don't enforce ACS check for functions of same device
  drm/xe/dmabuf: Don't migrate BO to System RAM while running in VF mode
  drm/xe/pf: Add a helper function to get a VF's backing object in LMEM
  drm/xe/bo: Create new dma_addr array for dmabuf BOs associated with
    VFs
  drm/xe/pt: Add an additional check for dmabuf BOs while updating PTEs

 drivers/gpu/drm/xe/xe_bo.c                 | 116 +++++++++++++++++++--
 drivers/gpu/drm/xe/xe_bo_types.h           |  11 +-
 drivers/gpu/drm/xe/xe_dma_buf.c            |   9 +-
 drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c |  23 ++++
 drivers/gpu/drm/xe/xe_gt_sriov_pf_config.h |   1 +
 drivers/gpu/drm/xe/xe_pt.c                 |   8 +-
 drivers/pci/p2pdma.c                       |  17 ++-
 7 files changed, 172 insertions(+), 13 deletions(-)