[0/6] Enable KFD to use render node BO mappings

Message

Felix Kuehling Jan. 12, 2023, 1:31 a.m. UTC

Rebased on latest amd-staging-drm-next. This is meant to be the final
review of this series, assuming no more issues are found.

This patch series enables KFD to interoperate more closely with DRM render
nodes. ROCm user mode already uses DRM render nodes to create its GPU VM
contexts and to CPU-map its GEM buffer objects. This patch series adds an
API to let KFD export its BOs as DMABufs, so they can be imported into
the DRM render nodes. This enables more flexible virtual memory mappings
using DRM_IOCTL_AMDGPU_GEM_VA.

Patches 1 and 2 deal with the exporting and importing of DMABufs.

The remaining patches let KFD validate and update GPUVM mappings managed
through render nodes.

The user mode side of this patch series can be seen in libhsakmt and
KFDTest where we improve integration with libdrm (initializing
amdgpu_device instances) to enable DMABuf imports into the render nodes
representing KFD GPU VM contexts. KFDTest is modified to test evictions
and validations of BOs mapped through amdgpu_bo_va_op:
https://github.com/fxkamd/ROCT-Thunk-Interface/commits/fxkamd/dmabuf

As a consequence, applications using Mesa and ROCm in the same process on
the same GPU will now share a single render node FD and GPUVM address
space.

The DMABuf export API will also be used later for upstream IPC and RDMA
implementations.

Felix Kuehling (6):
  drm/amdgpu: Generalize KFD dmabuf import
  drm/amdkfd: Implement DMA buf fd export from KFD
  drm/amdkfd: Improve amdgpu_vm_handle_moved
  drm/amdgpu: Attach eviction fence on alloc
  drm/amdgpu: update mappings not managed by KFD
  drm/amdgpu: Do bo_va ref counting for KFD BOs

 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h    |   2 +
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  | 196 ++++++++++++------
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c        |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  18 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |   3 +-
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      |  55 +++++
 include/uapi/linux/kfd_ioctl.h                |  14 +-
 8 files changed, 219 insertions(+), 77 deletions(-)

Comments

Christian König Jan. 12, 2023, 7:18 a.m. UTC | #1

Am 12.01.23 um 02:31 schrieb Felix Kuehling:
> Rebased on latest amd-staging-drm-next. This is meant to be the final
> review of this series, assuming no more issues are found.
>
> This patch series enables KFD to interoperate more closely with DRM render
> nodes. ROCm user mode already uses DRM render nodes to create its GPU VM
> contexts and to CPU-map its GEM buffer objects. This patch series adds an
> API to let KFD export its BOs as DMABufs, so they can be imported into
> the DRM render nodes. This enables more flexible virtual memory mappings
> using DRM_IOCTL_AMDGPU_GEM_VA.
>
> Patches 1 and 2 deal with the exporting and importing of DMABufs.
>
> The remaining patches let KFD validate and update GPUVM mappings managed
> through render nodes.
>
> The user mode side of this patch series can be seen in libhsakmt and
> KFDTest where we improve integration with libdrm (initializing
> amdgpu_device instances) to enable DMABuf imports into the render nodes
> representing KFD GPU VM contexts. KFDTest is modified to test evictions
> and validations of BOs mapped through amdgpu_bo_va_op:
> https://github.com/fxkamd/ROCT-Thunk-Interface/commits/fxkamd/dmabuf
>
> As a consequence, applications using Mesa and ROCm in the same process on
> the same GPU will now share a single render node FD and GPUVM address
> space.
>
> The DMABuf export API will also be used later for upstream IPC and RDMA
> implementations.

Nice, I don't have time to check everything in detail but at least from 
a high level skimming over this I can't see anything obvious wrong.

Feel free to add an Acked-by: Christian König <[e-mail]> 
to the series.

Christian.

>
> Felix Kuehling (6):
>    drm/amdgpu: Generalize KFD dmabuf import
>    drm/amdkfd: Implement DMA buf fd export from KFD
>    drm/amdkfd: Improve amdgpu_vm_handle_moved
>    drm/amdgpu: Attach eviction fence on alloc
>    drm/amdgpu: update mappings not managed by KFD
>    drm/amdgpu: Do bo_va ref counting for KFD BOs
>
>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h    |   2 +
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  | 196 ++++++++++++------
>   drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c        |   6 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |   2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  18 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |   3 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      |  55 +++++
>   include/uapi/linux/kfd_ioctl.h                |  14 +-
>   8 files changed, 219 insertions(+), 77 deletions(-)
>