[RFC,0/6] DRM resource management cgroup, try 2.

Message

Maarten Lankhorst June 27, 2024, 3:47 p.m. UTC

Hey,

A new version of my attempt at managing VRAM through cgroups.
Even though it's called the DRM resource management cgroup, it would be trivial
to rename it to devmem or whatever, since there is nothing DRM specific about it.

This series allows setting limits on VRAM similar to system memory,
with min/low/max limits.
This allows various cgroups to have their own limits for usage.

It sounds very abstract, but it can be used to prioritise the foreground
application (by setting low), or hard partition memory so multiple processes
sharing a single GPU use a proportional amount of memory each in a fair way,
or to prevent long running compute jobs from having their memory evicted.

This is a minimal proof of concept to get discussion going again. It works,
but it only tracks active use of VRAM. In the ideal world, we would track
it better in a way that also integrates better with the memory cgroup
controller. Ideally for every VRAM allocation, we would know we could push
it out to swap if needed, charging the original process not the process evicting.

I'm hoping to restart the discussion, so that we can plug the holes and finally move forward.

New in this version:
- Complete rewrite using page_counter.
- Support setting min/low/max, respected in the same way as memory cgroup.
  (Could be useful to add/allow high? To go over limit for temporary bindings
   during eviction on GART.)
- Locking reworked. Fastpath should now be lockless with RCU.
- Add a second implementation for AMD, to show how easy it is to make it work.
  (Should we completely move this to TTM instead?)
- TTM now always respects min/low when evicting, bailing out with -ENOSPC instead
  where required.

I'm hoping for some good feedback on the path forward for upstreaming. I feel this
version has a lot better chance of being upstreamed than the previous. It should
be a lot more scalable thanks to the usage of RCU and page_counter.

Cheers,
Maarten

Maarten Lankhorst (6):
  mm/page_counter: Move calculating protection values to page_counter
  drm/cgroup: Add memory accounting DRM cgroup
  drm/ttm: Handle cgroup based eviction in TTM
  drm/xe: Implement cgroup for vram
  drm/amdgpu: Add cgroups implementation
  drm/xe: Hack to test with mapped pages instead of vram.

 Documentation/admin-guide/cgroup-v2.rst       |  51 ++
 Documentation/gpu/drm-compute.rst             |  54 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   6 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c  |   6 +
 drivers/gpu/drm/ttm/tests/ttm_bo_test.c       |  18 +-
 drivers/gpu/drm/ttm/tests/ttm_resource_test.c |   2 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  38 +-
 drivers/gpu/drm/ttm/ttm_resource.c            |  28 +-
 drivers/gpu/drm/xe/xe_device.c                |   4 +
 drivers/gpu/drm/xe/xe_device_types.h          |   4 +
 drivers/gpu/drm/xe/xe_ttm_sys_mgr.c           |  14 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c          |  10 +
 include/drm/ttm/ttm_bo.h                      |   3 +-
 include/drm/ttm/ttm_resource.h                |  16 +-
 include/linux/cgroup_drm.h                    | 115 +++
 include/linux/cgroup_subsys.h                 |   4 +
 include/linux/page_counter.h                  |   4 +
 init/Kconfig                                  |   7 +
 kernel/cgroup/Makefile                        |   1 +
 kernel/cgroup/drm.c                           | 813 ++++++++++++++++++
 mm/memcontrol.c                               | 154 +---
 mm/page_counter.c                             | 173 ++++
 23 files changed, 1355 insertions(+), 172 deletions(-)
 create mode 100644 Documentation/gpu/drm-compute.rst
 create mode 100644 include/linux/cgroup_drm.h
 create mode 100644 kernel/cgroup/drm.c