mbox series

[0/7] Preemption support for A7XX

Message ID 20240815-preemption-a750-t-v1-0-7bda26c34037@gmail.com (mailing list archive)
Headers show
Series Preemption support for A7XX | expand

Message

Antonino Maniscalco Aug. 15, 2024, 6:26 p.m. UTC 
This series implements preemption for A7XX targets, which allows the GPU to
switch to an higher priority ring when work is pushed to it, reducing latency
for high priority submissions.

This series enables L1 preemption with skip_save_restore which requires
the following userspace patches to function:

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544

A flag is added to `msm_gem_submit` to only allow submissions from compatible
userspace to be preempted, therefore maintaining compatibility.

Some commits from this series are based on a previous series to enable
preemption on A6XX targets:

https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org

Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
---
Antonino Maniscalco (7):
      drm/msm: Fix bv_fence being used as bv_rptr
      drm/msm: Add submitqueue setup and close
      drm/msm: Add a `preempt_record_size` field
      drm/msm/A6xx: Implement preemption for A7XX targets
      drm/msm/A6xx: Add traces for preemption
      drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
      drm/msm/A6xx: Enable preemption for A7xx targets

 drivers/gpu/drm/msm/Makefile              |   1 +
 drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
 drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
 drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
 drivers/gpu/drm/msm/msm_gpu.h             |   7 +
 drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
 drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
 drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
 include/uapi/drm/msm_drm.h                |   5 +-
 11 files changed, 995 insertions(+), 16 deletions(-)
---
base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
change-id: 20240815-preemption-a750-t-fcee9a844b39

Best regards,

Comments

Rob Clark Aug. 16, 2024, 5:47 p.m. UTC | #1 
On Thu, Aug 15, 2024 at 11:27 AM Antonino Maniscalco
<antomani103@gmail.com> wrote:
>
> This series implements preemption for A7XX targets, which allows the GPU to
> switch to an higher priority ring when work is pushed to it, reducing latency
> for high priority submissions.
>
> This series enables L1 preemption with skip_save_restore which requires
> the following userspace patches to function:
>
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
>
> A flag is added to `msm_gem_submit` to only allow submissions from compatible
> userspace to be preempted, therefore maintaining compatibility.

I guess this last para is from an earlier iteration of this series?
Looks like instead you are making this a submitqueue flag (which is an
approach that I prefer)

BR,
-R

> Some commits from this series are based on a previous series to enable
> preemption on A6XX targets:
>
> https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
>
> Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
> ---
> Antonino Maniscalco (7):
>       drm/msm: Fix bv_fence being used as bv_rptr
>       drm/msm: Add submitqueue setup and close
>       drm/msm: Add a `preempt_record_size` field
>       drm/msm/A6xx: Implement preemption for A7XX targets
>       drm/msm/A6xx: Add traces for preemption
>       drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
>       drm/msm/A6xx: Enable preemption for A7xx targets
>
>  drivers/gpu/drm/msm/Makefile              |   1 +
>  drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
>  drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
>  drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
>  drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
>  drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
>  drivers/gpu/drm/msm/msm_gpu.h             |   7 +
>  drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
>  drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
>  drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
>  include/uapi/drm/msm_drm.h                |   5 +-
>  11 files changed, 995 insertions(+), 16 deletions(-)
> ---
> base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
> change-id: 20240815-preemption-a750-t-fcee9a844b39
>
> Best regards,
> --
> Antonino Maniscalco <antomani103@gmail.com>
>
Antonino Maniscalco Aug. 16, 2024, 11:42 p.m. UTC | #2 
On 8/16/24 7:47 PM, Rob Clark wrote:
> On Thu, Aug 15, 2024 at 11:27 AM Antonino Maniscalco
> <antomani103@gmail.com> wrote:
>>
>> This series implements preemption for A7XX targets, which allows the GPU to
>> switch to an higher priority ring when work is pushed to it, reducing latency
>> for high priority submissions.
>>
>> This series enables L1 preemption with skip_save_restore which requires
>> the following userspace patches to function:
>>
>> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
>>
>> A flag is added to `msm_gem_submit` to only allow submissions from compatible
>> userspace to be preempted, therefore maintaining compatibility.
> 
> I guess this last para is from an earlier iteration of this series?
> Looks like instead you are making this a submitqueue flag (which is an
> approach that I prefer)
> 
> BR,
> -R
> 

That is correct, I got confused on which one I ended up going with when 
I wrote that.

Thanks for spotting it!

>> Some commits from this series are based on a previous series to enable
>> preemption on A6XX targets:
>>
>> https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
>>
>> Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
>> ---
>> Antonino Maniscalco (7):
>>        drm/msm: Fix bv_fence being used as bv_rptr
>>        drm/msm: Add submitqueue setup and close
>>        drm/msm: Add a `preempt_record_size` field
>>        drm/msm/A6xx: Implement preemption for A7XX targets
>>        drm/msm/A6xx: Add traces for preemption
>>        drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
>>        drm/msm/A6xx: Enable preemption for A7xx targets
>>
>>   drivers/gpu/drm/msm/Makefile              |   1 +
>>   drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
>>   drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
>>   drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
>>   drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
>>   drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
>>   drivers/gpu/drm/msm/msm_gpu.h             |   7 +
>>   drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
>>   drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
>>   drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
>>   include/uapi/drm/msm_drm.h                |   5 +-
>>   11 files changed, 995 insertions(+), 16 deletions(-)
>> ---
>> base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
>> change-id: 20240815-preemption-a750-t-fcee9a844b39
>>
>> Best regards,
>> --
>> Antonino Maniscalco <antomani103@gmail.com>
>>

Best regards,
Neil Armstrong Aug. 23, 2024, 8:22 a.m. UTC | #3 
On 15/08/2024 20:26, Antonino Maniscalco wrote:
> This series implements preemption for A7XX targets, which allows the GPU to
> switch to an higher priority ring when work is pushed to it, reducing latency
> for high priority submissions.
> 
> This series enables L1 preemption with skip_save_restore which requires
> the following userspace patches to function:
> 
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
> 
> A flag is added to `msm_gem_submit` to only allow submissions from compatible
> userspace to be preempted, therefore maintaining compatibility.
> 
> Some commits from this series are based on a previous series to enable
> preemption on A6XX targets:
> 
> https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
> 
> Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
> ---
> Antonino Maniscalco (7):
>        drm/msm: Fix bv_fence being used as bv_rptr
>        drm/msm: Add submitqueue setup and close
>        drm/msm: Add a `preempt_record_size` field
>        drm/msm/A6xx: Implement preemption for A7XX targets
>        drm/msm/A6xx: Add traces for preemption
>        drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
>        drm/msm/A6xx: Enable preemption for A7xx targets
> 
>   drivers/gpu/drm/msm/Makefile              |   1 +
>   drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
>   drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
>   drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
>   drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
>   drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
>   drivers/gpu/drm/msm/msm_gpu.h             |   7 +
>   drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
>   drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
>   drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
>   include/uapi/drm/msm_drm.h                |   5 +-
>   11 files changed, 995 insertions(+), 16 deletions(-)
> ---
> base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
> change-id: 20240815-preemption-a750-t-fcee9a844b39
> 
> Best regards,

For what is worth, I've tested it on the SM8650 QRD with the Mesa 30544 MR & vkcube

Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8650-QRD

If you think of more tests to run, please tell me.

Neil
Connor Abbott Aug. 23, 2024, 9:54 a.m. UTC | #4 
On Fri, Aug 23, 2024 at 9:30 AM <neil.armstrong@linaro.org> wrote:
>
> On 15/08/2024 20:26, Antonino Maniscalco wrote:
> > This series implements preemption for A7XX targets, which allows the GPU to
> > switch to an higher priority ring when work is pushed to it, reducing latency
> > for high priority submissions.
> >
> > This series enables L1 preemption with skip_save_restore which requires
> > the following userspace patches to function:
> >
> > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
> >
> > A flag is added to `msm_gem_submit` to only allow submissions from compatible
> > userspace to be preempted, therefore maintaining compatibility.
> >
> > Some commits from this series are based on a previous series to enable
> > preemption on A6XX targets:
> >
> > https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
> >
> > Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
> > ---
> > Antonino Maniscalco (7):
> >        drm/msm: Fix bv_fence being used as bv_rptr
> >        drm/msm: Add submitqueue setup and close
> >        drm/msm: Add a `preempt_record_size` field
> >        drm/msm/A6xx: Implement preemption for A7XX targets
> >        drm/msm/A6xx: Add traces for preemption
> >        drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
> >        drm/msm/A6xx: Enable preemption for A7xx targets
> >
> >   drivers/gpu/drm/msm/Makefile              |   1 +
> >   drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
> >   drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
> >   drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
> >   drivers/gpu/drm/msm/msm_gpu.h             |   7 +
> >   drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
> >   drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
> >   drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
> >   include/uapi/drm/msm_drm.h                |   5 +-
> >   11 files changed, 995 insertions(+), 16 deletions(-)
> > ---
> > base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
> > change-id: 20240815-preemption-a750-t-fcee9a844b39
> >
> > Best regards,
>
> For what is worth, I've tested it on the SM8650 QRD with the Mesa 30544 MR & vkcube
>
> Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8650-QRD
>
> If you think of more tests to run, please tell me.
>
> Neil

Hi Neil,

I think it would help to test this on SM8550 and SM8450 too. I don't
have SM8450 to test with. Maybe also worth mentioning that there are
now vulkan CTS tests that try to test L1 preemption:
https://github.com/KhronosGroup/VK-GL-CTS/commit/7e0e4a000f34e748bb527b39f730f78b595140b9
although it's not in a released version yet.

Connor
Neil Armstrong Aug. 23, 2024, 10:44 a.m. UTC | #5 
On 23/08/2024 11:54, Connor Abbott wrote:
> On Fri, Aug 23, 2024 at 9:30 AM <neil.armstrong@linaro.org> wrote:
>>
>> On 15/08/2024 20:26, Antonino Maniscalco wrote:
>>> This series implements preemption for A7XX targets, which allows the GPU to
>>> switch to an higher priority ring when work is pushed to it, reducing latency
>>> for high priority submissions.
>>>
>>> This series enables L1 preemption with skip_save_restore which requires
>>> the following userspace patches to function:
>>>
>>> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
>>>
>>> A flag is added to `msm_gem_submit` to only allow submissions from compatible
>>> userspace to be preempted, therefore maintaining compatibility.
>>>
>>> Some commits from this series are based on a previous series to enable
>>> preemption on A6XX targets:
>>>
>>> https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
>>>
>>> Signed-off-by: Antonino Maniscalco <antomani103@gmail.com>
>>> ---
>>> Antonino Maniscalco (7):
>>>         drm/msm: Fix bv_fence being used as bv_rptr
>>>         drm/msm: Add submitqueue setup and close
>>>         drm/msm: Add a `preempt_record_size` field
>>>         drm/msm/A6xx: Implement preemption for A7XX targets
>>>         drm/msm/A6xx: Add traces for preemption
>>>         drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
>>>         drm/msm/A6xx: Enable preemption for A7xx targets
>>>
>>>    drivers/gpu/drm/msm/Makefile              |   1 +
>>>    drivers/gpu/drm/msm/adreno/a6xx_catalog.c |   3 +
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.c     | 339 ++++++++++++++++++++++-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.h     | 168 ++++++++++++
>>>    drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 441 ++++++++++++++++++++++++++++++
>>>    drivers/gpu/drm/msm/adreno/adreno_gpu.h   |   1 +
>>>    drivers/gpu/drm/msm/msm_gpu.h             |   7 +
>>>    drivers/gpu/drm/msm/msm_gpu_trace.h       |  28 ++
>>>    drivers/gpu/drm/msm/msm_ringbuffer.h      |   8 +
>>>    drivers/gpu/drm/msm/msm_submitqueue.c     |  10 +
>>>    include/uapi/drm/msm_drm.h                |   5 +-
>>>    11 files changed, 995 insertions(+), 16 deletions(-)
>>> ---
>>> base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
>>> change-id: 20240815-preemption-a750-t-fcee9a844b39
>>>
>>> Best regards,
>>
>> For what is worth, I've tested it on the SM8650 QRD with the Mesa 30544 MR & vkcube
>>
>> Tested-by: Neil Armstrong <neil.armstrong@linaro.org> # on SM8650-QRD
>>
>> If you think of more tests to run, please tell me.
>>
>> Neil
> 
> Hi Neil,
> 
> I think it would help to test this on SM8550 and SM8450 too. I don't
> have SM8450 to test with. Maybe also worth mentioning that there are
> now vulkan CTS tests that try to test L1 preemption:
> https://github.com/KhronosGroup/VK-GL-CTS/commit/7e0e4a000f34e748bb527b39f730f78b595140b9
> although it's not in a released version yet.

Sure, It's in my plans, I'm currently testing it on SM8550.

Thanks,
Neil
> 
> Connor