drm/sched: Document run_job() refcount hazard

Commit Message

Philipp Stanner Dec. 20, 2024, 12:45 p.m. UTC

From: Philipp Stanner <[e-mail]>

drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler.
That fence is signalled by the driver once the hardware completed the
associated job. The scheduler does not increment the reference count on
that fence, but implicitly expects to inherit this fence from run_job().

This is relatively subtle and prone to misunderstandings.

This implies that, to keep a reference for itself, a driver needs to
call dma_fence_get() in addition to dma_fence_init() in that callback.

It's further complicated by the fact that the scheduler even decrements
the refcount in drm_sched_run_job_work() since it created a new
reference in drm_sched_fence_scheduled(). It does, however, still use
its pointer to the fence after calling dma_fence_put() - which is safe
because of the aforementioned new reference, but actually still violates
the refcounting rules.

Improve the explanatory comment for that decrement.

Move the call to dma_fence_put() to the position behind the last usage
of the fence.

Document the necessity to increment the reference count in
drm_sched_backend_ops.run_job().

Cc: Christian König <[e-mail]>
Cc: Tvrtko Ursulin <[e-mail]>
Cc: Andrey Grodzovsky <[e-mail]>
Signed-off-by: Philipp Stanner <[e-mail]>
---
 drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
 include/drm/gpu_scheduler.h            | 20 ++++++++++++++++----
 2 files changed, 23 insertions(+), 7 deletions(-)

Comments

Christian König Dec. 20, 2024, 12:53 p.m. UTC | #1

Am 20.12.24 um 13:45 schrieb Philipp Stanner:
> From: Philipp Stanner <[e-mail]>
>
> drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler.
> That fence is signalled by the driver once the hardware completed the
> associated job. The scheduler does not increment the reference count on
> that fence, but implicitly expects to inherit this fence from run_job().
>
> This is relatively subtle and prone to misunderstandings.
>
> This implies that, to keep a reference for itself, a driver needs to
> call dma_fence_get() in addition to dma_fence_init() in that callback.
>
> It's further complicated by the fact that the scheduler even decrements
> the refcount in drm_sched_run_job_work() since it created a new
> reference in drm_sched_fence_scheduled(). It does, however, still use
> its pointer to the fence after calling dma_fence_put() - which is safe
> because of the aforementioned new reference, but actually still violates
> the refcounting rules.
>
> Improve the explanatory comment for that decrement.
>
> Move the call to dma_fence_put() to the position behind the last usage
> of the fence.
>
> Document the necessity to increment the reference count in
> drm_sched_backend_ops.run_job().
>
> Cc: Christian König <[e-mail]>
> Cc: Tvrtko Ursulin <[e-mail]>
> Cc: Andrey Grodzovsky <[e-mail]>
> Signed-off-by: Philipp Stanner <[e-mail]>
> ---
>   drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
>   include/drm/gpu_scheduler.h            | 20 ++++++++++++++++----
>   2 files changed, 23 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 7ce25281c74c..d6f8df39d848 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w)
>   	drm_sched_fence_scheduled(s_fence, fence);
>   
>   	if (!IS_ERR_OR_NULL(fence)) {
> -		/* Drop for original kref_init of the fence */
> -		dma_fence_put(fence);
> -
>   		r = dma_fence_add_callback(fence, &sched_job->cb,
>   					   drm_sched_job_done_cb);
>   		if (r == -ENOENT)
>   			drm_sched_job_done(sched_job, fence->error);
>   		else if (r)
>   			DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);
> +
> +		/*
> +		 * s_fence took a new reference to fence in the call to
> +		 * drm_sched_fence_scheduled() above. The reference passed by
> +		 * run_job() above is now not needed any longer. Drop it.
> +		 */
> +		dma_fence_put(fence);
>   	} else {
>   		drm_sched_job_done(sched_job, IS_ERR(fence) ?
>   				   PTR_ERR(fence) : 0);
> diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
> index 95e17504e46a..a1f5c9a14278 100644
> --- a/include/drm/gpu_scheduler.h
> +++ b/include/drm/gpu_scheduler.h
> @@ -420,10 +420,22 @@ struct drm_sched_backend_ops {
>   					 struct drm_sched_entity *s_entity);
>   
>   	/**
> -         * @run_job: Called to execute the job once all of the dependencies
> -         * have been resolved.  This may be called multiple times, if
> -	 * timedout_job() has happened and drm_sched_job_recovery()
> -	 * decides to try it again.
> +	 * @run_job: Called to execute the job once all of the dependencies
> +	 * have been resolved. This may be called multiple times, if
> +	 * timedout_job() has happened and drm_sched_job_recovery() decides to
> +	 * try it again.

Maybe we should improve that here as well while at it.

That drm_sched_job_recovery() can call this multiple times actually goes 
against the dma_fence rules since drivers can't easily allocate a new HW 
fence.

Something like "The deprecated drm_sched_job_recovery() function might 
call this again, but it is strongly advised to not be used because it 
violates dma_fence memory allocations rules."

On the other hand can of course be a separate patch.

> +	 *
> +	 * @sched_job: the job to run
> +	 *
> +	 * Returns: dma_fence the driver must signal once the hardware has
> +	 *	completed the job ("hardware fence").
> +	 *
> +	 * Note that the scheduler expects to 'inherit' its own reference to
> +	 * this fence from the callback. It does not invoke an extra
> +	 * dma_fence_get() on it. Consequently, this callback must return a
> +	 * fence whose refcount is at least 2: One for the scheduler's
> +	 * reference returned here, another one for the reference kept by the
> +	 * driver.

Well the driver actually doesn't need any extra reference. The scheduler 
just needs to guarantee that this reference isn't dropped before it is 
signaled.

Regards,
Christian.

>   	 */
>   	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
>

Danilo Krummrich Dec. 20, 2024, 1:16 p.m. UTC | #2

On Fri, Dec 20, 2024 at 01:45:15PM +0100, Philipp Stanner wrote:
> From: Philipp Stanner <[e-mail]>
> 
> drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler.
> That fence is signalled by the driver once the hardware completed the
> associated job. The scheduler does not increment the reference count on
> that fence, but implicitly expects to inherit this fence from run_job().
> 
> This is relatively subtle and prone to misunderstandings.
> 
> This implies that, to keep a reference for itself, a driver needs to
> call dma_fence_get() in addition to dma_fence_init() in that callback.

I think that's a bit too specific. It's just that the returned dma_fence pointer
of run_job() must be backed by a reference, otherwise it can't be valid.

Everything else is an implementation detail of the driver.

> 
> It's further complicated by the fact that the scheduler even decrements
> the refcount in drm_sched_run_job_work() since it created a new
> reference in drm_sched_fence_scheduled(). It does, however, still use

Those two are unrelated. The decrement comes from the reference count that has
to be taken to return the fence in run_job().

The reference count in drm_sched_fence_set_parent() is for s_fence->parent.

> its pointer to the fence after calling dma_fence_put() - which is safe
> because of the aforementioned new reference, but actually still violates
> the refcounting rules.
> 
> Improve the explanatory comment for that decrement.
> 
> Move the call to dma_fence_put() to the position behind the last usage
> of the fence.
> 
> Document the necessity to increment the reference count in
> drm_sched_backend_ops.run_job().
> 
> Cc: Christian König <[e-mail]>
> Cc: Tvrtko Ursulin <[e-mail]>
> Cc: Andrey Grodzovsky <[e-mail]>
> Signed-off-by: Philipp Stanner <[e-mail]>

You may want to add

Suggested-by: Danilo Krummrich <[e-mail]>

> ---
>  drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
>  include/drm/gpu_scheduler.h            | 20 ++++++++++++++++----
>  2 files changed, 23 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 7ce25281c74c..d6f8df39d848 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct work_struct *w)
>  	drm_sched_fence_scheduled(s_fence, fence);
>  
>  	if (!IS_ERR_OR_NULL(fence)) {
> -		/* Drop for original kref_init of the fence */
> -		dma_fence_put(fence);
> -
>  		r = dma_fence_add_callback(fence, &sched_job->cb,
>  					   drm_sched_job_done_cb);
>  		if (r == -ENOENT)
>  			drm_sched_job_done(sched_job, fence->error);
>  		else if (r)
>  			DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);
> +
> +		/*
> +		 * s_fence took a new reference to fence in the call to
> +		 * drm_sched_fence_scheduled() above. The reference passed by
> +		 * run_job() above is now not needed any longer. Drop it.
> +		 */

Taking a new reference in drm_sched_fence_set_parent() is not an argument for
dropping the reference taken by run_job() here. drm_sched_fence_set_parent()
takes this reference for s_fence->parent.

> +		dma_fence_put(fence);
>  	} else {
>  		drm_sched_job_done(sched_job, IS_ERR(fence) ?
>  				   PTR_ERR(fence) : 0);
> diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
> index 95e17504e46a..a1f5c9a14278 100644
> --- a/include/drm/gpu_scheduler.h
> +++ b/include/drm/gpu_scheduler.h
> @@ -420,10 +420,22 @@ struct drm_sched_backend_ops {
>  					 struct drm_sched_entity *s_entity);
>  
>  	/**
> -         * @run_job: Called to execute the job once all of the dependencies
> -         * have been resolved.  This may be called multiple times, if
> -	 * timedout_job() has happened and drm_sched_job_recovery()
> -	 * decides to try it again.
> +	 * @run_job: Called to execute the job once all of the dependencies
> +	 * have been resolved. This may be called multiple times, if
> +	 * timedout_job() has happened and drm_sched_job_recovery() decides to
> +	 * try it again.
> +	 *
> +	 * @sched_job: the job to run
> +	 *
> +	 * Returns: dma_fence the driver must signal once the hardware has
> +	 *	completed the job ("hardware fence").
> +	 *
> +	 * Note that the scheduler expects to 'inherit' its own reference to
> +	 * this fence from the callback. It does not invoke an extra

That's not wrong, but I wouldn't say the scheduler expects to inherit the
reference. When a function returns a fence pointer it *has* to make sure to take
it's own reference. Otherwise the pointer may be invalid once used by the
caller.

> +	 * dma_fence_get() on it. Consequently, this callback must return a
> +	 * fence whose refcount is at least 2: One for the scheduler's
> +	 * reference returned here, another one for the reference kept by the
> +	 * driver.
>  	 */
>  	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
>  
> -- 
> 2.47.1
>

Danilo Krummrich Dec. 20, 2024, 1:18 p.m. UTC | #3

On Fri, Dec 20, 2024 at 01:53:34PM +0100, Christian König wrote:
> Am 20.12.24 um 13:45 schrieb Philipp Stanner:
> > From: Philipp Stanner <[e-mail]>
> > 
> > drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler.
> > That fence is signalled by the driver once the hardware completed the
> > associated job. The scheduler does not increment the reference count on
> > that fence, but implicitly expects to inherit this fence from run_job().
> > 
> > This is relatively subtle and prone to misunderstandings.
> > 
> > This implies that, to keep a reference for itself, a driver needs to
> > call dma_fence_get() in addition to dma_fence_init() in that callback.
> > 
> > It's further complicated by the fact that the scheduler even decrements
> > the refcount in drm_sched_run_job_work() since it created a new
> > reference in drm_sched_fence_scheduled(). It does, however, still use
> > its pointer to the fence after calling dma_fence_put() - which is safe
> > because of the aforementioned new reference, but actually still violates
> > the refcounting rules.
> > 
> > Improve the explanatory comment for that decrement.
> > 
> > Move the call to dma_fence_put() to the position behind the last usage
> > of the fence.
> > 
> > Document the necessity to increment the reference count in
> > drm_sched_backend_ops.run_job().
> > 
> > Cc: Christian König <[e-mail]>
> > Cc: Tvrtko Ursulin <[e-mail]>
> > Cc: Andrey Grodzovsky <[e-mail]>
> > Signed-off-by: Philipp Stanner <[e-mail]>
> > ---
> >   drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
> >   include/drm/gpu_scheduler.h            | 20 ++++++++++++++++----
> >   2 files changed, 23 insertions(+), 7 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> > index 7ce25281c74c..d6f8df39d848 100644
> > --- a/drivers/gpu/drm/scheduler/sched_main.c
> > +++ b/drivers/gpu/drm/scheduler/sched_main.c
> > +	 *
> > +	 * @sched_job: the job to run
> > +	 *
> > +	 * Returns: dma_fence the driver must signal once the hardware has
> > +	 *	completed the job ("hardware fence").
> > +	 *
> > +	 * Note that the scheduler expects to 'inherit' its own reference to
> > +	 * this fence from the callback. It does not invoke an extra
> > +	 * dma_fence_get() on it. Consequently, this callback must return a
> > +	 * fence whose refcount is at least 2: One for the scheduler's
> > +	 * reference returned here, another one for the reference kept by the
> > +	 * driver.
> 
> Well the driver actually doesn't need any extra reference. The scheduler
> just needs to guarantee that this reference isn't dropped before it is
> signaled.

I think he means the reference the driver's fence context has to have in order
to signal that thing eventually.

> 
> Regards,
> Christian.
> 
> >   	 */
> >   	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
>

Christian König Dec. 20, 2024, 1:25 p.m. UTC | #4

Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
> On Fri, Dec 20, 2024 at 01:53:34PM +0100, Christian König wrote:
>> Am 20.12.24 um 13:45 schrieb Philipp Stanner:
>>> From: Philipp Stanner <[e-mail]>
>>>
>>> drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler.
>>> That fence is signalled by the driver once the hardware completed the
>>> associated job. The scheduler does not increment the reference count on
>>> that fence, but implicitly expects to inherit this fence from run_job().
>>>
>>> This is relatively subtle and prone to misunderstandings.
>>>
>>> This implies that, to keep a reference for itself, a driver needs to
>>> call dma_fence_get() in addition to dma_fence_init() in that callback.
>>>
>>> It's further complicated by the fact that the scheduler even decrements
>>> the refcount in drm_sched_run_job_work() since it created a new
>>> reference in drm_sched_fence_scheduled(). It does, however, still use
>>> its pointer to the fence after calling dma_fence_put() - which is safe
>>> because of the aforementioned new reference, but actually still violates
>>> the refcounting rules.
>>>
>>> Improve the explanatory comment for that decrement.
>>>
>>> Move the call to dma_fence_put() to the position behind the last usage
>>> of the fence.
>>>
>>> Document the necessity to increment the reference count in
>>> drm_sched_backend_ops.run_job().
>>>
>>> Cc: Christian König <[e-mail]>
>>> Cc: Tvrtko Ursulin <[e-mail]>
>>> Cc: Andrey Grodzovsky <[e-mail]>
>>> Signed-off-by: Philipp Stanner <[e-mail]>
>>> ---
>>>    drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
>>>    include/drm/gpu_scheduler.h            | 20 ++++++++++++++++----
>>>    2 files changed, 23 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
>>> index 7ce25281c74c..d6f8df39d848 100644
>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>> +	 *
>>> +	 * @sched_job: the job to run
>>> +	 *
>>> +	 * Returns: dma_fence the driver must signal once the hardware has
>>> +	 *	completed the job ("hardware fence").
>>> +	 *
>>> +	 * Note that the scheduler expects to 'inherit' its own reference to
>>> +	 * this fence from the callback. It does not invoke an extra
>>> +	 * dma_fence_get() on it. Consequently, this callback must return a
>>> +	 * fence whose refcount is at least 2: One for the scheduler's
>>> +	 * reference returned here, another one for the reference kept by the
>>> +	 * driver.
>> Well the driver actually doesn't need any extra reference. The scheduler
>> just needs to guarantee that this reference isn't dropped before it is
>> signaled.
> I think he means the reference the driver's fence context has to have in order
> to signal that thing eventually.

Yeah, but this is usually a weak reference. IIRC most drivers don't 
increment the reference count for the reference they keep to signal a fence.

It's expected that the consumers of the dma_fence keep the fence alive 
at least until it is signaled. That's why we have this nice warning in 
dma_fence_release().

On the other hand I completely agree it would be more defensive if 
drivers increment the reference count for the reference they keep for 
signaling.

So if we want to document that the fence reference count should at least 
be 2 we somehow need to enforce this with a warning for example.

Regards,
Christian.



>
>> Regards,
>> Christian.
>>
>>>    	 */
>>>    	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);

Philipp Stanner Dec. 20, 2024, 2:11 p.m. UTC | #5

On Fri, 2024-12-20 at 14:25 +0100, Christian König wrote:
> Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
> > On Fri, Dec 20, 2024 at 01:53:34PM +0100, Christian König wrote:
> > > Am 20.12.24 um 13:45 schrieb Philipp Stanner:
> > > > From: Philipp Stanner <[e-mail]>
> > > > 
> > > > drm_sched_backend_ops.run_job() returns a dma_fence for the
> > > > scheduler.
> > > > That fence is signalled by the driver once the hardware
> > > > completed the
> > > > associated job. The scheduler does not increment the reference
> > > > count on
> > > > that fence, but implicitly expects to inherit this fence from
> > > > run_job().
> > > > 
> > > > This is relatively subtle and prone to misunderstandings.
> > > > 
> > > > This implies that, to keep a reference for itself, a driver
> > > > needs to
> > > > call dma_fence_get() in addition to dma_fence_init() in that
> > > > callback.
> > > > 
> > > > It's further complicated by the fact that the scheduler even
> > > > decrements
> > > > the refcount in drm_sched_run_job_work() since it created a new
> > > > reference in drm_sched_fence_scheduled(). It does, however,
> > > > still use
> > > > its pointer to the fence after calling dma_fence_put() - which
> > > > is safe
> > > > because of the aforementioned new reference, but actually still
> > > > violates
> > > > the refcounting rules.
> > > > 
> > > > Improve the explanatory comment for that decrement.
> > > > 
> > > > Move the call to dma_fence_put() to the position behind the
> > > > last usage
> > > > of the fence.
> > > > 
> > > > Document the necessity to increment the reference count in
> > > > drm_sched_backend_ops.run_job().
> > > > 
> > > > Cc: Christian König <[e-mail]>
> > > > Cc: Tvrtko Ursulin <[e-mail]>
> > > > Cc: Andrey Grodzovsky <[e-mail]>
> > > > Signed-off-by: Philipp Stanner <[e-mail]>
> > > > ---
> > > >    drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++---
> > > >    include/drm/gpu_scheduler.h            | 20
> > > > ++++++++++++++++----
> > > >    2 files changed, 23 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c
> > > > b/drivers/gpu/drm/scheduler/sched_main.c
> > > > index 7ce25281c74c..d6f8df39d848 100644
> > > > --- a/drivers/gpu/drm/scheduler/sched_main.c
> > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c
> > > > +	 *
> > > > +	 * @sched_job: the job to run
> > > > +	 *
> > > > +	 * Returns: dma_fence the driver must signal once the
> > > > hardware has
> > > > +	 *	completed the job ("hardware fence").
> > > > +	 *
> > > > +	 * Note that the scheduler expects to 'inherit' its
> > > > own reference to
> > > > +	 * this fence from the callback. It does not invoke an
> > > > extra
> > > > +	 * dma_fence_get() on it. Consequently, this callback
> > > > must return a
> > > > +	 * fence whose refcount is at least 2: One for the
> > > > scheduler's
> > > > +	 * reference returned here, another one for the
> > > > reference kept by the
> > > > +	 * driver.
> > > Well the driver actually doesn't need any extra reference. The
> > > scheduler
> > > just needs to guarantee that this reference isn't dropped before
> > > it is
> > > signaled.
> > I think he means the reference the driver's fence context has to
> > have in order
> > to signal that thing eventually.
> 
> Yeah, but this is usually a weak reference. IIRC most drivers don't 
> increment the reference count for the reference they keep to signal a
> fence.
> 
> It's expected that the consumers of the dma_fence keep the fence
> alive 
> at least until it is signaled.

So are you saying that the driver having an extra reference (without
having obtained it with dma_fence_get()) is not an issue because the
driver is the one who will signal the fence [and then be done with it]?

>  That's why we have this nice warning in 
> dma_fence_release().
> 
> On the other hand I completely agree it would be more defensive if 
> drivers increment the reference count for the reference they keep for
> signaling.
> 
> So if we want to document that the fence reference count should at
> least 
> be 2 we somehow need to enforce this with a warning for example.

We could – but I'm not sure whether it really needs to be "enforced",
especially if it were only to be a minor issue, as you seem to hint at
above.
Document it is the minimum IMO


P.

> 
> Regards,
> Christian.
> 
> 
> 
> > 
> > > Regards,
> > > Christian.
> > > 
> > > >    	 */
> > > >    	struct dma_fence *(*run_job)(struct drm_sched_job
> > > > *sched_job);
>

Danilo Krummrich Dec. 20, 2024, 2:51 p.m. UTC | #6

On Fri, Dec 20, 2024 at 03:11:34PM +0100, Philipp Stanner wrote:
> On Fri, 2024-12-20 at 14:25 +0100, Christian König wrote:
> > Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
> > > On Fri, Dec 20, 2024 at 01:53:34PM +0100, Christian König wrote:
> > > > Am 20.12.24 um 13:45 schrieb Philipp Stanner:
> > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c
> > > > > b/drivers/gpu/drm/scheduler/sched_main.c
> > > > > index 7ce25281c74c..d6f8df39d848 100644
> > > > > --- a/drivers/gpu/drm/scheduler/sched_main.c
> > > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c
> > > > > +	 *
> > > > > +	 * @sched_job: the job to run
> > > > > +	 *
> > > > > +	 * Returns: dma_fence the driver must signal once the
> > > > > hardware has
> > > > > +	 *	completed the job ("hardware fence").
> > > > > +	 *
> > > > > +	 * Note that the scheduler expects to 'inherit' its
> > > > > own reference to
> > > > > +	 * this fence from the callback. It does not invoke an
> > > > > extra
> > > > > +	 * dma_fence_get() on it. Consequently, this callback
> > > > > must return a
> > > > > +	 * fence whose refcount is at least 2: One for the
> > > > > scheduler's
> > > > > +	 * reference returned here, another one for the
> > > > > reference kept by the
> > > > > +	 * driver.
> > > > Well the driver actually doesn't need any extra reference. The
> > > > scheduler
> > > > just needs to guarantee that this reference isn't dropped before
> > > > it is
> > > > signaled.
> > > I think he means the reference the driver's fence context has to
> > > have in order
> > > to signal that thing eventually.
> > 
> > Yeah, but this is usually a weak reference. IIRC most drivers don't 
> > increment the reference count for the reference they keep to signal a
> > fence.
> > 
> > It's expected that the consumers of the dma_fence keep the fence
> > alive 
> > at least until it is signaled.
> 
> So are you saying that the driver having an extra reference (without
> having obtained it with dma_fence_get()) is not an issue because the
> driver is the one who will signal the fence [and then be done with it]?

It's never a "real" issue if you have multiple pointers to a reference counted
object as long as you can ensure that you hold at least one reference for the
time you have pointers to the object.

But, that's bad design. For every pointer to an object a separate reference
should be taken.

Christian König Dec. 20, 2024, 3:25 p.m. UTC | #7

Am 20.12.24 um 15:51 schrieb Danilo Krummrich:
> On Fri, Dec 20, 2024 at 03:11:34PM +0100, Philipp Stanner wrote:
>> On Fri, 2024-12-20 at 14:25 +0100, Christian König wrote:
>>> Am 20.12.24 um 14:18 schrieb Danilo Krummrich:
>>>> On Fri, Dec 20, 2024 at 01:53:34PM +0100, Christian König wrote:
>>>>> Am 20.12.24 um 13:45 schrieb Philipp Stanner:
>>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>>>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>>>>> index 7ce25281c74c..d6f8df39d848 100644
>>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>>>>> +	 *
>>>>>> +	 * @sched_job: the job to run
>>>>>> +	 *
>>>>>> +	 * Returns: dma_fence the driver must signal once the
>>>>>> hardware has
>>>>>> +	 *	completed the job ("hardware fence").
>>>>>> +	 *
>>>>>> +	 * Note that the scheduler expects to 'inherit' its
>>>>>> own reference to
>>>>>> +	 * this fence from the callback. It does not invoke an
>>>>>> extra
>>>>>> +	 * dma_fence_get() on it. Consequently, this callback
>>>>>> must return a
>>>>>> +	 * fence whose refcount is at least 2: One for the
>>>>>> scheduler's
>>>>>> +	 * reference returned here, another one for the
>>>>>> reference kept by the
>>>>>> +	 * driver.
>>>>> Well the driver actually doesn't need any extra reference. The
>>>>> scheduler
>>>>> just needs to guarantee that this reference isn't dropped before
>>>>> it is
>>>>> signaled.
>>>> I think he means the reference the driver's fence context has to
>>>> have in order
>>>> to signal that thing eventually.
>>> Yeah, but this is usually a weak reference. IIRC most drivers don't
>>> increment the reference count for the reference they keep to signal a
>>> fence.
>>>
>>> It's expected that the consumers of the dma_fence keep the fence
>>> alive
>>> at least until it is signaled.
>> So are you saying that the driver having an extra reference (without
>> having obtained it with dma_fence_get()) is not an issue because the
>> driver is the one who will signal the fence [and then be done with it]?
> It's never a "real" issue if you have multiple pointers to a reference counted
> object as long as you can ensure that you hold at least one reference for the
> time you have pointers to the object.

Well, I'm not saying that this isn't an issue. I'm just pointing out 
that this is the current practice :)

> But, that's bad design. For every pointer to an object a separate reference
> should be taken.

Yeah, completely agree. Weak references are usually a bad idea if you 
don't absolutely need them for something.

Regards,
Christian.

Patch

diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index 7ce25281c74c..d6f8df39d848 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -1218,15 +1218,19 @@  static void drm_sched_run_job_work(struct work_struct *w)
 	drm_sched_fence_scheduled(s_fence, fence);
 
 	if (!IS_ERR_OR_NULL(fence)) {
-		/* Drop for original kref_init of the fence */
-		dma_fence_put(fence);
-
 		r = dma_fence_add_callback(fence, &sched_job->cb,
 					   drm_sched_job_done_cb);
 		if (r == -ENOENT)
 			drm_sched_job_done(sched_job, fence->error);
 		else if (r)
 			DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);
+
+		/*
+		 * s_fence took a new reference to fence in the call to
+		 * drm_sched_fence_scheduled() above. The reference passed by
+		 * run_job() above is now not needed any longer. Drop it.
+		 */
+		dma_fence_put(fence);
 	} else {
 		drm_sched_job_done(sched_job, IS_ERR(fence) ?
 				   PTR_ERR(fence) : 0);
diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
index 95e17504e46a..a1f5c9a14278 100644
--- a/include/drm/gpu_scheduler.h
+++ b/include/drm/gpu_scheduler.h
@@ -420,10 +420,22 @@  struct drm_sched_backend_ops {
 					 struct drm_sched_entity *s_entity);
 
 	/**
-         * @run_job: Called to execute the job once all of the dependencies
-         * have been resolved.  This may be called multiple times, if
-	 * timedout_job() has happened and drm_sched_job_recovery()
-	 * decides to try it again.
+	 * @run_job: Called to execute the job once all of the dependencies
+	 * have been resolved. This may be called multiple times, if
+	 * timedout_job() has happened and drm_sched_job_recovery() decides to
+	 * try it again.
+	 *
+	 * @sched_job: the job to run
+	 *
+	 * Returns: dma_fence the driver must signal once the hardware has
+	 *	completed the job ("hardware fence").
+	 *
+	 * Note that the scheduler expects to 'inherit' its own reference to
+	 * this fence from the callback. It does not invoke an extra
+	 * dma_fence_get() on it. Consequently, this callback must return a
+	 * fence whose refcount is at least 2: One for the scheduler's
+	 * reference returned here, another one for the reference kept by the
+	 * driver.
 	 */
 	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);