diff mbox series

drm/i915: Don't unwedge if reset is disabled

Message ID 20190905090949.30424-1-janusz.krzysztofik@linux.intel.com (mailing list archive)
State New, archived
Headers show
Series drm/i915: Don't unwedge if reset is disabled | expand

Commit Message

Janusz Krzysztofik Sept. 5, 2019, 9:09 a.m. UTC
When trying to reset a device with reset capability disabled or not
supported while rings are full of requests, it has been observed when
running in execlists submission mode that command stream buffer tail
tends to be incremented by apparently still running GPU regardless of
all requests being already cancelled and command stream buffer pointers
reset.  As a result, kernel panic on NULL pointer dereference occurs
when a trace_ports() helper is called with command stream buffer tail
incremented but request pointers being NULL during final
__intel_gt_set_wedged() operation called from intel_gt_reset().

Skip actual reset procedure if reset is disabled or not supported.

Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
---
 drivers/gpu/drm/i915/gt/intel_reset.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

Comments

Daniele Ceraolo Spurio Sept. 6, 2019, 10:28 p.m. UTC | #1
On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
> When trying to reset a device with reset capability disabled or not
> supported while rings are full of requests, it has been observed when
> running in execlists submission mode that command stream buffer tail
> tends to be incremented by apparently still running GPU regardless of
> all requests being already cancelled and command stream buffer pointers
> reset.  As a result, kernel panic on NULL pointer dereference occurs
> when a trace_ports() helper is called with command stream buffer tail
> incremented but request pointers being NULL during final
> __intel_gt_set_wedged() operation called from intel_gt_reset().
> 
> Skip actual reset procedure if reset is disabled or not supported.

This last sentence is a bit confusing. You're not skipping the reset 
procedure, you're skipping the attempt of unwedging and resetting again 
after a reset & wedge already happened.

> 
> Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
> Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
> ---
>   drivers/gpu/drm/i915/gt/intel_reset.c | 26 ++++++++++++++++++--------
>   1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
> index b9d84d52e986..d75da124e280 100644
> --- a/drivers/gpu/drm/i915/gt/intel_reset.c
> +++ b/drivers/gpu/drm/i915/gt/intel_reset.c
> @@ -932,25 +932,35 @@ void intel_gt_reset(struct intel_gt *gt,
>   	GEM_BUG_ON(!test_bit(I915_RESET_BACKOFF, &gt->reset.flags));
>   	mutex_lock(&gt->reset.mutex);
>   
> -	/* Clear any previous failed attempts at recovery. Time to try again. */
> -	if (!__intel_gt_unset_wedged(gt))
> -		goto unlock;
> -

Since you're anyway checking the wedged status and reset support 
multiple times, wouldn't it have been better to just add a single check 
at the beginning? e.g.

	/* we can't recover a wedged GT without reset */
	if (!intel_has_gpu_reset(gt->i915) && intel_gt_is_wedged(gt))
		goto unlock;

Daniele

>   	if (reason)
>   		dev_notice(gt->i915->drm.dev,
>   			   "Resetting chip for %s\n", reason);
> -	atomic_inc(&gt->i915->gpu_error.reset_count);
> -
> -	awake = reset_prepare(gt);
>   
>   	if (!intel_has_gpu_reset(gt->i915)) {
>   		if (i915_modparams.reset)
>   			dev_err(gt->i915->drm.dev, "GPU reset not supported\n");
>   		else
>   			DRM_DEBUG_DRIVER("GPU reset disabled\n");
> -		goto error;
> +
> +		/*
> +		 * Don't unwedge if reset is disabled or not supported
> +		 * because we can't guarantee what the hardware status is.
> +		 */
> +		if (intel_gt_is_wedged(gt))
> +			goto unlock;
>   	}
>   
> +	/* Clear any previous failed attempts at recovery. Time to try again. */
> +	if (!__intel_gt_unset_wedged(gt))
> +		goto unlock;
> +
> +	atomic_inc(&gt->i915->gpu_error.reset_count);
> +
> +	awake = reset_prepare(gt);
> +
> +	if (!intel_has_gpu_reset(gt->i915))
> +		goto error;
> +
>   	if (INTEL_INFO(gt->i915)->gpu_reset_clobbers_display)
>   		intel_runtime_pm_disable_interrupts(gt->i915);
>   
>
Chris Wilson Sept. 7, 2019, 8:39 a.m. UTC | #2
Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05)
> 
> 
> On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
> > When trying to reset a device with reset capability disabled or not
> > supported while rings are full of requests, it has been observed when
> > running in execlists submission mode that command stream buffer tail
> > tends to be incremented by apparently still running GPU regardless of
> > all requests being already cancelled and command stream buffer pointers
> > reset.  As a result, kernel panic on NULL pointer dereference occurs
> > when a trace_ports() helper is called with command stream buffer tail
> > incremented but request pointers being NULL during final
> > __intel_gt_set_wedged() operation called from intel_gt_reset().
> > 
> > Skip actual reset procedure if reset is disabled or not supported.
> 
> This last sentence is a bit confusing. You're not skipping the reset 
> procedure, you're skipping the attempt of unwedging and resetting again 
> after a reset & wedge already happened.

Loss of email over the last week, so jumping in at the end. My gut
response is that this is still just papering over the bug, as what you
say above makes no sense.
-Chris
Daniele Ceraolo Spurio Sept. 9, 2019, 4:27 p.m. UTC | #3
On 9/7/19 1:39 AM, Chris Wilson wrote:
> Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05)
>>
>>
>> On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
>>> When trying to reset a device with reset capability disabled or not
>>> supported while rings are full of requests, it has been observed when
>>> running in execlists submission mode that command stream buffer tail
>>> tends to be incremented by apparently still running GPU regardless of
>>> all requests being already cancelled and command stream buffer pointers
>>> reset.  As a result, kernel panic on NULL pointer dereference occurs
>>> when a trace_ports() helper is called with command stream buffer tail
>>> incremented but request pointers being NULL during final
>>> __intel_gt_set_wedged() operation called from intel_gt_reset().
>>>
>>> Skip actual reset procedure if reset is disabled or not supported.
>>
>> This last sentence is a bit confusing. You're not skipping the reset
>> procedure, you're skipping the attempt of unwedging and resetting again
>> after a reset & wedge already happened.
> 
> Loss of email over the last week, so jumping in at the end. My gut
> response is that this is still just papering over the bug, as what you
> say above makes no sense.
> -Chris
> 

The issue here is that if we don't reset the HW when we wedge, whatever 
was running on the engines might complete at any point after that, which 
generates an unexpected post-wedge CSB event that we don't handle 
gracefully when we unwedge. The CSB event might arrive at any time (even 
after the unwedge) or cause weird behavior on the first re-submission, 
so trying to handle it is not worth the effort IMO since having reset 
disabled is a debug-only use-case.

Daniele
Chris Wilson Sept. 9, 2019, 9:45 p.m. UTC | #4
Quoting Daniele Ceraolo Spurio (2019-09-09 17:27:47)
> 
> 
> On 9/7/19 1:39 AM, Chris Wilson wrote:
> > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05)
> >>
> >>
> >> On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
> >>> When trying to reset a device with reset capability disabled or not
> >>> supported while rings are full of requests, it has been observed when
> >>> running in execlists submission mode that command stream buffer tail
> >>> tends to be incremented by apparently still running GPU regardless of
> >>> all requests being already cancelled and command stream buffer pointers
> >>> reset.  As a result, kernel panic on NULL pointer dereference occurs
> >>> when a trace_ports() helper is called with command stream buffer tail
> >>> incremented but request pointers being NULL during final
> >>> __intel_gt_set_wedged() operation called from intel_gt_reset().
> >>>
> >>> Skip actual reset procedure if reset is disabled or not supported.
> >>
> >> This last sentence is a bit confusing. You're not skipping the reset
> >> procedure, you're skipping the attempt of unwedging and resetting again
> >> after a reset & wedge already happened.
> > 
> > Loss of email over the last week, so jumping in at the end. My gut
> > response is that this is still just papering over the bug, as what you
> > say above makes no sense.
> > -Chris
> > 
> 
> The issue here is that if we don't reset the HW when we wedge, whatever 
> was running on the engines might complete at any point after that, which 
> generates an unexpected post-wedge CSB event that we don't handle 
> gracefully when we unwedge.

Indeed, until we call reset_default_submission all those unexpected
interrupts are redirected to the nop_submission_tasklet.

I think it should be more along the lines of

 	struct intel_timeline *tl;
 	unsigned long flags;
+	bool ok;

 	if (!test_bit(I915_WEDGED, &gt->reset.flags))
 		return true;
@@ -838,7 +839,11 @@ static bool __intel_gt_unset_wedged(struct intel_gt *gt)
 	}
 	spin_unlock_irqrestore(&timelines->lock, flags);

-	intel_gt_sanitize(gt, false);
+	ok = false;
+	if (!INTEL_INFO(gt->i915)->gpu_reset_clobbers_display)
+		ok = __intel_gt_reset(gt, ALL_ENGINES) == 0;
+	if (!ok)
+		return false;

 	/*
 	 * Undo nop_submit_request. We prevent all new i915 requests from

For bonus points, gpu_reset_clobbers_display should take into account
whether the display is active.
-Chris
Chris Wilson Sept. 9, 2019, 9:48 p.m. UTC | #5
Quoting Chris Wilson (2019-09-07 09:39:52)
> Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05)
> > 
> > 
> > On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
> > > When trying to reset a device with reset capability disabled or not
> > > supported while rings are full of requests, it has been observed when
> > > running in execlists submission mode that command stream buffer tail
> > > tends to be incremented by apparently still running GPU regardless of
> > > all requests being already cancelled and command stream buffer pointers
> > > reset.  As a result, kernel panic on NULL pointer dereference occurs
> > > when a trace_ports() helper is called with command stream buffer tail
> > > incremented but request pointers being NULL during final
> > > __intel_gt_set_wedged() operation called from intel_gt_reset().
> > > 
> > > Skip actual reset procedure if reset is disabled or not supported.
> > 
> > This last sentence is a bit confusing. You're not skipping the reset 
> > procedure, you're skipping the attempt of unwedging and resetting again 
> > after a reset & wedge already happened.
> 
> Loss of email over the last week, so jumping in at the end. My gut
> response is that this is still just papering over the bug, as what you
> say above makes no sense.

So my gut response was to the run on sentence, when all you needed to
say that without a successful reset prior to calling
reset_default_submission, the engine may still generate CS events out of
the blue. And I think the patch should be written to require the
successful reset.
-Chris
Janusz Krzysztofik Sept. 13, 2019, 2:20 p.m. UTC | #6
On Monday, September 9, 2019 11:48:42 PM CEST Chris Wilson wrote:
> Quoting Chris Wilson (2019-09-07 09:39:52)
> > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05)
> > > 
> > > 
> > > On 9/5/19 2:09 AM, Janusz Krzysztofik wrote:
> > > > When trying to reset a device with reset capability disabled or not
> > > > supported while rings are full of requests, it has been observed when
> > > > running in execlists submission mode that command stream buffer tail
> > > > tends to be incremented by apparently still running GPU regardless of
> > > > all requests being already cancelled and command stream buffer 
pointers
> > > > reset.  As a result, kernel panic on NULL pointer dereference occurs
> > > > when a trace_ports() helper is called with command stream buffer tail
> > > > incremented but request pointers being NULL during final
> > > > __intel_gt_set_wedged() operation called from intel_gt_reset().
> > > > 
> > > > Skip actual reset procedure if reset is disabled or not supported.
> > > 
> > > This last sentence is a bit confusing. You're not skipping the reset 
> > > procedure, you're skipping the attempt of unwedging and resetting again 
> > > after a reset & wedge already happened.
> > 
> > Loss of email over the last week, so jumping in at the end. My gut
> > response is that this is still just papering over the bug, as what you
> > say above makes no sense.
> 
> So my gut response was to the run on sentence, when all you needed to
> say that without a successful reset prior to calling
> reset_default_submission, the engine may still generate CS events out of
> the blue. And I think the patch should be written to require the
> successful reset.

You are right, successful reset seems the only safe protection.

But anyway, while digging deeper waiting for your clarification of that gut 
respone ;-) , I've discovered that symptoms from which the issue can be 
predicted may be sometimes observed during reset_prepere() as failing 
intel_engine_stop_cs().  Checking for that failure alone may be too weak as it 
can probably happen to succeed regardless of the uncertain hardware status, 
but anyway, what do you think about modifying reset_prepare() so it may fail 
with an error propagated from functions it calls, then calling reset_prepare() 
at the beginning of intel_gt_reset() and skiping over 
__intel_gt_unset_wedgede() and further steps (do_reset(), ..., reset_finish()) 
if reset_prepare() fails?  Wouldn't that be a useful additional layer of 
protection?

If you think the idea is worth of being considered, please have a look at my 
first attempt sent to trybot already before your explanation arrived:
https://patchwork.freedesktop.org/patch/329840/?series=66447&rev=1
(don't complain on its commit message making no sense, please ;-) ).

Thanks,
Janusz

> -Chris
>
diff mbox series

Patch

diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c
index b9d84d52e986..d75da124e280 100644
--- a/drivers/gpu/drm/i915/gt/intel_reset.c
+++ b/drivers/gpu/drm/i915/gt/intel_reset.c
@@ -932,25 +932,35 @@  void intel_gt_reset(struct intel_gt *gt,
 	GEM_BUG_ON(!test_bit(I915_RESET_BACKOFF, &gt->reset.flags));
 	mutex_lock(&gt->reset.mutex);
 
-	/* Clear any previous failed attempts at recovery. Time to try again. */
-	if (!__intel_gt_unset_wedged(gt))
-		goto unlock;
-
 	if (reason)
 		dev_notice(gt->i915->drm.dev,
 			   "Resetting chip for %s\n", reason);
-	atomic_inc(&gt->i915->gpu_error.reset_count);
-
-	awake = reset_prepare(gt);
 
 	if (!intel_has_gpu_reset(gt->i915)) {
 		if (i915_modparams.reset)
 			dev_err(gt->i915->drm.dev, "GPU reset not supported\n");
 		else
 			DRM_DEBUG_DRIVER("GPU reset disabled\n");
-		goto error;
+
+		/*
+		 * Don't unwedge if reset is disabled or not supported
+		 * because we can't guarantee what the hardware status is.
+		 */
+		if (intel_gt_is_wedged(gt))
+			goto unlock;
 	}
 
+	/* Clear any previous failed attempts at recovery. Time to try again. */
+	if (!__intel_gt_unset_wedged(gt))
+		goto unlock;
+
+	atomic_inc(&gt->i915->gpu_error.reset_count);
+
+	awake = reset_prepare(gt);
+
+	if (!intel_has_gpu_reset(gt->i915))
+		goto error;
+
 	if (INTEL_INFO(gt->i915)->gpu_reset_clobbers_display)
 		intel_runtime_pm_disable_interrupts(gt->i915);