| Message ID | 20190905090949.30424-1-janusz.krzysztofik@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | drm/i915: Don't unwedge if reset is disabled | expand |
On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: > When trying to reset a device with reset capability disabled or not > supported while rings are full of requests, it has been observed when > running in execlists submission mode that command stream buffer tail > tends to be incremented by apparently still running GPU regardless of > all requests being already cancelled and command stream buffer pointers > reset. As a result, kernel panic on NULL pointer dereference occurs > when a trace_ports() helper is called with command stream buffer tail > incremented but request pointers being NULL during final > __intel_gt_set_wedged() operation called from intel_gt_reset(). > > Skip actual reset procedure if reset is disabled or not supported. This last sentence is a bit confusing. You're not skipping the reset procedure, you're skipping the attempt of unwedging and resetting again after a reset & wedge already happened. > > Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> > --- > drivers/gpu/drm/i915/gt/intel_reset.c | 26 ++++++++++++++++++-------- > 1 file changed, 18 insertions(+), 8 deletions(-) > > diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c > index b9d84d52e986..d75da124e280 100644 > --- a/drivers/gpu/drm/i915/gt/intel_reset.c > +++ b/drivers/gpu/drm/i915/gt/intel_reset.c > @@ -932,25 +932,35 @@ void intel_gt_reset(struct intel_gt *gt, > GEM_BUG_ON(!test_bit(I915_RESET_BACKOFF, >->reset.flags)); > mutex_lock(>->reset.mutex); > > - /* Clear any previous failed attempts at recovery. Time to try again. */ > - if (!__intel_gt_unset_wedged(gt)) > - goto unlock; > - Since you're anyway checking the wedged status and reset support multiple times, wouldn't it have been better to just add a single check at the beginning? e.g. /* we can't recover a wedged GT without reset */ if (!intel_has_gpu_reset(gt->i915) && intel_gt_is_wedged(gt)) goto unlock; Daniele > if (reason) > dev_notice(gt->i915->drm.dev, > "Resetting chip for %s\n", reason); > - atomic_inc(>->i915->gpu_error.reset_count); > - > - awake = reset_prepare(gt); > > if (!intel_has_gpu_reset(gt->i915)) { > if (i915_modparams.reset) > dev_err(gt->i915->drm.dev, "GPU reset not supported\n"); > else > DRM_DEBUG_DRIVER("GPU reset disabled\n"); > - goto error; > + > + /* > + * Don't unwedge if reset is disabled or not supported > + * because we can't guarantee what the hardware status is. > + */ > + if (intel_gt_is_wedged(gt)) > + goto unlock; > } > > + /* Clear any previous failed attempts at recovery. Time to try again. */ > + if (!__intel_gt_unset_wedged(gt)) > + goto unlock; > + > + atomic_inc(>->i915->gpu_error.reset_count); > + > + awake = reset_prepare(gt); > + > + if (!intel_has_gpu_reset(gt->i915)) > + goto error; > + > if (INTEL_INFO(gt->i915)->gpu_reset_clobbers_display) > intel_runtime_pm_disable_interrupts(gt->i915); > >
Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05) > > > On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: > > When trying to reset a device with reset capability disabled or not > > supported while rings are full of requests, it has been observed when > > running in execlists submission mode that command stream buffer tail > > tends to be incremented by apparently still running GPU regardless of > > all requests being already cancelled and command stream buffer pointers > > reset. As a result, kernel panic on NULL pointer dereference occurs > > when a trace_ports() helper is called with command stream buffer tail > > incremented but request pointers being NULL during final > > __intel_gt_set_wedged() operation called from intel_gt_reset(). > > > > Skip actual reset procedure if reset is disabled or not supported. > > This last sentence is a bit confusing. You're not skipping the reset > procedure, you're skipping the attempt of unwedging and resetting again > after a reset & wedge already happened. Loss of email over the last week, so jumping in at the end. My gut response is that this is still just papering over the bug, as what you say above makes no sense. -Chris
On 9/7/19 1:39 AM, Chris Wilson wrote: > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05) >> >> >> On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: >>> When trying to reset a device with reset capability disabled or not >>> supported while rings are full of requests, it has been observed when >>> running in execlists submission mode that command stream buffer tail >>> tends to be incremented by apparently still running GPU regardless of >>> all requests being already cancelled and command stream buffer pointers >>> reset. As a result, kernel panic on NULL pointer dereference occurs >>> when a trace_ports() helper is called with command stream buffer tail >>> incremented but request pointers being NULL during final >>> __intel_gt_set_wedged() operation called from intel_gt_reset(). >>> >>> Skip actual reset procedure if reset is disabled or not supported. >> >> This last sentence is a bit confusing. You're not skipping the reset >> procedure, you're skipping the attempt of unwedging and resetting again >> after a reset & wedge already happened. > > Loss of email over the last week, so jumping in at the end. My gut > response is that this is still just papering over the bug, as what you > say above makes no sense. > -Chris > The issue here is that if we don't reset the HW when we wedge, whatever was running on the engines might complete at any point after that, which generates an unexpected post-wedge CSB event that we don't handle gracefully when we unwedge. The CSB event might arrive at any time (even after the unwedge) or cause weird behavior on the first re-submission, so trying to handle it is not worth the effort IMO since having reset disabled is a debug-only use-case. Daniele
Quoting Daniele Ceraolo Spurio (2019-09-09 17:27:47) > > > On 9/7/19 1:39 AM, Chris Wilson wrote: > > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05) > >> > >> > >> On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: > >>> When trying to reset a device with reset capability disabled or not > >>> supported while rings are full of requests, it has been observed when > >>> running in execlists submission mode that command stream buffer tail > >>> tends to be incremented by apparently still running GPU regardless of > >>> all requests being already cancelled and command stream buffer pointers > >>> reset. As a result, kernel panic on NULL pointer dereference occurs > >>> when a trace_ports() helper is called with command stream buffer tail > >>> incremented but request pointers being NULL during final > >>> __intel_gt_set_wedged() operation called from intel_gt_reset(). > >>> > >>> Skip actual reset procedure if reset is disabled or not supported. > >> > >> This last sentence is a bit confusing. You're not skipping the reset > >> procedure, you're skipping the attempt of unwedging and resetting again > >> after a reset & wedge already happened. > > > > Loss of email over the last week, so jumping in at the end. My gut > > response is that this is still just papering over the bug, as what you > > say above makes no sense. > > -Chris > > > > The issue here is that if we don't reset the HW when we wedge, whatever > was running on the engines might complete at any point after that, which > generates an unexpected post-wedge CSB event that we don't handle > gracefully when we unwedge. Indeed, until we call reset_default_submission all those unexpected interrupts are redirected to the nop_submission_tasklet. I think it should be more along the lines of struct intel_timeline *tl; unsigned long flags; + bool ok; if (!test_bit(I915_WEDGED, >->reset.flags)) return true; @@ -838,7 +839,11 @@ static bool __intel_gt_unset_wedged(struct intel_gt *gt) } spin_unlock_irqrestore(&timelines->lock, flags); - intel_gt_sanitize(gt, false); + ok = false; + if (!INTEL_INFO(gt->i915)->gpu_reset_clobbers_display) + ok = __intel_gt_reset(gt, ALL_ENGINES) == 0; + if (!ok) + return false; /* * Undo nop_submit_request. We prevent all new i915 requests from For bonus points, gpu_reset_clobbers_display should take into account whether the display is active. -Chris
Quoting Chris Wilson (2019-09-07 09:39:52) > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05) > > > > > > On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: > > > When trying to reset a device with reset capability disabled or not > > > supported while rings are full of requests, it has been observed when > > > running in execlists submission mode that command stream buffer tail > > > tends to be incremented by apparently still running GPU regardless of > > > all requests being already cancelled and command stream buffer pointers > > > reset. As a result, kernel panic on NULL pointer dereference occurs > > > when a trace_ports() helper is called with command stream buffer tail > > > incremented but request pointers being NULL during final > > > __intel_gt_set_wedged() operation called from intel_gt_reset(). > > > > > > Skip actual reset procedure if reset is disabled or not supported. > > > > This last sentence is a bit confusing. You're not skipping the reset > > procedure, you're skipping the attempt of unwedging and resetting again > > after a reset & wedge already happened. > > Loss of email over the last week, so jumping in at the end. My gut > response is that this is still just papering over the bug, as what you > say above makes no sense. So my gut response was to the run on sentence, when all you needed to say that without a successful reset prior to calling reset_default_submission, the engine may still generate CS events out of the blue. And I think the patch should be written to require the successful reset. -Chris
On Monday, September 9, 2019 11:48:42 PM CEST Chris Wilson wrote: > Quoting Chris Wilson (2019-09-07 09:39:52) > > Quoting Daniele Ceraolo Spurio (2019-09-06 23:28:05) > > > > > > > > > On 9/5/19 2:09 AM, Janusz Krzysztofik wrote: > > > > When trying to reset a device with reset capability disabled or not > > > > supported while rings are full of requests, it has been observed when > > > > running in execlists submission mode that command stream buffer tail > > > > tends to be incremented by apparently still running GPU regardless of > > > > all requests being already cancelled and command stream buffer pointers > > > > reset. As a result, kernel panic on NULL pointer dereference occurs > > > > when a trace_ports() helper is called with command stream buffer tail > > > > incremented but request pointers being NULL during final > > > > __intel_gt_set_wedged() operation called from intel_gt_reset(). > > > > > > > > Skip actual reset procedure if reset is disabled or not supported. > > > > > > This last sentence is a bit confusing. You're not skipping the reset > > > procedure, you're skipping the attempt of unwedging and resetting again > > > after a reset & wedge already happened. > > > > Loss of email over the last week, so jumping in at the end. My gut > > response is that this is still just papering over the bug, as what you > > say above makes no sense. > > So my gut response was to the run on sentence, when all you needed to > say that without a successful reset prior to calling > reset_default_submission, the engine may still generate CS events out of > the blue. And I think the patch should be written to require the > successful reset. You are right, successful reset seems the only safe protection. But anyway, while digging deeper waiting for your clarification of that gut respone ;-) , I've discovered that symptoms from which the issue can be predicted may be sometimes observed during reset_prepere() as failing intel_engine_stop_cs(). Checking for that failure alone may be too weak as it can probably happen to succeed regardless of the uncertain hardware status, but anyway, what do you think about modifying reset_prepare() so it may fail with an error propagated from functions it calls, then calling reset_prepare() at the beginning of intel_gt_reset() and skiping over __intel_gt_unset_wedgede() and further steps (do_reset(), ..., reset_finish()) if reset_prepare() fails? Wouldn't that be a useful additional layer of protection? If you think the idea is worth of being considered, please have a look at my first attempt sent to trybot already before your explanation arrived: https://patchwork.freedesktop.org/patch/329840/?series=66447&rev=1 (don't complain on its commit message making no sense, please ;-) ). Thanks, Janusz > -Chris >
diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c index b9d84d52e986..d75da124e280 100644 --- a/drivers/gpu/drm/i915/gt/intel_reset.c +++ b/drivers/gpu/drm/i915/gt/intel_reset.c @@ -932,25 +932,35 @@ void intel_gt_reset(struct intel_gt *gt, GEM_BUG_ON(!test_bit(I915_RESET_BACKOFF, >->reset.flags)); mutex_lock(>->reset.mutex); - /* Clear any previous failed attempts at recovery. Time to try again. */ - if (!__intel_gt_unset_wedged(gt)) - goto unlock; - if (reason) dev_notice(gt->i915->drm.dev, "Resetting chip for %s\n", reason); - atomic_inc(>->i915->gpu_error.reset_count); - - awake = reset_prepare(gt); if (!intel_has_gpu_reset(gt->i915)) { if (i915_modparams.reset) dev_err(gt->i915->drm.dev, "GPU reset not supported\n"); else DRM_DEBUG_DRIVER("GPU reset disabled\n"); - goto error; + + /* + * Don't unwedge if reset is disabled or not supported + * because we can't guarantee what the hardware status is. + */ + if (intel_gt_is_wedged(gt)) + goto unlock; } + /* Clear any previous failed attempts at recovery. Time to try again. */ + if (!__intel_gt_unset_wedged(gt)) + goto unlock; + + atomic_inc(>->i915->gpu_error.reset_count); + + awake = reset_prepare(gt); + + if (!intel_has_gpu_reset(gt->i915)) + goto error; + if (INTEL_INFO(gt->i915)->gpu_reset_clobbers_display) intel_runtime_pm_disable_interrupts(gt->i915);
When trying to reset a device with reset capability disabled or not supported while rings are full of requests, it has been observed when running in execlists submission mode that command stream buffer tail tends to be incremented by apparently still running GPU regardless of all requests being already cancelled and command stream buffer pointers reset. As a result, kernel panic on NULL pointer dereference occurs when a trace_ports() helper is called with command stream buffer tail incremented but request pointers being NULL during final __intel_gt_set_wedged() operation called from intel_gt_reset(). Skip actual reset procedure if reset is disabled or not supported. Suggested-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com> --- drivers/gpu/drm/i915/gt/intel_reset.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)