| Message ID | 20220811210812.3239621-1-daniele.ceraolospurio@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | drm/i915/guc: clear stalled request after a reset | expand |
On 11/08/2022 22:08, Daniele Ceraolo Spurio wrote: > If the GuC CTs are full and we need to stall the request submission > while waiting for space, we save the stalled request and where the stall > occurred; when the CTs have space again we pick up the request submission > from where we left off. How serious is it? Statement always was CT buffers can never get full outside the pathological IGT test cases. So I am wondering if this is in the category of fix for correctness or actually the CT buffers can get full in normal use so it is imperative to fix. Regards, Tvrtko > If a full GT reset occurs, the state of all contexts is cleared and all > non-guilty requests are unsubmitted, therefore we need to restart the > stalled request submission from scratch. To make sure that we do so, > clear the saved request after a reset. > > Fixes note: the patch that introduced the bug is in 5.15, but no > officially supported platform had GuC submission enabled by default > in that kernel, so the backport to that particular version (and only > that one) can potentially be skipped. > > Fixes: 925dc1cf58ed ("drm/i915/guc: Implement GuC submission tasklet") > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> > Cc: Matthew Brost <matthew.brost@intel.com> > Cc: John Harrison <john.c.harrison@intel.com> > Cc: <stable@vger.kernel.org> # v5.15+ > --- > drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > index 0d17da77e787..0d56b615bf78 100644 > --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > @@ -4002,6 +4002,13 @@ static inline void guc_init_lrc_mapping(struct intel_guc *guc) > /* make sure all descriptors are clean... */ > xa_destroy(&guc->context_lookup); > > + /* > + * A reset might have occurred while we had a pending stalled request, > + * so make sure we clean that up. > + */ > + guc->stalled_request = NULL; > + guc->submission_stall_reason = STALL_NONE; > + > /* > * Some contexts might have been pinned before we enabled GuC > * submission, so we need to add them to the GuC bookeeping.
On 8/12/2022 12:29 AM, Tvrtko Ursulin wrote: > > On 11/08/2022 22:08, Daniele Ceraolo Spurio wrote: >> If the GuC CTs are full and we need to stall the request submission >> while waiting for space, we save the stalled request and where the stall >> occurred; when the CTs have space again we pick up the request >> submission >> from where we left off. > > How serious is it? Statement always was CT buffers can never get full > outside the pathological IGT test cases. So I am wondering if this is > in the category of fix for correctness or actually the CT buffers can > get full in normal use so it is imperative to fix. The CT buffers being full is indeed something that is normally only observed with IGTs that hammer the submission path, but it is still something that a user can do so IMO we do have to fix it. However, the bug is still extremely unlikely to happen out in the wild as it needs 2 relatively rare things to happen: - We need to hit the pathological case of the GuC CTs being full and the stall kicking in - Something needs to go wrong and escalated to a full GT reset The bug report that triggered my investigation into this came from what look like faulty HW: the HW seems to suddenly just stop with no errors anywhere, which leads to the buffers filling up because the GuC is no longer processing them, followed by a GT reset as we try to recover the HW. To replicate this locally I had to add a debugfs to kill the GuC in the middle of the test to simulate this "HW silently dies" scenario. Daniele > > Regards, > > Tvrtko > >> If a full GT reset occurs, the state of all contexts is cleared and all >> non-guilty requests are unsubmitted, therefore we need to restart the >> stalled request submission from scratch. To make sure that we do so, >> clear the saved request after a reset. >> >> Fixes note: the patch that introduced the bug is in 5.15, but no >> officially supported platform had GuC submission enabled by default >> in that kernel, so the backport to that particular version (and only >> that one) can potentially be skipped. >> >> Fixes: 925dc1cf58ed ("drm/i915/guc: Implement GuC submission tasklet") >> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> >> Cc: Matthew Brost <matthew.brost@intel.com> >> Cc: John Harrison <john.c.harrison@intel.com> >> Cc: <stable@vger.kernel.org> # v5.15+ >> --- >> drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >> b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >> index 0d17da77e787..0d56b615bf78 100644 >> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >> @@ -4002,6 +4002,13 @@ static inline void guc_init_lrc_mapping(struct >> intel_guc *guc) >> /* make sure all descriptors are clean... */ >> xa_destroy(&guc->context_lookup); >> + /* >> + * A reset might have occurred while we had a pending stalled >> request, >> + * so make sure we clean that up. >> + */ >> + guc->stalled_request = NULL; >> + guc->submission_stall_reason = STALL_NONE; >> + >> /* >> * Some contexts might have been pinned before we enabled GuC >> * submission, so we need to add them to the GuC bookeeping.
On 8/11/2022 14:08, Daniele Ceraolo Spurio wrote: > If the GuC CTs are full and we need to stall the request submission > while waiting for space, we save the stalled request and where the stall > occurred; when the CTs have space again we pick up the request submission > from where we left off. > > If a full GT reset occurs, the state of all contexts is cleared and all > non-guilty requests are unsubmitted, therefore we need to restart the > stalled request submission from scratch. To make sure that we do so, > clear the saved request after a reset. > > Fixes note: the patch that introduced the bug is in 5.15, but no > officially supported platform had GuC submission enabled by default > in that kernel, so the backport to that particular version (and only > that one) can potentially be skipped. > > Fixes: 925dc1cf58ed ("drm/i915/guc: Implement GuC submission tasklet") > Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> > Cc: Matthew Brost <matthew.brost@intel.com> > Cc: John Harrison <john.c.harrison@intel.com> > Cc: <stable@vger.kernel.org> # v5.15+ Seems like a good thing to do. Reviewed-by: John Harrison <John.C.Harrison@Intel.com> > --- > drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > index 0d17da77e787..0d56b615bf78 100644 > --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c > @@ -4002,6 +4002,13 @@ static inline void guc_init_lrc_mapping(struct intel_guc *guc) > /* make sure all descriptors are clean... */ > xa_destroy(&guc->context_lookup); > > + /* > + * A reset might have occurred while we had a pending stalled request, > + * so make sure we clean that up. > + */ > + guc->stalled_request = NULL; > + guc->submission_stall_reason = STALL_NONE; > + > /* > * Some contexts might have been pinned before we enabled GuC > * submission, so we need to add them to the GuC bookeeping.
On 12/08/2022 16:31, Ceraolo Spurio, Daniele wrote: > On 8/12/2022 12:29 AM, Tvrtko Ursulin wrote: >> >> On 11/08/2022 22:08, Daniele Ceraolo Spurio wrote: >>> If the GuC CTs are full and we need to stall the request submission >>> while waiting for space, we save the stalled request and where the stall >>> occurred; when the CTs have space again we pick up the request >>> submission >>> from where we left off. >> >> How serious is it? Statement always was CT buffers can never get full >> outside the pathological IGT test cases. So I am wondering if this is >> in the category of fix for correctness or actually the CT buffers can >> get full in normal use so it is imperative to fix. > > The CT buffers being full is indeed something that is normally only > observed with IGTs that hammer the submission path, but it is still > something that a user can do so IMO we do have to fix it. However, the > bug is still extremely unlikely to happen out in the wild as it needs 2 > relatively rare things to happen: > > - We need to hit the pathological case of the GuC CTs being full and the > stall kicking in > - Something needs to go wrong and escalated to a full GT reset > > The bug report that triggered my investigation into this came from what > look like faulty HW: the HW seems to suddenly just stop with no errors > anywhere, which leads to the buffers filling up because the GuC is no > longer processing them, followed by a GT reset as we try to recover the > HW. To replicate this locally I had to add a debugfs to kill the GuC in > the middle of the test to simulate this "HW silently dies" scenario. Ack. Given the activity around pin/unpin performance issues I was briefly worried this is another one which was "upgraded" from only IGT to something in the real world after all. Thanks for confirming that is not the case. Regards, Tvrtko > > Daniele > >> >> Regards, >> >> Tvrtko >> >>> If a full GT reset occurs, the state of all contexts is cleared and all >>> non-guilty requests are unsubmitted, therefore we need to restart the >>> stalled request submission from scratch. To make sure that we do so, >>> clear the saved request after a reset. >>> >>> Fixes note: the patch that introduced the bug is in 5.15, but no >>> officially supported platform had GuC submission enabled by default >>> in that kernel, so the backport to that particular version (and only >>> that one) can potentially be skipped. >>> >>> Fixes: 925dc1cf58ed ("drm/i915/guc: Implement GuC submission tasklet") >>> Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> >>> Cc: Matthew Brost <matthew.brost@intel.com> >>> Cc: John Harrison <john.c.harrison@intel.com> >>> Cc: <stable@vger.kernel.org> # v5.15+ >>> --- >>> drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >>> index 0d17da77e787..0d56b615bf78 100644 >>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c >>> @@ -4002,6 +4002,13 @@ static inline void guc_init_lrc_mapping(struct >>> intel_guc *guc) >>> /* make sure all descriptors are clean... */ >>> xa_destroy(&guc->context_lookup); >>> + /* >>> + * A reset might have occurred while we had a pending stalled >>> request, >>> + * so make sure we clean that up. >>> + */ >>> + guc->stalled_request = NULL; >>> + guc->submission_stall_reason = STALL_NONE; >>> + >>> /* >>> * Some contexts might have been pinned before we enabled GuC >>> * submission, so we need to add them to the GuC bookeeping. >
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index 0d17da77e787..0d56b615bf78 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -4002,6 +4002,13 @@ static inline void guc_init_lrc_mapping(struct intel_guc *guc) /* make sure all descriptors are clean... */ xa_destroy(&guc->context_lookup); + /* + * A reset might have occurred while we had a pending stalled request, + * so make sure we clean that up. + */ + guc->stalled_request = NULL; + guc->submission_stall_reason = STALL_NONE; + /* * Some contexts might have been pinned before we enabled GuC * submission, so we need to add them to the GuC bookeeping.
If the GuC CTs are full and we need to stall the request submission while waiting for space, we save the stalled request and where the stall occurred; when the CTs have space again we pick up the request submission from where we left off. If a full GT reset occurs, the state of all contexts is cleared and all non-guilty requests are unsubmitted, therefore we need to restart the stalled request submission from scratch. To make sure that we do so, clear the saved request after a reset. Fixes note: the patch that introduced the bug is in 5.15, but no officially supported platform had GuC submission enabled by default in that kernel, so the backport to that particular version (and only that one) can potentially be skipped. Fixes: 925dc1cf58ed ("drm/i915/guc: Implement GuC submission tasklet") Signed-off-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: John Harrison <john.c.harrison@intel.com> Cc: <stable@vger.kernel.org> # v5.15+ --- drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c | 7 +++++++ 1 file changed, 7 insertions(+)