| Message ID | 20230928132235.349258-1-andi.shyti@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RFC] drm/i915/gt: Force mcr lock takeover if hardware forgot to release it | expand |
On Thu, Sep 28, 2023 at 03:22:35PM +0200, Andi Shyti wrote: > While discussing with Nirmoy offline about this other way for > fixing lock contention, he was a bit sceptical about it. > > But why not? We know that if we fall into this case it's because > some hardware component has forgotten to release the lock within > 100ms. So that we have two possibilities, either bail out or > force the unlock. > > Forcing the unlock might not be respectful to the environment, > but, at the end, i915 should have the highest priority. > > Nirmoy's solution here[*] is to force the unlock during gt > resume, but what happens if meantime the hardware takes the lock > and doesn't release it? > > Open for opinions or profligate rejections :-) > > I'm also curious to see what CI has to say about. > > [*] https://patchwork.freedesktop.org/series/124397/ > As far as I can tell, this patch doesn't really do anything beneficial that I can see. We already unlock and proceed today if we hit a lock timeout: - intel_gt_mcr_lock - attempt to get lock - timeout, warn, add CI taint - perform MCR register access even if the lock failed - intel_gt_mcr_unlock - lock is released regardless of whether we obtained it successfully at the beginning, or whether someone else was still holding it With your patch, it looks like you're just adding an extra unlock/reacquire step before we move on which I don't think accomplishes anything. If someone else forgot to release the lock, then we're still protected from other agents, and we'll take care of releasing it ourselves once we're done. If the other agent actually is still using the lock and they're just going slower than we expected, then when they finally finish they're just going to blindly unlock; if we're in the middle of our critical section at that point, they'll release our lock the same way we released theirs. The main change here is that when we hit a timeout, your patch is giving other outside agents a chance to sneak in and re-grab the lock, further delaying our KMD acquisition. The real-world IFWI problems we saw, which Nirmoy's series is working around, is that some boot-time agent simply forgot to ever release the lock, leaving it locked "forever" so it makes sense to sanitize it initially. Load/resume is the only time when it's actually "safe" to reset/sanitize the lock like that. If we're getting MCR timeouts during regular driver operation (i.e., not during the beginning of driver load or resume), then it either means our timeout values are too quick (i.e., we're not giving external agents sufficient time to run their critical sections), or some piece of system firmware (e.g., pcode) has completely died in the middle of its critical section. In the former case, we probably need to adjust our timeout amount (and possibly work with those firmware teams to see if they can reduce the size of their critical sections). In the latter case, the system is going to be so badly broken that it doesn't really matter what we do; we're just not going to have a functioning system anymore at that point and it's not something the graphics driver has a way of recovering from. Matt > Signed-off-by: Andi Shyti <andi.shyti@linux.intel.com> > Cc: Nirmoy Das <nirmoy.das@intel.com> > Cc: Matt Roper <matthew.d.roper@intel.com> > --- > drivers/gpu/drm/i915/gt/intel_gt_mcr.c | 46 ++++++++++++++++---------- > 1 file changed, 28 insertions(+), 18 deletions(-) > > diff --git a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c > index bf4a933de03a..e3eb3c2ace68 100644 > --- a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c > +++ b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c > @@ -371,14 +371,34 @@ void intel_gt_mcr_lock(struct intel_gt *gt, unsigned long *flags) > > lockdep_assert_not_held(>->uncore->lock); > > - /* > - * Starting with MTL, we need to coordinate not only with other > - * driver threads, but also with hardware/firmware agents. A dedicated > - * locking register is used. > - */ > - if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) > - err = wait_for(intel_uncore_read_fw(gt->uncore, > - MTL_STEER_SEMAPHORE) == 0x1, 100); > + do { > + /* > + * Starting with MTL, we need to coordinate not only with other > + * driver threads, but also with hardware/firmware agents. A > + * dedicated locking register is used. > + */ > + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) > + err = wait_for(intel_uncore_read_fw(gt->uncore, > + MTL_STEER_SEMAPHORE) == 0x1, 100); > + else > + break; > + > + /* > + * In theory we should never fail to acquire the HW semaphore; > + * this would indicate some hardware/firmware is misbehaving and > + * not releasing it properly. > + */ > + if (err == -ETIMEDOUT) { > + gt_warn(gt, > + "hardware MCR steering semaphore timed out " > + "forcing lock takeover\n"); > + /* > + * Force lock takeover > + */ > + intel_uncore_write_fw(gt->uncore, > + MTL_STEER_SEMAPHORE, 0x1); > + } > + } while (err != -ETIMEDOUT); > > /* > * Even on platforms with a hardware lock, we'll continue to grab > @@ -389,16 +409,6 @@ void intel_gt_mcr_lock(struct intel_gt *gt, unsigned long *flags) > spin_lock_irqsave(>->mcr_lock, __flags); > > *flags = __flags; > - > - /* > - * In theory we should never fail to acquire the HW semaphore; this > - * would indicate some hardware/firmware is misbehaving and not > - * releasing it properly. > - */ > - if (err == -ETIMEDOUT) { > - gt_err_ratelimited(gt, "hardware MCR steering semaphore timed out"); > - add_taint_for_CI(gt->i915, TAINT_WARN); /* CI is now unreliable */ > - } > } > > /** > -- > 2.40.1 >
Hi Matt, > > While discussing with Nirmoy offline about this other way for > > fixing lock contention, he was a bit sceptical about it. > > > > But why not? We know that if we fall into this case it's because > > some hardware component has forgotten to release the lock within > > 100ms. So that we have two possibilities, either bail out or > > force the unlock. > > > > Forcing the unlock might not be respectful to the environment, > > but, at the end, i915 should have the highest priority. > > > > Nirmoy's solution here[*] is to force the unlock during gt > > resume, but what happens if meantime the hardware takes the lock > > and doesn't release it? > > > > Open for opinions or profligate rejections :-) > > > > I'm also curious to see what CI has to say about. > > > > [*] https://patchwork.freedesktop.org/series/124397/ > > > > As far as I can tell, this patch doesn't really do anything beneficial > that I can see. We already unlock and proceed today if we hit a lock > timeout: > > - intel_gt_mcr_lock > - attempt to get lock > - timeout, warn, add CI taint > - perform MCR register access even if the lock failed > - intel_gt_mcr_unlock > - lock is released regardless of whether we obtained it successfully > at the beginning, or whether someone else was still holding it > > With your patch, it looks like you're just adding an extra > unlock/reacquire step before we move on which I don't think accomplishes > anything. If someone else forgot to release the lock, then we're still > protected from other agents, and we'll take care of releasing it > ourselves once we're done. If the other agent actually is still using > the lock and they're just going slower than we expected, then when they > finally finish they're just going to blindly unlock; if we're in the > middle of our critical section at that point, they'll release our lock > the same way we released theirs. The main change here is that when we > hit a timeout, your patch is giving other outside agents a chance to sneak in > and re-grab the lock, further delaying our KMD acquisition. > > The real-world IFWI problems we saw, which Nirmoy's series is working > around, is that some boot-time agent simply forgot to ever release the > lock, leaving it locked "forever" so it makes sense to sanitize it > initially. Load/resume is the only time when it's actually "safe" to > reset/sanitize the lock like that. If we're getting MCR timeouts during > regular driver operation (i.e., not during the beginning of driver load > or resume), then it either means our timeout values are too quick (i.e., > we're not giving external agents sufficient time to run their critical > sections), or some piece of system firmware (e.g., pcode) has completely > died in the middle of its critical section. In the former case, we > probably need to adjust our timeout amount (and possibly work with those > firmware teams to see if they can reduce the size of their critical > sections). In the latter case, the system is going to be so badly > broken that it doesn't really matter what we do; we're just not going to > have a functioning system anymore at that point and it's not something > the graphics driver has a way of recovering from. Makes sense... thanks! Andi
diff --git a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c index bf4a933de03a..e3eb3c2ace68 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c @@ -371,14 +371,34 @@ void intel_gt_mcr_lock(struct intel_gt *gt, unsigned long *flags) lockdep_assert_not_held(>->uncore->lock); - /* - * Starting with MTL, we need to coordinate not only with other - * driver threads, but also with hardware/firmware agents. A dedicated - * locking register is used. - */ - if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) - err = wait_for(intel_uncore_read_fw(gt->uncore, - MTL_STEER_SEMAPHORE) == 0x1, 100); + do { + /* + * Starting with MTL, we need to coordinate not only with other + * driver threads, but also with hardware/firmware agents. A + * dedicated locking register is used. + */ + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) + err = wait_for(intel_uncore_read_fw(gt->uncore, + MTL_STEER_SEMAPHORE) == 0x1, 100); + else + break; + + /* + * In theory we should never fail to acquire the HW semaphore; + * this would indicate some hardware/firmware is misbehaving and + * not releasing it properly. + */ + if (err == -ETIMEDOUT) { + gt_warn(gt, + "hardware MCR steering semaphore timed out " + "forcing lock takeover\n"); + /* + * Force lock takeover + */ + intel_uncore_write_fw(gt->uncore, + MTL_STEER_SEMAPHORE, 0x1); + } + } while (err != -ETIMEDOUT); /* * Even on platforms with a hardware lock, we'll continue to grab @@ -389,16 +409,6 @@ void intel_gt_mcr_lock(struct intel_gt *gt, unsigned long *flags) spin_lock_irqsave(>->mcr_lock, __flags); *flags = __flags; - - /* - * In theory we should never fail to acquire the HW semaphore; this - * would indicate some hardware/firmware is misbehaving and not - * releasing it properly. - */ - if (err == -ETIMEDOUT) { - gt_err_ratelimited(gt, "hardware MCR steering semaphore timed out"); - add_taint_for_CI(gt->i915, TAINT_WARN); /* CI is now unreliable */ - } } /**
While discussing with Nirmoy offline about this other way for fixing lock contention, he was a bit sceptical about it. But why not? We know that if we fall into this case it's because some hardware component has forgotten to release the lock within 100ms. So that we have two possibilities, either bail out or force the unlock. Forcing the unlock might not be respectful to the environment, but, at the end, i915 should have the highest priority. Nirmoy's solution here[*] is to force the unlock during gt resume, but what happens if meantime the hardware takes the lock and doesn't release it? Open for opinions or profligate rejections :-) I'm also curious to see what CI has to say about. [*] https://patchwork.freedesktop.org/series/124397/ Signed-off-by: Andi Shyti <andi.shyti@linux.intel.com> Cc: Nirmoy Das <nirmoy.das@intel.com> Cc: Matt Roper <matthew.d.roper@intel.com> --- drivers/gpu/drm/i915/gt/intel_gt_mcr.c | 46 ++++++++++++++++---------- 1 file changed, 28 insertions(+), 18 deletions(-)