| Message ID | 303007688.lvsDLFNCpe@aspire.rjw.lan (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Rafael Wysocki |
| Headers | show |
On 3 January 2018 at 12:06, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > One of the limitations of pm_runtime_force_suspend/resume() is that > if a parent driver wants to use these functions, all of its child > drivers have to do that too because of the parent usage counter > manipulations necessary to get the correct state of the parent during > system-wide transitions to the working state (system resume). I understand your point, however this isn't describing the full story, because there are a some more alternatives when pm_runtime_force_suspend|resume() works well, when used for parent devices. Let me clarify, just to make sure we get this correct. 1) If the child device isn't managed by runtime PM at all (it always has runtime PM disabled), pm_runtime_force_suspend|resume() can be used for the parent device. This works because the runtime PM status if the child remains in the default status, RPM_SUSPENDED. 2) If, somehow, during system suspend/resume, the driver for the child make sure to synchronize the runtime PM status of the child device, according to the state of the HW, this should work too. I leave it to you how to take this fact into consideration, if and how to update the changelog. > However, that limitation turns out to be artificial, so remove it. Agree! > > Namely, pm_runtime_force_suspend() only needs to update the children > counter of its parent (if there's is a parent) when the device can > stay in suspend after the subsequent system resume transition, as > that counter is correct already otherwise. Now, if the parent's > children counter is not updated, it is not necessary to increment > the parent's usage counter in that case any more, as long as the > children counters of devices are checked along with their usage > counters in order to decide whether or not the devices may be left > in suspend after the subsequent system resume transition. > > Accordingly, modify pm_runtime_force_suspend() to only call > pm_runtime_set_suspended() for devices whose usage and children > counters are at the "no references" level (the runtime PM status > of the device needs to be updated to "suspended" anyway in case > this function is called once again for the same device during the > transition under way), drop the parent usage counter incrementation > from it and update pm_runtime_force_resume() to compensate for these > changes. > > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > --- > drivers/base/power/runtime.c | 74 +++++++++++++++++++------------------------ > 1 file changed, 34 insertions(+), 40 deletions(-) > > Index: linux-pm/drivers/base/power/runtime.c > =================================================================== > --- linux-pm.orig/drivers/base/power/runtime.c > +++ linux-pm/drivers/base/power/runtime.c > @@ -1613,17 +1613,28 @@ void pm_runtime_drop_link(struct device > spin_unlock_irq(&dev->power.lock); > } > > +static bool pm_runtime_need_not_resume(struct device *dev) > +{ > + return atomic_read(&dev->power.usage_count) <= 1 && > + atomic_read(&dev->power.child_count) == 0; We should take into account the ignore_children flag here, I think. Something like this: return atomic_read(&dev->power.usage_count) <= 1 && (atomic_read(&dev->power.child_count) == 0 || dev->power.ignore_children) > +} > + > /** > * pm_runtime_force_suspend - Force a device into suspend state if needed. > * @dev: Device to suspend. > * > * Disable runtime PM so we safely can check the device's runtime PM status and > - * if it is active, invoke it's .runtime_suspend callback to bring it into > - * suspend state. Keep runtime PM disabled to preserve the state unless we > - * encounter errors. > + * if it is active, invoke its ->runtime_suspend callback to suspend it and > + * change its runtime PM status field to RPM_SUSPENDED. Also, if the device's > + * usage and children counters don't indicate that the device was in use before > + * the system-wide transition under way, decrement its parent's children counter > + * (if there is a parent). Keep runtime PM disabled to preserve the state > + * unless we encounter errors. > * > * Typically this function may be invoked from a system suspend callback to make > - * sure the device is put into low power state. > + * sure the device is put into low power state and it should only be used during > + * system-wide PM transitions to sleep states. It assumes that the analogous > + * pm_runtime_force_resume() will be used to resume the device. > */ > int pm_runtime_force_suspend(struct device *dev) > { > @@ -1646,17 +1657,18 @@ int pm_runtime_force_suspend(struct devi > goto err; > > /* > - * Increase the runtime PM usage count for the device's parent, in case > - * when we find the device being used when system suspend was invoked. > - * This informs pm_runtime_force_resume() to resume the parent > - * immediately, which is needed to be able to resume its children, > - * when not deferring the resume to be managed via runtime PM. > + * If the device can stay in suspend after the system-wide transition > + * to the working state that will follow, drop the children counter of > + * its parent, but set its status to RPM_SUSPENDED anyway in case this > + * function will be called again for it in the meantime. > */ > - if (dev->parent && atomic_read(&dev->power.usage_count) > 1) > - pm_runtime_get_noresume(dev->parent); > + if (pm_runtime_need_not_resume(dev)) > + pm_runtime_set_suspended(dev); > + else > + __update_runtime_status(dev, RPM_SUSPENDED); > This is clever! I recall when working on commit 1d9174fbc55e ("PM / Runtime: Defer resuming of the device in pm_runtime_force_resume()"), that I wanted to take into account the dev->power.child_count, but couldn't figure out exactly how. :-) > - pm_runtime_set_suspended(dev); > return 0; > + > err: > pm_runtime_enable(dev); > return ret; > @@ -1669,13 +1681,9 @@ EXPORT_SYMBOL_GPL(pm_runtime_force_suspe > * > * Prior invoking this function we expect the user to have brought the device > * into low power state by a call to pm_runtime_force_suspend(). Here we reverse > - * those actions and brings the device into full power, if it is expected to be > - * used on system resume. To distinguish that, we check whether the runtime PM > - * usage count is greater than 1 (the PM core increases the usage count in the > - * system PM prepare phase), as that indicates a real user (such as a subsystem, > - * driver, userspace, etc.) is using it. If that is the case, the device is > - * expected to be used on system resume as well, so then we resume it. In the > - * other case, we defer the resume to be managed via runtime PM. > + * those actions and bring the device into full power, if it is expected to be > + * used on system resume. In the other case, we defer the resume to be managed > + * via runtime PM. > * > * Typically this function may be invoked from a system resume callback. > */ > @@ -1684,32 +1692,18 @@ int pm_runtime_force_resume(struct devic > int (*callback)(struct device *); > int ret = 0; > > - callback = RPM_GET_CALLBACK(dev, runtime_resume); > - > - if (!callback) { > - ret = -ENOSYS; > - goto out; > - } This make me realize that, I think this is probably for legacy reasons. I think we should allow the callback to be NULL, don't you think? Of course, this deserves its own change. > - > - if (!pm_runtime_status_suspended(dev)) > + if (!pm_runtime_status_suspended(dev) || pm_runtime_need_not_resume(dev)) > goto out; > > /* > - * Decrease the parent's runtime PM usage count, if we increased it > - * during system suspend in pm_runtime_force_suspend(). > - */ > - if (atomic_read(&dev->power.usage_count) > 1) { > - if (dev->parent) > - pm_runtime_put_noidle(dev->parent); > - } else { > - goto out; > - } > + * The value of the parent's children counter is correct already, so > + * just update the status of the device. > + */ > + __update_runtime_status(dev, RPM_ACTIVE); > > - ret = pm_runtime_set_active(dev); > - if (ret) > - goto out; > + callback = RPM_GET_CALLBACK(dev, runtime_resume); > > - ret = callback(dev); > + ret = callback ? callback(dev) : -ENOSYS; > if (ret) { > pm_runtime_set_suspended(dev); > goto out; > Thanks for looking into this! Kind regards Uffe
On Tuesday, January 9, 2018 7:08:21 AM CET Ulf Hansson wrote: > On 3 January 2018 at 12:06, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: > > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > > > One of the limitations of pm_runtime_force_suspend/resume() is that > > if a parent driver wants to use these functions, all of its child > > drivers have to do that too because of the parent usage counter > > manipulations necessary to get the correct state of the parent during > > system-wide transitions to the working state (system resume). > > I understand your point, however this isn't describing the full story, > because there are a some more alternatives when > pm_runtime_force_suspend|resume() works well, when used for parent > devices. Let me clarify, just to make sure we get this correct. > > 1) If the child device isn't managed by runtime PM at all (it always > has runtime PM disabled), pm_runtime_force_suspend|resume() can be > used for the parent device. This works because the runtime PM status > if the child remains in the default status, RPM_SUSPENDED. > > 2) If, somehow, during system suspend/resume, the driver for the child > make sure to synchronize the runtime PM status of the child device, > according to the state of the HW, this should work too. > > I leave it to you how to take this fact into consideration, if and how > to update the changelog. OK, let me see. > > However, that limitation turns out to be artificial, so remove it. > > Agree! > > > > > Namely, pm_runtime_force_suspend() only needs to update the children > > counter of its parent (if there's is a parent) when the device can > > stay in suspend after the subsequent system resume transition, as > > that counter is correct already otherwise. Now, if the parent's > > children counter is not updated, it is not necessary to increment > > the parent's usage counter in that case any more, as long as the > > children counters of devices are checked along with their usage > > counters in order to decide whether or not the devices may be left > > in suspend after the subsequent system resume transition. > > > > Accordingly, modify pm_runtime_force_suspend() to only call > > pm_runtime_set_suspended() for devices whose usage and children > > counters are at the "no references" level (the runtime PM status > > of the device needs to be updated to "suspended" anyway in case > > this function is called once again for the same device during the > > transition under way), drop the parent usage counter incrementation > > from it and update pm_runtime_force_resume() to compensate for these > > changes. > > > > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > --- > > drivers/base/power/runtime.c | 74 +++++++++++++++++++------------------------ > > 1 file changed, 34 insertions(+), 40 deletions(-) > > > > Index: linux-pm/drivers/base/power/runtime.c > > =================================================================== > > --- linux-pm.orig/drivers/base/power/runtime.c > > +++ linux-pm/drivers/base/power/runtime.c > > @@ -1613,17 +1613,28 @@ void pm_runtime_drop_link(struct device > > spin_unlock_irq(&dev->power.lock); > > } > > > > +static bool pm_runtime_need_not_resume(struct device *dev) > > +{ > > + return atomic_read(&dev->power.usage_count) <= 1 && > > + atomic_read(&dev->power.child_count) == 0; > > We should take into account the ignore_children flag here, I think. > Something like this: > > return atomic_read(&dev->power.usage_count) <= 1 && > (atomic_read(&dev->power.child_count) == 0 || > dev->power.ignore_children) The current code doesn't quite take ignore_children into account, however. Regardless of which way the change is made, there will be one corner case that is not going to be covered. First, if ignore_children is taken into account, some cases in which the current code increments the parent's usage counter will be treated as "need not resume". Second, if ignore_children is ignored, some cases in which the parent's usage counter is not incremented today will cause the parent to resume after the change. Frankly, I prefer to ignore ignore_children at least for the time being, because resuming the parent unnecessarily is not a tragedy (it will likely suspend shortly anyway), but if it is necessary to resume it and it is not be resumed, things will visibly break. So, I'd prefer to leave this patch as is and do a second one adding the ignore_children check. Then, if things break due to the second patch, it can be reverted easily (unless that can be fixed differently). > > +} > > + > > /** > > * pm_runtime_force_suspend - Force a device into suspend state if needed. > > * @dev: Device to suspend. > > * > > * Disable runtime PM so we safely can check the device's runtime PM status and > > - * if it is active, invoke it's .runtime_suspend callback to bring it into > > - * suspend state. Keep runtime PM disabled to preserve the state unless we > > - * encounter errors. > > + * if it is active, invoke its ->runtime_suspend callback to suspend it and > > + * change its runtime PM status field to RPM_SUSPENDED. Also, if the device's > > + * usage and children counters don't indicate that the device was in use before > > + * the system-wide transition under way, decrement its parent's children counter > > + * (if there is a parent). Keep runtime PM disabled to preserve the state > > + * unless we encounter errors. > > * > > * Typically this function may be invoked from a system suspend callback to make > > - * sure the device is put into low power state. > > + * sure the device is put into low power state and it should only be used during > > + * system-wide PM transitions to sleep states. It assumes that the analogous > > + * pm_runtime_force_resume() will be used to resume the device. > > */ > > int pm_runtime_force_suspend(struct device *dev) > > { > > @@ -1646,17 +1657,18 @@ int pm_runtime_force_suspend(struct devi > > goto err; > > > > /* > > - * Increase the runtime PM usage count for the device's parent, in case > > - * when we find the device being used when system suspend was invoked. > > - * This informs pm_runtime_force_resume() to resume the parent > > - * immediately, which is needed to be able to resume its children, > > - * when not deferring the resume to be managed via runtime PM. > > + * If the device can stay in suspend after the system-wide transition > > + * to the working state that will follow, drop the children counter of > > + * its parent, but set its status to RPM_SUSPENDED anyway in case this > > + * function will be called again for it in the meantime. > > */ > > - if (dev->parent && atomic_read(&dev->power.usage_count) > 1) > > - pm_runtime_get_noresume(dev->parent); > > + if (pm_runtime_need_not_resume(dev)) > > + pm_runtime_set_suspended(dev); > > + else > > + __update_runtime_status(dev, RPM_SUSPENDED); > > > > This is clever! > > I recall when working on commit 1d9174fbc55e ("PM / Runtime: Defer > resuming of the device in pm_runtime_force_resume()"), that I wanted > to take into account the dev->power.child_count, but couldn't figure > out exactly how. :-) > > > - pm_runtime_set_suspended(dev); > > return 0; > > + > > err: > > pm_runtime_enable(dev); > > return ret; > > @@ -1669,13 +1681,9 @@ EXPORT_SYMBOL_GPL(pm_runtime_force_suspe > > * > > * Prior invoking this function we expect the user to have brought the device > > * into low power state by a call to pm_runtime_force_suspend(). Here we reverse > > - * those actions and brings the device into full power, if it is expected to be > > - * used on system resume. To distinguish that, we check whether the runtime PM > > - * usage count is greater than 1 (the PM core increases the usage count in the > > - * system PM prepare phase), as that indicates a real user (such as a subsystem, > > - * driver, userspace, etc.) is using it. If that is the case, the device is > > - * expected to be used on system resume as well, so then we resume it. In the > > - * other case, we defer the resume to be managed via runtime PM. > > + * those actions and bring the device into full power, if it is expected to be > > + * used on system resume. In the other case, we defer the resume to be managed > > + * via runtime PM. > > * > > * Typically this function may be invoked from a system resume callback. > > */ > > @@ -1684,32 +1692,18 @@ int pm_runtime_force_resume(struct devic > > int (*callback)(struct device *); > > int ret = 0; > > > > - callback = RPM_GET_CALLBACK(dev, runtime_resume); > > - > > - if (!callback) { > > - ret = -ENOSYS; > > - goto out; > > - } > > This make me realize that, I think this is probably for legacy reasons. > > I think we should allow the callback to be NULL, don't you think? Well, it probably doesn't matter too much, but OK. > Of course, this deserves its own change. Right. > > - > > - if (!pm_runtime_status_suspended(dev)) > > + if (!pm_runtime_status_suspended(dev) || pm_runtime_need_not_resume(dev)) > > goto out; > > > > /* > > - * Decrease the parent's runtime PM usage count, if we increased it > > - * during system suspend in pm_runtime_force_suspend(). > > - */ > > - if (atomic_read(&dev->power.usage_count) > 1) { > > - if (dev->parent) > > - pm_runtime_put_noidle(dev->parent); > > - } else { > > - goto out; > > - } > > + * The value of the parent's children counter is correct already, so > > + * just update the status of the device. > > + */ > > + __update_runtime_status(dev, RPM_ACTIVE); > > > > - ret = pm_runtime_set_active(dev); > > - if (ret) > > - goto out; > > + callback = RPM_GET_CALLBACK(dev, runtime_resume); > > > > - ret = callback(dev); > > + ret = callback ? callback(dev) : -ENOSYS; > > if (ret) { > > pm_runtime_set_suspended(dev); > > goto out; > > > > Thanks for looking into this! No problem. Thanks, Rafael
Hi Rafael, On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > One of the limitations of pm_runtime_force_suspend/resume() is that > if a parent driver wants to use these functions, all of its child > drivers have to do that too because of the parent usage counter > manipulations necessary to get the correct state of the parent during > system-wide transitions to the working state (system resume). > However, that limitation turns out to be artificial, so remove it. > > Namely, pm_runtime_force_suspend() only needs to update the children > counter of its parent (if there's is a parent) when the device can > stay in suspend after the subsequent system resume transition, as > that counter is correct already otherwise. Now, if the parent's > children counter is not updated, it is not necessary to increment > the parent's usage counter in that case any more, as long as the > children counters of devices are checked along with their usage > counters in order to decide whether or not the devices may be left > in suspend after the subsequent system resume transition. > > Accordingly, modify pm_runtime_force_suspend() to only call > pm_runtime_set_suspended() for devices whose usage and children > counters are at the "no references" level (the runtime PM status > of the device needs to be updated to "suspended" anyway in case > this function is called once again for the same device during the > transition under way), drop the parent usage counter incrementation > from it and update pm_runtime_force_resume() to compensate for these > changes. > > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> This patch causes a regression during system resume on Renesas Salvator-XS with R-Car H3 ES2.0: SError Interrupt on CPU3, code 0xbf000002 -- SError SError Interrupt on CPU2, code 0xbf000002 -- SError CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT) Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT) Workqueue: events_unbound async_run_entry_fn Workqueue: events_unbound async_run_entry_fn pstate: 60000005 (nZCv daif -PAN -UAO) pstate: 60000005 (nZCv daif -PAN -UAO) pc : rcar_gen3_phy_usb2_init+0x34/0xf8 pc : rcar_gen3_phy_usb2_init+0x34/0xf8 lr : phy_init+0x64/0xcc lr : phy_init+0x64/0xcc ... Kernel panic - not syncing: Asynchronous SError Interrupt Note that before, it printed a warning instead: Enabling runtime PM for inactive device (ee0a0200.usb-phy) with active children WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 pm_runtime_enable+0x94/0xd8 Reverting commit 0408584d580d4a2c ("PM / runtime: Rework pm_runtime_force_suspend/resume()") fixes the crash. Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM deployment and fix an issue" (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead does not fix the crash. With more debug code added, it seems the EHCI module clocks (701-703) are enabled earlier than before. I guess this triggers the workqueue to perform an operation while another related device (HSUSB 704?) is still disabled? Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
[...] >> > +++ linux-pm/drivers/base/power/runtime.c >> > @@ -1613,17 +1613,28 @@ void pm_runtime_drop_link(struct device >> > spin_unlock_irq(&dev->power.lock); >> > } >> > >> > +static bool pm_runtime_need_not_resume(struct device *dev) >> > +{ >> > + return atomic_read(&dev->power.usage_count) <= 1 && >> > + atomic_read(&dev->power.child_count) == 0; >> >> We should take into account the ignore_children flag here, I think. >> Something like this: >> >> return atomic_read(&dev->power.usage_count) <= 1 && >> (atomic_read(&dev->power.child_count) == 0 || >> dev->power.ignore_children) > > The current code doesn't quite take ignore_children into account, however. > > Regardless of which way the change is made, there will be one corner case > that is not going to be covered. First, if ignore_children is taken into > account, some cases in which the current code increments the parent's > usage counter will be treated as "need not resume". Second, if ignore_children > is ignored, some cases in which the parent's usage counter is not incremented > today will cause the parent to resume after the change. > > Frankly, I prefer to ignore ignore_children at least for the time being, > because resuming the parent unnecessarily is not a tragedy (it will likely > suspend shortly anyway), but if it is necessary to resume it and it is not > be resumed, things will visibly break. > > So, I'd prefer to leave this patch as is and do a second one adding the > ignore_children check. Then, if things break due to the second patch, it > can be reverted easily (unless that can be fixed differently). Okay, that seems like a reasonable approach. Feel free to add: Reviewed-by: Ulf Hansson <ulf.hansson@linaro.org> [...] Kind regards Uffe
On 9 January 2018 at 14:37, Geert Uytterhoeven <geert@linux-m68k.org> wrote: > Hi Rafael, > > On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> One of the limitations of pm_runtime_force_suspend/resume() is that >> if a parent driver wants to use these functions, all of its child >> drivers have to do that too because of the parent usage counter >> manipulations necessary to get the correct state of the parent during >> system-wide transitions to the working state (system resume). >> However, that limitation turns out to be artificial, so remove it. >> >> Namely, pm_runtime_force_suspend() only needs to update the children >> counter of its parent (if there's is a parent) when the device can >> stay in suspend after the subsequent system resume transition, as >> that counter is correct already otherwise. Now, if the parent's >> children counter is not updated, it is not necessary to increment >> the parent's usage counter in that case any more, as long as the >> children counters of devices are checked along with their usage >> counters in order to decide whether or not the devices may be left >> in suspend after the subsequent system resume transition. >> >> Accordingly, modify pm_runtime_force_suspend() to only call >> pm_runtime_set_suspended() for devices whose usage and children >> counters are at the "no references" level (the runtime PM status >> of the device needs to be updated to "suspended" anyway in case >> this function is called once again for the same device during the >> transition under way), drop the parent usage counter incrementation >> from it and update pm_runtime_force_resume() to compensate for these >> changes. >> >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > This patch causes a regression during system resume on Renesas Salvator-XS > with R-Car H3 ES2.0: > > SError Interrupt on CPU3, code 0xbf000002 -- SError > SError Interrupt on CPU2, code 0xbf000002 -- SError > CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Workqueue: events_unbound async_run_entry_fn > Workqueue: events_unbound async_run_entry_fn > pstate: 60000005 (nZCv daif -PAN -UAO) > pstate: 60000005 (nZCv daif -PAN -UAO) > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > lr : phy_init+0x64/0xcc > lr : phy_init+0x64/0xcc > ... > Kernel panic - not syncing: Asynchronous SError Interrupt > > Note that before, it printed a warning instead: > > Enabling runtime PM for inactive device (ee0a0200.usb-phy) with > active children > WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 > pm_runtime_enable+0x94/0xd8 > > Reverting commit 0408584d580d4a2c ("PM / runtime: Rework > pm_runtime_force_suspend/resume()") fixes the crash. > > Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM > deployment and fix an issue" > (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead > does not fix the crash. What happens if you apply both the revert and the above series? Does the WARNING still disappear? Or have something else been changed? > > With more debug code added, it seems the EHCI module clocks (701-703) are > enabled earlier than before. I guess this triggers the workqueue to perform > an operation while another related device (HSUSB 704?) is still disabled? Huh, this seems broken in several aspects. Could this be the classic case of having the wrong suspend/resume order of devices? [...] Kind regards Uffe
Hi Ulf, On Tue, Jan 9, 2018 at 3:27 PM, Ulf Hansson <ulf.hansson@linaro.org> wrote: > On 9 January 2018 at 14:37, Geert Uytterhoeven <geert@linux-m68k.org> wrote: >> On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>> One of the limitations of pm_runtime_force_suspend/resume() is that >>> if a parent driver wants to use these functions, all of its child >>> drivers have to do that too because of the parent usage counter >>> manipulations necessary to get the correct state of the parent during >>> system-wide transitions to the working state (system resume). >>> However, that limitation turns out to be artificial, so remove it. >>> >>> Namely, pm_runtime_force_suspend() only needs to update the children >>> counter of its parent (if there's is a parent) when the device can >>> stay in suspend after the subsequent system resume transition, as >>> that counter is correct already otherwise. Now, if the parent's >>> children counter is not updated, it is not necessary to increment >>> the parent's usage counter in that case any more, as long as the >>> children counters of devices are checked along with their usage >>> counters in order to decide whether or not the devices may be left >>> in suspend after the subsequent system resume transition. >>> >>> Accordingly, modify pm_runtime_force_suspend() to only call >>> pm_runtime_set_suspended() for devices whose usage and children >>> counters are at the "no references" level (the runtime PM status >>> of the device needs to be updated to "suspended" anyway in case >>> this function is called once again for the same device during the >>> transition under way), drop the parent usage counter incrementation >>> from it and update pm_runtime_force_resume() to compensate for these >>> changes. >>> >>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> This patch causes a regression during system resume on Renesas Salvator-XS >> with R-Car H3 ES2.0: >> >> SError Interrupt on CPU3, code 0xbf000002 -- SError >> SError Interrupt on CPU2, code 0xbf000002 -- SError >> CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Workqueue: events_unbound async_run_entry_fn >> Workqueue: events_unbound async_run_entry_fn >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> lr : phy_init+0x64/0xcc >> lr : phy_init+0x64/0xcc >> ... >> Kernel panic - not syncing: Asynchronous SError Interrupt >> >> Note that before, it printed a warning instead: >> >> Enabling runtime PM for inactive device (ee0a0200.usb-phy) with >> active children >> WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 >> pm_runtime_enable+0x94/0xd8 >> >> Reverting commit 0408584d580d4a2c ("PM / runtime: Rework >> pm_runtime_force_suspend/resume()") fixes the crash. >> >> Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM >> deployment and fix an issue" >> (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead >> does not fix the crash. > > What happens if you apply both the revert and the above series? > > Does the WARNING still disappear? Or have something else been changed? The warning doesn't seem to go away in that case. Although Shimoda-san reported before it does. So something else has changed? >> With more debug code added, it seems the EHCI module clocks (701-703) are >> enabled earlier than before. I guess this triggers the workqueue to perform >> an operation while another related device (HSUSB 704?) is still disabled? > > Huh, this seems broken in several aspects. > > Could this be the classic case of having the wrong suspend/resume > order of devices? Possibly. /me no USB expert. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
On Tue, Jan 9, 2018 at 2:37 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: > Hi Rafael, > > On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> One of the limitations of pm_runtime_force_suspend/resume() is that >> if a parent driver wants to use these functions, all of its child >> drivers have to do that too because of the parent usage counter >> manipulations necessary to get the correct state of the parent during >> system-wide transitions to the working state (system resume). >> However, that limitation turns out to be artificial, so remove it. >> >> Namely, pm_runtime_force_suspend() only needs to update the children >> counter of its parent (if there's is a parent) when the device can >> stay in suspend after the subsequent system resume transition, as >> that counter is correct already otherwise. Now, if the parent's >> children counter is not updated, it is not necessary to increment >> the parent's usage counter in that case any more, as long as the >> children counters of devices are checked along with their usage >> counters in order to decide whether or not the devices may be left >> in suspend after the subsequent system resume transition. >> >> Accordingly, modify pm_runtime_force_suspend() to only call >> pm_runtime_set_suspended() for devices whose usage and children >> counters are at the "no references" level (the runtime PM status >> of the device needs to be updated to "suspended" anyway in case >> this function is called once again for the same device during the >> transition under way), drop the parent usage counter incrementation >> from it and update pm_runtime_force_resume() to compensate for these >> changes. >> >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > This patch causes a regression during system resume on Renesas Salvator-XS > with R-Car H3 ES2.0: I have dropped it for now, but we need to address the underlying issue. > SError Interrupt on CPU3, code 0xbf000002 -- SError > SError Interrupt on CPU2, code 0xbf000002 -- SError > CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Workqueue: events_unbound async_run_entry_fn > Workqueue: events_unbound async_run_entry_fn > pstate: 60000005 (nZCv daif -PAN -UAO) > pstate: 60000005 (nZCv daif -PAN -UAO) > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > lr : phy_init+0x64/0xcc > lr : phy_init+0x64/0xcc > ... > Kernel panic - not syncing: Asynchronous SError Interrupt > > Note that before, it printed a warning instead: > > Enabling runtime PM for inactive device (ee0a0200.usb-phy) with > active children > WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 > pm_runtime_enable+0x94/0xd8 > > Reverting commit 0408584d580d4a2c ("PM / runtime: Rework > pm_runtime_force_suspend/resume()") fixes the crash. > > Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM > deployment and fix an issue" > (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead > does not fix the crash. OK > With more debug code added, it seems the EHCI module clocks (701-703) are > enabled earlier than before. I guess this triggers the workqueue to perform > an operation while another related device (HSUSB 704?) is still disabled? Probably. Likely a device that wasn't resumed before resumes now and that causes the issue to appear. I'm wondering if adding the ignore_children check to the patch helps. If not, there clearly is a resume ordering issue that is papered over by the current code. Thanks, Rafael
Hi Rafael, CC usb On Tue, Jan 9, 2018 at 4:00 PM, Rafael J. Wysocki <rafael@kernel.org> wrote: > On Tue, Jan 9, 2018 at 2:37 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: >> On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>> >>> One of the limitations of pm_runtime_force_suspend/resume() is that >>> if a parent driver wants to use these functions, all of its child >>> drivers have to do that too because of the parent usage counter >>> manipulations necessary to get the correct state of the parent during >>> system-wide transitions to the working state (system resume). >>> However, that limitation turns out to be artificial, so remove it. >>> >>> Namely, pm_runtime_force_suspend() only needs to update the children >>> counter of its parent (if there's is a parent) when the device can >>> stay in suspend after the subsequent system resume transition, as >>> that counter is correct already otherwise. Now, if the parent's >>> children counter is not updated, it is not necessary to increment >>> the parent's usage counter in that case any more, as long as the >>> children counters of devices are checked along with their usage >>> counters in order to decide whether or not the devices may be left >>> in suspend after the subsequent system resume transition. >>> >>> Accordingly, modify pm_runtime_force_suspend() to only call >>> pm_runtime_set_suspended() for devices whose usage and children >>> counters are at the "no references" level (the runtime PM status >>> of the device needs to be updated to "suspended" anyway in case >>> this function is called once again for the same device during the >>> transition under way), drop the parent usage counter incrementation >>> from it and update pm_runtime_force_resume() to compensate for these >>> changes. >>> >>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> This patch causes a regression during system resume on Renesas Salvator-XS >> with R-Car H3 ES2.0: > > I have dropped it for now, but we need to address the underlying issue. > >> SError Interrupt on CPU3, code 0xbf000002 -- SError >> SError Interrupt on CPU2, code 0xbf000002 -- SError >> CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Workqueue: events_unbound async_run_entry_fn >> Workqueue: events_unbound async_run_entry_fn >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> lr : phy_init+0x64/0xcc >> lr : phy_init+0x64/0xcc >> ... >> Kernel panic - not syncing: Asynchronous SError Interrupt >> >> Note that before, it printed a warning instead: >> >> Enabling runtime PM for inactive device (ee0a0200.usb-phy) with >> active children >> WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 >> pm_runtime_enable+0x94/0xd8 >> >> Reverting commit 0408584d580d4a2c ("PM / runtime: Rework >> pm_runtime_force_suspend/resume()") fixes the crash. >> >> Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM >> deployment and fix an issue" >> (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead >> does not fix the crash. > > OK > >> With more debug code added, it seems the EHCI module clocks (701-703) are >> enabled earlier than before. I guess this triggers the workqueue to perform >> an operation while another related device (HSUSB 704?) is still disabled? > > Probably. > > Likely a device that wasn't resumed before resumes now and that causes > the issue to appear. > > I'm wondering if adding the ignore_children check to the patch helps. > If not, there clearly is a resume ordering issue that is papered over > by the current code. Something fishy is going on. Status of the USB PHYs differ before/after system suspend, according to /sys/kernel/debug/pm_genpd/pm_genpd_summary: - /devices/platform/soc/ee0a0200.usb-phy active - /devices/platform/soc/ee0c0200.usb-phy active - /devices/platform/soc/ee080200.usb-phy active + /devices/platform/soc/ee0a0200.usb-phy suspended + /devices/platform/soc/ee0c0200.usb-phy suspended + /devices/platform/soc/ee080200.usb-phy suspended Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
On Tue, Jan 9, 2018 at 4:30 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: > Hi Rafael, > > CC usb > > On Tue, Jan 9, 2018 at 4:00 PM, Rafael J. Wysocki <rafael@kernel.org> wrote: >> On Tue, Jan 9, 2018 at 2:37 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: >>> On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >>>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>>> >>>> One of the limitations of pm_runtime_force_suspend/resume() is that >>>> if a parent driver wants to use these functions, all of its child >>>> drivers have to do that too because of the parent usage counter >>>> manipulations necessary to get the correct state of the parent during >>>> system-wide transitions to the working state (system resume). >>>> However, that limitation turns out to be artificial, so remove it. >>>> >>>> Namely, pm_runtime_force_suspend() only needs to update the children >>>> counter of its parent (if there's is a parent) when the device can >>>> stay in suspend after the subsequent system resume transition, as >>>> that counter is correct already otherwise. Now, if the parent's >>>> children counter is not updated, it is not necessary to increment >>>> the parent's usage counter in that case any more, as long as the >>>> children counters of devices are checked along with their usage >>>> counters in order to decide whether or not the devices may be left >>>> in suspend after the subsequent system resume transition. >>>> >>>> Accordingly, modify pm_runtime_force_suspend() to only call >>>> pm_runtime_set_suspended() for devices whose usage and children >>>> counters are at the "no references" level (the runtime PM status >>>> of the device needs to be updated to "suspended" anyway in case >>>> this function is called once again for the same device during the >>>> transition under way), drop the parent usage counter incrementation >>>> from it and update pm_runtime_force_resume() to compensate for these >>>> changes. >>>> >>>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>> >>> This patch causes a regression during system resume on Renesas Salvator-XS >>> with R-Car H3 ES2.0: >> >> I have dropped it for now, but we need to address the underlying issue. >> >>> SError Interrupt on CPU3, code 0xbf000002 -- SError >>> SError Interrupt on CPU2, code 0xbf000002 -- SError >>> CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted >>> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >>> CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted >>> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >>> Hardware name: Renesas Salvator-X 2nd version board based on >>> r8a7795 ES2.0+ (DT) >>> Hardware name: Renesas Salvator-X 2nd version board based on >>> r8a7795 ES2.0+ (DT) >>> Workqueue: events_unbound async_run_entry_fn >>> Workqueue: events_unbound async_run_entry_fn >>> pstate: 60000005 (nZCv daif -PAN -UAO) >>> pstate: 60000005 (nZCv daif -PAN -UAO) >>> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >>> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >>> lr : phy_init+0x64/0xcc >>> lr : phy_init+0x64/0xcc >>> ... >>> Kernel panic - not syncing: Asynchronous SError Interrupt >>> >>> Note that before, it printed a warning instead: >>> >>> Enabling runtime PM for inactive device (ee0a0200.usb-phy) with >>> active children >>> WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 >>> pm_runtime_enable+0x94/0xd8 >>> >>> Reverting commit 0408584d580d4a2c ("PM / runtime: Rework >>> pm_runtime_force_suspend/resume()") fixes the crash. >>> >>> Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM >>> deployment and fix an issue" >>> (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead >>> does not fix the crash. >> >> OK >> >>> With more debug code added, it seems the EHCI module clocks (701-703) are >>> enabled earlier than before. I guess this triggers the workqueue to perform >>> an operation while another related device (HSUSB 704?) is still disabled? >> >> Probably. >> >> Likely a device that wasn't resumed before resumes now and that causes >> the issue to appear. >> >> I'm wondering if adding the ignore_children check to the patch helps. >> If not, there clearly is a resume ordering issue that is papered over >> by the current code. > > Something fishy is going on. Status of the USB PHYs differ before/after > system suspend, according to /sys/kernel/debug/pm_genpd/pm_genpd_summary: > > - /devices/platform/soc/ee0a0200.usb-phy active > - /devices/platform/soc/ee0c0200.usb-phy active > - /devices/platform/soc/ee080200.usb-phy active > + /devices/platform/soc/ee0a0200.usb-phy suspended > + /devices/platform/soc/ee0c0200.usb-phy suspended > + /devices/platform/soc/ee080200.usb-phy suspended Yeah. That's because of the pm_runtime_force_suspend/resume() called by genpd. These functions generally may cause devices active before system suspend to be left in suspend after it. That generally is a good idea if the device was not really in use before the system suspend, but there is a problem that the driver of it may not be prepared for that to happen (and also the way to determine the "not really in use" case may not be as expected by the driver). Thanks, Rafael
On Tue, Jan 9, 2018 at 2:37 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: > Hi Rafael, > > On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> One of the limitations of pm_runtime_force_suspend/resume() is that >> if a parent driver wants to use these functions, all of its child >> drivers have to do that too because of the parent usage counter >> manipulations necessary to get the correct state of the parent during >> system-wide transitions to the working state (system resume). >> However, that limitation turns out to be artificial, so remove it. >> >> Namely, pm_runtime_force_suspend() only needs to update the children >> counter of its parent (if there's is a parent) when the device can >> stay in suspend after the subsequent system resume transition, as >> that counter is correct already otherwise. Now, if the parent's >> children counter is not updated, it is not necessary to increment >> the parent's usage counter in that case any more, as long as the >> children counters of devices are checked along with their usage >> counters in order to decide whether or not the devices may be left >> in suspend after the subsequent system resume transition. >> >> Accordingly, modify pm_runtime_force_suspend() to only call >> pm_runtime_set_suspended() for devices whose usage and children >> counters are at the "no references" level (the runtime PM status >> of the device needs to be updated to "suspended" anyway in case >> this function is called once again for the same device during the >> transition under way), drop the parent usage counter incrementation >> from it and update pm_runtime_force_resume() to compensate for these >> changes. >> >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > This patch causes a regression during system resume on Renesas Salvator-XS > with R-Car H3 ES2.0: > > SError Interrupt on CPU3, code 0xbf000002 -- SError > SError Interrupt on CPU2, code 0xbf000002 -- SError > CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted > 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Workqueue: events_unbound async_run_entry_fn > Workqueue: events_unbound async_run_entry_fn > pstate: 60000005 (nZCv daif -PAN -UAO) > pstate: 60000005 (nZCv daif -PAN -UAO) > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > pc : rcar_gen3_phy_usb2_init+0x34/0xf8 > lr : phy_init+0x64/0xcc > lr : phy_init+0x64/0xcc > ... > Kernel panic - not syncing: Asynchronous SError Interrupt > > Note that before, it printed a warning instead: > > Enabling runtime PM for inactive device (ee0a0200.usb-phy) with > active children > WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 > pm_runtime_enable+0x94/0xd8 > > Reverting commit 0408584d580d4a2c ("PM / runtime: Rework > pm_runtime_force_suspend/resume()") fixes the crash. It looks like what happens without the Ulf's patch is as follows. usb-phy has children with runtime PM enabled that are not in the domain, so without the $subject patch the pm_runtime_force_suspend() in genpd_finish_suspend() checks the usage counter of usb-phy and since it is 1, the parent's usage counter is not incremented and genpd_runtime_suspend() is run for usb-phy. On resume, the pm_runtime_force_resume() in genpd_resume_noirq() finds that the usage counter of usb-phy is 1, so the parent's usage counter is not decremented (correctly) and the function returns (arguably incorrectly if the runtime PM status of the children is "active", because it is necessary to resume the children in that case, but the children have no PM callbacks and even if they had had them, they would have been run later anyway). The parent of usb-phy is skipped by the pm_runtime_force_resume() too, because its usage counter is 1 when it is checked by this function. With the $subject patch pm_runtime_force_suspend() in genpd_finish_suspend() calls pm_runtime_need_not_resume() and that returns "false" for usb-phy if the runtime PM status or at least one of its children is "active" (which it is for the "phy" devices). That is correct, but for this reason the parent's children counter is not decremented and both usb-phy and its parent will be resumed by the subsequent pm_runtime_force_resume(). The pm_runtime_force_resume() in genpd_resume(_noirq) now finds that pm_runtime_need_not_resume() returns "false" for both usb-phy and its parent and attempts to resume them both via genpd_runtime_resume() which is too early, because stuff they depend on hasn't been resumed yet. That triggers the crash (so https://patchwork.kernel.org/patch/10152767/ will cause the crash to happen too). In conclusion, without the $subject patch it all works pretty much by accident, basically because the pm_runtime_force_resume() inadvertently decides to skip the resume of some devices which avoids the premature execution of genpd_runtime_resume() for them. > Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM > deployment and fix an issue" > (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead > does not fix the crash. I'm not sure why the crash is still there in this case, but quite likely it is related (again) to something with a child outside of the domain that causes its parent to be treated as a device that must be resumed and that triggers a premature execution of genpd_runtime_resume() in genpd_resume_noirq() for the parent (because something the parents depends on quite likely is outside of the domain and will be resumed later). Granted, the (indirect) invocation of genpd_runtime_suspend() in genpd_finish_suspend() is too late for drivers that are not expecting it to be called then (but also harmless if the drivers check the status of the devices) and the (indirect) invocation of genpd_runtime_resume() from genpd_resume_noirq() is too early, especially if there are dependencies on devices outside the domains handled by genpd. So something like https://patchwork.kernel.org/patch/10152873/ is needed IMO. Thanks, Rafael
On Tue, Jan 9, 2018 at 7:46 PM, Rafael J. Wysocki <rafael@kernel.org> wrote: > On Tue, Jan 9, 2018 at 2:37 PM, Geert Uytterhoeven <geert@linux-m68k.org> wrote: >> Hi Rafael, >> >> On Wed, Jan 3, 2018 at 12:06 PM, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>> [cut] >>> >>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> This patch causes a regression during system resume on Renesas Salvator-XS >> with R-Car H3 ES2.0: >> >> SError Interrupt on CPU3, code 0xbf000002 -- SError >> SError Interrupt on CPU2, code 0xbf000002 -- SError >> CPU: 3 PID: 1769 Comm: kworker/u16:13 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> CPU: 2 PID: 1774 Comm: kworker/u16:18 Not tainted >> 4.15.0-rc7-arm64-renesas-05338-gf14cf570a813c9ca-dirty #97 >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Hardware name: Renesas Salvator-X 2nd version board based on >> r8a7795 ES2.0+ (DT) >> Workqueue: events_unbound async_run_entry_fn >> Workqueue: events_unbound async_run_entry_fn >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pstate: 60000005 (nZCv daif -PAN -UAO) >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> pc : rcar_gen3_phy_usb2_init+0x34/0xf8 >> lr : phy_init+0x64/0xcc >> lr : phy_init+0x64/0xcc >> ... >> Kernel panic - not syncing: Asynchronous SError Interrupt >> >> Note that before, it printed a warning instead: >> >> Enabling runtime PM for inactive device (ee0a0200.usb-phy) with >> active children >> WARNING: CPU: 0 PID: 1741 at drivers/base/power/runtime.c:1300 >> pm_runtime_enable+0x94/0xd8 >> >> Reverting commit 0408584d580d4a2c ("PM / runtime: Rework >> pm_runtime_force_suspend/resume()") fixes the crash. > > It looks like what happens without the Ulf's patch is as follows. > > usb-phy has children with runtime PM enabled that are not in the > domain, so without the $subject patch the pm_runtime_force_suspend() > in genpd_finish_suspend() checks the usage counter of usb-phy and > since it is 1, the parent's usage counter is not incremented and > genpd_runtime_suspend() is run for usb-phy. On resume, the > pm_runtime_force_resume() in genpd_resume_noirq() finds that the usage > counter of usb-phy is 1, so the parent's usage counter is not > decremented (correctly) and the function returns (arguably incorrectly > if the runtime PM status of the children is "active", because it is > necessary to resume the children in that case, but the children have > no PM callbacks and even if they had had them, they would have been > run later anyway). The parent of usb-phy is skipped by the > pm_runtime_force_resume() too, because its usage counter is 1 when it > is checked by this function. And note that it shouldn't be skipped by pm_runtime_force_resume() in principle, because there are active children under usb-phy. > With the $subject patch pm_runtime_force_suspend() in > genpd_finish_suspend() calls pm_runtime_need_not_resume() and that > returns "false" for usb-phy if the runtime PM status or at least one > of its children is "active" (which it is for the "phy" devices). That > is correct, but for this reason the parent's children counter is not > decremented and both usb-phy and its parent will be resumed by the > subsequent pm_runtime_force_resume(). > > The pm_runtime_force_resume() in genpd_resume(_noirq) now finds that > pm_runtime_need_not_resume() returns "false" for both usb-phy and its > parent and attempts to resume them both via genpd_runtime_resume() > which is too early, because stuff they depend on hasn't been resumed > yet. That triggers the crash (so > https://patchwork.kernel.org/patch/10152767/ will cause the crash to > happen too). > > In conclusion, without the $subject patch it all works pretty much by > accident, basically because the pm_runtime_force_resume() > inadvertently decides to skip the resume of some devices which avoids > the premature execution of genpd_runtime_resume() for them. > >> Note that applying Ulf's "[PATCH v2 0/3] phy: core: Re-work runtime PM >> deployment and fix an issue" >> (https://www.spinics.net/lists/linux-renesas-soc/msg21719.html) instead >> does not fix the crash. > > I'm not sure why the crash is still there in this case, It is there, because usb-phy itself is now reference-counted by the phy layer, so its usage counter is greater than 1 in genpd_finish_suspend() and its parent's children counter is not decremented then. Next, on resume, the pm_runtime_force_resume() in genpd_resume_noirq() will attempt to resume both usb-phy and its parent via genpd_runtime_suspend() and that (again) is too early. I'm not sure, however, why the crash isn't there with https://www.spinics.net/lists/linux-renesas-soc/msg21719.html alone, because in theory it should be there too in that case. Thanks, Rafael
On Tue, Jan 9, 2018 at 7:08 AM, Ulf Hansson <ulf.hansson@linaro.org> wrote: > On 3 January 2018 at 12:06, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> >> One of the limitations of pm_runtime_force_suspend/resume() is that >> if a parent driver wants to use these functions, all of its child >> drivers have to do that too because of the parent usage counter >> manipulations necessary to get the correct state of the parent during >> system-wide transitions to the working state (system resume). > > I understand your point, however this isn't describing the full story, > because there are a some more alternatives when > pm_runtime_force_suspend|resume() works well, when used for parent > devices. Let me clarify, just to make sure we get this correct. > > 1) If the child device isn't managed by runtime PM at all (it always > has runtime PM disabled), pm_runtime_force_suspend|resume() can be > used for the parent device. This works because the runtime PM status > if the child remains in the default status, RPM_SUSPENDED. Say the child is resumed every time during system-wide resume (which is the only possibility for drivers that don't support runtime PM) and it needs the parent to be accessible for that. If the parent uses pm_runtime_force_suspend|resume(), the child needs to bump up the usage counter of the parent during system-wide suspend to prevent pm_runtime_force_resume() from leaving the parent in suspend during the subsequent system-wide resume and that obviously is not going to happen if the child driver doesn't support runtime PM. Note that the $subject patch doesn't really change anything in that respect, because the children counter of the parent will be zero (or at least will not cover the child in question) then. Of course, this means that it is only safe to use pm_runtime_force_suspend|resume() in parent drivers if all of their child drivers support runtime PM. > 2) If, somehow, during system suspend/resume, the driver for the child > make sure to synchronize the runtime PM status of the child device, > according to the state of the HW, this should work too. Not really. Again, problematic is the case when the child is resumed during system-wide resume and needs the parent to be accessible for that. Note that the runtime PM status of the child doesn't matter for pm_runtime_force_resume() running for the parent. What matters in there is the parent's usage counter alone. If that usage counter is not greater than 1, pm_runtime_force_resume() will leave the parent in suspend and note that it must run before the child's system-wide resume callbacks for the ordering between the parent resume and the child resume to be correct. This means that the parent will still be suspended when the child's system-wide resume callbacks run and that is a problem if the parent has to be accessible for them to succeed. Thus the runtime PM status of the child has no bearing on whether or not this is going to work. The child has to bump up the parent's usage counter during system-wide suspend to ensure that it will not be left in suspend by the subsequent pm_runtime_force_resume() and that practically requires the child driver to also use pm_runtime_force_suspend|resume(). Now, the $subject patch changes the situation here for child drivers that only resume their devices during system-wide resume if they were "active" before the preceding system-wide suspend, because in that case the children counter of the parent will be nonzero. If the child resumes during system-wide resume even though it was suspended before the preceding system-wide suspend, pm_runtime_force_resume() running for its parent won't realize that the parent needs to resume too even after this patch. I generally think that leaving devices in suspend during system-wide resume is a very aggressive optimization and it should be done with extra care. It generally is not sufficient to rely on information coming from the runtime PM framework to do that safely in all cases. IMO it should only be done for drivers that explicitly expect it to be done and only if all of the drivers depending on them also expect it to be done. Otherwise it always will be potentially unsafe. Thanks, Rafael
On 12 January 2018 at 02:55, Rafael J. Wysocki <rafael@kernel.org> wrote: > On Tue, Jan 9, 2018 at 7:08 AM, Ulf Hansson <ulf.hansson@linaro.org> wrote: >> On 3 January 2018 at 12:06, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: >>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>> >>> One of the limitations of pm_runtime_force_suspend/resume() is that >>> if a parent driver wants to use these functions, all of its child >>> drivers have to do that too because of the parent usage counter >>> manipulations necessary to get the correct state of the parent during >>> system-wide transitions to the working state (system resume). >> >> I understand your point, however this isn't describing the full story, >> because there are a some more alternatives when >> pm_runtime_force_suspend|resume() works well, when used for parent >> devices. Let me clarify, just to make sure we get this correct. >> >> 1) If the child device isn't managed by runtime PM at all (it always >> has runtime PM disabled), pm_runtime_force_suspend|resume() can be >> used for the parent device. This works because the runtime PM status >> if the child remains in the default status, RPM_SUSPENDED. > > Say the child is resumed every time during system-wide resume (which > is the only possibility for drivers that don't support runtime PM) and > it needs the parent to be accessible for that. > > If the parent uses pm_runtime_force_suspend|resume(), the child needs > to bump up the usage counter of the parent during system-wide suspend > to prevent pm_runtime_force_resume() from leaving the parent in > suspend during the subsequent system-wide resume and that obviously is > not going to happen if the child driver doesn't support runtime PM. My take on this, is that this isn't a system suspend/resume specific issue, thus also not something that pm_runtime_force_suspend|resume() should need need to care about. The reason is why I think so is because, normally during runtime, the runtime PM core don't prevent parents from being runtime suspended, in case their children isn't managed by runtime PM. In other words, the problem exists already and isn't specific to system suspend. > > Note that the $subject patch doesn't really change anything in that > respect, because the children counter of the parent will be zero (or > at least will not cover the child in question) then. > > Of course, this means that it is only safe to use > pm_runtime_force_suspend|resume() in parent drivers if all of their > child drivers support runtime PM. I understand your concern, however, hopefully the about comment can address it!? > >> 2) If, somehow, during system suspend/resume, the driver for the child >> make sure to synchronize the runtime PM status of the child device, >> according to the state of the HW, this should work too. > > Not really. > > Again, problematic is the case when the child is resumed during > system-wide resume and needs the parent to be accessible for that. > > Note that the runtime PM status of the child doesn't matter for > pm_runtime_force_resume() running for the parent. What matters in > there is the parent's usage counter alone. If that usage counter is > not greater than 1, pm_runtime_force_resume() will leave the parent in > suspend and note that it must run before the child's system-wide > resume callbacks for the ordering between the parent resume and the > child resume to be correct. This means that the parent will still be > suspended when the child's system-wide resume callbacks run and that > is a problem if the parent has to be accessible for them to succeed. > > Thus the runtime PM status of the child has no bearing on whether or > not this is going to work. The child has to bump up the parent's > usage counter during system-wide suspend to ensure that it will not be > left in suspend by the subsequent pm_runtime_force_resume() and that > practically requires the child driver to also use > pm_runtime_force_suspend|resume(). > Yeah, you are right! I was thinking about the special case when the child driver has called pm_runtime_get_sync() during ->probe(), then during system suspend it puts the child device into low power state and updates the runtime PM status of the child device, by using pm_runtime_set_suspended(). In system resume the child driver then reverse those actions. Anyway, let's forget about these cases, as they are rather specific. Instead I think it's fair to say that currently: "If the child device is managed by runtime PM, and the parent driver wants to use pm_runtime_force_suspend|resume() for the parent device, so must the child driver for the child device" - and subject patch removes that artificial limitation. > Now, the $subject patch changes the situation here for child drivers > that only resume their devices during system-wide resume if they were > "active" before the preceding system-wide suspend, because in that > case the children counter of the parent will be nonzero. If the child > resumes during system-wide resume even though it was suspended before > the preceding system-wide suspend, pm_runtime_force_resume() running > for its parent won't realize that the parent needs to resume too even > after this patch. > > I generally think that leaving devices in suspend during system-wide > resume is a very aggressive optimization and it should be done with > extra care. It generally is not sufficient to rely on information > coming from the runtime PM framework to do that safely in all cases. > IMO it should only be done for drivers that explicitly expect it to be > done and only if all of the drivers depending on them also expect it > to be done. Otherwise it always will be potentially unsafe. That's true, but I think your concern is related to the case when the child isn't managed by runtime PM, while the parent is? In regards to $subject patch, I really think it replaces the crappy magic I invented to try to deal with child devices, with something correct and clever. I would really appreciate it getting applied. Kind regards Uffe
On Friday, January 12, 2018 2:46:12 PM CET Ulf Hansson wrote: > On 12 January 2018 at 02:55, Rafael J. Wysocki <rafael@kernel.org> wrote: > > On Tue, Jan 9, 2018 at 7:08 AM, Ulf Hansson <ulf.hansson@linaro.org> wrote: > >> On 3 January 2018 at 12:06, Rafael J. Wysocki <rjw@rjwysocki.net> wrote: > >>> From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > >>> > >>> One of the limitations of pm_runtime_force_suspend/resume() is that > >>> if a parent driver wants to use these functions, all of its child > >>> drivers have to do that too because of the parent usage counter > >>> manipulations necessary to get the correct state of the parent during > >>> system-wide transitions to the working state (system resume). > >> > >> I understand your point, however this isn't describing the full story, > >> because there are a some more alternatives when > >> pm_runtime_force_suspend|resume() works well, when used for parent > >> devices. Let me clarify, just to make sure we get this correct. > >> > >> 1) If the child device isn't managed by runtime PM at all (it always > >> has runtime PM disabled), pm_runtime_force_suspend|resume() can be > >> used for the parent device. This works because the runtime PM status > >> if the child remains in the default status, RPM_SUSPENDED. > > > > Say the child is resumed every time during system-wide resume (which > > is the only possibility for drivers that don't support runtime PM) and > > it needs the parent to be accessible for that. > > > > If the parent uses pm_runtime_force_suspend|resume(), the child needs > > to bump up the usage counter of the parent during system-wide suspend > > to prevent pm_runtime_force_resume() from leaving the parent in > > suspend during the subsequent system-wide resume and that obviously is > > not going to happen if the child driver doesn't support runtime PM. > > My take on this, is that this isn't a system suspend/resume specific > issue, thus also not something that pm_runtime_force_suspend|resume() > should need need to care about. > > The reason is why I think so is because, normally during runtime, the > runtime PM core don't prevent parents from being runtime suspended, in > case their children isn't managed by runtime PM. In other words, the > problem exists already and isn't specific to system suspend. Well, you said above that "If the child device isn't managed by runtime PM at all (it always has runtime PM disabled), pm_runtime_force_suspend|resume() can be used for the parent device", but that's not the case according to my analysis and I was responding specifically to that statement. For pm_runtime_force_suspend|resume() to be used safely in a parent driver, all of its child drivers must in fact support runtime PM as stated below (and, of course, that's related to the fact that runtime PM can only be safely supported by the parent driver, in general, if all of its child drivers support it). > > > > Note that the $subject patch doesn't really change anything in that > > respect, because the children counter of the parent will be zero (or > > at least will not cover the child in question) then. > > > > Of course, this means that it is only safe to use > > pm_runtime_force_suspend|resume() in parent drivers if all of their > > child drivers support runtime PM. > > I understand your concern, however, hopefully the about comment can address it!? This is an observation, not a concern. It is sort of obvious that the driver using pm_runtime_force_suspend|resume() must support runtime PM and for that to really work all of its child drivers must support runtime PM too (in general). I just don't know why you were arguing otherwise to start with. :-) > > > >> 2) If, somehow, during system suspend/resume, the driver for the child > >> make sure to synchronize the runtime PM status of the child device, > >> according to the state of the HW, this should work too. > > > > Not really. > > > > Again, problematic is the case when the child is resumed during > > system-wide resume and needs the parent to be accessible for that. > > > > Note that the runtime PM status of the child doesn't matter for > > pm_runtime_force_resume() running for the parent. What matters in > > there is the parent's usage counter alone. If that usage counter is > > not greater than 1, pm_runtime_force_resume() will leave the parent in > > suspend and note that it must run before the child's system-wide > > resume callbacks for the ordering between the parent resume and the > > child resume to be correct. This means that the parent will still be > > suspended when the child's system-wide resume callbacks run and that > > is a problem if the parent has to be accessible for them to succeed. > > > > Thus the runtime PM status of the child has no bearing on whether or > > not this is going to work. The child has to bump up the parent's > > usage counter during system-wide suspend to ensure that it will not be > > left in suspend by the subsequent pm_runtime_force_resume() and that > > practically requires the child driver to also use > > pm_runtime_force_suspend|resume(). > > > > Yeah, you are right! > > I was thinking about the special case when the child driver has called > pm_runtime_get_sync() during ->probe(), then during system suspend it > puts the child device into low power state and updates the runtime PM > status of the child device, by using pm_runtime_set_suspended(). In > system resume the child driver then reverse those actions. > > Anyway, let's forget about these cases, as they are rather specific. > Instead I think it's fair to say that currently: > > "If the child device is managed by runtime PM, and the parent driver > wants to use pm_runtime_force_suspend|resume() for the parent device, > so must the child driver for the child device" - and subject patch > removes that artificial limitation. This isn't entirely accurate, because all of that implicitly assumes runtime PM support in all of the involved drivers anyway. > > Now, the $subject patch changes the situation here for child drivers > > that only resume their devices during system-wide resume if they were > > "active" before the preceding system-wide suspend, because in that > > case the children counter of the parent will be nonzero. If the child > > resumes during system-wide resume even though it was suspended before > > the preceding system-wide suspend, pm_runtime_force_resume() running > > for its parent won't realize that the parent needs to resume too even > > after this patch. > > > > I generally think that leaving devices in suspend during system-wide > > resume is a very aggressive optimization and it should be done with > > extra care. It generally is not sufficient to rely on information > > coming from the runtime PM framework to do that safely in all cases. > > IMO it should only be done for drivers that explicitly expect it to be > > done and only if all of the drivers depending on them also expect it > > to be done. Otherwise it always will be potentially unsafe. > > That's true, but I think your concern is related to the case when the > child isn't managed by runtime PM, while the parent is? No, that case is not interesting at all (again, runtime PM must be supported by all of the involved drivers for this to work at all). Anyway, my point is that optimizations like this one only work under specific assumptions, but here there's nothing to ensure that these assumptions hold. It is sort of like a loaded gun with no safety. > In regards to $subject patch, I really think it replaces the crappy > magic I invented to try to deal with child devices, with something > correct and clever. I would really appreciate it getting applied. Sure, but that requires genpd to be fixed too. Thanks, Rafael
Index: linux-pm/drivers/base/power/runtime.c =================================================================== --- linux-pm.orig/drivers/base/power/runtime.c +++ linux-pm/drivers/base/power/runtime.c @@ -1613,17 +1613,28 @@ void pm_runtime_drop_link(struct device spin_unlock_irq(&dev->power.lock); } +static bool pm_runtime_need_not_resume(struct device *dev) +{ + return atomic_read(&dev->power.usage_count) <= 1 && + atomic_read(&dev->power.child_count) == 0; +} + /** * pm_runtime_force_suspend - Force a device into suspend state if needed. * @dev: Device to suspend. * * Disable runtime PM so we safely can check the device's runtime PM status and - * if it is active, invoke it's .runtime_suspend callback to bring it into - * suspend state. Keep runtime PM disabled to preserve the state unless we - * encounter errors. + * if it is active, invoke its ->runtime_suspend callback to suspend it and + * change its runtime PM status field to RPM_SUSPENDED. Also, if the device's + * usage and children counters don't indicate that the device was in use before + * the system-wide transition under way, decrement its parent's children counter + * (if there is a parent). Keep runtime PM disabled to preserve the state + * unless we encounter errors. * * Typically this function may be invoked from a system suspend callback to make - * sure the device is put into low power state. + * sure the device is put into low power state and it should only be used during + * system-wide PM transitions to sleep states. It assumes that the analogous + * pm_runtime_force_resume() will be used to resume the device. */ int pm_runtime_force_suspend(struct device *dev) { @@ -1646,17 +1657,18 @@ int pm_runtime_force_suspend(struct devi goto err; /* - * Increase the runtime PM usage count for the device's parent, in case - * when we find the device being used when system suspend was invoked. - * This informs pm_runtime_force_resume() to resume the parent - * immediately, which is needed to be able to resume its children, - * when not deferring the resume to be managed via runtime PM. + * If the device can stay in suspend after the system-wide transition + * to the working state that will follow, drop the children counter of + * its parent, but set its status to RPM_SUSPENDED anyway in case this + * function will be called again for it in the meantime. */ - if (dev->parent && atomic_read(&dev->power.usage_count) > 1) - pm_runtime_get_noresume(dev->parent); + if (pm_runtime_need_not_resume(dev)) + pm_runtime_set_suspended(dev); + else + __update_runtime_status(dev, RPM_SUSPENDED); - pm_runtime_set_suspended(dev); return 0; + err: pm_runtime_enable(dev); return ret; @@ -1669,13 +1681,9 @@ EXPORT_SYMBOL_GPL(pm_runtime_force_suspe * * Prior invoking this function we expect the user to have brought the device * into low power state by a call to pm_runtime_force_suspend(). Here we reverse - * those actions and brings the device into full power, if it is expected to be - * used on system resume. To distinguish that, we check whether the runtime PM - * usage count is greater than 1 (the PM core increases the usage count in the - * system PM prepare phase), as that indicates a real user (such as a subsystem, - * driver, userspace, etc.) is using it. If that is the case, the device is - * expected to be used on system resume as well, so then we resume it. In the - * other case, we defer the resume to be managed via runtime PM. + * those actions and bring the device into full power, if it is expected to be + * used on system resume. In the other case, we defer the resume to be managed + * via runtime PM. * * Typically this function may be invoked from a system resume callback. */ @@ -1684,32 +1692,18 @@ int pm_runtime_force_resume(struct devic int (*callback)(struct device *); int ret = 0; - callback = RPM_GET_CALLBACK(dev, runtime_resume); - - if (!callback) { - ret = -ENOSYS; - goto out; - } - - if (!pm_runtime_status_suspended(dev)) + if (!pm_runtime_status_suspended(dev) || pm_runtime_need_not_resume(dev)) goto out; /* - * Decrease the parent's runtime PM usage count, if we increased it - * during system suspend in pm_runtime_force_suspend(). - */ - if (atomic_read(&dev->power.usage_count) > 1) { - if (dev->parent) - pm_runtime_put_noidle(dev->parent); - } else { - goto out; - } + * The value of the parent's children counter is correct already, so + * just update the status of the device. + */ + __update_runtime_status(dev, RPM_ACTIVE); - ret = pm_runtime_set_active(dev); - if (ret) - goto out; + callback = RPM_GET_CALLBACK(dev, runtime_resume); - ret = callback(dev); + ret = callback ? callback(dev) : -ENOSYS; if (ret) { pm_runtime_set_suspended(dev); goto out;