mbox series

[v3,00/14] thermal: Rework binding cooling devices to trip points

Message ID 2205737.irdbgypaU6@rjwysocki.net (mailing list archive)
Headers show
Series thermal: Rework binding cooling devices to trip points | expand

Message

Rafael J. Wysocki Aug. 19, 2024, 3:49 p.m. UTC
Hi Everyone,

This is one more update of

https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r

the cover letter of which was sent separately by mistake:

https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/

and it has been updated once already:

https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/

Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
in the v2), and two more that would need to be rebased significantly, either
because of dropping the other broken patch or because of the recent Bang-bang
governor fixes:

https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/

The remaining 14 patches, 2 of which have been slightly rebased and the rest
is mostly unchanged (except for some very minor subject and changelog fixes),
is not expected to be controversial and are targeting 6.12, on top of the
current linux-next material.

The original motivation for this series quoted below has not changed:

 The code for binding cooling devices to trip points (and unbinding them from
 trip point) is one of the murkiest pieces of the thermal subsystem.  It is
 convoluted, bloated with unnecessary code doing questionable things, and it
 works backwards.

 The idea is to bind cooling devices to trip points in accordance with some
 information known to the thermal zone owner (thermal driver).  This information
 is not known to the thermal core when the thermal zone is registered, so the
 driver needs to be involved, but instead of just asking the driver whether
 or not the given cooling device should be bound to a given trip point, the
 thermal core expects the driver to carry out all of the binding process
 including calling functions specifically provided by the core for this
 purpose which is cumbersome and counter-intuitive.

 Because the driver has no information regarding the representation of the trip
 points at the core level, it is forced to walk them (and it has to avoid some
 locking traps while doing this), or it needs to make questionable assumptions
 regarding the ordering of the trips in the core.  There are drivers doing both
 these things.

The first 5 patches in the series are preliminary.

Patch [06/14] introduces a new .should_bind() callback for thermal zones and
patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
.unbind() callbacks which allows them to be simplified quite a bit.

The other patches [08,13-14/14] get rid of code that becomes unused after the
previous changes and do some cleanups on top of that.

The entire series along with 2 patches on top of it (that were present in the
v2 of this set of patches) is available in the thermal-core-testing git branch:

https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing

(note that this branch is going to be rebased shortly on top of 6.11-rc4
and the thermal control material in linux-next).

Thanks!

Comments

Nícolas F. R. A. Prado Aug. 24, 2024, 6:45 p.m. UTC | #1
On Mon, Aug 19, 2024 at 05:49:07PM +0200, Rafael J. Wysocki wrote:
> Hi Everyone,
> 
> This is one more update of
> 
> https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r
> 
> the cover letter of which was sent separately by mistake:
> 
> https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/
> 
> and it has been updated once already:
> 
> https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/
> 
> Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
> in the v2), and two more that would need to be rebased significantly, either
> because of dropping the other broken patch or because of the recent Bang-bang
> governor fixes:
> 
> https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/
> 
> The remaining 14 patches, 2 of which have been slightly rebased and the rest
> is mostly unchanged (except for some very minor subject and changelog fixes),
> is not expected to be controversial and are targeting 6.12, on top of the
> current linux-next material.
> 
> The original motivation for this series quoted below has not changed:
> 
>  The code for binding cooling devices to trip points (and unbinding them from
>  trip point) is one of the murkiest pieces of the thermal subsystem.  It is
>  convoluted, bloated with unnecessary code doing questionable things, and it
>  works backwards.
> 
>  The idea is to bind cooling devices to trip points in accordance with some
>  information known to the thermal zone owner (thermal driver).  This information
>  is not known to the thermal core when the thermal zone is registered, so the
>  driver needs to be involved, but instead of just asking the driver whether
>  or not the given cooling device should be bound to a given trip point, the
>  thermal core expects the driver to carry out all of the binding process
>  including calling functions specifically provided by the core for this
>  purpose which is cumbersome and counter-intuitive.
> 
>  Because the driver has no information regarding the representation of the trip
>  points at the core level, it is forced to walk them (and it has to avoid some
>  locking traps while doing this), or it needs to make questionable assumptions
>  regarding the ordering of the trips in the core.  There are drivers doing both
>  these things.
> 
> The first 5 patches in the series are preliminary.
> 
> Patch [06/14] introduces a new .should_bind() callback for thermal zones and
> patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
> .unbind() callbacks which allows them to be simplified quite a bit.
> 
> The other patches [08,13-14/14] get rid of code that becomes unused after the
> previous changes and do some cleanups on top of that.
> 
> The entire series along with 2 patches on top of it (that were present in the
> v2 of this set of patches) is available in the thermal-core-testing git branch:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing
> 
> (note that this branch is going to be rebased shortly on top of 6.11-rc4
> and the thermal control material in linux-next).
> 
> Thanks!

Hi,

KernelCI has identified a boot regression originating from this series. I've
verified that reverting the series fixes the issue.

Affected platforms:
* mt8195-cherry-tomato-r2
* mt8192-asurada-spherion-r0
* mt8183-kukui-jacuzzi-juniper-sku16
* mt8186-corsola-steelix-sku131072
* sc7180-trogdor-kingoftown
* sc7180-trogdor-lazor-limozeen

Relevant log from mt8195-cherry-tomato-r2 (with additional debug configs
enabled):

[   11.326726] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1578
[   11.335294] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 165, name: udevd
[   11.342944] preempt_count: 1, expected: 0
[   11.346943] RCU nest depth: 0, expected: 0
[   11.351028] 4 locks held by udevd/165:
[   11.354766]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
[   11.363207]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
[   11.374248]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
[   11.383896]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
[   11.392418] irq event stamp: 173740
[   11.395895] hardirqs last  enabled at (173739): [<ffffc208ecde804c>] _raw_spin_unlock_irqrestore+0x84/0x90
[   11.405537] hardirqs last disabled at (173740): [<ffffc208ecde6f7c>] _raw_spin_lock_irqsave+0xe0/0xf4
[   11.414742] softirqs last  enabled at (172404): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
[   11.423517] softirqs last disabled at (172393): [<ffffc208e961097c>] __do_softirq+0x14/0x20
[   11.431857] CPU: 5 UID: 0 PID: 165 Comm: udevd Not tainted 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
[   11.441670] Hardware name: Acer Tomato (rev2) board (DT)
[   11.446970] Call trace:
[   11.449407]  dump_backtrace+0x98/0xf0
[   11.453059]  show_stack+0x18/0x24
[   11.456364]  dump_stack_lvl+0x90/0xd0
[   11.460018]  dump_stack+0x1c/0x28
[   11.463322]  __might_resched+0x358/0x570
[   11.467234]  __might_sleep+0xa4/0x16c
[   11.470885]  down_write+0x8c/0x21c
[   11.474277]  kernfs_remove+0x64/0x98
[   11.477844]  sysfs_remove_dir+0xa8/0xe8
[   11.481669]  __kobject_del+0xb0/0x27c
[   11.485321]  kobject_release+0xfc/0x134
[   11.489146]  kobject_put+0xb0/0x130
[   11.492624]  of_node_put+0x18/0x28
[   11.496016]  of_get_next_child+0x64/0xc4
[   11.499929]  thermal_of_should_bind+0x154/0x390
[   11.504449]  thermal_zone_cdev_binding.part.0+0x174/0x280
[   11.509836]  thermal_zone_device_register_with_trips+0x914/0xcd8
[   11.515831]  thermal_of_zone_register+0x284/0x464
[   11.520523]  devm_thermal_of_zone_register+0x80/0xf4
[   11.525476]  lvts_domain_init+0x500/0x760 [lvts_thermal]
[   11.530785]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
[   11.535565]  platform_probe+0xc4/0x214
[   11.539303]  really_probe+0x188/0x5d0
[   11.542954]  __driver_probe_device+0x160/0x2e8
[   11.547386]  driver_probe_device+0x5c/0x298
[   11.551558]  __driver_attach+0x13c/0x4a8
[   11.555470]  bus_for_each_dev+0xf8/0x180
[   11.559383]  driver_attach+0x3c/0x58
[   11.562947]  bus_add_driver+0x1c4/0x458
[   11.566772]  driver_register+0xf4/0x3c0
[   11.570598]  __platform_driver_register+0x60/0x88
[   11.575291]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
[   11.580593]  do_one_initcall+0xcc/0x284
[   11.584418]  do_init_module+0x278/0x740
[   11.588244]  load_module+0xed8/0x1434
[   11.591897]  init_module_from_file+0xdc/0x1fc
[   11.596243]  idempotent_init_module+0x2bc/0x604
[   11.600762]  __arm64_sys_finit_module+0xac/0x100
[   11.605368]  invoke_syscall+0x6c/0x258
[   11.609107]  el0_svc_common.constprop.0+0xac/0x230
[   11.613886]  do_el0_svc+0x40/0x58
[   11.617190]  el0_svc+0x48/0xb8
[   11.620234]  el0t_64_sync_handler+0x100/0x12c
[   11.624580]  el0t_64_sync+0x190/0x194
[   11.628233]
[   11.629713] =============================
[   11.633708] [ BUG: Invalid wait context ]
[   11.637705] 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628 Tainted: G        W
[   11.645953] -----------------------------
[   11.649950] udevd/165 is trying to lock:
[   11.653859] ffff4dc880881148 (&root->kernfs_rwsem){++++}-{3:3}, at: kernfs_remove+0x64/0x98
[   11.662200] other info that might help us debug this:
[   11.667238] context-{4:4}
[   11.669846] 4 locks held by udevd/165:
[   11.673582]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
[   11.682009]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
[   11.693041]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
[   11.702684]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
[   11.711199] stack backtrace:
[   11.714067] CPU: 5 UID: 0 PID: 165 Comm: udevd Tainted: G        W          6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
[   11.725355] Tainted: [W]=WARN
[   11.728310] Hardware name: Acer Tomato (rev2) board (DT)
[   11.733608] Call trace:
[   11.736041]  dump_backtrace+0x98/0xf0
[   11.739692]  show_stack+0x18/0x24
[   11.742994]  dump_stack_lvl+0x90/0xd0
[   11.746645]  dump_stack+0x1c/0x28
[   11.749948]  __lock_acquire+0x10f8/0x2710
[   11.753948]  lock_acquire.part.0+0x218/0x518
[   11.758206]  lock_acquire+0x90/0xb4
[   11.761683]  down_write+0xb4/0x21c
[   11.765074]  kernfs_remove+0x64/0x98
[   11.768637]  sysfs_remove_dir+0xa8/0xe8
[   11.772461]  __kobject_del+0xb0/0x27c
[   11.776111]  kobject_release+0xfc/0x134
[   11.779935]  kobject_put+0xb0/0x130
[   11.783413]  of_node_put+0x18/0x28
[   11.786803]  of_get_next_child+0x64/0xc4
[   11.790714]  thermal_of_should_bind+0x154/0x390
[   11.795231]  thermal_zone_cdev_binding.part.0+0x174/0x280
[   11.800617]  thermal_zone_device_register_with_trips+0x914/0xcd8
[   11.806609]  thermal_of_zone_register+0x284/0x464
[   11.811301]  devm_thermal_of_zone_register+0x80/0xf4
[   11.816253]  lvts_domain_init+0x500/0x760 [lvts_thermal]
[   11.821553]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
[   11.826332]  platform_probe+0xc4/0x214
[   11.830069]  really_probe+0x188/0x5d0
[   11.833719]  __driver_probe_device+0x160/0x2e8
[   11.838150]  driver_probe_device+0x5c/0x298
[   11.842320]  __driver_attach+0x13c/0x4a8
[   11.846230]  bus_for_each_dev+0xf8/0x180
[   11.850141]  driver_attach+0x3c/0x58
[   11.853704]  bus_add_driver+0x1c4/0x458
[   11.857529]  driver_register+0xf4/0x3c0
[   11.861352]  __platform_driver_register+0x60/0x88
[   11.866043]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
[   11.871342]  do_one_initcall+0xcc/0x284
[   11.875166]  do_init_module+0x278/0x740
[   11.878990]  load_module+0xed8/0x1434
[   11.882641]  init_module_from_file+0xdc/0x1fc
[   11.886986]  idempotent_init_module+0x2bc/0x604
[   11.891504]  __arm64_sys_finit_module+0xac/0x100
[   11.896109]  invoke_syscall+0x6c/0x258
[   11.899846]  el0_svc_common.constprop.0+0xac/0x230
[   11.904624]  do_el0_svc+0x40/0x58
[   11.907927]  el0_svc+0x48/0xb8
[   11.910969]  el0t_64_sync_handler+0x100/0x12c
[   11.915314]  el0t_64_sync+0x190/0x194
[   36.261761] watchdog: Watchdog detected hard LOCKUP on cpu 0
[   36.267414] Modules linked in: cbmem cros_ec_lid_angle cros_ec_sensors(+) cros_ec_sensors_core pcie_mediatek_gen3 sbs_battery cros_kbd_led_backlight industrialio_triggered_buffer kfifo_buf cros_ec_chardev cros_ec_rpmsg lvts_thermal(+) cros_ec_typec leds_cros_ec mtk_svs snd_sof_mt8195 mtk_adsp_common snd_sof_xtensa_dsp snd_sof_of mt6577_auxadc snd_soc_mt8195_afe snd_sof snd_sof_utils mtk_scp mtk_rpmsg mtk_scp_ipi pwm_bl mtk_wdt coreboot_table backlight mt8195_mt6359 ramoops reed_solomon
[   36.310414] irq event stamp: 197347
[   36.313890] hardirqs last  enabled at (197347): [<ffffc208ecdc994c>] exit_to_kernel_mode+0x38/0x118
[   36.322923] hardirqs last disabled at (197346): [<ffffc208ecdcac44>] el1_interrupt+0x24/0x54
[   36.331347] softirqs last  enabled at (197268): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
[   36.340117] softirqs last disabled at (197263): [<ffffc208e961097c>] __do_softirq+0x14/0x20

Full log at http://0x0.st/XyID.txt

Let me know if you need any more information.

#regzbot introduced: next-20240821..next-20240822
#regzbot title: Hang during boot in sysfs_remove_dir() called by thermal_of_zone_register()

Thanks,
Nícolas
Rafael J. Wysocki Aug. 26, 2024, 9:58 a.m. UTC | #2
On Sat, Aug 24, 2024 at 8:45 PM Nícolas F. R. A. Prado
<nfraprado@collabora.com> wrote:
>
> On Mon, Aug 19, 2024 at 05:49:07PM +0200, Rafael J. Wysocki wrote:
> > Hi Everyone,
> >
> > This is one more update of
> >
> > https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r
> >
> > the cover letter of which was sent separately by mistake:
> >
> > https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/
> >
> > and it has been updated once already:
> >
> > https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/
> >
> > Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
> > in the v2), and two more that would need to be rebased significantly, either
> > because of dropping the other broken patch or because of the recent Bang-bang
> > governor fixes:
> >
> > https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/
> >
> > The remaining 14 patches, 2 of which have been slightly rebased and the rest
> > is mostly unchanged (except for some very minor subject and changelog fixes),
> > is not expected to be controversial and are targeting 6.12, on top of the
> > current linux-next material.
> >
> > The original motivation for this series quoted below has not changed:
> >
> >  The code for binding cooling devices to trip points (and unbinding them from
> >  trip point) is one of the murkiest pieces of the thermal subsystem.  It is
> >  convoluted, bloated with unnecessary code doing questionable things, and it
> >  works backwards.
> >
> >  The idea is to bind cooling devices to trip points in accordance with some
> >  information known to the thermal zone owner (thermal driver).  This information
> >  is not known to the thermal core when the thermal zone is registered, so the
> >  driver needs to be involved, but instead of just asking the driver whether
> >  or not the given cooling device should be bound to a given trip point, the
> >  thermal core expects the driver to carry out all of the binding process
> >  including calling functions specifically provided by the core for this
> >  purpose which is cumbersome and counter-intuitive.
> >
> >  Because the driver has no information regarding the representation of the trip
> >  points at the core level, it is forced to walk them (and it has to avoid some
> >  locking traps while doing this), or it needs to make questionable assumptions
> >  regarding the ordering of the trips in the core.  There are drivers doing both
> >  these things.
> >
> > The first 5 patches in the series are preliminary.
> >
> > Patch [06/14] introduces a new .should_bind() callback for thermal zones and
> > patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
> > .unbind() callbacks which allows them to be simplified quite a bit.
> >
> > The other patches [08,13-14/14] get rid of code that becomes unused after the
> > previous changes and do some cleanups on top of that.
> >
> > The entire series along with 2 patches on top of it (that were present in the
> > v2 of this set of patches) is available in the thermal-core-testing git branch:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing
> >
> > (note that this branch is going to be rebased shortly on top of 6.11-rc4
> > and the thermal control material in linux-next).
> >
> > Thanks!
>
> Hi,
>
> KernelCI has identified a boot regression originating from this series. I've
> verified that reverting the series fixes the issue.

Thanks for the report!

There was a bug in the original patch [12/14] that would cause
symptoms like what you are observing to appear, which was reported on
Friday and has since been fixed in the tree.  Please see:

https://lore.kernel.org/linux-pm/CAJZ5v0iw7uXE_cfU5VXOjFDg9GM8Hu0+hKxqfzU3v0OM5KK9oQ@mail.gmail.com/

You probably have not tested the fixed tree yet, so please let
kernelci run again on it and if the issue is still there, please let
me know.


> Affected platforms:
> * mt8195-cherry-tomato-r2
> * mt8192-asurada-spherion-r0
> * mt8183-kukui-jacuzzi-juniper-sku16
> * mt8186-corsola-steelix-sku131072
> * sc7180-trogdor-kingoftown
> * sc7180-trogdor-lazor-limozeen
>
> Relevant log from mt8195-cherry-tomato-r2 (with additional debug configs
> enabled):
>
> [   11.326726] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1578
> [   11.335294] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 165, name: udevd
> [   11.342944] preempt_count: 1, expected: 0
> [   11.346943] RCU nest depth: 0, expected: 0
> [   11.351028] 4 locks held by udevd/165:
> [   11.354766]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
> [   11.363207]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
> [   11.374248]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
> [   11.383896]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
> [   11.392418] irq event stamp: 173740
> [   11.395895] hardirqs last  enabled at (173739): [<ffffc208ecde804c>] _raw_spin_unlock_irqrestore+0x84/0x90
> [   11.405537] hardirqs last disabled at (173740): [<ffffc208ecde6f7c>] _raw_spin_lock_irqsave+0xe0/0xf4
> [   11.414742] softirqs last  enabled at (172404): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
> [   11.423517] softirqs last disabled at (172393): [<ffffc208e961097c>] __do_softirq+0x14/0x20
> [   11.431857] CPU: 5 UID: 0 PID: 165 Comm: udevd Not tainted 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
> [   11.441670] Hardware name: Acer Tomato (rev2) board (DT)
> [   11.446970] Call trace:
> [   11.449407]  dump_backtrace+0x98/0xf0
> [   11.453059]  show_stack+0x18/0x24
> [   11.456364]  dump_stack_lvl+0x90/0xd0
> [   11.460018]  dump_stack+0x1c/0x28
> [   11.463322]  __might_resched+0x358/0x570
> [   11.467234]  __might_sleep+0xa4/0x16c
> [   11.470885]  down_write+0x8c/0x21c
> [   11.474277]  kernfs_remove+0x64/0x98
> [   11.477844]  sysfs_remove_dir+0xa8/0xe8
> [   11.481669]  __kobject_del+0xb0/0x27c
> [   11.485321]  kobject_release+0xfc/0x134
> [   11.489146]  kobject_put+0xb0/0x130
> [   11.492624]  of_node_put+0x18/0x28
> [   11.496016]  of_get_next_child+0x64/0xc4
> [   11.499929]  thermal_of_should_bind+0x154/0x390
> [   11.504449]  thermal_zone_cdev_binding.part.0+0x174/0x280
> [   11.509836]  thermal_zone_device_register_with_trips+0x914/0xcd8
> [   11.515831]  thermal_of_zone_register+0x284/0x464
> [   11.520523]  devm_thermal_of_zone_register+0x80/0xf4
> [   11.525476]  lvts_domain_init+0x500/0x760 [lvts_thermal]
> [   11.530785]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
> [   11.535565]  platform_probe+0xc4/0x214
> [   11.539303]  really_probe+0x188/0x5d0
> [   11.542954]  __driver_probe_device+0x160/0x2e8
> [   11.547386]  driver_probe_device+0x5c/0x298
> [   11.551558]  __driver_attach+0x13c/0x4a8
> [   11.555470]  bus_for_each_dev+0xf8/0x180
> [   11.559383]  driver_attach+0x3c/0x58
> [   11.562947]  bus_add_driver+0x1c4/0x458
> [   11.566772]  driver_register+0xf4/0x3c0
> [   11.570598]  __platform_driver_register+0x60/0x88
> [   11.575291]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
> [   11.580593]  do_one_initcall+0xcc/0x284
> [   11.584418]  do_init_module+0x278/0x740
> [   11.588244]  load_module+0xed8/0x1434
> [   11.591897]  init_module_from_file+0xdc/0x1fc
> [   11.596243]  idempotent_init_module+0x2bc/0x604
> [   11.600762]  __arm64_sys_finit_module+0xac/0x100
> [   11.605368]  invoke_syscall+0x6c/0x258
> [   11.609107]  el0_svc_common.constprop.0+0xac/0x230
> [   11.613886]  do_el0_svc+0x40/0x58
> [   11.617190]  el0_svc+0x48/0xb8
> [   11.620234]  el0t_64_sync_handler+0x100/0x12c
> [   11.624580]  el0t_64_sync+0x190/0x194
> [   11.628233]
> [   11.629713] =============================
> [   11.633708] [ BUG: Invalid wait context ]
> [   11.637705] 6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628 Tainted: G        W
> [   11.645953] -----------------------------
> [   11.649950] udevd/165 is trying to lock:
> [   11.653859] ffff4dc880881148 (&root->kernfs_rwsem){++++}-{3:3}, at: kernfs_remove+0x64/0x98
> [   11.662200] other info that might help us debug this:
> [   11.667238] context-{4:4}
> [   11.669846] 4 locks held by udevd/165:
> [   11.673582]  #0: ffff4dc8825db0f8 (&dev->mutex){....}-{3:3}, at: __driver_attach+0x130/0x4a8
> [   11.682009]  #1: ffffc208f386c3c8 (thermal_list_lock){+.+.}-{3:3}, at: thermal_zone_device_register_with_trips+0x85c/0xcd8
> [   11.693041]  #2: ffff4dc7dc3586f0 (&tz->lock){+.+.}-{3:3}, at: thermal_zone_cdev_binding.part.0+0x98/0x280
> [   11.702684]  #3: ffffc208f39b7b78 (devtree_lock){....}-{2:2}, at: of_get_next_child+0x2c/0xc4
> [   11.711199] stack backtrace:
> [   11.714067] CPU: 5 UID: 0 PID: 165 Comm: udevd Tainted: G        W          6.11.0-rc4-next-20240822-00002-gfbbbf9faa56a #628
> [   11.725355] Tainted: [W]=WARN
> [   11.728310] Hardware name: Acer Tomato (rev2) board (DT)
> [   11.733608] Call trace:
> [   11.736041]  dump_backtrace+0x98/0xf0
> [   11.739692]  show_stack+0x18/0x24
> [   11.742994]  dump_stack_lvl+0x90/0xd0
> [   11.746645]  dump_stack+0x1c/0x28
> [   11.749948]  __lock_acquire+0x10f8/0x2710
> [   11.753948]  lock_acquire.part.0+0x218/0x518
> [   11.758206]  lock_acquire+0x90/0xb4
> [   11.761683]  down_write+0xb4/0x21c
> [   11.765074]  kernfs_remove+0x64/0x98
> [   11.768637]  sysfs_remove_dir+0xa8/0xe8
> [   11.772461]  __kobject_del+0xb0/0x27c
> [   11.776111]  kobject_release+0xfc/0x134
> [   11.779935]  kobject_put+0xb0/0x130
> [   11.783413]  of_node_put+0x18/0x28
> [   11.786803]  of_get_next_child+0x64/0xc4
> [   11.790714]  thermal_of_should_bind+0x154/0x390
> [   11.795231]  thermal_zone_cdev_binding.part.0+0x174/0x280
> [   11.800617]  thermal_zone_device_register_with_trips+0x914/0xcd8
> [   11.806609]  thermal_of_zone_register+0x284/0x464
> [   11.811301]  devm_thermal_of_zone_register+0x80/0xf4
> [   11.816253]  lvts_domain_init+0x500/0x760 [lvts_thermal]
> [   11.821553]  lvts_probe+0x1b4/0x3ac [lvts_thermal]
> [   11.826332]  platform_probe+0xc4/0x214
> [   11.830069]  really_probe+0x188/0x5d0
> [   11.833719]  __driver_probe_device+0x160/0x2e8
> [   11.838150]  driver_probe_device+0x5c/0x298
> [   11.842320]  __driver_attach+0x13c/0x4a8
> [   11.846230]  bus_for_each_dev+0xf8/0x180
> [   11.850141]  driver_attach+0x3c/0x58
> [   11.853704]  bus_add_driver+0x1c4/0x458
> [   11.857529]  driver_register+0xf4/0x3c0
> [   11.861352]  __platform_driver_register+0x60/0x88
> [   11.866043]  lvts_driver_init+0x20/0x1000 [lvts_thermal]
> [   11.871342]  do_one_initcall+0xcc/0x284
> [   11.875166]  do_init_module+0x278/0x740
> [   11.878990]  load_module+0xed8/0x1434
> [   11.882641]  init_module_from_file+0xdc/0x1fc
> [   11.886986]  idempotent_init_module+0x2bc/0x604
> [   11.891504]  __arm64_sys_finit_module+0xac/0x100
> [   11.896109]  invoke_syscall+0x6c/0x258
> [   11.899846]  el0_svc_common.constprop.0+0xac/0x230
> [   11.904624]  do_el0_svc+0x40/0x58
> [   11.907927]  el0_svc+0x48/0xb8
> [   11.910969]  el0t_64_sync_handler+0x100/0x12c
> [   11.915314]  el0t_64_sync+0x190/0x194
> [   36.261761] watchdog: Watchdog detected hard LOCKUP on cpu 0
> [   36.267414] Modules linked in: cbmem cros_ec_lid_angle cros_ec_sensors(+) cros_ec_sensors_core pcie_mediatek_gen3 sbs_battery cros_kbd_led_backlight industrialio_triggered_buffer kfifo_buf cros_ec_chardev cros_ec_rpmsg lvts_thermal(+) cros_ec_typec leds_cros_ec mtk_svs snd_sof_mt8195 mtk_adsp_common snd_sof_xtensa_dsp snd_sof_of mt6577_auxadc snd_soc_mt8195_afe snd_sof snd_sof_utils mtk_scp mtk_rpmsg mtk_scp_ipi pwm_bl mtk_wdt coreboot_table backlight mt8195_mt6359 ramoops reed_solomon
> [   36.310414] irq event stamp: 197347
> [   36.313890] hardirqs last  enabled at (197347): [<ffffc208ecdc994c>] exit_to_kernel_mode+0x38/0x118
> [   36.322923] hardirqs last disabled at (197346): [<ffffc208ecdcac44>] el1_interrupt+0x24/0x54
> [   36.331347] softirqs last  enabled at (197268): [<ffffc208e978bb20>] handle_softirqs+0x534/0x874
> [   36.340117] softirqs last disabled at (197263): [<ffffc208e961097c>] __do_softirq+0x14/0x20
>
> Full log at http://0x0.st/XyID.txt
>
> Let me know if you need any more information.
>
> #regzbot introduced: next-20240821..next-20240822
> #regzbot title: Hang during boot in sysfs_remove_dir() called by thermal_of_zone_register()
>
> Thanks,
> Nícolas
>
Nícolas F. R. A. Prado Aug. 30, 2024, 1:55 p.m. UTC | #3
On Mon, Aug 26, 2024 at 11:58:12AM +0200, Rafael J. Wysocki wrote:
> On Sat, Aug 24, 2024 at 8:45 PM Nícolas F. R. A. Prado
> <nfraprado@collabora.com> wrote:
> >
> > On Mon, Aug 19, 2024 at 05:49:07PM +0200, Rafael J. Wysocki wrote:
> > > Hi Everyone,
> > >
> > > This is one more update of
> > >
> > > https://lore.kernel.org/linux-pm/3134863.CbtlEUcBR6@rjwysocki.net/#r
> > >
> > > the cover letter of which was sent separately by mistake:
> > >
> > > https://lore.kernel.org/linux-pm/CAJZ5v0jo5vh2uD5t4GqBnN0qukMBG_ty33PB=NiEqigqxzBcsw@mail.gmail.com/
> > >
> > > and it has been updated once already:
> > >
> > > https://lore.kernel.org/linux-pm/114901234.nniJfEyVGO@rjwysocki.net/
> > >
> > > Relative to the v2 above it drops 3 patches, one because it was broken ([04/17
> > > in the v2), and two more that would need to be rebased significantly, either
> > > because of dropping the other broken patch or because of the recent Bang-bang
> > > governor fixes:
> > >
> > > https://lore.kernel.org/linux-pm/1903691.tdWV9SEqCh@rjwysocki.net/
> > >
> > > The remaining 14 patches, 2 of which have been slightly rebased and the rest
> > > is mostly unchanged (except for some very minor subject and changelog fixes),
> > > is not expected to be controversial and are targeting 6.12, on top of the
> > > current linux-next material.
> > >
> > > The original motivation for this series quoted below has not changed:
> > >
> > >  The code for binding cooling devices to trip points (and unbinding them from
> > >  trip point) is one of the murkiest pieces of the thermal subsystem.  It is
> > >  convoluted, bloated with unnecessary code doing questionable things, and it
> > >  works backwards.
> > >
> > >  The idea is to bind cooling devices to trip points in accordance with some
> > >  information known to the thermal zone owner (thermal driver).  This information
> > >  is not known to the thermal core when the thermal zone is registered, so the
> > >  driver needs to be involved, but instead of just asking the driver whether
> > >  or not the given cooling device should be bound to a given trip point, the
> > >  thermal core expects the driver to carry out all of the binding process
> > >  including calling functions specifically provided by the core for this
> > >  purpose which is cumbersome and counter-intuitive.
> > >
> > >  Because the driver has no information regarding the representation of the trip
> > >  points at the core level, it is forced to walk them (and it has to avoid some
> > >  locking traps while doing this), or it needs to make questionable assumptions
> > >  regarding the ordering of the trips in the core.  There are drivers doing both
> > >  these things.
> > >
> > > The first 5 patches in the series are preliminary.
> > >
> > > Patch [06/14] introduces a new .should_bind() callback for thermal zones and
> > > patches [07,09-12/14] modifies drivers to use it instead of the .bind() and
> > > .unbind() callbacks which allows them to be simplified quite a bit.
> > >
> > > The other patches [08,13-14/14] get rid of code that becomes unused after the
> > > previous changes and do some cleanups on top of that.
> > >
> > > The entire series along with 2 patches on top of it (that were present in the
> > > v2 of this set of patches) is available in the thermal-core-testing git branch:
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git/log/?h=thermal-core-testing
> > >
> > > (note that this branch is going to be rebased shortly on top of 6.11-rc4
> > > and the thermal control material in linux-next).
> > >
> > > Thanks!
> >
> > Hi,
> >
> > KernelCI has identified a boot regression originating from this series. I've
> > verified that reverting the series fixes the issue.
> 
> Thanks for the report!
> 
> There was a bug in the original patch [12/14] that would cause
> symptoms like what you are observing to appear, which was reported on
> Friday and has since been fixed in the tree.  Please see:
> 
> https://lore.kernel.org/linux-pm/CAJZ5v0iw7uXE_cfU5VXOjFDg9GM8Hu0+hKxqfzU3v0OM5KK9oQ@mail.gmail.com/
> 
> You probably have not tested the fixed tree yet, so please let
> kernelci run again on it and if the issue is still there, please let
> me know.

Indeed it has been fixed.

#regzbot fix: 'thermal/of: Use the .should_bind() thermal zone callback'

Thanks,
Nícolas