diff mbox

[RFC] PCI: rockchip: fix system hang up if activate CONFIG_DEBUG_SHIRQ

Message ID 1502353273-123788-1-git-send-email-shawn.lin@rock-chips.com (mailing list archive)
State New, archived
Headers show

Commit Message

Shawn Lin Aug. 10, 2017, 8:21 a.m. UTC
With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
would still access the irq handler registed as a shard irq.
Per the comment within the function of __free_irq, it says
"It's a shared IRQ -- the driver ought to be prepared for
an IRQ event to happen even now it's being freed". However
when failing to probe the driver, it may disable the clock
for accessing the register and the following check for shared
irq state would call the irq handler which accesses the register
w/o the clk enabled. That will hang the system forever.

With adding some dump_stack we could see how that happened.

calling  rockchip_pcie_driver_init+0x0/0x28 @ 1
rockchip-pcie f8000000.pcie: no vpcie3v3 regulator found
rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found
rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found
rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!
CPU: 0 PID: 1 Comm: swapper/0 Not tainted
4.13.0-rc3-next-20170807-ARCH+ #189
Hardware name: Firefly-RK3399 Board (DT)
Call trace:
[<ffff000008089bf0>] dump_backtrace+0x0/0x250
[<ffff000008089eb0>] show_stack+0x20/0x28
[<ffff000008c3313c>] dump_stack+0x90/0xb0
[<ffff000008632ad4>] rockchip_pcie_read.isra.11+0x54/0x58
[<ffff0000086334fc>] rockchip_pcie_client_irq_handler+0x30/0x1a0
[<ffff00000813ce98>] __free_irq+0x1c8/0x2dc
[<ffff00000813d044>] free_irq+0x44/0x74
[<ffff0000081415fc>] devm_irq_release+0x24/0x2c
[<ffff00000877429c>] release_nodes+0x1d8/0x30c
[<ffff000008774838>] devres_release_all+0x3c/0x5c
[<ffff00000876f19c>] driver_probe_device+0x244/0x494
[<ffff00000876f50c>] __driver_attach+0x120/0x124
[<ffff00000876cb80>] bus_for_each_dev+0x6c/0xac
[<ffff00000876e984>] driver_attach+0x2c/0x34
[<ffff00000876e3a4>] bus_add_driver+0x244/0x2b0
[<ffff000008770264>] driver_register+0x70/0x110
[<ffff0000087718b4>] platform_driver_register+0x60/0x6c
[<ffff0000091eb108>] rockchip_pcie_driver_init+0x20/0x28
[<ffff000008083a2c>] do_one_initcall+0xc8/0x130
[<ffff0000091a0ea8>] kernel_init_freeable+0x1a0/0x238
[<ffff000008c461cc>] kernel_init+0x18/0x108
[<ffff0000080836c0>] ret_from_fork+0x10/0x50

In order to fix this, we check the clk state before accessing the
pcie register in rockchip_pcie_read, but don't touch rockchip_pcie_write
as no case to trigger that from write routine.

Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>

---
Hi Bjorn, Thomas and Marc,

This fix looks more like a hack, but I don't know the legit way to
deal with that case. Just quick look into the drivers/pci/host, and
I see almost all drivers register shard irq and also disable the clk
in their probe's error handle path. So I guess this is NOT pcie-rockchip
specified even not pcie host drivers specified, but folks just luckily didn't
trip over it.

Could you give some suggestion on this?

 drivers/pci/host/pcie-rockchip.c | 4 ++++
 1 file changed, 4 insertions(+)

Comments

JeffyChen Aug. 10, 2017, 8:39 a.m. UTC | #1
Hi shawn,

On 08/10/2017 04:21 PM, Shawn Lin wrote:
> With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
> would still access the irq handler registed as a shard irq.
> Per the comment within the function of __free_irq, it says
> "It's a shared IRQ -- the driver ought to be prepared for
> an IRQ event to happen even now it's being freed". However
> when failing to probe the driver, it may disable the clock
> for accessing the register and the following check for shared
> irq state would call the irq handler which accesses the register
> w/o the clk enabled. That will hang the system forever.

i think this extra irq call is to make sure it's safe to get a shared 
irq when we are freeing it, and we would not get this irq after freed it.

so maybe just call devm_free_irq before disable clks(and other required 
resources for the irq handler)? and also do it in the rockchip_pcie_remove.
Shawn Lin Aug. 10, 2017, 9:14 a.m. UTC | #2
Hi Jeffy

On 2017/8/10 16:39, jeffy wrote:
> Hi shawn,
> 
> On 08/10/2017 04:21 PM, Shawn Lin wrote:
>> With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
>> would still access the irq handler registed as a shard irq.
>> Per the comment within the function of __free_irq, it says
>> "It's a shared IRQ -- the driver ought to be prepared for
>> an IRQ event to happen even now it's being freed". However
>> when failing to probe the driver, it may disable the clock
>> for accessing the register and the following check for shared
>> irq state would call the irq handler which accesses the register
>> w/o the clk enabled. That will hang the system forever.
> 
> i think this extra irq call is to make sure it's safe to get a shared 
> irq when we are freeing it, and we would not get this irq after freed it.
> 
> so maybe just call devm_free_irq before disable clks(and other required 
> resources for the irq handler)? and also do it in the rockchip_pcie_remove.

That works since we free it and the following devm_irq_release would
bail out early. But I know sure if it is the real intention of  the
devm_  irq stuff,  although devm_free_irq should be called to free the
IRQ separately if we want. At least that is so explicit that folks will
know that, as more from the top view, it seems only a little drivers
there call devm_free_irq for this case by looking into some more
drivers(Still no any PCIe drivers do that).

Not sure if their systems are robust enough that could access the
register w/o clk, even without power domain enabled.

> 
> 
>
JeffyChen Aug. 10, 2017, 9:25 a.m. UTC | #3
Hi shawn,

On 08/10/2017 05:14 PM, Shawn Lin wrote:
> Hi Jeffy
>
> On 2017/8/10 16:39, jeffy wrote:
>> Hi shawn,
>>
>> On 08/10/2017 04:21 PM, Shawn Lin wrote:
>>> With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
>>> would still access the irq handler registed as a shard irq.
>>> Per the comment within the function of __free_irq, it says
>>> "It's a shared IRQ -- the driver ought to be prepared for
>>> an IRQ event to happen even now it's being freed". However
>>> when failing to probe the driver, it may disable the clock
>>> for accessing the register and the following check for shared
>>> irq state would call the irq handler which accesses the register
>>> w/o the clk enabled. That will hang the system forever.
>>
>> i think this extra irq call is to make sure it's safe to get a shared
>> irq when we are freeing it, and we would not get this irq after freed it.
>>
>> so maybe just call devm_free_irq before disable clks(and other
>> required resources for the irq handler)? and also do it in the
>> rockchip_pcie_remove.
>
> That works since we free it and the following devm_irq_release would
> bail out early. But I know sure if it is the real intention of  the
> devm_  irq stuff,  although devm_free_irq should be called to free the
> IRQ separately if we want. At least that is so explicit that folks will
> know that, as more from the top view, it seems only a little drivers
> there call devm_free_irq for this case by looking into some more
> drivers(Still no any PCIe drivers do that).
>
> Not sure if their systems are robust enough that could access the
> register w/o clk, even without power domain enabled.

maybe we can use request_irq & free_irq directly?
Heiko Stuebner Aug. 10, 2017, 9:27 a.m. UTC | #4
Hi Shawn,

Am Donnerstag, 10. August 2017, 16:21:13 CEST schrieb Shawn Lin:
> With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
> would still access the irq handler registed as a shard irq.
> Per the comment within the function of __free_irq, it says
> "It's a shared IRQ -- the driver ought to be prepared for
> an IRQ event to happen even now it's being freed". However
> when failing to probe the driver, it may disable the clock
> for accessing the register and the following check for shared
> irq state would call the irq handler which accesses the register
> w/o the clk enabled. That will hang the system forever.

The key point would be to fix the ordering. So you could also just use
devm_add_action to move the clock-disabling into a callback that
gets run at the appropriate time, as devm does the shutdown in the
exact opposite order this should fix your problem.

So until all clocks could be enabled, you would disable them manually
and after that rely on the devm_action to fire at the appropriate time.

ret = clk_prepare_enable(clk1);
if (ret < 0 )
	return ret;

ret = clk_prepare_enable(clk2);
if (ret < 0) {
	clk_disable_unprepare(clk1);
	return ret;
}

devm_add_action_or_reset(dev, rk_pcie_disable_clocks);

The rockchip crypto driver [0] shows how this could be done.


Heiko

[0] http://elixir.free-electrons.com/linux/latest/source/drivers/crypto/rockchip/rk3288_crypto.c#L307

> 
> With adding some dump_stack we could see how that happened.
> 
> calling  rockchip_pcie_driver_init+0x0/0x28 @ 1
> rockchip-pcie f8000000.pcie: no vpcie3v3 regulator found
> rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found
> rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found
> rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!
> CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 4.13.0-rc3-next-20170807-ARCH+ #189
> Hardware name: Firefly-RK3399 Board (DT)
> Call trace:
> [<ffff000008089bf0>] dump_backtrace+0x0/0x250
> [<ffff000008089eb0>] show_stack+0x20/0x28
> [<ffff000008c3313c>] dump_stack+0x90/0xb0
> [<ffff000008632ad4>] rockchip_pcie_read.isra.11+0x54/0x58
> [<ffff0000086334fc>] rockchip_pcie_client_irq_handler+0x30/0x1a0
> [<ffff00000813ce98>] __free_irq+0x1c8/0x2dc
> [<ffff00000813d044>] free_irq+0x44/0x74
> [<ffff0000081415fc>] devm_irq_release+0x24/0x2c
> [<ffff00000877429c>] release_nodes+0x1d8/0x30c
> [<ffff000008774838>] devres_release_all+0x3c/0x5c
> [<ffff00000876f19c>] driver_probe_device+0x244/0x494
> [<ffff00000876f50c>] __driver_attach+0x120/0x124
> [<ffff00000876cb80>] bus_for_each_dev+0x6c/0xac
> [<ffff00000876e984>] driver_attach+0x2c/0x34
> [<ffff00000876e3a4>] bus_add_driver+0x244/0x2b0
> [<ffff000008770264>] driver_register+0x70/0x110
> [<ffff0000087718b4>] platform_driver_register+0x60/0x6c
> [<ffff0000091eb108>] rockchip_pcie_driver_init+0x20/0x28
> [<ffff000008083a2c>] do_one_initcall+0xc8/0x130
> [<ffff0000091a0ea8>] kernel_init_freeable+0x1a0/0x238
> [<ffff000008c461cc>] kernel_init+0x18/0x108
> [<ffff0000080836c0>] ret_from_fork+0x10/0x50
> 
> In order to fix this, we check the clk state before accessing the
> pcie register in rockchip_pcie_read, but don't touch rockchip_pcie_write
> as no case to trigger that from write routine.
> 
> Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
> 
> ---
> Hi Bjorn, Thomas and Marc,
> 
> This fix looks more like a hack, but I don't know the legit way to
> deal with that case. Just quick look into the drivers/pci/host, and
> I see almost all drivers register shard irq and also disable the clk
> in their probe's error handle path. So I guess this is NOT pcie-rockchip
> specified even not pcie host drivers specified, but folks just luckily didn't
> trip over it.
> 
> Could you give some suggestion on this?
> 
>  drivers/pci/host/pcie-rockchip.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/pci/host/pcie-rockchip.c b/drivers/pci/host/pcie-rockchip.c
> index 39aafe2..daaa868b 100644
> --- a/drivers/pci/host/pcie-rockchip.c
> +++ b/drivers/pci/host/pcie-rockchip.c
> @@ -17,6 +17,7 @@
>  
>  #include <linux/bitrev.h>
>  #include <linux/clk.h>
> +#include <linux/clk-provider.h>
>  #include <linux/delay.h>
>  #include <linux/gpio/consumer.h>
>  #include <linux/init.h>
> @@ -252,6 +253,9 @@ struct rockchip_pcie {
>  
>  static u32 rockchip_pcie_read(struct rockchip_pcie *rockchip, u32 reg)
>  {
> +	if (!__clk_is_enabled(rockchip->hclk_pcie))
> +		return 0;
> +
>  	return readl(rockchip->apb_base + reg);
>  }
>  
>
JeffyChen Aug. 10, 2017, 9:40 a.m. UTC | #5
Hi Heiko,

On 08/10/2017 05:27 PM, Heiko Stuebner wrote:
> Hi Shawn,
>
> Am Donnerstag, 10. August 2017, 16:21:13 CEST schrieb Shawn Lin:
>> >With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
>> >would still access the irq handler registed as a shard irq.
>> >Per the comment within the function of __free_irq, it says
>> >"It's a shared IRQ -- the driver ought to be prepared for
>> >an IRQ event to happen even now it's being freed". However
>> >when failing to probe the driver, it may disable the clock
>> >for accessing the register and the following check for shared
>> >irq state would call the irq handler which accesses the register
>> >w/o the clk enabled. That will hang the system forever.
> The key point would be to fix the ordering. So you could also just use
> devm_add_action to move the clock-disabling into a callback that
> gets run at the appropriate time, as devm does the shutdown in the
> exact opposite order this should fix your problem.
>
> So until all clocks could be enabled, you would disable them manually
> and after that rely on the devm_action to fire at the appropriate time.
>
> ret = clk_prepare_enable(clk1);
> if (ret < 0 )
> 	return ret;
>
> ret = clk_prepare_enable(clk2);
> if (ret < 0) {
> 	clk_disable_unprepare(clk1);
> 	return ret;
> }
>
> devm_add_action_or_reset(dev, rk_pcie_disable_clocks);

right, and we should also move request irq after this, so the devres 
core can release them in reverse.

and we should double check would those clks be the only thing that irq 
handler required...
>
> The rockchip crypto driver [0] shows how this could be done.
>
>
> Heiko
>
> [0]http://elixir.free-electrons.com/linux/latest/source/drivers/crypto/rockchip/rk3288_crypto.c#L307
>
Shawn Lin Aug. 10, 2017, 11:04 a.m. UTC | #6
Hi Heiko

On 2017/8/10 17:27, Heiko Stuebner wrote:
> Hi Shawn,
> 
> Am Donnerstag, 10. August 2017, 16:21:13 CEST schrieb Shawn Lin:
>> With CONFIG_DEBUG_SHIRQ enabled, the irq tear down routine
>> would still access the irq handler registed as a shard irq.
>> Per the comment within the function of __free_irq, it says
>> "It's a shared IRQ -- the driver ought to be prepared for
>> an IRQ event to happen even now it's being freed". However
>> when failing to probe the driver, it may disable the clock
>> for accessing the register and the following check for shared
>> irq state would call the irq handler which accesses the register
>> w/o the clk enabled. That will hang the system forever.
> 
> The key point would be to fix the ordering. So you could also just use
> devm_add_action to move the clock-disabling into a callback that
> gets run at the appropriate time, as devm does the shutdown in the
> exact opposite order this should fix your problem.
> 
> So until all clocks could be enabled, you would disable them manually
> and after that rely on the devm_action to fire at the appropriate time.
> 
> ret = clk_prepare_enable(clk1);
> if (ret < 0 )
> 	return ret;
> 
> ret = clk_prepare_enable(clk2);
> if (ret < 0) {
> 	clk_disable_unprepare(clk1);
> 	return ret;
> }
> 
> devm_add_action_or_reset(dev, rk_pcie_disable_clocks);
> 

That seems great! I will respin v2 for that.

> The rockchip crypto driver [0] shows how this could be done.
> 
> 
> Heiko
> 
> [0] http://elixir.free-electrons.com/linux/latest/source/drivers/crypto/rockchip/rk3288_crypto.c#L307
> 
>>
>> With adding some dump_stack we could see how that happened.
>>
>> calling  rockchip_pcie_driver_init+0x0/0x28 @ 1
>> rockchip-pcie f8000000.pcie: no vpcie3v3 regulator found
>> rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found
>> rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found
>> rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!
>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted
>> 4.13.0-rc3-next-20170807-ARCH+ #189
>> Hardware name: Firefly-RK3399 Board (DT)
>> Call trace:
>> [<ffff000008089bf0>] dump_backtrace+0x0/0x250
>> [<ffff000008089eb0>] show_stack+0x20/0x28
>> [<ffff000008c3313c>] dump_stack+0x90/0xb0
>> [<ffff000008632ad4>] rockchip_pcie_read.isra.11+0x54/0x58
>> [<ffff0000086334fc>] rockchip_pcie_client_irq_handler+0x30/0x1a0
>> [<ffff00000813ce98>] __free_irq+0x1c8/0x2dc
>> [<ffff00000813d044>] free_irq+0x44/0x74
>> [<ffff0000081415fc>] devm_irq_release+0x24/0x2c
>> [<ffff00000877429c>] release_nodes+0x1d8/0x30c
>> [<ffff000008774838>] devres_release_all+0x3c/0x5c
>> [<ffff00000876f19c>] driver_probe_device+0x244/0x494
>> [<ffff00000876f50c>] __driver_attach+0x120/0x124
>> [<ffff00000876cb80>] bus_for_each_dev+0x6c/0xac
>> [<ffff00000876e984>] driver_attach+0x2c/0x34
>> [<ffff00000876e3a4>] bus_add_driver+0x244/0x2b0
>> [<ffff000008770264>] driver_register+0x70/0x110
>> [<ffff0000087718b4>] platform_driver_register+0x60/0x6c
>> [<ffff0000091eb108>] rockchip_pcie_driver_init+0x20/0x28
>> [<ffff000008083a2c>] do_one_initcall+0xc8/0x130
>> [<ffff0000091a0ea8>] kernel_init_freeable+0x1a0/0x238
>> [<ffff000008c461cc>] kernel_init+0x18/0x108
>> [<ffff0000080836c0>] ret_from_fork+0x10/0x50
>>
>> In order to fix this, we check the clk state before accessing the
>> pcie register in rockchip_pcie_read, but don't touch rockchip_pcie_write
>> as no case to trigger that from write routine.
>>
>> Signed-off-by: Shawn Lin <shawn.lin@rock-chips.com>
>>
>> ---
>> Hi Bjorn, Thomas and Marc,
>>
>> This fix looks more like a hack, but I don't know the legit way to
>> deal with that case. Just quick look into the drivers/pci/host, and
>> I see almost all drivers register shard irq and also disable the clk
>> in their probe's error handle path. So I guess this is NOT pcie-rockchip
>> specified even not pcie host drivers specified, but folks just luckily didn't
>> trip over it.
>>
>> Could you give some suggestion on this?
>>
>>   drivers/pci/host/pcie-rockchip.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/pci/host/pcie-rockchip.c b/drivers/pci/host/pcie-rockchip.c
>> index 39aafe2..daaa868b 100644
>> --- a/drivers/pci/host/pcie-rockchip.c
>> +++ b/drivers/pci/host/pcie-rockchip.c
>> @@ -17,6 +17,7 @@
>>   
>>   #include <linux/bitrev.h>
>>   #include <linux/clk.h>
>> +#include <linux/clk-provider.h>
>>   #include <linux/delay.h>
>>   #include <linux/gpio/consumer.h>
>>   #include <linux/init.h>
>> @@ -252,6 +253,9 @@ struct rockchip_pcie {
>>   
>>   static u32 rockchip_pcie_read(struct rockchip_pcie *rockchip, u32 reg)
>>   {
>> +	if (!__clk_is_enabled(rockchip->hclk_pcie))
>> +		return 0;
>> +
>>   	return readl(rockchip->apb_base + reg);
>>   }
>>   
>>
> 
> 
> 
> 
>
diff mbox

Patch

diff --git a/drivers/pci/host/pcie-rockchip.c b/drivers/pci/host/pcie-rockchip.c
index 39aafe2..daaa868b 100644
--- a/drivers/pci/host/pcie-rockchip.c
+++ b/drivers/pci/host/pcie-rockchip.c
@@ -17,6 +17,7 @@ 
 
 #include <linux/bitrev.h>
 #include <linux/clk.h>
+#include <linux/clk-provider.h>
 #include <linux/delay.h>
 #include <linux/gpio/consumer.h>
 #include <linux/init.h>
@@ -252,6 +253,9 @@  struct rockchip_pcie {
 
 static u32 rockchip_pcie_read(struct rockchip_pcie *rockchip, u32 reg)
 {
+	if (!__clk_is_enabled(rockchip->hclk_pcie))
+		return 0;
+
 	return readl(rockchip->apb_base + reg);
 }