| Message ID | 20210702045120.22855-1-aaron.ma@canonical.com (mailing list archive) |
|---|---|
| State | Awaiting Upstream |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [1/2] igc: don't rd/wr iomem when PCI is removed | expand |
| Context | Check | Description |
|---|---|---|
| netdev/cover_letter | success | Link |
| netdev/fixes_present | success | Link |
| netdev/patch_count | success | Link |
| netdev/tree_selection | success | Guessed tree name to be net-next |
| netdev/subject_prefix | warning | Target tree name not specified in the subject |
| netdev/cc_maintainers | success | CCed 6 of 6 maintainers |
| netdev/source_inline | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Link |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/kdoc | success | Errors and warnings before: 0 this patch: 0 |
| netdev/verify_fixes | success | Link |
| netdev/checkpatch | warning | CHECK: Alignment should match open parenthesis |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/header_inline | success | Link |
+ Bjorn, Krzysztof and linux-pci On Friday 02 July 2021 12:51:19 Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) Hello! This code pattern and commit message looks like that we could use pci_dev_is_disconnected() helper function for checking if device is still connected or was disconnected. Apparently pci_dev_is_disconnected() is defined only in private header file drivers/pci/pci.h and not in public include/linux/pci.h. Aaron: can you check if pci_dev_is_disconnected() is really something which should be used and it helps you? Bjorn, Krzysztof: what do you think about lifting helper function pci_dev_is_disconnected() to be available to all drivers and not only in PCI subsystem? I think that such helper function makes driver code more readable and can be useful also for other drivers which are checking if return value is all F's. > + return 0; Aaron: should not you return all F's on error? Because few lines below in this function is returned value with all F's when PCIe link lost. > + > value = readl(&hw_addr[reg]); Anyway, this code looks to be racy. When pci_channel_io_perm_failure is set (e.g. by hotplug interrupt) after checking for pdev->error_state and prior executing above readl() then mentioned fatal error still occurs. > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
On 7/4/21 10:28 PM, Pali Rohár wrote: > + Bjorn, Krzysztof and linux-pci > > On Friday 02 July 2021 12:51:19 Aaron Ma wrote: >> Check PCI state when rd/wr iomem. >> Implement wr32 function as rd32 too. >> >> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: >> Trace: >> BUG: unable to handle page fault for address: 000000000000b604 >> Oops: 0000 [#1] SMP NOPTI >> RIP: 0010:igc_rd32+0x1c/0x90 [igc] >> Call Trace: >> igc_ptp_suspend+0x6c/0xa0 [igc] >> igc_ptp_stop+0x12/0x50 [igc] >> igc_remove+0x7f/0x1c0 [igc] >> pci_device_remove+0x3e/0xb0 >> __device_release_driver+0x181/0x240 >> >> Signed-off-by: Aaron Ma <aaron.ma@canonical.com> >> --- >> drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ >> drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- >> 2 files changed, 18 insertions(+), 5 deletions(-) >> >> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c >> index f1adf154ec4a..606b72cb6193 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_main.c >> +++ b/drivers/net/ethernet/intel/igc/igc_main.c >> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> u32 value = 0; >> >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) > > Hello! This code pattern and commit message looks like that we could use > pci_dev_is_disconnected() helper function for checking if device is > still connected or was disconnected. > > Apparently pci_dev_is_disconnected() is defined only in private header > file drivers/pci/pci.h and not in public include/linux/pci.h. > > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? > Hi Pali, How about using pci_channel_offline instead? It's ready and also safe for frozen state, and verified on hw. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? > > I think that such helper function makes driver code more readable and > can be useful also for other drivers which are checking if return value > is all F's. > >> + return 0; > > Aaron: should not you return all F's on error? Because few lines below > in this function is returned value with all F's when PCIe link lost. > If you agree with the above change, I can fix it to "return -1" in v2. Thanks for your comments, Aaron >> + >> value = readl(&hw_addr[reg]); > > Anyway, this code looks to be racy. When pci_channel_io_perm_failure is > set (e.g. by hotplug interrupt) after checking for pdev->error_state and > prior executing above readl() then mentioned fatal error still occurs. > >> >> /* reads should not return all F's */ >> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> return value; >> } >> >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) >> +{ >> + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); >> + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> + >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) >> + return; >> + >> + writel((val), &hw_addr[(reg)]); >> +} >> + >> int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) >> { >> struct igc_mac_info *mac = &adapter->hw.mac; >> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h >> index cc174853554b..eb4be87d0e8b 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_regs.h >> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h >> @@ -260,13 +260,10 @@ struct igc_hw; >> u32 igc_rd32(struct igc_hw *hw, u32 reg); >> >> /* write operations, indexed using DWORDS */ >> -#define wr32(reg, val) \ >> -do { \ >> - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ >> - writel((val), &hw_addr[(reg)]); \ >> -} while (0) >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); >> >> #define rd32(reg) (igc_rd32(hw, reg)) >> +#define wr32(reg, val) (igc_wr32(hw, reg, val)) >> >> #define wrfl() ((void)rd32(IGC_STATUS)) >> >> -- >> 2.30.2 >>
On Fri, 2 Jul 2021 at 14:53, Aaron Ma <aaron.ma@canonical.com> wrote: > > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- Drive-by, but won't this add a lot of overhead on every register access? has this been benchmarked with lots of small network transfers or anything? Dave. > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return 0; > + > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
Hi Pali, [...] > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? While having a closer look, I've noticed that quite a few of the network drivers handle this somewhat, as I see that a lot of them have some sort of I/O error handles set where a check for "pci_channel_io_perm_failure" seem to be having place. This is also true for this driver looking at the igc_io_error_detected(). Is this not working for the igc driver? Or is this for something completely different? Having said all that, I am not an expert in network drivers, so pardon me if I am asking about something completely different, and I apologise if that is the case. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? No objections from me, if we believe it's useful and that it might encourage people to use a common API. Currently, I can see at least five potential users of this helper. Krzysztof
On 7/5/21 3:47 PM, Dave Airlie wrote: > Drive-by, but won't this add a lot of overhead on every register > access? has this been benchmarked with lots of small network transfers > or anything? > iperf3 is tested, the result is the same as before. Due to the registers are rd/wr even after error_handler and remove. Didn't find better fix. Please let me know if you have any idea. Thanks, Aaron > Dave.
On Tuesday 06 July 2021 01:02:12 Krzysztof Wilczyński wrote: > Hi Pali, > > [...] > > Aaron: can you check if pci_dev_is_disconnected() is really something > > which should be used and it helps you? > > While having a closer look, I've noticed that quite a few of the network > drivers handle this somewhat, as I see that a lot of them have some sort > of I/O error handles set where a check for "pci_channel_io_perm_failure" > seem to be having place. This is also true for this driver looking at > the igc_io_error_detected(). > > Is this not working for the igc driver? Or is this for something > completely different? I guess that this callback is called when Bridge receive some kind of fatal error. Non-AER-aware bridges probably do not have to inform system that error happened and kernel would not call this callback. So I guess it depends on to which "machine" you need this network adapter. So in my opinion this callback is there for PCI subsystem to inform driver that error happened and let driver to do any hw specific recovery if it is possible. But I think problem described here can be slightly different. It is needed to check if device is still alive or was disconnected. > Having said all that, I am not an expert in network drivers, so pardon > me if I am asking about something completely different, and I apologise > if that is the case. > > > Bjorn, Krzysztof: what do you think about lifting helper function > > pci_dev_is_disconnected() to be available to all drivers and not only in > > PCI subsystem? > > No objections from me, if we believe it's useful and that it might > encourage people to use a common API. Currently, I can see at least > five potential users of this helper. > > Krzysztof
On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return 0; I don't think this solves the problem. - Driver calls igc_rd32(). - "if (pci_channel_io_perm_failure)" evaluates to false (error_state does not indicate an error). - Device is unplugged. - igc_rd32() calls readl(), which performs MMIO read, which fails because the device is no longer present. readl() returns ~0 on most platforms. - Same page fault occurs. The only way is to check *after* the MMIO read to see whether an error occurred. On most platforms that means checking for ~0 data. If you see that, a PCI error *may* have occurred. If you know that ~0 can never be valid, e.g., if you're reading a register where ~0 is not a valid value, you know for sure that an error has occurred. If ~0 might be a valid value, e.g., if you're reading a buffer that contains arbitrary data, you have to look harder. You might read a register than cannot contain ~0, and see if you get the data you expect. Or you might read the Vendor ID or something from config space. > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > Check PCI state when rd/wr iomem. > > Implement wr32 function as rd32 too. > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > Trace: > > BUG: unable to handle page fault for address: 000000000000b604 > > Oops: 0000 [#1] SMP NOPTI > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > Call Trace: > > igc_ptp_suspend+0x6c/0xa0 [igc] > > igc_ptp_stop+0x12/0x50 [igc] > > igc_remove+0x7f/0x1c0 [igc] > > pci_device_remove+0x3e/0xb0 > > __device_release_driver+0x181/0x240 > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > --- > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > index f1adf154ec4a..606b72cb6193 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > u32 value = 0; > > > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return 0; > > I don't think this solves the problem. > > - Driver calls igc_rd32(). > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > does not indicate an error). > > - Device is unplugged. > > - igc_rd32() calls readl(), which performs MMIO read, which fails > because the device is no longer present. readl() returns ~0 on > most platforms. > > - Same page fault occurs. Hi Bjorn! I think that backtrace show that this error happens when PCIe hotplug get interrupt that device was unplugged and PCIe hotplug code calls remove/unbind procedure to stop unplugged driver. And in this case really does not make sense to try issuing MMIO read, device is already unplugged. I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when this unplug interrupt happens and pci_dev_set_disconnected() just sets pci_channel_io_perm_failure flag. drivers/pci/pci.h provides function pci_dev_is_disconnected() which checks if that flag pci_channel_io_perm_failure is set. So I think that pci_dev_is_disconnected() is useful and could be exported also to drivers (like this one) so they can check if pci_dev_set_disconnected() was called in past and PCI driver is now in unbind/cleanup/remove state because PCIe device is already disconnected and not accessible anymore. But maybe this check should be on other place in driver unbound procedure and not in general MMIO read function? > The only way is to check *after* the MMIO read to see whether an error > occurred. On most platforms that means checking for ~0 data. If you > see that, a PCI error *may* have occurred. > > If you know that ~0 can never be valid, e.g., if you're reading a > register where ~0 is not a valid value, you know for sure that an > error has occurred. > > If ~0 might be a valid value, e.g., if you're reading a buffer that > contains arbitrary data, you have to look harder. You might read a > register than cannot contain ~0, and see if you get the data you > expect. Or you might read the Vendor ID or something from config > space. > > > value = readl(&hw_addr[reg]); > > > > /* reads should not return all F's */ > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > return value; > > } > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > +{ > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > + > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return; > > + > > + writel((val), &hw_addr[(reg)]); > > +} > > + > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > { > > struct igc_mac_info *mac = &adapter->hw.mac; > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > index cc174853554b..eb4be87d0e8b 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > @@ -260,13 +260,10 @@ struct igc_hw; > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > /* write operations, indexed using DWORDS */ > > -#define wr32(reg, val) \ > > -do { \ > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > - writel((val), &hw_addr[(reg)]); \ > > -} while (0) > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > -- > > 2.30.2 > >
On Wed, Jul 07, 2021 at 11:53:37PM +0200, Pali Rohár wrote: > On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > > Check PCI state when rd/wr iomem. > > > Implement wr32 function as rd32 too. > > > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > > Trace: > > > BUG: unable to handle page fault for address: 000000000000b604 > > > Oops: 0000 [#1] SMP NOPTI > > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > > Call Trace: > > > igc_ptp_suspend+0x6c/0xa0 [igc] > > > igc_ptp_stop+0x12/0x50 [igc] > > > igc_remove+0x7f/0x1c0 [igc] > > > pci_device_remove+0x3e/0xb0 > > > __device_release_driver+0x181/0x240 > > > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > > --- > > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > > index f1adf154ec4a..606b72cb6193 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > u32 value = 0; > > > > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return 0; > > > > I don't think this solves the problem. > > > > - Driver calls igc_rd32(). > > > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > > does not indicate an error). > > > > - Device is unplugged. > > > > - igc_rd32() calls readl(), which performs MMIO read, which fails > > because the device is no longer present. readl() returns ~0 on > > most platforms. > > > > - Same page fault occurs. > > Hi Bjorn! I think that backtrace show that this error happens when PCIe > hotplug get interrupt that device was unplugged and PCIe hotplug code > calls remove/unbind procedure to stop unplugged driver. > > And in this case really does not make sense to try issuing MMIO read, > device is already unplugged. > > I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when > this unplug interrupt happens and pci_dev_set_disconnected() just sets > pci_channel_io_perm_failure flag. > > drivers/pci/pci.h provides function pci_dev_is_disconnected() which > checks if that flag pci_channel_io_perm_failure is set. > > So I think that pci_dev_is_disconnected() is useful and could be > exported also to drivers (like this one) so they can check if > pci_dev_set_disconnected() was called in past and PCI driver is now in > unbind/cleanup/remove state because PCIe device is already disconnected > and not accessible anymore. > > But maybe this check should be on other place in driver unbound > procedure and not in general MMIO read function? If we add the check as proposed in this patch, I think people will read it and think this is the correct way to avoid MMIO errors. It does happen to avoid some MMIO errors, but it cannot avoid them all, so it's not a complete solution and it gives a false sense of security. A complete solution requires a test *after* the MMIO read. If you have the test after the read, you don't really need one before. Sure, testing before means you can avoid one MMIO read failure in some cases. But avoiding that failure costs quite a lot in code clutter. > > The only way is to check *after* the MMIO read to see whether an error > > occurred. On most platforms that means checking for ~0 data. If you > > see that, a PCI error *may* have occurred. > > > > If you know that ~0 can never be valid, e.g., if you're reading a > > register where ~0 is not a valid value, you know for sure that an > > error has occurred. > > > > If ~0 might be a valid value, e.g., if you're reading a buffer that > > contains arbitrary data, you have to look harder. You might read a > > register than cannot contain ~0, and see if you get the data you > > expect. Or you might read the Vendor ID or something from config > > space. > > > > > value = readl(&hw_addr[reg]); > > > > > > /* reads should not return all F's */ > > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > return value; > > > } > > > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > > +{ > > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > + > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return; > > > + > > > + writel((val), &hw_addr[(reg)]); > > > +} > > > + > > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > > { > > > struct igc_mac_info *mac = &adapter->hw.mac; > > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > > index cc174853554b..eb4be87d0e8b 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > > @@ -260,13 +260,10 @@ struct igc_hw; > > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > > > /* write operations, indexed using DWORDS */ > > > -#define wr32(reg, val) \ > > > -do { \ > > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > > - writel((val), &hw_addr[(reg)]); \ > > > -} while (0) > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > > > -- > > > 2.30.2 > > >
On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > If we add the check as proposed in this patch, I think people will > read it and think this is the correct way to avoid MMIO errors. It > does happen to avoid some MMIO errors, but it cannot avoid them all, > so it's not a complete solution and it gives a false sense of > security. I think it's helpful to classify MMIO errors as either benign or poisonous with the poison MMIOs causing some kind of crash. Most of the discussions about pci_dev_is_disconnected(), including this one, seem to stem from people trying to use it to avoid the poison case. I agree that using pci_dev_is_disconnected() that way is hacky and doesn't really fix the problem, but considering poison MMIOs usually stem from broken hardware or firmware maybe we should allow it anyway. We can't do anything better and it's an improvement compared to crashing. > A complete solution requires a test *after* the MMIO read. If you > have the test after the read, you don't really need one before. Sure, > testing before means you can avoid one MMIO read failure in some > cases. But avoiding that failure costs quite a lot in code clutter. It's not that much clutter if the checks are buried in the MMIO helpers which most drivers define. Speaking of which: > u32 igc_rd32(struct igc_hw *hw, u32 reg) > { > struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > if (!(~value) && (!reg || !(~readl(hw_addr)))) { > struct net_device *netdev = igc->netdev; > > hw->hw_addr = NULL; > netif_device_detach(netdev); > netdev_err(netdev, "PCIe link lost, device now detached\n"); > WARN(pci_device_is_present(igc->pdev), > "igc: Failed to read reg 0x%x!\n", reg); > } > > return value; > } I think I found where that page fault is coming from. I wonder if we should provide drivers some way of invoking the error recovery mechanisms manually or even just flagging itself as broken. Right now even if the driver bothers with synchronous error detection the driver can't really do anything other than parking itself and hoping AER/EEH recovery kicks in. Oliver
On Thu, Jul 08, 2021 at 12:04:02PM +1000, Oliver O'Halloran wrote: > On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > If we add the check as proposed in this patch, I think people will > > read it and think this is the correct way to avoid MMIO errors. It > > does happen to avoid some MMIO errors, but it cannot avoid them all, > > so it's not a complete solution and it gives a false sense of > > security. > > I think it's helpful to classify MMIO errors as either benign or > poisonous with the poison MMIOs causing some kind of crash. Most of > the discussions about pci_dev_is_disconnected(), including this one, > seem to stem from people trying to use it to avoid the poison case. I > agree that using pci_dev_is_disconnected() that way is hacky and > doesn't really fix the problem, but considering poison MMIOs usually > stem from broken hardware or firmware maybe we should allow it > anyway. We can't do anything better and it's an improvement compared > to crashing. Apologies for rehashing what's probably obvious to everybody but me. I'm trying to get a better handle on benign vs poisonous errors. MMIO means CPU reads or writes to the device. In PCI, writes are posted and don't receive a response, so a driver will never see writel() return an error (although an error may be reported asynchronously via AER or similar). So I think we're mostly talking about CPU reads here. We expect a PCI response containing the data. Sometimes there's no response or an error response. The behavior of the host bridge in these error cases is not defined by PCI, so what the CPU sees is not consistent across platforms. In some cases, the bridge handles this as a catastrophic error that forces a system restart. But in most cases, at least on x86, the bridge logs an error and fabricates ~0 data so the CPU read can complete. Then it's up to software to recognize that an error occurred and decide what to do about it. Is this a benign or a poisonous error? I'd say this is a benign error. It certainly can't be ignored, but as long as the driver recognizes the error, it should be able to deal with it without crashing the whole system and forcing a restart. Bjorn
On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > *snip* > > Apologies for rehashing what's probably obvious to everybody but me. > I'm trying to get a better handle on benign vs poisonous errors. > > MMIO means CPU reads or writes to the device. In PCI, writes are > posted and don't receive a response, so a driver will never see > writel() return an error (although an error may be reported > asynchronously via AER or similar). > > So I think we're mostly talking about CPU reads here. We expect a PCI > response containing the data. Sometimes there's no response or an > error response. The behavior of the host bridge in these error cases > is not defined by PCI, so what the CPU sees is not consistent across > platforms. In some cases, the bridge handles this as a catastrophic > error that forces a system restart. > > But in most cases, at least on x86, the bridge logs an error and > fabricates ~0 data so the CPU read can complete. Then it's up to > software to recognize that an error occurred and decide what to do > about it. Is this a benign or a poisonous error? > > I'd say this is a benign error. It certainly can't be ignored, but as > long as the driver recognizes the error, it should be able to deal > with it without crashing the whole system and forcing a restart. I was thinking more in terms of what the driver author sees rather than what's happening on the CPU side. The crash seen in the OP appears to be because the code is "doing an MMIO." However, the reasons for the crash have nothing to do with the actual mechanics of the operation (which should be benign). The point I was making is that the pattern of: if (is_disconnected()) return failure; return do_mmio_read(addr); does have some utility as a last-ditch attempt to prevent crashes in the face of obnoxious bridges or bad hardware. Granted, that should be a platform concern rather than something that should ever appear in driver code, but considering drivers open-code readl()/writel() calls there's not really any place to put that sort of workaround. That all said, the case in the OP is due to an entirely avoidable driver bug and that sort of hack is absolutely the wrong thing to do. Oliver
On Monday 19 July 2021 02:31:10 Oliver O'Halloran wrote: > On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > *snip* > > > > Apologies for rehashing what's probably obvious to everybody but me. > > I'm trying to get a better handle on benign vs poisonous errors. > > > > MMIO means CPU reads or writes to the device. In PCI, writes are > > posted and don't receive a response, so a driver will never see > > writel() return an error (although an error may be reported > > asynchronously via AER or similar). > > > > So I think we're mostly talking about CPU reads here. We expect a PCI > > response containing the data. Sometimes there's no response or an > > error response. The behavior of the host bridge in these error cases > > is not defined by PCI, so what the CPU sees is not consistent across > > platforms. In some cases, the bridge handles this as a catastrophic > > error that forces a system restart. > > > > But in most cases, at least on x86, the bridge logs an error and > > fabricates ~0 data so the CPU read can complete. Then it's up to > > software to recognize that an error occurred and decide what to do > > about it. Is this a benign or a poisonous error? > > > > I'd say this is a benign error. It certainly can't be ignored, but as > > long as the driver recognizes the error, it should be able to deal > > with it without crashing the whole system and forcing a restart. > > I was thinking more in terms of what the driver author sees rather > than what's happening on the CPU side. The crash seen in the OP > appears to be because the code is "doing an MMIO." However, the > reasons for the crash have nothing to do with the actual mechanics of > the operation (which should be benign). The point I was making is that > the pattern of: > > if (is_disconnected()) > return failure; > return do_mmio_read(addr); > > does have some utility as a last-ditch attempt to prevent crashes in > the face of obnoxious bridges or bad hardware. Granted, that should be > a platform concern rather than something that should ever appear in > driver code, but considering drivers open-code readl()/writel() calls > there's not really any place to put that sort of workaround. > > That all said, the case in the OP is due to an entirely avoidable > driver bug and that sort of hack is absolutely the wrong thing to do. > > Oliver And do we have some solution for this kind of issue? There are more PCIe controllers / platforms which do not like MMIO read/write operation when card / link is not connected. If we do not provide a way how to solve these problems then we can expect that people would just hack ethernet / wifi / ... device drivers which are currently crashing by patches like in this thread. Maybe PCI subsystem could provide wrapper function which implements above pattern and which can be used by device drivers?
On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > And do we have some solution for this kind of issue? There are more PCIe > controllers / platforms which do not like MMIO read/write operation when > card / link is not connected. Do you have some actual examples? The few times I've seen those crashes were due to broken firmware-first error handling. The AER notifications would be escalated into some kind of ACPI error which the kernel didn't have a good way of dealing with so it panicked instead. Assuming it is a real problem then as Bjorn pointed out this sort of hack doesn't really fix the issue because hotplug and AER notifications are fundamentally asynchronous. If the driver is actively using the device when the error / removal happens then the pci_dev_is_disconnected() check will pass and the MMIO will go through. If the MMIO is poisonous because of dumb hardware then this sort of hack will only paper over the issue. > If we do not provide a way how to solve these problems then we can > expect that people would just hack ethernet / wifi / ... device drivers > which are currently crashing by patches like in this thread. > > Maybe PCI subsystem could provide wrapper function which implements > above pattern and which can be used by device drivers? We could do that and I think there was a proposal to add some pci_readl(pdev, <addr>) style wrappers at one point. On powerpc there's hooks in the arch provided MMIO functions to detect error responses and kick off the error handling machinery when a problem is detected. Those hooks are mainly there to help the platform detect errors though and they don't make life much easier for drivers. Due to locking concerns the driver's .error_detected() callback cannot be called in the MMIO hook so even when the platform detects errors synchronously the driver notifications must happen asynchronously. In the meanwhile the driver still needs to handle the 0xFFs response safely and there's not much we can do from the platform side to help there. Oliver
On Monday 19 July 2021 12:49:18 Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. I have experience and examples with pci aardvark controller. When card is disconnected it sends synchronous abort to CPU when doing MMIO read operation. One example is in this linux-usb thread: https://lore.kernel.org/linux-usb/20210505120117.4wpmo6fhvzznf3wv@pali/t/#u I can trigger this issue at least for xhci, nvme and ath drivers. > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. In case of pci aardvark it is not AER notification. And for MMIO read it is synchronous abort. Anyway, hotplug events are really asynchronous, but there is main issue that this hotplug disconnect event instruct device driver to "unbind" and e.g. these ethernet or usb controllers try to do MMIO operations in their cleanup / remove / unbind phase, even when card is already "disconnected" in PCI subsystem. > If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver
On Mon, Jul 19, 2021 at 12:49:18PM +1000, Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. > > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. Obviously this wouldn't help user-space mmaps, but in the kernel, Documentation/driver-api/device-io.rst [1] does say that drivers are supposed to use readl() et al even though on most arches it "works" to just dereference the result of ioremap(), so maybe we could make a useful wrapper. Seems like we should do *something*, even if it's just a generic #define and some examples. I took a stab at this [2] a couple years ago, but it was only for the PCI core, and it didn't go anywhere. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/device-io.rst?id=v5.13#n160 [2] https://lore.kernel.org/linux-pci/20190822200551.129039-1-helgaas@kernel.org/ > On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index f1adf154ec4a..606b72cb6193 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); u32 value = 0; + if (igc->pdev && + igc->pdev->error_state == pci_channel_io_perm_failure) + return 0; + value = readl(&hw_addr[reg]); /* reads should not return all F's */ @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) return value; } +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) +{ + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); + + if (igc->pdev && + igc->pdev->error_state == pci_channel_io_perm_failure) + return; + + writel((val), &hw_addr[(reg)]); +} + int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) { struct igc_mac_info *mac = &adapter->hw.mac; diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h index cc174853554b..eb4be87d0e8b 100644 --- a/drivers/net/ethernet/intel/igc/igc_regs.h +++ b/drivers/net/ethernet/intel/igc/igc_regs.h @@ -260,13 +260,10 @@ struct igc_hw; u32 igc_rd32(struct igc_hw *hw, u32 reg); /* write operations, indexed using DWORDS */ -#define wr32(reg, val) \ -do { \ - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ - writel((val), &hw_addr[(reg)]); \ -} while (0) +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); #define rd32(reg) (igc_rd32(hw, reg)) +#define wr32(reg, val) (igc_wr32(hw, reg, val)) #define wrfl() ((void)rd32(IGC_STATUS))
Check PCI state when rd/wr iomem. Implement wr32 function as rd32 too. When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: Trace: BUG: unable to handle page fault for address: 000000000000b604 Oops: 0000 [#1] SMP NOPTI RIP: 0010:igc_rd32+0x1c/0x90 [igc] Call Trace: igc_ptp_suspend+0x6c/0xa0 [igc] igc_ptp_stop+0x12/0x50 [igc] igc_remove+0x7f/0x1c0 [igc] pci_device_remove+0x3e/0xb0 __device_release_driver+0x181/0x240 Signed-off-by: Aaron Ma <aaron.ma@canonical.com> --- drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- 2 files changed, 18 insertions(+), 5 deletions(-)