| Message ID | 20230824044645.423378-1-kai.heng.feng@canonical.com (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | [1/3] PCI: Add helper to check if any of ancestor device support D3cold | expand |
Hi, On Thu, Aug 24, 2023 at 12:46:43PM +0800, Kai-Heng Feng wrote: > In addition to nearest upstream bridge, driver may want to know if the > entire hierarchy can be powered off to perform different action. > > So walk higher up the hierarchy to find out if any device has valid > _PR3. I'm not entirely sure this is good idea. The drivers should expect that the power will be turned off pretty soon after device enters D3hot. Also _PR3 is not PCI concept it's ACPI concept so API like this would only work on systems with ACPI.
Hi, On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg <mika.westerberg@linux.intel.com> wrote: > > Hi, > > On Thu, Aug 24, 2023 at 12:46:43PM +0800, Kai-Heng Feng wrote: > > In addition to nearest upstream bridge, driver may want to know if the > > entire hierarchy can be powered off to perform different action. > > > > So walk higher up the hierarchy to find out if any device has valid > > _PR3. > > I'm not entirely sure this is good idea. The drivers should expect that > the power will be turned off pretty soon after device enters D3hot. Also > _PR3 is not PCI concept it's ACPI concept so API like this would only > work on systems with ACPI. IIUC, Bjorn wants to limit the AER/DPC disablement when device power is really off. Is "the power will be turned off pretty soon after device enters D3hot" applicable to most devices? Since config space is still accessible when device is in D3hot. Unless there are cases when device firmware behave differently to D3hot? Then maybe it's better to disable AER for both D3hot, D3cold and system S3. Kai-Heng
On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > Hi, > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > <mika.westerberg@linux.intel.com> wrote: > > > > Hi, > > > > On Thu, Aug 24, 2023 at 12:46:43PM +0800, Kai-Heng Feng wrote: > > > In addition to nearest upstream bridge, driver may want to know if the > > > entire hierarchy can be powered off to perform different action. > > > > > > So walk higher up the hierarchy to find out if any device has valid > > > _PR3. > > > > I'm not entirely sure this is good idea. The drivers should expect that > > the power will be turned off pretty soon after device enters D3hot. Also > > _PR3 is not PCI concept it's ACPI concept so API like this would only > > work on systems with ACPI. > > IIUC, Bjorn wants to limit the AER/DPC disablement when device power > is really off. > Is "the power will be turned off pretty soon after device enters > D3hot" applicable to most devices? Since config space is still > accessible when device is in D3hot. Well the device may be part of a topology, say Thunderbolt/USB4 (but can be NVMe or similar) where it initially goes into D3hot but in the end the whole topology is put into D3cold. The device driver really should expect that this happens always and not try to distinguish between the D3hot or D3cold. > Unless there are cases when device firmware behave differently to > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > and system S3. Yes, this makes sense.
On Fri, Aug 25, 2023 at 1:29 PM Mika Westerberg <mika.westerberg@linux.intel.com> wrote: > > On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > > Hi, > > > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > > <mika.westerberg@linux.intel.com> wrote: > > > > > > Hi, > > > > > > On Thu, Aug 24, 2023 at 12:46:43PM +0800, Kai-Heng Feng wrote: > > > > In addition to nearest upstream bridge, driver may want to know if the > > > > entire hierarchy can be powered off to perform different action. > > > > > > > > So walk higher up the hierarchy to find out if any device has valid > > > > _PR3. > > > > > > I'm not entirely sure this is good idea. The drivers should expect that > > > the power will be turned off pretty soon after device enters D3hot. Also > > > _PR3 is not PCI concept it's ACPI concept so API like this would only > > > work on systems with ACPI. > > > > IIUC, Bjorn wants to limit the AER/DPC disablement when device power > > is really off. > > Is "the power will be turned off pretty soon after device enters > > D3hot" applicable to most devices? Since config space is still > > accessible when device is in D3hot. > > Well the device may be part of a topology, say Thunderbolt/USB4 (but can > be NVMe or similar) where it initially goes into D3hot but in the end > the whole topology is put into D3cold. The device driver really should > expect that this happens always and not try to distinguish between the > D3hot or D3cold. What if the device is not in such topology? There are cases that the rootport doesn't have Power Resources associated so the rootport also stays in D3hot. I think what Bjorn suggested is to keep AER enabled for D3hot, and only disable it for D3cold and S3. > > > Unless there are cases when device firmware behave differently to > > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > > and system S3. > > Yes, this makes sense. I agree that differentiate between D3hot and D3cold unnecessarily make things more complicated, but Bjorn suggested errors reported by AER under D3hot should still be recorded. Do you have more compelling data to persuade Bjorn that AER should be disabled for both D3 states? Kai-Heng
On Fri, Aug 25, 2023 at 01:43:08PM +0800, Kai-Heng Feng wrote: > On Fri, Aug 25, 2023 at 1:29 PM Mika Westerberg > <mika.westerberg@linux.intel.com> wrote: > > > > On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > > > Hi, > > > > > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > > > <mika.westerberg@linux.intel.com> wrote: > > > > > > > > Hi, > > > > > > > > On Thu, Aug 24, 2023 at 12:46:43PM +0800, Kai-Heng Feng wrote: > > > > > In addition to nearest upstream bridge, driver may want to know if the > > > > > entire hierarchy can be powered off to perform different action. > > > > > > > > > > So walk higher up the hierarchy to find out if any device has valid > > > > > _PR3. > > > > > > > > I'm not entirely sure this is good idea. The drivers should expect that > > > > the power will be turned off pretty soon after device enters D3hot. Also > > > > _PR3 is not PCI concept it's ACPI concept so API like this would only > > > > work on systems with ACPI. > > > > > > IIUC, Bjorn wants to limit the AER/DPC disablement when device power > > > is really off. > > > Is "the power will be turned off pretty soon after device enters > > > D3hot" applicable to most devices? Since config space is still > > > accessible when device is in D3hot. > > > > Well the device may be part of a topology, say Thunderbolt/USB4 (but can > > be NVMe or similar) where it initially goes into D3hot but in the end > > the whole topology is put into D3cold. The device driver really should > > expect that this happens always and not try to distinguish between the > > D3hot or D3cold. > > What if the device is not in such topology? There are cases that the > rootport doesn't have Power Resources associated so the rootport also > stays in D3hot. Yes and this is why the driver should not care. Otherwise it just complicates things. For instance, a root port may have _PR3 that allows the whole thing to enter L2/3 (D3cold) but some device has d3cold_allowed set to false or there may be a device that does not support PME on D3cold which prevents the power to be turned off. > I think what Bjorn suggested is to keep AER enabled for D3hot, and > only disable it for D3cold and S3. > > > > > > Unless there are cases when device firmware behave differently to > > > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > > > and system S3. > > > > Yes, this makes sense. > > I agree that differentiate between D3hot and D3cold unnecessarily make > things more complicated, but Bjorn suggested errors reported by AER > under D3hot should still be recorded. > Do you have more compelling data to persuade Bjorn that AER should be > disabled for both D3 states? Is there even an AER error that can happen when a device is in D3hot (link is in L1) or D3cold (link is in L2/3)? I'm not an expert in AER but AFAICT these errors are reported when the device is in active state not when it is in low power state. All that said, Bjorn is the maintainer so he gets to decide. If this is what he suggested then I'm fine with it. Just wanted to mention this. I too want to see this issue fixed, one way or another ;-) Thanks for working on this BTW!
On Fri, Aug 25, 2023 at 09:39:48AM +0300, Mika Westerberg wrote: > On Fri, Aug 25, 2023 at 01:43:08PM +0800, Kai-Heng Feng wrote: > > On Fri, Aug 25, 2023 at 1:29 PM Mika Westerberg > > <mika.westerberg@linux.intel.com> wrote: > > > On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > > > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > > > > <mika.westerberg@linux.intel.com> wrote: > > I think what Bjorn suggested is to keep AER enabled for D3hot, and > > only disable it for D3cold and S3. > > > > > > Unless there are cases when device firmware behave differently to > > > > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > > > > and system S3. > > > > > > Yes, this makes sense. > > > > I agree that differentiate between D3hot and D3cold unnecessarily make > > things more complicated, but Bjorn suggested errors reported by AER > > under D3hot should still be recorded. > > Do you have more compelling data to persuade Bjorn that AER should be > > disabled for both D3 states? > > Is there even an AER error that can happen when a device is in D3hot > (link is in L1) or D3cold (link is in L2/3)? I'm not an expert in AER > but AFAICT these errors are reported when the device is in active state > not when it is in low power state. I don't think a device in D3cold can signal its own errors. But the link transition to L2/L3 as a device goes to D3cold may cause the bridge above to log an error. And of course a config access to a device in D3cold will probably result in an Unsupported Request being logged by the bridge above it. I think these are the sorts of errors we do need to avoid or ignore somehow. But Configuration and Message requests definitely happen in D3hot, and they can cause errors reported via AER. The spec (r6.0, sec 2.2.8) recommends that Messages be handled the same in D0-D3hot. PTM is an example of where we had errors being reported at suspend/ resume because we had it configured incorrectly. If we disabled AER in D3hot we might not learn about that kind of configuration problem. That's what makes me think there's some value in keeping AER enabled in D3hot. Bjorn
On Sat, Aug 26, 2023 at 9:11 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Fri, Aug 25, 2023 at 09:39:48AM +0300, Mika Westerberg wrote: > > On Fri, Aug 25, 2023 at 01:43:08PM +0800, Kai-Heng Feng wrote: > > > On Fri, Aug 25, 2023 at 1:29 PM Mika Westerberg > > > <mika.westerberg@linux.intel.com> wrote: > > > > On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > > > > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > > > > > <mika.westerberg@linux.intel.com> wrote: > > > > I think what Bjorn suggested is to keep AER enabled for D3hot, and > > > only disable it for D3cold and S3. > > > > > > > > Unless there are cases when device firmware behave differently to > > > > > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > > > > > and system S3. > > > > > > > > Yes, this makes sense. > > > > > > I agree that differentiate between D3hot and D3cold unnecessarily make > > > things more complicated, but Bjorn suggested errors reported by AER > > > under D3hot should still be recorded. > > > Do you have more compelling data to persuade Bjorn that AER should be > > > disabled for both D3 states? > > > > Is there even an AER error that can happen when a device is in D3hot > > (link is in L1) or D3cold (link is in L2/3)? I'm not an expert in AER > > but AFAICT these errors are reported when the device is in active state > > not when it is in low power state. > > I don't think a device in D3cold can signal its own errors. But the > link transition to L2/L3 as a device goes to D3cold may cause the > bridge above to log an error. And of course a config access to a > device in D3cold will probably result in an Unsupported Request being > logged by the bridge above it. I think these are the sorts of errors > we do need to avoid or ignore somehow. In addition to that, we can't really control what device behaves during the D3hot (L2) transition. The kernel can't control what the firmware on the device may respond. > > But Configuration and Message requests definitely happen in D3hot, and > they can cause errors reported via AER. The spec (r6.0, sec 2.2.8) > recommends that Messages be handled the same in D0-D3hot. > > PTM is an example of where we had errors being reported at suspend/ > resume because we had it configured incorrectly. If we disabled AER > in D3hot we might not learn about that kind of configuration problem. > That's what makes me think there's some value in keeping AER enabled > in D3hot. In this particular case, the firmware of the device gets power cycled and starts sending PTM because of that. For this case, we want to know the error happens, but in the meantime there's nothing much can be done. So is it reasonable to log Corrected Errors, but skipping the AER reset? Kai-Heng > > Bjorn
On Mon, Aug 28, 2023 at 03:29:08PM +0800, Kai-Heng Feng wrote: > On Sat, Aug 26, 2023 at 9:11 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Fri, Aug 25, 2023 at 09:39:48AM +0300, Mika Westerberg wrote: > > > On Fri, Aug 25, 2023 at 01:43:08PM +0800, Kai-Heng Feng wrote: > > > > On Fri, Aug 25, 2023 at 1:29 PM Mika Westerberg > > > > <mika.westerberg@linux.intel.com> wrote: > > > > > On Thu, Aug 24, 2023 at 09:46:00PM +0800, Kai-Heng Feng wrote: > > > > > > On Thu, Aug 24, 2023 at 7:57 PM Mika Westerberg > > > > > > <mika.westerberg@linux.intel.com> wrote: > > > > > > I think what Bjorn suggested is to keep AER enabled for D3hot, and > > > > only disable it for D3cold and S3. > > > > > > > > > > Unless there are cases when device firmware behave differently to > > > > > > D3hot? Then maybe it's better to disable AER for both D3hot, D3cold > > > > > > and system S3. > > > > > > > > > > Yes, this makes sense. > > > > > > > > I agree that differentiate between D3hot and D3cold unnecessarily make > > > > things more complicated, but Bjorn suggested errors reported by AER > > > > under D3hot should still be recorded. > > > > Do you have more compelling data to persuade Bjorn that AER should be > > > > disabled for both D3 states? > > > > > > Is there even an AER error that can happen when a device is in D3hot > > > (link is in L1) or D3cold (link is in L2/3)? I'm not an expert in AER > > > but AFAICT these errors are reported when the device is in active state > > > not when it is in low power state. > > > > I don't think a device in D3cold can signal its own errors. But the > > link transition to L2/L3 as a device goes to D3cold may cause the > > bridge above to log an error. And of course a config access to a > > device in D3cold will probably result in an Unsupported Request being > > logged by the bridge above it. I think these are the sorts of errors > > we do need to avoid or ignore somehow. > > In addition to that, we can't really control what device behaves > during the D3hot (L2) transition. I don't think a link in L2 (main power off) can lead to a device in D3hot, can it? I assume that a device in D3hot can be returned to D0 by a config write to the PM CSR, and the link must be usable for that. > The kernel can't control what the firmware on the device may > respond. The kernel can't directly control the internal behavior of the device, but the behavior that's observable on the PCIe link should always conform to the spec. Do you see devices where a transition to D3hot may cause some kind of non-compliant behavior on the link? If so, then I guess we have to consider whether to quirk them avoid D3hot completely or to work around it somehow. > > But Configuration and Message requests definitely happen in D3hot, and > > they can cause errors reported via AER. The spec (r6.0, sec 2.2.8) > > recommends that Messages be handled the same in D0-D3hot. > > > > PTM is an example of where we had errors being reported at suspend/ > > resume because we had it configured incorrectly. If we disabled AER > > in D3hot we might not learn about that kind of configuration problem. > > That's what makes me think there's some value in keeping AER enabled > > in D3hot. > > In this particular case, the firmware of the device gets power cycled > and starts sending PTM because of that. > For this case, we want to know the error happens, but in the meantime > there's nothing much can be done. So simply putting the device in D3hot restarts firmware on the device? And it starts sending PTM requests after the restart? I *assume* that at least it only sends the PTM requests if the PTM Enable bit is set, right? That shouldn't cause us trouble unless we configured something wrong. Bjorn
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 60230da957e0..79961936ed1b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -6500,6 +6500,22 @@ bool pci_pr3_present(struct pci_dev *pdev) acpi_has_method(adev->handle, "_PR3"); } EXPORT_SYMBOL_GPL(pci_pr3_present); + +bool pci_ancestor_pr3_present(struct pci_dev *pdev) +{ + struct pci_dev *parent = pdev; + + if (acpi_disabled) + return false; + + while ((parent = pci_upstream_bridge(parent))) { + if (pci_pr3_present(pdev)) + return true; + } + + return false; +} +EXPORT_SYMBOL_GPL(pci_ancestor_pr3_present); #endif /** diff --git a/include/linux/pci.h b/include/linux/pci.h index c69a2cc1f412..1f30c20753a3 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2512,10 +2512,12 @@ struct irq_domain *pci_host_bridge_acpi_msi_domain(struct pci_bus *bus); void pci_msi_register_fwnode_provider(struct fwnode_handle *(*fn)(struct device *)); bool pci_pr3_present(struct pci_dev *pdev); +bool pci_ancestor_pr3_present(struct pci_dev *pdev); #else static inline struct irq_domain * pci_host_bridge_acpi_msi_domain(struct pci_bus *bus) { return NULL; } static inline bool pci_pr3_present(struct pci_dev *pdev) { return false; } +static inline bool pci_ancestor_pr3_present(struct pci_dev *pdev) { return false; } #endif #ifdef CONFIG_EEH
In addition to nearest upstream bridge, driver may want to know if the entire hierarchy can be powered off to perform different action. So walk higher up the hierarchy to find out if any device has valid _PR3. The user will be introduced in next patch. Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com> --- drivers/pci/pci.c | 16 ++++++++++++++++ include/linux/pci.h | 2 ++ 2 files changed, 18 insertions(+)