| Message ID | 20230708214457.1229-2-mario.limonciello@amd.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | Fix suspend issues with AMD root ports | expand |
On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote: > commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") > sets the policy that all PCIe ports are allowed to use D3. When > the system is suspended if the port is not power manageable by the > platform and won't be used for wakeup via a PME this sets up the > policy for these ports to go into D3hot. > > This policy generally makes sense from an OSPM perspective but it leads > to problems with wakeup from suspend on laptops with AMD chips: > > - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup > interrupt. > - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. > > Add a quirk for the PCI device ID used by the problematic root port on > both chips to ensure that these root ports are not put into D3hot at > suspend. What is problematic about these root ports? Is this a hardware erratum? Some corner of the ACPI spec that allows undefined behavior? Does AMD have any guidance about generic ways to use D3, or does AMD expect to add quirks piecemeal as problems are discovered? How does Windows handle all this? Adding quirks as we discover random devices that don't behave correctly for reasons unknown is not very sustainable. Bjorn > Cc: stable@vger.kernel.org # 6.1+ > Reported-by: Iain Lane <iain@orangesquash.org.uk> > Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121 > Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") > Signed-off-by: Mario Limonciello <mario.limonciello@amd.com> > --- > drivers/pci/quirks.c | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index 321156ca273d5..e0346073e5855 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev) > DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL, > PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C, > quirk_apple_poweroff_thunderbolt); > + > +/* > + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot > + * may cause problems when the system attempts wake up from s2idle. > + * > + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup > + * interrupt. > + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. > + */ > +static void quirk_ryzen_rp_d3(struct pci_dev *pdev) > +{ > + if (!acpi_pci_power_manageable(pdev)) > + pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3; > +} > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3); > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3); > #endif > > /* > -- > 2.34.1 >
On 7/10/2023 14:32, Bjorn Helgaas wrote: > On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote: >> commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") >> sets the policy that all PCIe ports are allowed to use D3. When >> the system is suspended if the port is not power manageable by the >> platform and won't be used for wakeup via a PME this sets up the >> policy for these ports to go into D3hot. >> >> This policy generally makes sense from an OSPM perspective but it leads >> to problems with wakeup from suspend on laptops with AMD chips: >> >> - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup >> interrupt. >> - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. >> >> Add a quirk for the PCI device ID used by the problematic root port on >> both chips to ensure that these root ports are not put into D3hot at >> suspend. > > What is problematic about these root ports? Is this a hardware > erratum? I mentioned some of this in earlier versions; but the problem is deeper in the platform firmware and only happens when the root ports are in D3hot across an s2idle cycle. When looked at independently they can be moved in and out of D3hot no problem. From the perspective of hardware designers they say this is a software bug that Linux puts PCI RP into D3hot over Modern Standby when they don't specify for this to be done. I don't expect any erratum for it. > Some corner of the ACPI spec that allows undefined behavior? > These root ports are not reported as power manageable by ACPI. As I mentioned in the cover letter how the OSPM handles this case is outside of any spec AFAICT. > Does AMD have any guidance about generic ways to use D3, or does AMD > expect to add quirks piecemeal as problems are discovered? How does > Windows handle all this? > Windows doesn't put root ports into D3hot over suspend unless they are power managed by ACPI (as described in section 7.3 of the ACPI spec). https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/device-power-management-objects.html So on Windows these ports are all in D0 and none of the issues happen. Linux if PCI devices aren't power managed by ACPI will either follow deepest state it can wake from PME or fall back to D3hot. > Adding quirks as we discover random devices that don't behave > correctly for reasons unknown is not very sustainable. I don't disagree but in v5 I tried to align this to the Windows behavior and it wasn't accepted. If you still think v5 is the better approach I'm fine to try to do that again and rewrite the commit message. > > Bjorn > >> Cc: stable@vger.kernel.org # 6.1+ >> Reported-by: Iain Lane <iain@orangesquash.org.uk> >> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121 >> Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") >> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com> >> --- >> drivers/pci/quirks.c | 16 ++++++++++++++++ >> 1 file changed, 16 insertions(+) >> >> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >> index 321156ca273d5..e0346073e5855 100644 >> --- a/drivers/pci/quirks.c >> +++ b/drivers/pci/quirks.c >> @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev) >> DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL, >> PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C, >> quirk_apple_poweroff_thunderbolt); >> + >> +/* >> + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot >> + * may cause problems when the system attempts wake up from s2idle. >> + * >> + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup >> + * interrupt. >> + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. >> + */ >> +static void quirk_ryzen_rp_d3(struct pci_dev *pdev) >> +{ >> + if (!acpi_pci_power_manageable(pdev)) >> + pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3; >> +} >> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3); >> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3); >> #endif >> >> /* >> -- >> 2.34.1 >>
[+cc Mika, author of 9d26d3a8f1b0] On Mon, Jul 10, 2023 at 02:44:23PM -0500, Limonciello, Mario wrote: > On 7/10/2023 14:32, Bjorn Helgaas wrote: > > On Sat, Jul 08, 2023 at 04:44:57PM -0500, Mario Limonciello wrote: > > > commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") > > > sets the policy that all PCIe ports are allowed to use D3. When > > > the system is suspended if the port is not power manageable by the > > > platform and won't be used for wakeup via a PME this sets up the > > > policy for these ports to go into D3hot. > > > > > > This policy generally makes sense from an OSPM perspective but it leads > > > to problems with wakeup from suspend on laptops with AMD chips: > > > > > > - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup > > > interrupt. > > > - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. > > > > > > Add a quirk for the PCI device ID used by the problematic root port on > > > both chips to ensure that these root ports are not put into D3hot at > > > suspend. > > > > What is problematic about these root ports? Is this a hardware > > erratum? > > I mentioned some of this in earlier versions; but the problem is deeper > in the platform firmware and only happens when the root ports are in D3hot > across an s2idle cycle. > > When looked at independently they can be moved in and out of D3hot no > problem. > > From the perspective of hardware designers they say this is a software bug > that Linux puts PCI RP into D3hot over Modern Standby when they don't > specify for this to be done. I don't expect any erratum for it. It sounds like there's someplace the hardware designers specify how this should work? Where is that? "Modern Standby" doesn't mean anything to me, but maybe there's some spec that covers it? > > Some corner of the ACPI spec that allows undefined behavior? > > These root ports are not reported as power manageable by ACPI. > > As I mentioned in the cover letter how the OSPM handles this case is outside > of any spec AFAICT. If it's truly outside the spec, then I don't think Linux should use D3 until we have some standardized way to tell whether it's safe. > > Does AMD have any guidance about generic ways to use D3, or does AMD > > expect to add quirks piecemeal as problems are discovered? How does > > Windows handle all this? > > Windows doesn't put root ports into D3hot over suspend unless they are power > managed by ACPI (as described in section 7.3 of the ACPI spec). > > https://uefi.org/htmlspecs/ACPI_Spec_6_4_html/07_Power_and_Performance_Mgmt/device-power-management-objects.html > > So on Windows these ports are all in D0 and none of the issues happen. Maybe this is the clue we need. My eyes glaze over when reading that section, but if we can come up with a commit log that starts with a sentence from that section and connects the dots all the way to the platform_pci_power_manageable() implementation, that might be a good argument that 9d26d3a8f1b0 was a little too aggressive. > Linux if PCI devices aren't power managed by ACPI will either follow deepest > state it can wake from PME or fall back to D3hot. > > > Adding quirks as we discover random devices that don't behave > > correctly for reasons unknown is not very sustainable. > > I don't disagree but in v5 I tried to align this to the Windows behavior and > it wasn't accepted. I like the fact that v5 ([1] for anybody following along at home) is very generic: @@ bool pci_bridge_d3_possible(struct pci_dev *bridge) ... + if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT && + !platform_pci_power_manageable(bridge)) + return false; My objection was that we didn't have a clear connection to any specs, so even though the code is perfectly generic, the commit log mentioned specific cases (USB keyboard/mouse on xHCI device connected to USB-C on AMD USB4 router). But maybe we *could* make a convincing generic commit log. I guess we'd also have to explain the PCI_EXP_TYPE_ROOT_PORT part of the patch. Bjorn [1] https://lore.kernel.org/r/20230530163947.230418-2-mario.limonciello@amd.com > > > Cc: stable@vger.kernel.org # 6.1+ > > > Reported-by: Iain Lane <iain@orangesquash.org.uk> > > > Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121 > > > Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") > > > Signed-off-by: Mario Limonciello <mario.limonciello@amd.com> > > > --- > > > drivers/pci/quirks.c | 16 ++++++++++++++++ > > > 1 file changed, 16 insertions(+) > > > > > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > > > index 321156ca273d5..e0346073e5855 100644 > > > --- a/drivers/pci/quirks.c > > > +++ b/drivers/pci/quirks.c > > > @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev) > > > DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL, > > > PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C, > > > quirk_apple_poweroff_thunderbolt); > > > + > > > +/* > > > + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot > > > + * may cause problems when the system attempts wake up from s2idle. > > > + * > > > + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup > > > + * interrupt. > > > + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. > > > + */ > > > +static void quirk_ryzen_rp_d3(struct pci_dev *pdev) > > > +{ > > > + if (!acpi_pci_power_manageable(pdev)) > > > + pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3; > > > +} > > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3); > > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3); > > > #endif > > > /* > > > -- > > > 2.34.1 > > > >
On 7/10/2023 3:33 PM, Bjorn Helgaas wrote: > > It sounds like there's someplace the hardware designers specify how > this should work? Where is that? "Modern Standby" doesn't mean > anything to me, but maybe there's some spec that covers it? https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/modern-standby It quickly devolves into Microsoft specific stuff though and I can't find anything interesting to our specific issue. > Maybe this is the clue we need. My eyes glaze over when reading that > section, but if we can come up with a commit log that starts with a > sentence from that section and connects the dots all the way to the > platform_pci_power_manageable() implementation, that might be a good > argument that 9d26d3a8f1b0 was a little too aggressive. Yeah. > I like the fact that v5 ([1] for anybody following along at home) is > very generic: > > @@ bool pci_bridge_d3_possible(struct pci_dev *bridge) > ... > + if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT && > + !platform_pci_power_manageable(bridge)) > + return false; > > My objection was that we didn't have a clear connection to any specs, > so even though the code is perfectly generic, the commit log mentioned > specific cases (USB keyboard/mouse on xHCI device connected to USB-C > on AMD USB4 router). > > But maybe we *could* make a convincing generic commit log. I guess > we'd also have to explain the PCI_EXP_TYPE_ROOT_PORT part of the > patch. OK, I'll take a stab at rewriting the v5 commit message to be more generic as you suggested as a v7. We might be able to drop the PCI_EXP_TYPE_ROOT_PORT part well but I would be more worried about regressions from this.
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 321156ca273d5..e0346073e5855 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3867,6 +3867,22 @@ static void quirk_apple_poweroff_thunderbolt(struct pci_dev *dev) DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CACTUS_RIDGE_4C, quirk_apple_poweroff_thunderbolt); + +/* + * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot + * may cause problems when the system attempts wake up from s2idle. + * + * On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup + * interrupt. + * On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. + */ +static void quirk_ryzen_rp_d3(struct pci_dev *pdev) +{ + if (!acpi_pci_power_manageable(pdev)) + pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3; +} +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14b9, quirk_ryzen_rp_d3); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14eb, quirk_ryzen_rp_d3); #endif /*
commit 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") sets the policy that all PCIe ports are allowed to use D3. When the system is suspended if the port is not power manageable by the platform and won't be used for wakeup via a PME this sets up the policy for these ports to go into D3hot. This policy generally makes sense from an OSPM perspective but it leads to problems with wakeup from suspend on laptops with AMD chips: - On family 19h model 44h (PCI 0x14b9) this manifests as a missing wakeup interrupt. - On family 19h model 74h (PCI 0x14eb) this manifests as a system hang. Add a quirk for the PCI device ID used by the problematic root port on both chips to ensure that these root ports are not put into D3hot at suspend. Cc: stable@vger.kernel.org # 6.1+ Reported-by: Iain Lane <iain@orangesquash.org.uk> Closes: https://forums.lenovo.com/t5/Ubuntu/Z13-can-t-resume-from-suspend-with-external-USB-keyboard/m-p/5217121 Fixes: 9d26d3a8f1b0 ("PCI: Put PCIe ports into D3 during suspend") Signed-off-by: Mario Limonciello <mario.limonciello@amd.com> --- drivers/pci/quirks.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)