| Message ID | 1893355.EP2830DdO9@kreacher (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | nvme-pci: Do not prevent PCI bus-level PM from being used | expand |
On Wednesday, August 7, 2019 11:53:44 AM CEST Rafael J. Wysocki wrote: > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > One of the modifications made by commit d916b1be94b6 ("nvme-pci: use > host managed power state for suspend") was adding a pci_save_state() > call to nvme_suspend() in order to prevent the PCI bus-level PM from > being applied to the suspended NVMe devices, but if ASPM is not > enabled for the target NVMe device, that causes its PCIe link to stay > up and the platform may not be able to get into its optimum low-power > state because of that. > > For example, if ASPM is disabled for the NVMe drive (PC401 NVMe SK > hynix 256GB) in my Dell XPS13 9380, leaving it in D0 during > suspend-to-idle prevents the SoC from reaching package idle states > deeper than PC3, which is way insufficient for system suspend. > > To address this shortcoming, make nvme_suspend() check if ASPM is > enabled for the target device and fall back to full device shutdown > and PCI bus-level PM if that is not the case. > > Fixes: d916b1be94b6 ("nvme-pci: use host managed power state for suspend") > Link: https://lore.kernel.org/linux-pm/2763495.NmdaWeg79L@kreacher/T/#t > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > --- I should have used a better subject for this patch. I'll resend it with a changed subject later, but for now I would like to collect opinions about it (if any). Cheers!
> + if (pm_suspend_via_firmware() || !ctrl->npss || !pcie_aspm_enabled(pdev)) { > + mutex_lock(&aspm_lock); > + aspm_enabled = bridge->link_state ? bridge->link_state->aspm_enabled : 0; Please fix the overly long lines..
On Wed, Aug 07, 2019 at 02:53:44AM -0700, Rafael J. Wysocki wrote: > From: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > One of the modifications made by commit d916b1be94b6 ("nvme-pci: use > host managed power state for suspend") was adding a pci_save_state() > call to nvme_suspend() in order to prevent the PCI bus-level PM from > being applied to the suspended NVMe devices, but if ASPM is not > enabled for the target NVMe device, that causes its PCIe link to stay > up and the platform may not be able to get into its optimum low-power > state because of that. > > For example, if ASPM is disabled for the NVMe drive (PC401 NVMe SK > hynix 256GB) in my Dell XPS13 9380, leaving it in D0 during > suspend-to-idle prevents the SoC from reaching package idle states > deeper than PC3, which is way insufficient for system suspend. > > To address this shortcoming, make nvme_suspend() check if ASPM is > enabled for the target device and fall back to full device shutdown > and PCI bus-level PM if that is not the case. > > Fixes: d916b1be94b6 ("nvme-pci: use host managed power state for suspend") > Link: https://lore.kernel.org/linux-pm/2763495.NmdaWeg79L@kreacher/T/#t > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Thanks for tracking down the cause. Sounds like your earlier assumption on ASPM's involvement was spot on. > +/* > + * pcie_aspm_enabled - Return the mask of enabled ASPM link states. > + * @pci_device: Target device. > + */ > +u32 pcie_aspm_enabled(struct pci_dev *pci_device) > +{ > + struct pci_dev *bridge = pci_device->bus->self; You may want use pci_upstream_bridge() instead, just in case someone calls this on a virtual function's pci_dev. > + u32 aspm_enabled; > + > + mutex_lock(&aspm_lock); > + aspm_enabled = bridge->link_state ? bridge->link_state->aspm_enabled : 0; > + mutex_unlock(&aspm_lock); > + > + return aspm_enabled; > +}
Index: linux-pm/drivers/nvme/host/pci.c =================================================================== --- linux-pm.orig/drivers/nvme/host/pci.c +++ linux-pm/drivers/nvme/host/pci.c @@ -2846,7 +2846,7 @@ static int nvme_resume(struct device *de struct nvme_dev *ndev = pci_get_drvdata(to_pci_dev(dev)); struct nvme_ctrl *ctrl = &ndev->ctrl; - if (pm_resume_via_firmware() || !ctrl->npss || + if (ndev->last_ps == U32_MAX || nvme_set_power_state(ctrl, ndev->last_ps) != 0) nvme_reset_ctrl(ctrl); return 0; @@ -2859,6 +2859,8 @@ static int nvme_suspend(struct device *d struct nvme_ctrl *ctrl = &ndev->ctrl; int ret = -EBUSY; + ndev->last_ps = U32_MAX; + /* * The platform does not remove power for a kernel managed suspend so * use host managed nvme power settings for lowest idle power if @@ -2866,8 +2868,13 @@ static int nvme_suspend(struct device *d * shutdown. But if the firmware is involved after the suspend or the * device does not support any non-default power states, shut down the * device fully. + * + * If ASPM is not enabled for the device, shut down the device and allow + * the PCI bus layer to put it into D3 in order to take the PCIe link + * down, so as to allow the platform to achieve its minimum low-power + * state (which may not be possible if the link is up). */ - if (pm_suspend_via_firmware() || !ctrl->npss) { + if (pm_suspend_via_firmware() || !ctrl->npss || !pcie_aspm_enabled(pdev)) { nvme_dev_disable(ndev, true); return 0; } @@ -2880,9 +2887,8 @@ static int nvme_suspend(struct device *d ctrl->state != NVME_CTRL_ADMIN_ONLY) goto unfreeze; - ndev->last_ps = 0; ret = nvme_get_power_state(ctrl, &ndev->last_ps); - if (ret < 0) + if (ret < 0 || ndev->last_ps == U32_MAX) goto unfreeze; ret = nvme_set_power_state(ctrl, ctrl->npss); Index: linux-pm/drivers/pci/pcie/aspm.c =================================================================== --- linux-pm.orig/drivers/pci/pcie/aspm.c +++ linux-pm/drivers/pci/pcie/aspm.c @@ -1170,6 +1170,23 @@ static int pcie_aspm_get_policy(char *bu module_param_call(policy, pcie_aspm_set_policy, pcie_aspm_get_policy, NULL, 0644); +/* + * pcie_aspm_enabled - Return the mask of enabled ASPM link states. + * @pci_device: Target device. + */ +u32 pcie_aspm_enabled(struct pci_dev *pci_device) +{ + struct pci_dev *bridge = pci_device->bus->self; + u32 aspm_enabled; + + mutex_lock(&aspm_lock); + aspm_enabled = bridge->link_state ? bridge->link_state->aspm_enabled : 0; + mutex_unlock(&aspm_lock); + + return aspm_enabled; +} + + #ifdef CONFIG_PCIEASPM_DEBUG static ssize_t link_state_show(struct device *dev, struct device_attribute *attr, Index: linux-pm/include/linux/pci.h =================================================================== --- linux-pm.orig/include/linux/pci.h +++ linux-pm/include/linux/pci.h @@ -1567,8 +1567,10 @@ extern bool pcie_ports_native; #ifdef CONFIG_PCIEASPM bool pcie_aspm_support_enabled(void); +u32 pcie_aspm_enabled(struct pci_dev *pci_device); #else static inline bool pcie_aspm_support_enabled(void) { return false; } +static inline u32 pcie_aspm_enabled(struct pci_dev *pci_device) { return 0; } #endif #ifdef CONFIG_PCIEAER