From patchwork Thu Mar 22 19:36:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 10302187 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 601CF60386 for ; Thu, 22 Mar 2018 19:36:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50EBF28801 for ; Thu, 22 Mar 2018 19:36:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 45D5828914; Thu, 22 Mar 2018 19:36:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4FF628801 for ; Thu, 22 Mar 2018 19:36:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751796AbeCVTgd (ORCPT ); Thu, 22 Mar 2018 15:36:33 -0400 Received: from mail.kernel.org ([198.145.29.99]:57442 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751761AbeCVTgc (ORCPT ); Thu, 22 Mar 2018 15:36:32 -0400 Received: from localhost (unknown [69.55.156.246]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8279921749; Thu, 22 Mar 2018 19:36:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8279921749 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=helgaas@kernel.org Date: Thu, 22 Mar 2018 14:36:30 -0500 From: Bjorn Helgaas To: Lukas Wunner Cc: Mika Westerberg , Bjorn Helgaas , "Rafael J . Wysocki" , Len Brown , Keith Busch , Linux PCI , ACPI Devel Maling List Subject: Re: [PATCH 1/2] PCI/DPC: Disable interrupt generation during suspend Message-ID: <20180322193630.GB252023@bhelgaas-glaptop.roam.corp.google.com> References: <20180314114125.71132-1-mika.westerberg@linux.intel.com> <20180314120547.GB2703@lahna.fi.intel.com> <20180314123332.GC19651@wunner.de> <20180320104508.GF2703@lahna.fi.intel.com> <20180320113556.GA24197@wunner.de> <20180322104517.GA20389@wunner.de> <20180322165317.GI2703@lahna.fi.intel.com> <20180322173903.GA15503@wunner.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180322173903.GA15503@wunner.de> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Mar 22, 2018 at 06:39:03PM +0100, Lukas Wunner wrote: > On Thu, Mar 22, 2018 at 06:53:17PM +0200, Mika Westerberg wrote: > > On Thu, Mar 22, 2018 at 11:45:17AM +0100, Lukas Wunner wrote: > > > Now I've thought of one. > > > > > > The port may have more children besides the port service devices, > > > namely all the PCI devices below the port. The PM core doesn't > > > impose a specific ordering on suspend/resume but will try to > > > parallelize among all the children. > > > > > > Usually that's not what you want. On resume, you want to resume > > > the port itself (including its port services) *before* resuming > > > the PCI child devices. And the other way round on suspend. > > > > That's a good point. > > > > So I guess there is no way avoiding adding suspend_late/resume_early > > callbacks to the pcie port service structure. I'll do that in the next > > revision. > > Well, there *are* ways to avoid it but they might not be better. > > Iterating over the port services' callbacks is equivalent to ordering > the port service devices after the port's PCI device but before its > PCI child devices in devices_kset. > > That can also be achieved by adding a device link from every PCI child > device (consumer) to every port service device (provider). The result > however is a combinatorial explosion. Say you've got 64 down stream > bridges in a PCIe switch and the upstream bridge has 3 port services, > that's 3 x 64 = 192 device links. That's probably clumsier than > iterating over the port services. I hope we can avoid adding suspend_late/resume_early callbacks in struct pcie_port_service_driver, and I also hope we can avoid adding device links. Those both sound pretty complicated. Can you do something like the patch below, which does something similar for PME? commit 6c4dfc1389e1 Author: Bjorn Helgaas Date: Fri Mar 9 11:06:54 2018 -0600 PCI/PM: Clear PCIe PME Status bit in core, not PCIe port driver fe31e69740ed ("PCI/PCIe: Clear Root PME Status bits early during system resume") added a .resume_noirq() callback to the PCIe port driver to clear the PME Status bit during resume to work around a BIOS issue. The BIOS evidently enabled PME interrupts for ACPI-based runtime wakeups but did not clear the PME Status bit during resume, which meant PMEs after resume did not trigger interrupts because PME Status did not transition from cleared to set. The fix was in the PCIe port driver, so it worked when CONFIG_PCIEPORTBUS was set. But I think we *always* want the fix because the platform may use PME interrupts even if Linux is built without the PCIe port driver. Move the fix from the port driver to the PCI core so we can work around this "PME doesn't work after waking from a sleep state" issue regardless of CONFIG_PCIEPORTBUS. Signed-off-by: Bjorn Helgaas diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 3bed6beda051..e561fa0f456c 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -525,6 +525,18 @@ static void pci_pm_default_resume_early(struct pci_dev *pci_dev) pci_fixup_device(pci_fixup_resume_early, pci_dev); } +static void pcie_pme_root_status_cleanup(struct pci_dev *pci_dev) +{ + /* + * Some BIOSes forget to clear Root PME Status bits after system + * wakeup, which breaks ACPI-based runtime wakeup on PCI Express. + * Clear those bits now just in case (shouldn't hurt). + */ + if (pci_is_pcie(pci_dev) && + pci_pcie_type(pci_dev) == PCI_EXP_TYPE_ROOT_PORT) + pcie_clear_root_pme_status(pci_dev); +} + /* * Default "suspend" method for devices that have no driver provided suspend, * or not even a driver at all (second part). @@ -873,6 +885,8 @@ static int pci_pm_resume_noirq(struct device *dev) if (pci_has_legacy_pm_support(pci_dev)) return pci_legacy_resume_early(dev); + pcie_pme_root_status_cleanup(pci_dev); + if (drv && drv->pm && drv->pm->resume_noirq) error = drv->pm->resume_noirq(dev); diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c index d6f10a97d400..ec9e936c2a5b 100644 --- a/drivers/pci/pcie/portdrv_pci.c +++ b/drivers/pci/pcie/portdrv_pci.c @@ -61,20 +61,6 @@ static int pcie_portdrv_restore_config(struct pci_dev *dev) } #ifdef CONFIG_PM -static int pcie_port_resume_noirq(struct device *dev) -{ - struct pci_dev *pdev = to_pci_dev(dev); - - /* - * Some BIOSes forget to clear Root PME Status bits after system wakeup - * which breaks ACPI-based runtime wakeup on PCI Express, so clear those - * bits now just in case (shouldn't hurt). - */ - if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT) - pcie_clear_root_pme_status(pdev); - return 0; -} - static int pcie_port_runtime_suspend(struct device *dev) { return to_pci_dev(dev)->bridge_d3 ? 0 : -EBUSY; @@ -102,7 +88,6 @@ static const struct dev_pm_ops pcie_portdrv_pm_ops = { .thaw = pcie_port_device_resume, .poweroff = pcie_port_device_suspend, .restore = pcie_port_device_resume, - .resume_noirq = pcie_port_resume_noirq, .runtime_suspend = pcie_port_runtime_suspend, .runtime_resume = pcie_port_runtime_resume, .runtime_idle = pcie_port_runtime_idle,