From patchwork Mon Mar 2 18:44:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416093 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66E2492A for ; Mon, 2 Mar 2020 18:45:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4608421739 for ; Mon, 2 Mar 2020 18:45:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="u1HGjJo1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727372AbgCBSpx (ORCPT ); Mon, 2 Mar 2020 13:45:53 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56061 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbgCBSpx (ORCPT ); Mon, 2 Mar 2020 13:45:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174753; x=1614710753; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SXJONBdcDukonc95OMH4eVS6Q98vylWW4QxZSA+zyc4=; b=u1HGjJo1RaNze1C56b/3IprbDCuWg2yjPzhBDNm92ZhqfpQvmLykHWau fGhF7y0LOxPi/p8ihKtnx2UmEBqLBMOOJ7Xlj3qtmmovtuJiR2OpG9kVM 821EVEBzyTkbZ30ufw5zk6yZhG7l4+isDELzUgE3ayTHnVoPqcSSaC8Wb 4=; IronPort-SDR: uj46hkOYu2rGamAQBTIbse7bMfFb7S7z6rgp9/EaUJW7hBitSpO+zr4/SiTDfq/45ZG/0PNQ30 Y/Bxx+nW4jBw== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19141882" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-81e76b79.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:45:39 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-81e76b79.us-west-2.amazon.com (Postfix) with ESMTPS id 24D64A04F0; Mon, 2 Mar 2020 18:45:38 +0000 (UTC) Received: from EX13D12EUA003.ant.amazon.com (10.43.165.147) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:18 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUA003.ant.amazon.com (10.43.165.147) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:16 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:13 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 01/17] PCI: Fall back to slot/bus reset if softer methods timeout Date: Mon, 2 Mar 2020 19:44:13 +0100 Message-ID: <20200302184429.12880-2-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Previously, if a device never came back (i.e., never started returning Successful Completions for Configuration Requests) after a reset, they would return -ENOTTY, causing __pci_reset_function_locked to move on to the next reset method. However, up until slot/bus reset, all of them rely on the device being responsive and are therefore not safe to attempt in this situation. This patch introduces ETIMEDOUT as a new, specially handled return value for the reset methods (which all end in "return pci_dev_wait"), to allow skipping to slot/bus reset where appropriate. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index d828ca835a98..ac8504d75c32 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1043,7 +1043,7 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout) if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", delay - 1, reset_type); - return -ENOTTY; + return -ETIMEDOUT; } if (delay > 1000) @@ -4845,6 +4845,7 @@ static int pci_parent_bus_reset(struct pci_dev *dev, int probe) if (probe) return 0; + /* XXX: Shouldn't this lock the bus? */ return pci_bridge_secondary_bus_reset(dev->bus->self); } @@ -4979,25 +4980,37 @@ int __pci_reset_function_locked(struct pci_dev *dev) /* * A reset method returns -ENOTTY if it doesn't support this device * and we should try the next method. + * It returns -ETIMEDOUT if the device never became responsive after + * the reset: we jump to the reset types that do not rely on config + * space access (if any are left). * - * If it returns 0 (success), we're finished. If it returns any - * other error, we're also finished: this indicates that further - * reset mechanisms might be broken on the device. + * If it returns 0 (success), we are finished. + * If it returns any other error, we are finished (something must have + * went terribly wrong and it is not safe to continue reset attempts). */ rc = pci_dev_specific_reset(dev, 0); + if (rc == -ETIMEDOUT) + goto unresponsive; if (rc != -ENOTTY) return rc; if (pcie_has_flr(dev)) { rc = pcie_flr(dev); + if (rc == -ETIMEDOUT) + goto unresponsive; if (rc != -ENOTTY) return rc; } rc = pci_af_flr(dev, 0); + if (rc == -ETIMEDOUT) + goto unresponsive; if (rc != -ENOTTY) return rc; rc = pci_pm_reset(dev, 0); + if (rc == -ETIMEDOUT) + goto unresponsive; if (rc != -ENOTTY) return rc; +unresponsive: rc = pci_dev_reset_slot_function(dev, 0); if (rc != -ENOTTY) return rc; From patchwork Mon Mar 2 18:44:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416097 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 434EF924 for ; Mon, 2 Mar 2020 18:46:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 224842072A for ; Mon, 2 Mar 2020 18:46:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="IV3XC7b4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727389AbgCBSqE (ORCPT ); Mon, 2 Mar 2020 13:46:04 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56139 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727408AbgCBSqE (ORCPT ); Mon, 2 Mar 2020 13:46:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174764; x=1614710764; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Cbndvv3qCssTnIgDUBDG4qXe8hnk+TGR5JEIQeTw7KA=; b=IV3XC7b4yxa/uuwT8RCDYI8b9rJGw3oFS3yQgDX/YPICCfLqYsc7dJQD Lmngum1Ilr6di9yW5n7w7BpOrKZCJKODhQXZzT0T/diZtmHSBRhbCIEF7 GkTH8FD745UFQCRMw+hUY+f+wweGKFZl74+0F2w+24kjSpslua0ctYIpL U=; IronPort-SDR: 6emHHV9YeIwwbj1EpazE7fNk6bjsuDJUO68Jb04s8JeO4FswiiJeMOdV+nN9ruk0XH0f0cjo5t IdPvnLiWbJnw== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19142009" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com (Postfix) with ESMTPS id 1EBA6A3155; Mon, 2 Mar 2020 18:46:00 +0000 (UTC) Received: from EX13D12EUA003.ant.amazon.com (10.43.165.147) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:22 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUA003.ant.amazon.com (10.43.165.147) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:21 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:17 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 02/17] PCI: Remove unused PCI_PM_BUS_WAIT Date: Mon, 2 Mar 2020 19:44:14 +0100 Message-ID: <20200302184429.12880-3-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov The last usage of this constant was removed by: commit 476e7faefc43 ("PCI PM: Do not wait for buses in B2 or B3 during resume") Signed-off-by: Stanislav Spassov --- drivers/pci/pci.h | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 6394e7746fb5..659ab3f9042a 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -46,7 +46,6 @@ int pci_bus_error_reset(struct pci_dev *dev); #define PCI_PM_D2_DELAY 200 #define PCI_PM_D3_WAIT 10 #define PCI_PM_D3COLD_WAIT 100 -#define PCI_PM_BUS_WAIT 50 /** * struct pci_platform_pm_ops - Firmware PM callbacks From patchwork Mon Mar 2 18:44:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416095 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B525924 for ; Mon, 2 Mar 2020 18:46:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2A6DA21775 for ; Mon, 2 Mar 2020 18:46:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="fc1R++Dc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727234AbgCBSqD (ORCPT ); Mon, 2 Mar 2020 13:46:03 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:59876 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbgCBSqD (ORCPT ); Mon, 2 Mar 2020 13:46:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174763; x=1614710763; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Gz6pfLMfGNZpaFxwlrBNkkqcP/vdNODzuYge/T20qto=; b=fc1R++DcWp4I0bKQrCwHpQUmlU7Ar446zXnEdPzYlFiF0I2qTta7ZEPD RbDJw//FH99ckAHLF4BAsLahsfIyDEW0Hsf9HDYD1TmbYlHd+ESnwfEF9 NYo4Oxm/ByNZH56HmPej/iHmPySFlNzKfqjg4bsP+12FaptuZ2ZX6SCA+ U=; IronPort-SDR: acV4vYPGOERTwoFEx1C+N37C0lqz95/RVPtoJJr7BW5kbi6VKBBaxZrsjNglZVrGfTo4Py4bv5 w3VTj94zsjkw== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="20320559" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 02 Mar 2020 18:46:03 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com (Postfix) with ESMTPS id 704A3C5CE9; Mon, 2 Mar 2020 18:46:01 +0000 (UTC) Received: from EX13D12EUA002.ant.amazon.com (10.43.165.103) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:27 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUA002.ant.amazon.com (10.43.165.103) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:25 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:22 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 03/17] PCI: Use pci_bridge_wait_for_secondary_bus after SBR Date: Mon, 2 Mar 2020 19:44:15 +0100 Message-ID: <20200302184429.12880-4-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov So far, pci_bridge_wait_for_secondary_bus() was only invoked by PM code after (runtime) resume of devices, but it naturally makes sense for handling post-SBR waiting as well. It uses the PCI_PM_D3COLD_WAIT value (100ms), potentially overridden on a per-device basis to a lower-value, as the basis for determining how long to wait, and handles special cases such as legacy PCI devices (requiring Trhfa), and the different starting points for the waiting time depending on PCIe port speed. On PCI Express, there will be cases where the new code sleeps far less than the 1s being replaced by this patch. This should be okay, because PCI Express Base Specification Revision 5.0 Version 1.0 (May 22, 2019) in Section 6.6.1 "Conventional Reset" only notes 100ms as the minimum waiting time. After this time, the OS is permitted to issue Configuration Requests, but it is possible that the device responds with Configuration Request Retry Status (CRS) Completions, rather than Successful Completion. Returning CRS can go on for up to 1 second after a Conventional Reset (such as SBR) before the OS can consider the device broken. This additional wait is handled by pci_dev_wait. Currently, the only callchain that lands in the function modified by this patch starts at pci_bridge_secondary_bus_reset which invokes one out of two versions of pcibios_reset_secondary_bus that both end with a call to pci_reset_secondary_bus. Afterwards, pci_bridge_secondary_bus_reset always invokes pci_dev_wait. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index ac8504d75c32..c1a866f733e9 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4800,14 +4800,7 @@ void pci_reset_secondary_bus(struct pci_dev *dev) ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); - /* - * Trhfa for conventional PCI is 2^25 clock cycles. - * Assuming a minimum 33MHz clock this results in a 1s - * delay before we can consider subordinate devices to - * be re-initialized. PCIe has some ways to shorten this, - * but we don't make use of them yet. - */ - ssleep(1); + pci_bridge_wait_for_secondary_bus(dev); } void __weak pcibios_reset_secondary_bus(struct pci_dev *dev) From patchwork Mon Mar 2 18:44:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416099 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D34B117E0 for ; Mon, 2 Mar 2020 18:46:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B2F6021739 for ; Mon, 2 Mar 2020 18:46:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="nimA25pl" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727408AbgCBSqF (ORCPT ); Mon, 2 Mar 2020 13:46:05 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56139 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbgCBSqF (ORCPT ); Mon, 2 Mar 2020 13:46:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174765; x=1614710765; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7WM9yX1OJkKXJJaN0Bul5znh9nhHO2FbDkre7Td3gO0=; b=nimA25plg3bHpNwbxFtmaKE//1TLen4iP1OZU7uH6tVA2ukRsf3d/fR3 2J5C0BrtpEhw5PkvNJhBxSdgGTxRCZR/SvLt+8+Hguni5srCGvrwaaHJ2 zd6gh2RtpViwb+yyccv4QMdwa6VOaitOO2BZR0SQtM9KpbnyTrjxkmmGi A=; IronPort-SDR: SiPSt2/PaDD1mUduWE3ZZcCXhAAoOWBUkvnPEFRKkR8bW3RmxLCE6FiVp7G5844uo/I33L7l/p ClNfH9QdN6HA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19142020" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-22cc717f.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:04 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-22cc717f.us-west-2.amazon.com (Postfix) with ESMTPS id 816DAA244C; Mon, 2 Mar 2020 18:46:02 +0000 (UTC) Received: from EX13D04EUB001.ant.amazon.com (10.43.166.190) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:31 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUB001.ant.amazon.com (10.43.166.190) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:30 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:26 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 04/17] PCI: Do not override delay for D0->D3hot transition Date: Mon, 2 Mar 2020 19:44:16 +0100 Message-ID: <20200302184429.12880-5-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Both specifications that document mechanisms for overriding the D3hot->D0 waiting time only speak of this specific direction. Nothing is mentioned about the opposite (D*->D3hot) except for the default value of 10ms in PCI Express Base Specification r5.0 (May 22, 2019), Section 5.9 "State Transition Recovery Time Requirements". Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c1a866f733e9..03103bb15b42 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4589,7 +4589,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D3hot; pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr); - pci_dev_d3_sleep(dev); + msleep(PCI_PM_D3_WAIT); csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D0; From patchwork Mon Mar 2 18:44:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416115 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18316924 for ; Mon, 2 Mar 2020 18:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E1CF82166E for ; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="iaLZ8YO9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727487AbgCBSqa (ORCPT ); Mon, 2 Mar 2020 13:46:30 -0500 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:47672 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727414AbgCBSqa (ORCPT ); Mon, 2 Mar 2020 13:46:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174788; x=1614710788; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hin5pnYLRZeeL4R19g+s7EtcXGJQv8ECPkAJzZhQii0=; b=iaLZ8YO9yi3EyEzGjcj6jkH75QaLaaUJR5VUCkIUq4qKtnOq4DtgSvy/ NSJ5I6rf/k07kMNuLBbki/9yXE7uqdWlNehpBWiZjLssHoLt3z0AfxMGZ oZGTYIoPwufjAtt3+rslJ3ssa//h6dy4W+Lj+JifUVgdqpwfxNiU+yznO Q=; IronPort-SDR: 27xnXP7QY/NNUZkoRY+upazzb2xDuwAwFj4CUT5r+SxptPa1Hlbs7KGcU+S8hc0OuVoziZ633k acE+vVzcbqcQ== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="20582195" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:04 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com (Postfix) with ESMTPS id 41592A2CB5; Mon, 2 Mar 2020 18:46:03 +0000 (UTC) Received: from EX13D12EUA004.ant.amazon.com (10.43.165.162) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:35 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUA004.ant.amazon.com (10.43.165.162) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:34 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:30 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 05/17] PCI: Fix handling of _DSM 8 (avoiding reset delays) Date: Mon, 2 Mar 2020 19:44:17 +0100 Message-ID: <20200302184429.12880-6-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov On Sx Resume (boot from ACPI S5, resume from ACPI S4 or S3), platform firmware may enforce that sufficient time has passed to cover the mandatory (per PCI Express Specification) power-on reset delays of all PCI devices before control is handed over to the operating system. To avoid duplicated waiting by the OS, the firmware uses _DSM 8 in the ACPI scope of a PCI host bus to inform the OS that for the whole PCI subsystem underneath that host bridge, power-on reset delays are being covered by firmware. Previously, the kernel used _DSM 8 to override the d3cold_delay value stored in struct pci_dev to 0. However, the assumption that the delay is covered by firmware when _DSM 8 returns 1 only holds for the firmware-initiated resets as part of Sx Resume flows. If an OS has the ability to apply Conventional Reset to devices without system firmware involvement, then it still needs to adhere to the usual reset delays. Runtime Device Power Management triggering a D3cold->D0 transition is an existing example of the kernel not involving system firmware. Another example is the OS triggering Secondary Bus Reset by setting the SBR bit in the Bridge Control register of any Port device. In preparation for future work to improve post-reset delays, this patch preserves the value of d3cold_delay even when _DSM 8 returns 1. The decision not to wait after resume is instead taken at a higher level in pci_bridge_wait_for_secondary_bus based on a new parameter that indicates whether the waiting is due to an Sx Resume event. Signed-off-by: Stanislav Spassov --- drivers/pci/pci-acpi.c | 65 ++++++++++++++++++++++++++++++---------- drivers/pci/pci-driver.c | 4 +-- drivers/pci/pci.c | 11 +++++-- drivers/pci/pci.h | 2 +- include/linux/pci-acpi.h | 8 ++--- include/linux/pci.h | 4 ++- 6 files changed, 67 insertions(+), 27 deletions(-) diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c index 0c02d500158f..a8fa13d6089d 100644 --- a/drivers/pci/pci-acpi.c +++ b/drivers/pci/pci-acpi.c @@ -1120,21 +1120,35 @@ void acpi_pci_add_bus(struct pci_bus *bus) acpi_pci_slot_enumerate(bus); acpiphp_enumerate_slots(bus); - /* - * For a host bridge, check its _DSM for function 8 and if - * that is available, mark it in pci_host_bridge. + /* For a host bridge, check _DSM function 8 and if that returns 1, + * mark it in the pci_host_bridge. + * + * Function 8, "Avoid Power-On Reset Delay Duplication on Sx Resume" + * applies to the entire hierarchy below a PCI host bridge. + * If it returns one, the OS may assume that all devices in the + * hierarchy have already completed power-on reset delays + * before FW handed over control to the OS on Sx Resume (such as boot + * from ACPI S5, or resume from ACPI S4 or S3 states). + * This _DSM is applicable whether reductions in device readiness + * timing, via Readiness Notification or _DSM function 9, are available + * or not. + * + * This _DSM function is defined by the PCI Firmware Specification + * Revision 3.2 (January 26, 2015), after originally introduced by a + * draft ECN of January 28, 2014, titled "ACPI additions for FW latency + * optimizations." */ if (!pci_is_root_bus(bus)) return; obj = acpi_evaluate_dsm(ACPI_HANDLE(bus->bridge), &pci_acpi_dsm_guid, 3, - RESET_DELAY_DSM, NULL); + IGNORE_RESET_DELAY_ON_SX_RESUME_DSM, NULL); if (!obj) return; if (obj->type == ACPI_TYPE_INTEGER && obj->integer.value == 1) { bridge = pci_find_host_bridge(bus); - bridge->ignore_reset_delay = 1; + bridge->ignore_reset_delay_on_sx_resume = 1; } ACPI_FREE(obj); } @@ -1168,19 +1182,38 @@ static struct acpi_device *acpi_pci_find_companion(struct device *dev) * @handle: ACPI handle of this device * * Update the d3_delay and d3cold_delay of a PCI device from the ACPI _DSM - * control method of either the device itself or the PCI host bridge. - * - * Function 8, "Reset Delay," applies to the entire hierarchy below a PCI - * host bridge. If it returns one, the OS may assume that all devices in - * the hierarchy have already completed power-on reset delays. + * Function 9 of the device, and cache the parent host bridge's flag for + * ignoring reset delay upon Sx Resume (the flag is originally set in + * acpi_pci_add_bus through _DSM Function 8). * * Function 9, "Device Readiness Durations," applies only to the object * where it is located. It returns delay durations required after various - * events if the device requires less time than the spec requires. Delays - * from this function take precedence over the Reset Delay function. + * events if the device requires less time than the spec requires. + * Values provided by this function can only be used to lower (reduce) the + * latency required by specification or values discovered from device. + * + * This _DSM function is defined by the PCI Firmware Specification Rev 3.2 + * (January 26, 2015), after originally introduced by a draft ECN of + * January 28, 2014, titled "ACPI additions for FW latency optimizations." * - * These _DSM functions are defined by the draft ECN of January 28, 2014, - * titled "ACPI additions for FW latency optimizations." + * XXX The PCI Firmware Specification contradicts itself by stating, in addition + * to the above "can only be used to lower (reduce)", that also: + * Values must be interpreted as overriding any Configuration Ready indicator + * from hardware, whether increasing or decreasing required delays. This + * includes ignoring FRS and DRS notifications where overridden by this + * _DSM function, as well as ignoring values specified in the Readiness Time + * Reporting Extended Capability, if present. + * Meanwhile, the PCI Express Base Specification Revision 5.0 Version 1.0 + * (22 May 2019) states in section 7.9.17 Readiness Time Reporting Extended + * Capability: + * Software is permitted to issue requests upon the earliest of: + * - Receiving a Readiness Notification messages + * - Waiting the appropriate time as per relevant specifications + * - Waiting the time indicated in the associated field of this capability + * - Waiting the time defined by system software or firmware + * The kernel does not yet support Readiness Notifications, and does not yet + * use a Readiness Time Reporting capability if present, so we do not need to + * worry about the prioritization for now. */ static void pci_acpi_optimize_delay(struct pci_dev *pdev, acpi_handle handle) @@ -1189,8 +1222,8 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, int value; union acpi_object *obj, *elements; - if (bridge->ignore_reset_delay) - pdev->d3cold_delay = 0; + pdev->ignore_reset_delay_on_sx_resume = + bridge->ignore_reset_delay_on_sx_resume; obj = acpi_evaluate_dsm(handle, &pci_acpi_dsm_guid, 3, FUNCTION_DELAY_DSM, NULL); diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 0454ca0e4e3f..7e8ca8115c4f 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -917,7 +917,7 @@ static int pci_pm_resume_noirq(struct device *dev) pcie_pme_root_status_cleanup(pci_dev); if (!skip_bus_pm && prev_state == PCI_D3cold) - pci_bridge_wait_for_secondary_bus(pci_dev); + pci_bridge_wait_for_secondary_bus(pci_dev, true); if (pci_has_legacy_pm_support(pci_dev)) return 0; @@ -1321,7 +1321,7 @@ static int pci_pm_runtime_resume(struct device *dev) pci_pm_default_resume(pci_dev); if (prev_state == PCI_D3cold) - pci_bridge_wait_for_secondary_bus(pci_dev); + pci_bridge_wait_for_secondary_bus(pci_dev, false); if (pm && pm->runtime_resume) error = pm->runtime_resume(dev); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 03103bb15b42..4899b12b5a38 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2843,6 +2843,7 @@ void pci_pm_init(struct pci_dev *dev) } dev->pm_cap = pm; + dev->ignore_reset_delay_on_sx_resume = 0; dev->d3_delay = PCI_PM_D3_WAIT; dev->d3cold_delay = PCI_PM_D3COLD_WAIT; dev->bridge_d3 = pci_bridge_d3_possible(dev); @@ -4698,7 +4699,7 @@ static int pci_bus_max_d3cold_delay(const struct pci_bus *bus) * conventional PCI it means Tpvrh + Trhfa specified in PCI 3.0 section * 4.3.2. */ -void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) +void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) { struct pci_dev *child; int delay; @@ -4723,7 +4724,11 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) } /* Take d3cold_delay requirements into account */ - delay = pci_bus_max_d3cold_delay(dev->subordinate); + if (sx_resume && dev->ignore_reset_delay_on_sx_resume) + delay = 0; + else + delay = pci_bus_max_d3cold_delay(dev->subordinate); + if (!delay) { up_read(&pci_bus_sem); return; @@ -4800,7 +4805,7 @@ void pci_reset_secondary_bus(struct pci_dev *dev) ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); - pci_bridge_wait_for_secondary_bus(dev); + pci_bridge_wait_for_secondary_bus(dev, false); } void __weak pcibios_reset_secondary_bus(struct pci_dev *dev) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 659ab3f9042a..c4c3ba926f45 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -107,7 +107,7 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev); void pci_free_cap_save_buffers(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); -void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev); +void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume); static inline void pci_wakeup_event(struct pci_dev *dev) { diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h index 62b7fdcc661c..99865773d65e 100644 --- a/include/linux/pci-acpi.h +++ b/include/linux/pci-acpi.h @@ -107,10 +107,10 @@ static inline void acpiphp_check_host_bridge(struct acpi_device *adev) { } #endif extern const guid_t pci_acpi_dsm_guid; -#define IGNORE_PCI_BOOT_CONFIG_DSM 0x05 -#define DEVICE_LABEL_DSM 0x07 -#define RESET_DELAY_DSM 0x08 -#define FUNCTION_DELAY_DSM 0x09 +#define IGNORE_PCI_BOOT_CONFIG_DSM 0x05 +#define DEVICE_LABEL_DSM 0x07 +#define IGNORE_RESET_DELAY_ON_SX_RESUME_DSM 0x08 +#define FUNCTION_DELAY_DSM 0x09 #else /* CONFIG_ACPI */ static inline void acpi_pci_add_bus(struct pci_bus *bus) { } diff --git a/include/linux/pci.h b/include/linux/pci.h index 3840a541a9de..22a5b7164c32 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -354,6 +354,8 @@ struct pci_dev { user sysfs */ unsigned int clear_retrain_link:1; /* Need to clear Retrain Link bit manually */ + unsigned int ignore_reset_delay_on_sx_resume:1; /* Cached value from + pci_host_bridge */ unsigned int d3_delay; /* D3->D0 transition time in ms */ unsigned int d3cold_delay; /* D3cold->D0 transition time in ms */ @@ -503,7 +505,7 @@ struct pci_host_bridge { void (*release_fn)(struct pci_host_bridge *); void *release_data; struct msi_controller *msi; - unsigned int ignore_reset_delay:1; /* For entire hierarchy */ + unsigned int ignore_reset_delay_on_sx_resume:1; /* For entire hierarchy */ unsigned int no_ext_tags:1; /* No Extended Tags */ unsigned int native_aer:1; /* OS may use PCIe AER */ unsigned int native_pcie_hotplug:1; /* OS may use PCIe hotplug */ From patchwork Mon Mar 2 18:44:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416117 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1B0F924 for ; Mon, 2 Mar 2020 18:46:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C018A2166E for ; Mon, 2 Mar 2020 18:46:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="X0X/Z6W1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727509AbgCBSqb (ORCPT ); Mon, 2 Mar 2020 13:46:31 -0500 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:47672 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqb (ORCPT ); Mon, 2 Mar 2020 13:46:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174790; x=1614710790; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xEtEknNf0OvkG/Jr4HiAOZ9QFQWLGLKe0eJdRuCqzpA=; b=X0X/Z6W1V5SYR2gzWAwTyv+yowNNAea7giG3mC7fIzWngj9hLgenkF3I 8/p4lhz8TDNY+pIXCgnlPH/MPxDe7UXshA/Gxi/veiXLlQKJCmpQfE6Gb UO5sxp/LhZZGd6UO6BltPton0CzKxG30GH2UlwOyynKCmfeRh1YWaM7WL M=; IronPort-SDR: m9r/eMg8Nk5FAUcH142fnKXd5lkAs8VzuDiP8d6YM4XJtBbqxjVo3rWMjXMKrBsEKfx9Fu2f+0 5em+LSvR7URQ== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="20582197" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-22cc717f.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:05 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-22cc717f.us-west-2.amazon.com (Postfix) with ESMTPS id 5A5C9A2687; Mon, 2 Mar 2020 18:46:04 +0000 (UTC) Received: from EX13D04EUA003.ant.amazon.com (10.43.165.148) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:40 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA003.ant.amazon.com (10.43.165.148) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:38 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:35 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 06/17] PCI: Fix us->ms conversion in pci_acpi_optimize_delay Date: Mon, 2 Mar 2020 19:44:18 +0100 Message-ID: <20200302184429.12880-7-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov _DSM Function 9 returns device readiness durations in microseconds. Without this fix, integer truncation could cause msleep()-ing for up to 999us less than actually requested by the firmware. Specifically, if the firmware specifies a 500us delay, msleep(0) would be invoked by the users of the delay values optimized here. Signed-off-by: Stanislav Spassov Reported-by: kbuild test robot Reported-by: kbuild test robot --- drivers/pci/pci-acpi.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c index a8fa13d6089d..508924377bff 100644 --- a/drivers/pci/pci-acpi.c +++ b/drivers/pci/pci-acpi.c @@ -1219,6 +1219,11 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, acpi_handle handle) { struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus); + /* + * _DSM 9 provides values in microseconds, but the kernel uses msleep() + * when waiting, so the code below rounds up when setting value in ms + */ + u64 value_us; int value; union acpi_object *obj, *elements; @@ -1233,12 +1238,18 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, if (obj->type == ACPI_TYPE_PACKAGE && obj->package.count == 5) { elements = obj->package.elements; if (elements[0].type == ACPI_TYPE_INTEGER) { - value = (int)elements[0].integer.value / 1000; + value_us = elements[0].integer.value; + value = (int)(value_us / 1000); + if (value_us % 1000 > 0) + value++; if (value < PCI_PM_D3COLD_WAIT) pdev->d3cold_delay = value; } if (elements[3].type == ACPI_TYPE_INTEGER) { - value = (int)elements[3].integer.value / 1000; + value_us = elements[3].integer.value; + value = (int)(value_us / 1000); + if (value_us % 1000 > 0) + value++; if (value < PCI_PM_D3_WAIT) pdev->d3_delay = value; } From patchwork Mon Mar 2 18:44:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416101 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1495D924 for ; Mon, 2 Mar 2020 18:46:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DA7192166E for ; Mon, 2 Mar 2020 18:46:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="dJvrESnf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727388AbgCBSqK (ORCPT ); Mon, 2 Mar 2020 13:46:10 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56169 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSqK (ORCPT ); Mon, 2 Mar 2020 13:46:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174769; x=1614710769; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KYFiznVd4GCcTMs7pWxyPrB6Z/Apm+l21BN4Ow2lEUM=; b=dJvrESnfUHGzyGe8JyArK0ptoGI0P/VC2NSJ07gCEZI6HsdEUlyLStDa 0x3XqwIXimfYYIr/GQ46VOA6VG2UM54nF3C/uRv6prgMe0gPKFNq+R9g+ Pv+LZHqkpUIUYPjT+GxFGRl+CxomKNKGmFRxTUEcBwnwml3jBGiE4IyU8 8=; IronPort-SDR: ciTfSjOJpvQcJeKfMANr2WgiWQxTHYrJ1xEoIVfrSI5VeBG2AUnBYIe/T+uWGUVkaLV84yT6gl FsW7YasdpxKA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19142040" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:08 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-4e7c8266.us-west-2.amazon.com (Postfix) with ESMTPS id 5C77CA187F; Mon, 2 Mar 2020 18:46:06 +0000 (UTC) Received: from EX13D12EUA001.ant.amazon.com (10.43.165.48) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:44 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUA001.ant.amazon.com (10.43.165.48) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:42 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:39 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 07/17] PCI: Clean up and document PM/reset delays Date: Mon, 2 Mar 2020 19:44:19 +0100 Message-ID: <20200302184429.12880-8-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov ...and replace several "magic numbers" scattered throughout the code. Signed-off-by: Stanislav Spassov Reported-by: kbuild test robot Reported-by: kbuild test robot --- drivers/pci/iov.c | 4 +-- drivers/pci/pci-acpi.c | 4 +-- drivers/pci/pci.c | 21 +++--------- drivers/pci/pci.h | 72 ++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 78 insertions(+), 23 deletions(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 4d1f392b05f9..d4e4a0c0a97f 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -524,7 +524,7 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; pci_cfg_access_lock(dev); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); - msleep(100); + msleep(PCI_VF_ENABLE_DELAY); pci_cfg_access_unlock(dev); rc = sriov_add_vfs(dev, initial); @@ -735,7 +735,7 @@ static void sriov_restore_state(struct pci_dev *dev) pci_iov_set_numvfs(dev, iov->num_VFs); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); if (iov->ctrl & PCI_SRIOV_CTRL_VFE) - msleep(100); + msleep(PCI_VF_ENABLE_DELAY); } /** diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c index 508924377bff..66e8f8199ce0 100644 --- a/drivers/pci/pci-acpi.c +++ b/drivers/pci/pci-acpi.c @@ -1242,7 +1242,7 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, value = (int)(value_us / 1000); if (value_us % 1000 > 0) value++; - if (value < PCI_PM_D3COLD_WAIT) + if (value < PCI_RESET_DELAY) pdev->d3cold_delay = value; } if (elements[3].type == ACPI_TYPE_INTEGER) { @@ -1250,7 +1250,7 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, value = (int)(value_us / 1000); if (value_us % 1000 > 0) value++; - if (value < PCI_PM_D3_WAIT) + if (value < PCI_PM_D3HOT_DELAY) pdev->d3_delay = value; } } diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 4899b12b5a38..aaef00578487 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2844,8 +2844,8 @@ void pci_pm_init(struct pci_dev *dev) dev->pm_cap = pm; dev->ignore_reset_delay_on_sx_resume = 0; - dev->d3_delay = PCI_PM_D3_WAIT; - dev->d3cold_delay = PCI_PM_D3COLD_WAIT; + dev->d3_delay = PCI_PM_D3HOT_DELAY; + dev->d3cold_delay = PCI_RESET_DELAY; dev->bridge_d3 = pci_bridge_d3_possible(dev); dev->d3cold_allowed = true; @@ -4500,12 +4500,7 @@ int pcie_flr(struct pci_dev *dev) if (dev->imm_ready) return 0; - /* - * Per PCIe r4.0, sec 6.6.2, a device must complete an FLR within - * 100ms, but may silently discard requests while the FLR is in - * progress. Wait 100ms before trying to access the device. - */ - msleep(100); + msleep(PCI_FLR_DELAY); return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS); } @@ -4544,13 +4539,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe) if (dev->imm_ready) return 0; - /* - * Per Advanced Capabilities for Conventional PCI ECN, 13 April 2006, - * updated 27 July 2006; a device must complete an FLR within - * 100ms, but may silently discard requests while the FLR is in - * progress. Wait 100ms before trying to access the device. - */ - msleep(100); + msleep(PCI_FLR_DELAY); return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS); } @@ -4590,7 +4579,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D3hot; pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr); - msleep(PCI_PM_D3_WAIT); + msleep(PCI_PM_D3HOT_DELAY); csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D0; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index c4c3ba926f45..9b5dd6ea2f52 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -43,9 +43,75 @@ int pci_probe_reset_function(struct pci_dev *dev); int pci_bridge_secondary_bus_reset(struct pci_dev *dev); int pci_bus_error_reset(struct pci_dev *dev); -#define PCI_PM_D2_DELAY 200 -#define PCI_PM_D3_WAIT 10 -#define PCI_PM_D3COLD_WAIT 100 +/* + * These constants represent the minimum amounts of time mandated by the + * PCI Express Base specification that software needs to wait after + * various PCI device events involving (re-)initialization. Only after + * the appropriate delay has elapsed, is software permitted to issue + * Configuration Requests targeting the affected device. + * + * Relevant sections in PCI Express Base Specification r5.0 (May 22, 2019): + * - 6.6.1 "Conventional Reset" for PCI_RESET_DELAY and PCI_DL_UP_DELAY + * - 6.6.2 "Function Level Reset" for PCI_FLR_DELAY + * - 5.9 "State Transition Recovery Time Requirements" for PCI_PM_D3HOT_DELAY + * and PCI_PM_D2_DELAY + * - 9.3.3.3.1 "VF Enable" for PCI_VF_ENABLE_DELAY + * + * There are mechanisms to reduce some of the delay values for specific devices: + * - a device may expose the Readiness Time Reporting Extended Capability from: + * PCI Express Base Specification r4.0 (September 27, 2017), sec 7.9.17 + * (This is currently not supported by the kernel.) + * - system firmware may provide overrides using an ACPI _DSM Function 9: + * PCI Firmware Specification r3.2 (January 26, 2015), sec 4.6.9 + * (see pci_acpi_optimize_delay) + * + * Unless overridden by _DSM Function 9, other mechanisms may be used to reduce + * or completely avoid some of the delays: + * - Readiness Notifications (DRS and FRS) + * - the Immediate Readiness bit of the Status Register in the PCI header + * - the Immediate_Readiness_on_Return_to_D0 in the Power Management + * Capabilities Register in the PCI Power Management Capability + * (None of these are currently supported by the kernel.) + * + * Note: While devices are required to be responsive to Configuration + * Requests after these delays, they may not respond with Successful + * Completion status until they complete potentially lengthy internal + * initialization sequences. Instead, devices respond with Configuration + * Request Retry Status (CRS) Completions. Therefore, additional waiting + * is necessary as handled by pci_dev_wait(). + */ +/* + * Conventional (non-FLR) reset delay, including D3cold->D0 transitions, + * Secondary Bus Reset, and any platform-specific means of triggering + * a Conventional Reset. + * + * According to PCI Firmware spec r3.2, sec 4.6.9, for devices beneath + * downstream ports supporting the Data Link Layer Active Reporting + * capability, this delay should not be used (see PCI_DL_UP_DELAY). + */ +#define PCI_RESET_DELAY 100 +/* + * Post-DL_Up (Data Link Layer Active) delay applicable for devices immediately + * under a Downstream Port that is capable of reporting Data Link Layer Ready. + * Not to be confused with how much time it takes for the link itself to become + * active (see pcie_wait_for_link_delay). + */ +#define PCI_DL_UP_DELAY 100 +/* + * Post-FLR delay + * Also applies to legacy devices supporting AF_FLR per Advanced Capabilities + * for Conventional PCI ECN, 13 April 2006, updated 27 July 2006) + */ +#define PCI_FLR_DELAY 100 +/* + * D0/D1/D2->D3hot and D3hot->D0 delay + * The specifications do *not* mention overridability of the ->D3hot direction + */ +#define PCI_PM_D3HOT_DELAY 10 +/* Post-VF_Enable delay */ +#define PCI_VF_ENABLE_DELAY 100 +/* D0/D1->D2 and D2->D0 delay */ +#define PCI_PM_D2_DELAY 200 /** * struct pci_platform_pm_ops - Firmware PM callbacks From patchwork Mon Mar 2 18:44:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416103 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 914A5924 for ; Mon, 2 Mar 2020 18:46:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5D22821739 for ; Mon, 2 Mar 2020 18:46:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="JU/ZRLQE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727468AbgCBSqM (ORCPT ); Mon, 2 Mar 2020 13:46:12 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:59922 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSqM (ORCPT ); Mon, 2 Mar 2020 13:46:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174772; x=1614710772; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+4JurlrGb3t9r87PNaUmllkXZG1O6GKaNJNLth7Yflg=; b=JU/ZRLQEdlDWgf8kfJhCv9Zf03rB/GoP5XUszCbFy/QFyrPYxxKtquKK gLeqyi2T72uygkuiUt9ys/Y01acozf43gS/SV58SNro9poQMucSIDTqJM BmO9Z06R9gep0lyTywYsr6q0Kr0v5OgUVztmBiKzLFohsG3MREJRPxBCA A=; IronPort-SDR: b4/wf/BUbq/dEooBe1eRIsHhTCa2DwbQ6ATFygt3ABE7nDwTymPKobmyyelsvzXx5lrlHIID15 0Jy6NzlCM/6Q== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="20320602" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 02 Mar 2020 18:46:11 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com (Postfix) with ESMTPS id E2537A2362; Mon, 2 Mar 2020 18:46:09 +0000 (UTC) Received: from EX13D04EUA001.ant.amazon.com (10.43.165.136) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:48 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA001.ant.amazon.com (10.43.165.136) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:47 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:43 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 08/17] PCI: Add more delay overrides to struct pci_dev Date: Mon, 2 Mar 2020 19:44:20 +0100 Message-ID: <20200302184429.12880-9-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Almost all of the PCI_*_DELAY constants declared in drivers/pci/pci.h can be overridden on a per-device basis, as explained in the constants' comments. To allow storing per-device values, this patch introduces a type to describe the various "initialization events": enum pci_init_event and an array in struct pci_dev, with an override for each event. This array is initially populated with the PCI_*_DELAY constants, but can be later overridden by pci_acpi_optimize_delay. Quirks and drivers may also end up tweaking the values. The new array replaces two previous members: d3_delay and d3cold_delay Direct mentions of PCI_*_DELAY are replaced with the per-device override where applicable. Signed-off-by: Stanislav Spassov --- Documentation/power/pci.rst | 4 ++- arch/x86/pci/intel_mid_pci.c | 2 +- drivers/hid/intel-ish-hid/ipc/ipc.c | 2 +- drivers/mfd/intel-lpss-pci.c | 2 +- drivers/net/ethernet/marvell/sky2.c | 2 +- drivers/pci/iov.c | 4 +-- drivers/pci/pci-acpi.c | 42 ++++++++++++++++------------- drivers/pci/pci.c | 36 ++++++++++++++++++------- drivers/pci/quirks.c | 9 ++++--- include/linux/pci.h | 35 ++++++++++++++++++++++-- 10 files changed, 97 insertions(+), 41 deletions(-) diff --git a/Documentation/power/pci.rst b/Documentation/power/pci.rst index 0924d29636ad..8136c8a4b150 100644 --- a/Documentation/power/pci.rst +++ b/Documentation/power/pci.rst @@ -320,7 +320,9 @@ that these callbacks operate on:: unsigned int d2_support:1; /* Low power state D2 is supported */ unsigned int no_d1d2:1; /* D1 and D2 are forbidden */ unsigned int wakeup_prepared:1; /* Device prepared for wake up */ - unsigned int d3_delay; /* D3->D0 transition time in ms */ + unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time + after various events + in ms */ ... }; diff --git a/arch/x86/pci/intel_mid_pci.c b/arch/x86/pci/intel_mid_pci.c index 00c62115f39c..b42b14cd0e00 100644 --- a/arch/x86/pci/intel_mid_pci.c +++ b/arch/x86/pci/intel_mid_pci.c @@ -322,7 +322,7 @@ static void pci_d3delay_fixup(struct pci_dev *dev) */ if (type1_access_ok(dev->bus->number, dev->devfn, PCI_DEVICE_ID)) return; - dev->d3_delay = 0; + dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] = 0; } DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_d3delay_fixup); diff --git a/drivers/hid/intel-ish-hid/ipc/ipc.c b/drivers/hid/intel-ish-hid/ipc/ipc.c index 8f8dfdf64833..a209b954fbe8 100644 --- a/drivers/hid/intel-ish-hid/ipc/ipc.c +++ b/drivers/hid/intel-ish-hid/ipc/ipc.c @@ -755,7 +755,7 @@ static int _ish_hw_reset(struct ishtp_device *dev) csr |= PCI_D3hot; pci_write_config_word(pdev, pdev->pm_cap + PCI_PM_CTRL, csr); - mdelay(pdev->d3_delay); + mdelay(pdev->delay[PCI_INIT_EVENT_D3HOT_TO_D0]); csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D0; diff --git a/drivers/mfd/intel-lpss-pci.c b/drivers/mfd/intel-lpss-pci.c index c40a6c7d0cf8..5b9e458d0ab1 100644 --- a/drivers/mfd/intel-lpss-pci.c +++ b/drivers/mfd/intel-lpss-pci.c @@ -35,7 +35,7 @@ static int intel_lpss_pci_probe(struct pci_dev *pdev, info->mem = &pdev->resource[0]; info->irq = pdev->irq; - pdev->d3cold_delay = 0; + pdev->delay[PCI_INIT_EVENT_RESET] = 0; /* Probably it is enough to set this for iDMA capable devices only */ pci_set_master(pdev); diff --git a/drivers/net/ethernet/marvell/sky2.c b/drivers/net/ethernet/marvell/sky2.c index ebfd0ceac884..f808562e389d 100644 --- a/drivers/net/ethernet/marvell/sky2.c +++ b/drivers/net/ethernet/marvell/sky2.c @@ -5100,7 +5100,7 @@ static int sky2_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_WORK(&hw->restart_work, sky2_restart); pci_set_drvdata(pdev, hw); - pdev->d3_delay = 300; + pdev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] = 300; return 0; diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index d4e4a0c0a97f..f71fc28b69e6 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -524,7 +524,7 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE; pci_cfg_access_lock(dev); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); - msleep(PCI_VF_ENABLE_DELAY); + msleep(dev->delay[PCI_INIT_EVENT_VF_ENABLE]); pci_cfg_access_unlock(dev); rc = sriov_add_vfs(dev, initial); @@ -735,7 +735,7 @@ static void sriov_restore_state(struct pci_dev *dev) pci_iov_set_numvfs(dev, iov->num_VFs); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); if (iov->ctrl & PCI_SRIOV_CTRL_VFE) - msleep(PCI_VF_ENABLE_DELAY); + msleep(dev->delay[PCI_INIT_EVENT_VF_ENABLE]); } /** diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c index 66e8f8199ce0..4b5c29d2a5ab 100644 --- a/drivers/pci/pci-acpi.c +++ b/drivers/pci/pci-acpi.c @@ -1177,11 +1177,11 @@ static struct acpi_device *acpi_pci_find_companion(struct device *dev) } /** - * pci_acpi_optimize_delay - optimize PCI D3 and D3cold delay from ACPI - * @pdev: the PCI device whose delay is to be updated + * pci_acpi_optimize_delay - optimize PCI readiness delays from ACPI + * @pdev: the PCI device whose delays are to be updated * @handle: ACPI handle of this device * - * Update the d3_delay and d3cold_delay of a PCI device from the ACPI _DSM + * Update the readiness delays of a PCI device from the ACPI _DSM * Function 9 of the device, and cache the parent host bridge's flag for * ignoring reset delay upon Sx Resume (the flag is originally set in * acpi_pci_add_bus through _DSM Function 8). @@ -1226,6 +1226,7 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, u64 value_us; int value; union acpi_object *obj, *elements; + int i; pdev->ignore_reset_delay_on_sx_resume = bridge->ignore_reset_delay_on_sx_resume; @@ -1237,21 +1238,26 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, if (obj->type == ACPI_TYPE_PACKAGE && obj->package.count == 5) { elements = obj->package.elements; - if (elements[0].type == ACPI_TYPE_INTEGER) { - value_us = elements[0].integer.value; - value = (int)(value_us / 1000); - if (value_us % 1000 > 0) - value++; - if (value < PCI_RESET_DELAY) - pdev->d3cold_delay = value; - } - if (elements[3].type == ACPI_TYPE_INTEGER) { - value_us = elements[3].integer.value; - value = (int)(value_us / 1000); - if (value_us % 1000 > 0) - value++; - if (value < PCI_PM_D3HOT_DELAY) - pdev->d3_delay = value; + for (i = 0; i < 5; i++) { + if (elements[i].type == ACPI_TYPE_INTEGER) { + value_us = elements[i].integer.value; + value = (int)(value_us / 1000); + if (value_us % 1000 > 0) + value++; + /* + * XXX This relies on the initial values in the + * delay array being set using the PCI_*_DELAY + * macros in drivers/pci/pci.h + * Once the kernel has support for Readiness + * Time Reporting Extended Capability, this + * needs fixing to honor prioritization of + * overrides. Also, a flag would need to be + * set to disable the use of Readiness + * Notifications at some point. + */ + if (value < pdev->delay[i]) + pdev->delay[i] = value; + } } } ACPI_FREE(obj); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index aaef00578487..ba54164652cc 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -43,6 +43,15 @@ const char *pci_power_names[] = { }; EXPORT_SYMBOL_GPL(pci_power_names); +const char *pci_init_event_names[] = { + [PCI_INIT_EVENT_RESET] = "conventional reset", + [PCI_INIT_EVENT_DL_UP] = "DL Up", + [PCI_INIT_EVENT_FLR] = "FLR", + [PCI_INIT_EVENT_D3HOT_TO_D0] = "PM D3hot->D0", + [PCI_INIT_EVENT_VF_ENABLE] = "VF Enable", +}; +EXPORT_SYMBOL_GPL(pci_init_event_names); + int isa_dma_bridge_buggy; EXPORT_SYMBOL(isa_dma_bridge_buggy); @@ -66,7 +75,7 @@ struct pci_pme_device { static void pci_dev_d3_sleep(struct pci_dev *dev) { - unsigned int delay = dev->d3_delay; + unsigned int delay = dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0]; if (delay < pci_pm_d3_delay) delay = pci_pm_d3_delay; @@ -2844,8 +2853,11 @@ void pci_pm_init(struct pci_dev *dev) dev->pm_cap = pm; dev->ignore_reset_delay_on_sx_resume = 0; - dev->d3_delay = PCI_PM_D3HOT_DELAY; - dev->d3cold_delay = PCI_RESET_DELAY; + dev->delay[PCI_INIT_EVENT_RESET] = PCI_RESET_DELAY; + dev->delay[PCI_INIT_EVENT_DL_UP] = PCI_DL_UP_DELAY; + dev->delay[PCI_INIT_EVENT_FLR] = PCI_FLR_DELAY; + dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] = PCI_PM_D3HOT_DELAY; + dev->delay[PCI_INIT_EVENT_VF_ENABLE] = PCI_VF_ENABLE_DELAY; dev->bridge_d3 = pci_bridge_d3_possible(dev); dev->d3cold_allowed = true; @@ -4500,7 +4512,7 @@ int pcie_flr(struct pci_dev *dev) if (dev->imm_ready) return 0; - msleep(PCI_FLR_DELAY); + msleep(dev->delay[PCI_INIT_EVENT_FLR]); return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS); } @@ -4539,7 +4551,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe) if (dev->imm_ready) return 0; - msleep(PCI_FLR_DELAY); + msleep(dev->delay[PCI_INIT_EVENT_FLR]); return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS); } @@ -4556,7 +4568,9 @@ static int pci_af_flr(struct pci_dev *dev, int probe) * * NOTE: This causes the caller to sleep for twice the device power transition * cooldown period, which for the D0->D3hot and D3hot->D0 transitions is 10 ms - * by default (i.e. unless the @dev's d3_delay field has a different value). + * by default (i.e. unless the @dev's delay[PCI_INIT_EVENT_D3HOT_TO_D0] field + * has a different value). + * * Moreover, only devices in D0 can be reset by this function. */ static int pci_pm_reset(struct pci_dev *dev, int probe) @@ -4666,12 +4680,14 @@ static int pci_bus_max_d3cold_delay(const struct pci_bus *bus) const struct pci_dev *pdev; int min_delay = 100; int max_delay = 0; + int delay; list_for_each_entry(pdev, &bus->devices, bus_list) { - if (pdev->d3cold_delay < min_delay) - min_delay = pdev->d3cold_delay; - if (pdev->d3cold_delay > max_delay) - max_delay = pdev->d3cold_delay; + delay = pdev->delay[PCI_INIT_EVENT_RESET]; + if (delay < min_delay) + min_delay = delay; + if (delay > max_delay) + max_delay = delay; } return max(min_delay, max_delay); diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 29f473ebf20f..12f22af0cbef 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -1873,12 +1873,13 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x260b, quirk_intel_pcie_pm); static void quirk_d3hot_delay(struct pci_dev *dev, unsigned int delay) { - if (dev->d3_delay >= delay) + + if (dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] >= delay) return; - dev->d3_delay = delay; + dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] = delay; pci_info(dev, "extending delay after power-on from D3hot to %d msec\n", - dev->d3_delay); + dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0]); } static void quirk_radeon_pm(struct pci_dev *dev) @@ -3310,7 +3311,7 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0152, disable_igfx_irq); */ static void quirk_remove_d3_delay(struct pci_dev *dev) { - dev->d3_delay = 0; + dev->delay[PCI_INIT_EVENT_D3HOT_TO_D0] = 0; } /* C600 Series devices do not need 10ms d3_delay */ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0412, quirk_remove_d3_delay); diff --git a/include/linux/pci.h b/include/linux/pci.h index 22a5b7164c32..16dbfff2092e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -268,6 +268,36 @@ enum pci_bus_speed { enum pci_bus_speed pcie_get_speed_cap(struct pci_dev *dev); enum pcie_link_width pcie_get_width_cap(struct pci_dev *dev); +/* + * The first five constants correspond to delays specified in both: + * PCI Firmware Specification Rev. 3.2 (January 26, 2015), + * Section 4.6.9. "_DSM for Specifying Device Readiness Durations", and + * PCI Express Base Specification Revision 5.0 Version 1.0 (May 22, 2019) + * Section 7.9.17 "Readiness Time Reporting Extended Capability" + * + * The code assumes these constants are in the same order as in the + * PCI Firmware Specification. + */ +enum pci_init_event { + PCI_INIT_EVENT_RESET = 0, /* D3cold->D0, SBR */ + PCI_INIT_EVENT_DL_UP = 1, + PCI_INIT_EVENT_FLR = 2, + PCI_INIT_EVENT_D3HOT_TO_D0 = 3, + PCI_INIT_EVENT_VF_ENABLE = 4, + PCI_INIT_EVENT_COUNT /* Keep this as last element */ +}; + +/* Remember to update this when the list above changes! */ +extern const char *pci_init_event_names[]; + +static inline const char *pci_init_event_name(enum pci_init_event event) +{ + if (event >= PCI_INIT_EVENT_COUNT) + return ""; + else + return pci_init_event_names[event]; +} + struct pci_cap_saved_data { u16 cap_nr; bool cap_extended; @@ -356,8 +386,9 @@ struct pci_dev { bit manually */ unsigned int ignore_reset_delay_on_sx_resume:1; /* Cached value from pci_host_bridge */ - unsigned int d3_delay; /* D3->D0 transition time in ms */ - unsigned int d3cold_delay; /* D3cold->D0 transition time in ms */ + unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time + after various events + in ms */ #ifdef CONFIG_PCIEASPM struct pcie_link_state *link_state; /* ASPM link state */ From patchwork Mon Mar 2 18:44:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD5D792A for ; Mon, 2 Mar 2020 18:46:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9D89E21D56 for ; Mon, 2 Mar 2020 18:46:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="SeVtVykV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727470AbgCBSqP (ORCPT ); Mon, 2 Mar 2020 13:46:15 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56169 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSqP (ORCPT ); Mon, 2 Mar 2020 13:46:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174774; x=1614710774; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=f1H7XOSMeRzetd59oqKV+/fQ2SI/DghKGu3MCc/evPI=; b=SeVtVykV3Se9bO7q9/pPOpSrUiVjXXVN15Aqa8tkSXagcE8/frsWKwrQ RVLv/WQQBZKJYFw7v5uO8z4oRbggCWuVA5ELdk4uGya6+kwhf7cwZMT1t blhRgH829pQFb2E+MGBI7Fo2z5Hm9Q0sdpGuXrjvdIPLmoQ11H56sO1fm s=; IronPort-SDR: OvBufc5a0yvD6kkq3CEIV1a6yG1A8jMz8YtYtEmb7c7x3pfoUTdi73owiBfMlN3EN/emwADcu7 KrUq0V3tmtKQ== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19142063" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:12 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id 470E6A2279; Mon, 2 Mar 2020 18:46:11 +0000 (UTC) Received: from EX13D12EUC001.ant.amazon.com (10.43.164.45) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:53 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUC001.ant.amazon.com (10.43.164.45) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:51 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:48 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 09/17] PCI: Generalize pci_bus_max_d3cold_delay to pci_bus_max_delay Date: Mon, 2 Mar 2020 19:44:21 +0100 Message-ID: <20200302184429.12880-10-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov This allows determining the maximum of any of the several delay values stored in struct pci_dev. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index ba54164652cc..e4840dbf2d1c 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4669,21 +4669,26 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active) } /* - * Find maximum D3cold delay required by all the devices on the bus. The - * spec says 100 ms, but firmware can lower it and we allow drivers to - * increase it as well. + * Find maximum delay required by all the devices on the bus after the + * given initialization event. * * Called with @pci_bus_sem locked for reading. + * + * XXX: It is not clear if this should descend down across bridges (if any) */ -static int pci_bus_max_d3cold_delay(const struct pci_bus *bus) +static int pci_bus_max_delay(const struct pci_bus *bus, + enum pci_init_event event, int default_delay) { const struct pci_dev *pdev; - int min_delay = 100; + int min_delay = default_delay; int max_delay = 0; int delay; + if (event >= PCI_INIT_EVENT_COUNT) + return default_delay; + list_for_each_entry(pdev, &bus->devices, bus_list) { - delay = pdev->delay[PCI_INIT_EVENT_RESET]; + delay = pdev->delay[event]; if (delay < min_delay) min_delay = delay; if (delay > max_delay) @@ -4728,11 +4733,13 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) return; } - /* Take d3cold_delay requirements into account */ + /* Take delay requirements into account */ if (sx_resume && dev->ignore_reset_delay_on_sx_resume) delay = 0; else - delay = pci_bus_max_d3cold_delay(dev->subordinate); + delay = pci_bus_max_delay(dev->subordinate, + PCI_INIT_EVENT_RESET, + PCI_RESET_DELAY); if (!delay) { up_read(&pci_bus_sem); From patchwork Mon Mar 2 18:44:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416107 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F3B992A for ; Mon, 2 Mar 2020 18:46:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3CCFB21739 for ; Mon, 2 Mar 2020 18:46:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="PqqlUm9D" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727450AbgCBSqQ (ORCPT ); Mon, 2 Mar 2020 13:46:16 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:52790 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSqQ (ORCPT ); Mon, 2 Mar 2020 13:46:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174776; x=1614710776; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9xqMD1N8mF24q9FbAWh4uDveX6H7dYjjMS/baY775kQ=; b=PqqlUm9DcbDVe2Q1ybxrTGCRrfxUJ+JfF/JeWV9X4xnP8EK3A1/6M+6B DO4gXf+lGlptRUkfURV6JKHD2N4yHVofEUVsRQYIo05OeSeEVVFwMO1pY hxcpyqNbx7RFtjlJ38oD47DqRdKsuiVR3leSWXDACH7yCJ8ZpAwc3ARq/ s=; IronPort-SDR: kO8HjqOLM8oqROWJwjcu+g6YR1IS+rg38zj8EgQHECQo0zDNMYwRDEeOe6+ymECVf12FftcEMI 39xDTktKKExA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="28691717" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 02 Mar 2020 18:46:14 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com (Postfix) with ESMTPS id 15E97C5D53; Mon, 2 Mar 2020 18:46:13 +0000 (UTC) Received: from EX13D04EUB002.ant.amazon.com (10.43.166.51) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:45:57 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUB002.ant.amazon.com (10.43.166.51) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:45:55 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:52 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 10/17] PCI: Use correct delay in pci_bridge_wait_for_secondary_bus Date: Mon, 2 Mar 2020 19:44:22 +0100 Message-ID: <20200302184429.12880-11-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov PCI Express Base Specification r5.0 (May 22, 2019) details the rules for device reset in Section 6.6. For a Downstream Port that does not support Link speeds greater than 5.0 GT/s, the minimum waiting period before software is permitted to send a Configuration Request after a Conventional Reset is 100ms (PCI_RESET_DELAY). For a Downstream Port that supports Link speeds greater than 5.0 GT/s (such ports are required to be Data Link Layer Link Active Reporting capable), the period is again 100ms but measured after the link has become active (PCI_DL_UP_DELAY). The delays for both cases above can be overridden independently, and pci_bridge_wait_for_secondary_bus should use the appropriate one. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index e4840dbf2d1c..7e08c5f38190 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4736,6 +4736,12 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) /* Take delay requirements into account */ if (sx_resume && dev->ignore_reset_delay_on_sx_resume) delay = 0; + else if (pcie_downstream_port(dev) && + pcie_get_speed_cap(dev) > PCIE_SPEED_5_0GT && + dev->link_active_reporting) + delay = pci_bus_max_delay(dev->subordinate, + PCI_INIT_EVENT_DL_UP, + PCI_DL_UP_DELAY); else delay = pci_bus_max_delay(dev->subordinate, PCI_INIT_EVENT_RESET, From patchwork Mon Mar 2 18:44:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416119 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F51E92A for ; Mon, 2 Mar 2020 18:46:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F2D712166E for ; Mon, 2 Mar 2020 18:46:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="CFvShSpU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727414AbgCBSqc (ORCPT ); Mon, 2 Mar 2020 13:46:32 -0500 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:29635 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqc (ORCPT ); Mon, 2 Mar 2020 13:46:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174791; x=1614710791; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JNPVRg0IXhKYsKVgVhd2+6miS3vS/5AexheE4GJwtE0=; b=CFvShSpUB6L2Vzn9WCH0wcg+AWdySCvbd1NjIW8bSkW7Olxn+08MyWtE d8tbi5CBw3nZPLK2JTvAQ4fadg7yKPSvDNYcw0muEUGz4cVNOrdO95VHt O4/7YxgbUDWzTDIrFJN4ZtLnAF+Amc4JEZrxH1mLJZQfsSkUc06LSlfK0 M=; IronPort-SDR: gmPq8usxjQZUxBcpQlmHbvtrPA4MqGtxcnJb/0gk8XT+ptiT2oj8HqgeIjiNx6hKu5glDVGdDG 9TEjHhJ6vlqw== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19686304" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2b-baacba05.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 02 Mar 2020 18:46:15 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-baacba05.us-west-2.amazon.com (Postfix) with ESMTPS id 6EB81A1ECE; Mon, 2 Mar 2020 18:46:14 +0000 (UTC) Received: from EX13D12EUC002.ant.amazon.com (10.43.164.134) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:01 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUC002.ant.amazon.com (10.43.164.134) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:00 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:45:56 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 11/17] PCI: Refactor pci_dev_wait to remove timeout parameter Date: Mon, 2 Mar 2020 19:44:23 +0100 Message-ID: <20200302184429.12880-12-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Currently, all callers supply the same value, and in the future pci_dev_wait itself could determine the appropriate timeout based on values stored in struct pci_dev, and the reset type. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 7e08c5f38190..9435e2b19f7b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1030,8 +1030,9 @@ void pci_wakeup_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_wakeup, NULL); } -static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout) +static int pci_dev_wait(struct pci_dev *dev, char *reset_type) { + int timeout = PCIE_RESET_READY_POLL_MS; int delay = 1; u32 id; @@ -4514,7 +4515,7 @@ int pcie_flr(struct pci_dev *dev) msleep(dev->delay[PCI_INIT_EVENT_FLR]); - return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS); + return pci_dev_wait(dev, "FLR"); } EXPORT_SYMBOL_GPL(pcie_flr); @@ -4553,7 +4554,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe) msleep(dev->delay[PCI_INIT_EVENT_FLR]); - return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS); + return pci_dev_wait(dev, "AF_FLR"); } /** @@ -4600,7 +4601,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr); pci_dev_d3_sleep(dev); - return pci_dev_wait(dev, "PM D3hot->D0", PCIE_RESET_READY_POLL_MS); + return pci_dev_wait(dev, "PM D3hot->D0"); } /** @@ -4842,7 +4843,7 @@ int pci_bridge_secondary_bus_reset(struct pci_dev *dev) { pcibios_reset_secondary_bus(dev); - return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS); + return pci_dev_wait(dev, "bus reset"); } EXPORT_SYMBOL_GPL(pci_bridge_secondary_bus_reset); From patchwork Mon Mar 2 18:44:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416109 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF296924 for ; Mon, 2 Mar 2020 18:46:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8ED012166E for ; Mon, 2 Mar 2020 18:46:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="UHQDXTTG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726451AbgCBSqW (ORCPT ); Mon, 2 Mar 2020 13:46:22 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:52790 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSqV (ORCPT ); Mon, 2 Mar 2020 13:46:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174781; x=1614710781; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=boi/zS9lyKDU1FKDZKAWPh9nzOZgJ/80+f/6R2sG+NE=; b=UHQDXTTGR2HVCCG+SEAVqlFlL41TZssT6vt7ifvmKYUafAB4HIM16LSJ R21HHYnMZbXXclFpwUUo92pGDeTxN/kfqJyoNj4+Y/aSM8RxYBhZT1dc4 YjEYCGhCuxUq6zpQL0Bhf2g9L7F4pATE7Ob2XO7AUBZtSok7Mju6/VNHN g=; IronPort-SDR: fos/NqpK5c3g5RO2Kjrji5filQUsg2iLwvOVjiYcfYnbG4a/tcCqkK6p/wLYSGRvnoSqK22Hbo X3n7rNsQAwYA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="28691747" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-baacba05.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 02 Mar 2020 18:46:20 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-baacba05.us-west-2.amazon.com (Postfix) with ESMTPS id 0AA16A18ED; Mon, 2 Mar 2020 18:46:18 +0000 (UTC) Received: from EX13D04EUA003.ant.amazon.com (10.43.165.148) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:05 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA003.ant.amazon.com (10.43.165.148) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:04 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:00 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 12/17] PCI: Refactor pci_dev_wait to take pci_init_event Date: Mon, 2 Mar 2020 19:44:24 +0100 Message-ID: <20200302184429.12880-13-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Knowing what kind of event knocked the device out could be useful not only for log output, but also to use different timeout/waiting behavior. Note: we do lose some specificity in log output due to the aliasing of FLR and AF_FLR, but it is doubtful the distinction is worthwhile. Also, "bus reset" does not exactly match the more generic name for PCI_INIT_EVENT_RESET, which could break programs that scrape kernel output for overly specific patterns. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 9435e2b19f7b..5d62d4841d68 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1030,8 +1030,9 @@ void pci_wakeup_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_wakeup, NULL); } -static int pci_dev_wait(struct pci_dev *dev, char *reset_type) +static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) { + const char *event_name = pci_init_event_name(event); int timeout = PCIE_RESET_READY_POLL_MS; int delay = 1; u32 id; @@ -1052,13 +1053,13 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type) while (id == ~0) { if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", - delay - 1, reset_type); + delay - 1, event_name); return -ETIMEDOUT; } if (delay > 1000) pci_info(dev, "not ready %dms after %s; waiting\n", - delay - 1, reset_type); + delay - 1, event_name); msleep(delay); delay *= 2; @@ -1067,7 +1068,7 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type) if (delay > 1000) pci_info(dev, "ready %dms after %s\n", delay - 1, - reset_type); + event_name); return 0; } @@ -4515,7 +4516,7 @@ int pcie_flr(struct pci_dev *dev) msleep(dev->delay[PCI_INIT_EVENT_FLR]); - return pci_dev_wait(dev, "FLR"); + return pci_dev_wait(dev, PCI_INIT_EVENT_FLR); } EXPORT_SYMBOL_GPL(pcie_flr); @@ -4554,7 +4555,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe) msleep(dev->delay[PCI_INIT_EVENT_FLR]); - return pci_dev_wait(dev, "AF_FLR"); + return pci_dev_wait(dev, PCI_INIT_EVENT_FLR); } /** @@ -4601,7 +4602,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr); pci_dev_d3_sleep(dev); - return pci_dev_wait(dev, "PM D3hot->D0"); + return pci_dev_wait(dev, PCI_INIT_EVENT_D3HOT_TO_D0); } /** @@ -4843,7 +4844,7 @@ int pci_bridge_secondary_bus_reset(struct pci_dev *dev) { pcibios_reset_secondary_bus(dev); - return pci_dev_wait(dev, "bus reset"); + return pci_dev_wait(dev, PCI_INIT_EVENT_RESET); } EXPORT_SYMBOL_GPL(pci_bridge_secondary_bus_reset); From patchwork Mon Mar 2 18:44:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416111 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32C32924 for ; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 11ECE2166E for ; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="f6y485F3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727479AbgCBSq3 (ORCPT ); Mon, 2 Mar 2020 13:46:29 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:56252 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727126AbgCBSq3 (ORCPT ); Mon, 2 Mar 2020 13:46:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174788; x=1614710788; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zo+xgKja6kmPxHfkNHFXpC45vKnXxIgJgU/su5JdReI=; b=f6y485F3J3AonfQxysgWrgIhGTyTFE13yarn7e9QVJerGwcK/cXBOhgE SIMKFRV9nqid8Wh3GP7XEd484iYaPQkHul+ChwdyZOxa0LGT/DG4+Rnuv nI9/HQB0bLQpXdQlwDz6hUr2XKleSo4LEKIlm5DBYhNyjXlq/4LEcJX3T I=; IronPort-SDR: 7NsYMCMRNrT/UHe7DWpaREhel35xf/uLqGgyEbI40WuPU2ES+S0dS0606b9McECDA/HIeXIlum ovC5OrGm0ZDA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19142125" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 02 Mar 2020 18:46:26 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com (Postfix) with ESMTPS id BB6F7A16C0; Mon, 2 Mar 2020 18:46:24 +0000 (UTC) Received: from EX13D12EUC004.ant.amazon.com (10.43.164.129) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:10 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUC004.ant.amazon.com (10.43.164.129) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:08 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:05 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 13/17] PCI: Cache CRS Software Visibiliy in struct pci_dev Date: Mon, 2 Mar 2020 19:44:25 +0100 Message-ID: <20200302184429.12880-14-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov Arguably, since CRS SV is a capability of Root Ports and determines how Root Ports will handle incoming CRS Completions, it makes more sense to store this flag in the struct pci bus that represents the port's secondary bus, and to cache it in any buses further down the hierarchy. However, storing the flag in struct pci_dev allows individual devices to be marked as not supporting CRS properly even when CRS SV is enabled on their parent Root Port. This way, future code that relies on the new flag will not be misled that it is safe to probe a device by relying on CRS SV to not cause platform timeouts (See the note on CRS SV on p. 553 of PCI Express Base Specification r5.0 from May 22, 2019). Note: Endpoints integrated into the Root Complex (RCiEP) may also be capable of using CRS. In that case, the software visibility is controlled using a Root Complex Register Block (RCRB). This is currently not supported by the kernel. The code introduced here would simply not set the newly introduced flag for RCiEP as for those bus->self is NULL. Signed-off-by: Stanislav Spassov --- drivers/pci/probe.c | 8 +++++++- include/linux/pci.h | 3 +++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 512cb4312ddd..eeff8a07f330 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1080,9 +1080,11 @@ static void pci_enable_crs(struct pci_dev *pdev) /* Enable CRS Software Visibility if supported */ pcie_capability_read_word(pdev, PCI_EXP_RTCAP, &root_cap); - if (root_cap & PCI_EXP_RTCAP_CRSVIS) + if (root_cap & PCI_EXP_RTCAP_CRSVIS) { pcie_capability_set_word(pdev, PCI_EXP_RTCTL, PCI_EXP_RTCTL_CRSSVE); + pdev->crssv_enabled = true; + } } static unsigned int pci_scan_child_bus_extend(struct pci_bus *bus, @@ -2414,6 +2416,10 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) list_add_tail(&dev->bus_list, &bus->devices); up_write(&pci_bus_sem); + /* Propagate CRS Software Visibility bit from the parent bridge */ + if (bus->self) + dev->crssv_enabled = bus->self->crssv_enabled; + ret = pcibios_add_device(dev); WARN_ON(ret < 0); diff --git a/include/linux/pci.h b/include/linux/pci.h index 16dbfff2092e..1763e98625b9 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -386,6 +386,9 @@ struct pci_dev { bit manually */ unsigned int ignore_reset_delay_on_sx_resume:1; /* Cached value from pci_host_bridge */ + unsigned int crssv_enabled:1; /* Configuration Request Retry + Status Software Visibility + enabled on (parent) bridge */ unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time after various events in ms */ From patchwork Mon Mar 2 18:44:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416113 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5C3E417E0 for ; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3B0ED2467B for ; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="R7Xhc8a0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727126AbgCBSq3 (ORCPT ); Mon, 2 Mar 2020 13:46:29 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:60001 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSq3 (ORCPT ); Mon, 2 Mar 2020 13:46:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174789; x=1614710789; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KTszniR1B8GtjTWJyYnBLjmuCmS7g4iBL/oem8tvWpM=; b=R7Xhc8a0BVfVrJgcIwPCUMdXkuCnN6IOZLiT7nYHDoYXT/AcTFbeX5Mw KCBd0TAy5q7/rwYcbMeRjqTCYJoQuuo7AhVggN/cmHy4KDwq8npjsDEY6 vMbDDB9g1Rgl+SPf+6cBtT1YoWDHXtuuoPIZ392TPaDjvwGFRp65CgoOI Y=; IronPort-SDR: CsBh9RpLxJSDTeJxmQhYUyM9m+zUrLKDcODM49vr0zRg2/Sl1rpsHprDpARYMM/1BCn34wRz93 +HKsndJKn+0w== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="20320689" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-55156cd4.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 02 Mar 2020 18:46:29 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-55156cd4.us-west-2.amazon.com (Postfix) with ESMTPS id 66B9FA22E6; Mon, 2 Mar 2020 18:46:27 +0000 (UTC) Received: from EX13D04EUA001.ant.amazon.com (10.43.165.136) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:14 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA001.ant.amazon.com (10.43.165.136) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:13 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:09 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 14/17] PCI: Introduce per-device reset_ready_poll override Date: Mon, 2 Mar 2020 19:44:26 +0100 Message-ID: <20200302184429.12880-15-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov A broken device may never become responsive after reset, hence the need for a timeout. However, waiting for too long can have unintended side effects such as triggering hung task timeouts for processes waiting on a lock held during the reset. Locks that are shared across multiple devices, such as VFIO's per-bus reflck, are especially problematic, because a single broken VF can cause hangs for processes working with other VFs on the same bus. To allow lowering the global default post-reset timeout, while still accommodating devices that require more time, this patch introduces a per-device override that can be configured via a quirk. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 5 +---- drivers/pci/pci.h | 3 +++ drivers/pci/probe.c | 2 ++ include/linux/pci.h | 3 +++ 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 5d62d4841d68..e81fd3b53bd0 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -157,9 +157,6 @@ static int __init pcie_port_pm_setup(char *str) } __setup("pcie_port_pm=", pcie_port_pm_setup); -/* Time to wait after a reset for device to become responsive */ -#define PCIE_RESET_READY_POLL_MS 60000 - /** * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children * @bus: pointer to PCI bus structure to search @@ -1033,7 +1030,7 @@ void pci_wakeup_bus(struct pci_bus *bus) static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) { const char *event_name = pci_init_event_name(event); - int timeout = PCIE_RESET_READY_POLL_MS; + int timeout = dev->reset_ready_poll_ms; int delay = 1; u32 id; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 9b5dd6ea2f52..d8043d4dbe2f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -113,6 +113,9 @@ int pci_bus_error_reset(struct pci_dev *dev); /* D0/D1->D2 and D2->D0 delay */ #define PCI_PM_D2_DELAY 200 +/* Time to wait after a reset for device to become responsive */ +#define PCIE_RESET_READY_POLL_MS 60000 + /** * struct pci_platform_pm_ops - Firmware PM callbacks * diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index eeff8a07f330..50b7219406ed 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2168,6 +2168,8 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) if (!dev) return NULL; + dev->reset_ready_poll_ms = PCIE_RESET_READY_POLL_MS; + INIT_LIST_HEAD(&dev->bus_list); dev->dev.type = &pci_dev_type; dev->bus = pci_bus_get(bus); diff --git a/include/linux/pci.h b/include/linux/pci.h index 1763e98625b9..978ede89741e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -392,6 +392,9 @@ struct pci_dev { unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time after various events in ms */ + unsigned int reset_ready_poll_ms; /* Timeout for polling after + reset before the device is + deemed broken. */ #ifdef CONFIG_PCIEASPM struct pcie_link_state *link_state; /* ASPM link state */ From patchwork Mon Mar 2 18:44:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416125 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77EC8924 for ; Mon, 2 Mar 2020 18:46:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 523482166E for ; Mon, 2 Mar 2020 18:46:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="XZ83bkh/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727491AbgCBSqp (ORCPT ); Mon, 2 Mar 2020 13:46:45 -0500 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:28549 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqp (ORCPT ); Mon, 2 Mar 2020 13:46:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174805; x=1614710805; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j4m+zq3gvOi1+9w1LAeWgb10VdG3HMJ9oTkeRMpYQqM=; b=XZ83bkh/UiSlgodKJr6iACXYlsvPllf7JHYQCXNunZLvxhGfEQX2X/QP Z1tpdzw862NqTEFcDNovAzDlHncD4pUU0PveLXXvC7kqtov1aKIqKj5YE bBM4CufEI8f/lFPYduVUQXI10iX0IoTCLJ0RdU+ygbnlaNmaF2/70P1Le w=; IronPort-SDR: QYi+yVJYza5I0nW9OxccCu2TeKpOQTURYlJ1FqvP2FmlJlHIzLBrinx+nPnaWhNW3vzfRQyuL2 aVCiRk6CwUDg== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19428575" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 02 Mar 2020 18:46:32 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-6f38efd9.us-west-2.amazon.com (Postfix) with ESMTPS id 4EFE1A36F9; Mon, 2 Mar 2020 18:46:30 +0000 (UTC) Received: from EX13D04EUA002.ant.amazon.com (10.43.165.75) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:18 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA002.ant.amazon.com (10.43.165.75) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:17 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:13 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 15/17] PCI: Refactor polling loop out of pci_dev_wait Date: Mon, 2 Mar 2020 19:44:27 +0100 Message-ID: <20200302184429.12880-16-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov This patch does not (intentionally) introduce any observable difference in runtime behavior. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 71 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index e81fd3b53bd0..f1ba931b0ead 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1027,27 +1027,28 @@ void pci_wakeup_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_wakeup, NULL); } -static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) +/* + * Performs DWORD Configuration Reads at a specific offset until the value read + * (with mask applied) is not equal to bad_value. + */ +static inline int pci_dev_poll_until_not_equal(struct pci_dev *dev, int where, + u32 mask, u32 bad_value, + const char *event_name, + int timeout, int *waited, + u32 *final_value) { - const char *event_name = pci_init_event_name(event); - int timeout = dev->reset_ready_poll_ms; int delay = 1; - u32 id; + u32 value; - /* - * After reset, the device should not silently discard config - * requests, but it may still indicate that it needs more time by - * responding to them with CRS completions. The Root Port will - * generally synthesize ~0 data to complete the read (except when - * CRS SV is enabled and the read was for the Vendor ID; in that - * case it synthesizes 0x0001 data). - * - * Wait for the device to return a non-CRS completion. Read the - * Command register instead of Vendor ID so we don't have to - * contend with the CRS SV value. - */ - pci_read_config_dword(dev, PCI_COMMAND, &id); - while (id == ~0) { + if (!event_name) + event_name = ""; + + if (waited) + delay = *waited + 1; + + pci_read_config_dword(dev, where, &value); + + while ((value & mask) == bad_value) { if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", delay - 1, event_name); @@ -1060,16 +1061,44 @@ static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) msleep(delay); delay *= 2; - pci_read_config_dword(dev, PCI_COMMAND, &id); + + pci_read_config_dword(dev, where, &value); } if (delay > 1000) - pci_info(dev, "ready %dms after %s\n", delay - 1, - event_name); + pci_info(dev, "ready %dms after %s\n", delay - 1, event_name); + + if (waited) + *waited = delay - 1; + + if (final_value) + *final_value = value; return 0; } +static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) +{ + const char *event_name = pci_init_event_name(event); + int timeout = dev->reset_ready_poll_ms; + + /* + * After reset, the device should not silently discard config + * requests, but it may still indicate that it needs more time by + * responding to them with CRS completions. The Root Port will + * generally synthesize ~0 data to complete the read (except when + * CRS SV is enabled and the read was for the Vendor ID; in that + * case it synthesizes 0x0001 data). + * + * Wait for the device to return a non-CRS completion. Read the + * Command register instead of Vendor ID so we don't have to + * contend with the CRS SV value. + */ + return pci_dev_poll_until_not_equal(dev, PCI_COMMAND, ~0, ~0, + event_name, timeout, NULL, + NULL); +} + /** * pci_power_up - Put the given device into D0 * @dev: PCI device to power up From patchwork Mon Mar 2 18:44:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416121 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 878E592A for ; Mon, 2 Mar 2020 18:46:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5AA6B21739 for ; Mon, 2 Mar 2020 18:46:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="QQhiJ//B" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727481AbgCBSqi (ORCPT ); Mon, 2 Mar 2020 13:46:38 -0500 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:21256 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqi (ORCPT ); Mon, 2 Mar 2020 13:46:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174797; x=1614710797; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+xq6lveZ+prywBAcKUivfYzgKg22lx6AhvILIKrWiGg=; b=QQhiJ//BlWDktyspSr4EhPMqYDQnPbg7r15qz+hvkLTwggfuHzxPpkpc imBGGJdCdJmClaMzDIUbAzTpHa17UZzJv1J9TlrpQH69Z7Sx0dH8rHqIt O27+lUtdwVRYDvfx6dErQNwmsXQg3u7tpjLzjiMNsWl6Gs+3LwJmkG+Vj 4=; IronPort-SDR: KH34hA9bGk7InqJD/pmZSs9iwTKAeIjEaMou9EBY1q0unkHGGYAdD3lB/dlqion3mAJ4kNISJg 0dkQuKVAuhRA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="30074795" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-2225282c.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 02 Mar 2020 18:46:35 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-2225282c.us-west-2.amazon.com (Postfix) with ESMTPS id 0221BA3608; Mon, 2 Mar 2020 18:46:33 +0000 (UTC) Received: from EX13D12EUC001.ant.amazon.com (10.43.164.45) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:23 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUC001.ant.amazon.com (10.43.164.45) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:21 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:18 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 16/17] PCI: Add CRS handling to pci_dev_wait() Date: Mon, 2 Mar 2020 19:44:28 +0100 Message-ID: <20200302184429.12880-17-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov The PCI Express specification dictates minimal amounts of time that the host needs to wait after triggering different kinds of resets before it is allowed to attempt accessing the device. After this waiting period, devices are required to be responsive to Configuration Space reads. However, if a device needs more time to actually complete the reset operation internally, it may respond to the read with a Completion Request Retry Status (CRS), and keep doing so on subsequent reads for as long as necessary. If the device is broken, it may even keep responding with CRS indefinitely. The specification also mandates that any Root Port that supports CRS and has CRS Software Visibility (CRS SV) enabled will synthesize the special value 0x0001 for the Vendor ID and set any other bits to 1 upon receiving a CRS Completion for a Configuration Read Request that includes both bytes of the Vendor ID (offset 0). IF CRS is supported by Root Port but CRS SV is not enabled, the request is retried autonomosly by the Root Port. Platform-specific configuration registers may exist to limit the number of or time taken by such retries. If CRS is not supported, or a different register (not Vendor ID) is polled, or the device is responding with CA/UR Completions (rather than CRS), the behavior is platform-dependent, but generally the Root Port synthesizes ~0 to complete the software read. Previously, pci_dev_wait() avoided taking advantage of CRS. However, on platforms where no limit/timeout can be configured as explained above, a device responding with CRS for too long (e.g. because it is stuck and cannot complete its reset) may trigger more severe error conditions (e.g. TOR timeout, 3-strike CPU CATERR), because the Root Port never reports back to the lower-level component requesting the transaction. This patch introduces special handling when CRS is available, and otherwise falls back to the previous behavior of polling COMMAND. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 52 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index f1ba931b0ead..1a504419e0de 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1081,18 +1081,54 @@ static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) { const char *event_name = pci_init_event_name(event); int timeout = dev->reset_ready_poll_ms; + int waited = 0; + int rc = 0; + /* * After reset, the device should not silently discard config * requests, but it may still indicate that it needs more time by - * responding to them with CRS completions. The Root Port will - * generally synthesize ~0 data to complete the read (except when - * CRS SV is enabled and the read was for the Vendor ID; in that - * case it synthesizes 0x0001 data). - * - * Wait for the device to return a non-CRS completion. Read the - * Command register instead of Vendor ID so we don't have to - * contend with the CRS SV value. + * responding to them with CRS completions. For such completions: + * - If CRS SV is enabled on the Root Port, and the read request + * covers both bytes of the Vendor ID register, the Root Port + * will synthesize the value 0x0001 (and set any extra requested + * bytes to 0xff) + * - If CRS SV is not enabled on the Root Port, the Root Port must + * re-issue the Configuration Request as a new Request. + * Depending on platform-specific Root Complex configurations, + * the Root Port may stop retrying after a set number of attempts, + * or a configured timeout is hit, or continue indefinitely + * (ultimately resulting in non-PCI-specific platform errors, such as + * a TOR timeout). + */ + if (dev->crssv_enabled) { + u32 id; + + rc = pci_dev_poll_until_not_equal(dev, PCI_VENDOR_ID, 0xffff, + 0x0001, event_name, timeout, + &waited, &id); + if (rc) + return rc; + + /* + * If Vendor/Device ID is valid, the device must be ready. + * Note: SR-IOV VFs return ~0 for reads to Vendor/Device + * ID and will not be recognized as ready by this check. + */ + if (id != 0x0000ffff && id != 0xffff0000 && + id != 0x00000000 && id != 0xffffffff) + return 0; + } + + /* + * Root Ports will generally indicate error scenarios (e.g. + * internal timeouts, or received Completion with CA/UR) by + * synthesizing an 'all bits set' value (~0). + * In case CRS is not supported/enabled, as well as for SR-IOV VFs, + * fall back to polling a different register that cannot validly + * contain ~0. As of PCIe 5.0, bits 11-15 of COMMAND are still RsvdP + * and must return 0 when read. + * XXX: These bits might become meaningful in the future */ return pci_dev_poll_until_not_equal(dev, PCI_COMMAND, ~0, ~0, event_name, timeout, NULL, From patchwork Mon Mar 2 18:44:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 11416123 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 742ED924 for ; Mon, 2 Mar 2020 18:46:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5306021739 for ; Mon, 2 Mar 2020 18:46:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="fJDl1vO1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727471AbgCBSqn (ORCPT ); Mon, 2 Mar 2020 13:46:43 -0500 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:29684 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqn (ORCPT ); Mon, 2 Mar 2020 13:46:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174802; x=1614710802; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NwisI3KTsm/OiywhsK6dC5HAgpMIY4oSgnpSsD8E7Hk=; b=fJDl1vO1VWOPAVnCabzvma9nw7x4ynek+S5hf9HsY7OXe4xOyuR8g1Zh ZCpvHX4HYYS3I2t2H+NcE7CNPX0FCdYNzQlXX2ObTjVdhPrnYZcK52Fdv 7d4VQUTCVIqwQrmzNaMAQfWSJnT/tG/NKm0fiuVbPq1RBDsreE9rYBtl8 A=; IronPort-SDR: 8M46f1gUsbFNBiNCQv3yws6knTb1k/rL/YJjs6SRPDXnIFzBrKs1nZJr7kpyJb0FeUsqx5ITY1 xCwgykWY5OHw== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="19686382" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2a-53356bf6.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 02 Mar 2020 18:46:41 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-53356bf6.us-west-2.amazon.com (Postfix) with ESMTPS id 5587CA25C4; Mon, 2 Mar 2020 18:46:40 +0000 (UTC) Received: from EX13D04EUA003.ant.amazon.com (10.43.165.148) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:27 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D04EUA003.ant.amazon.com (10.43.165.148) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:26 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:22 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nhe?= =?utf-8?q?rr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 17/17] PCI: Lower PCIE_RESET_READY_POLL_MS from 1m to 1s Date: Mon, 2 Mar 2020 19:44:29 +0100 Message-ID: <20200302184429.12880-18-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov PCI Express Base specification r5.0 (May 22, 2019), sec 6.6.1 mentions on more than one occasion that the appropriate waiting time before deeming a device broken if it is not able to return Successful Completion for valid Configuration Requests is 1 second after a Conventional Reset (which should be the lengthiest of resets). For devices that take longer than 1s to complete initialization, quirks can override the waiting time via the reset_ready_poll_ms field in struct pci_dev. Note: This timeout is used in pci_dev_wait for the polling that happens after we have already waited for the required post-reset times mandated by the spec. All devices are expected to be responsive to Configuration Requests at that point. "Completing initialization" here means that the device is not only responsive, but actually returns Successful Completions rather than CRS Completions (or any other error). Signed-off-by: Stanislav Spassov --- drivers/pci/pci.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index d8043d4dbe2f..1c6722b5c3ee 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -113,8 +113,11 @@ int pci_bus_error_reset(struct pci_dev *dev); /* D0/D1->D2 and D2->D0 delay */ #define PCI_PM_D2_DELAY 200 -/* Time to wait after a reset for device to become responsive */ -#define PCIE_RESET_READY_POLL_MS 60000 +/* + * Time to wait (in addition to the delays above) for a device to start + * returning Successful Completions before OS can deem it broken + */ +#define PCIE_RESET_READY_POLL_MS 1000 /** * struct pci_platform_pm_ops - Firmware PM callbacks