From patchwork Sun Jun 11 17:19:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D191C7EE45 for ; Sun, 11 Jun 2023 17:19:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229807AbjFKRTW (ORCPT ); Sun, 11 Jun 2023 13:19:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233530AbjFKRTV (ORCPT ); Sun, 11 Jun 2023 13:19:21 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 72CBBE5F; Sun, 11 Jun 2023 10:19:18 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 0EA1A9200B4; Sun, 11 Jun 2023 19:19:15 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 0B5499200B3; Sun, 11 Jun 2023 18:19:15 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:14 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 01/14] PCI: pciehp: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use `link_active_reporting' to determine whether Data Link Layer Link Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki Reviewed-by: Lukas Wunner --- NB this has been compile-tested only with PPC64LE and x86-64 configurations. No change from v8. Changes from v7: - Add Reviewed-by: tag by Lukas Wunner. - Reorder from 6/7. No change from v6. New change in v6. --- drivers/pci/hotplug/pciehp_hpc.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) linux-pcie-link-active-reporting-hpc.diff Index: linux-macro/drivers/pci/hotplug/pciehp_hpc.c =================================================================== --- linux-macro.orig/drivers/pci/hotplug/pciehp_hpc.c +++ linux-macro/drivers/pci/hotplug/pciehp_hpc.c @@ -984,7 +984,7 @@ static inline int pcie_hotplug_depth(str struct controller *pcie_init(struct pcie_device *dev) { struct controller *ctrl; - u32 slot_cap, slot_cap2, link_cap; + u32 slot_cap, slot_cap2; u8 poweron; struct pci_dev *pdev = dev->port; struct pci_bus *subordinate = pdev->subordinate; @@ -1030,9 +1030,6 @@ struct controller *pcie_init(struct pcie if (dmi_first_match(inband_presence_disabled_dmi_table)) ctrl->inband_presence_disabled = 1; - /* Check if Data Link Layer Link Active Reporting is implemented */ - pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP, &link_cap); - /* Clear all remaining event bits in Slot Status register. */ pcie_capability_write_word(pdev, PCI_EXP_SLTSTA, PCI_EXP_SLTSTA_ABP | PCI_EXP_SLTSTA_PFD | @@ -1051,7 +1048,7 @@ struct controller *pcie_init(struct pcie FLAG(slot_cap, PCI_EXP_SLTCAP_EIP), FLAG(slot_cap, PCI_EXP_SLTCAP_NCCS), FLAG(slot_cap2, PCI_EXP_SLTCAP2_IBPD), - FLAG(link_cap, PCI_EXP_LNKCAP_DLLLARC), + FLAG(pdev->link_active_reporting, true), pdev->broken_cmd_compl ? " (with Cmd Compl erratum)" : ""); /* From patchwork Sun Jun 11 17:19:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 736F0C7EE37 for ; Sun, 11 Jun 2023 17:19:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233373AbjFKRTX (ORCPT ); Sun, 11 Jun 2023 13:19:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233665AbjFKRTV (ORCPT ); Sun, 11 Jun 2023 13:19:21 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 57FD6E68; Sun, 11 Jun 2023 10:19:20 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id B62CE9200BC; Sun, 11 Jun 2023 19:19:19 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id AEEC09200B3; Sun, 11 Jun 2023 18:19:19 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:19 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 02/14] PCI: Export PCIe link retrain timeout In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Convert LINK_RETRAIN_TIMEOUT from jiffies to milliseconds, accordingly rename to PCIE_LINK_RETRAIN_TIMEOUT_MS, and make available via "pci.h" for PCI drivers to use. Use in `pcie_wait_for_link_delay'. Signed-off-by: Maciej W. Rozycki --- Changes from v8: - Convert LINK_RETRAIN_TIMEOUT from jiffies to milliseconds, rename it to PCIE_LINK_RETRAIN_TIMEOUT_MS rather than PCIE_LINK_RETRAIN_TIMEOUT, and adjust its use accordingly. - Also replace hardcoded 1000 in `pcie_wait_for_link_delay'. - Correct the change heading, s/PCI/PCIe/ for the link reference. Changes from v7: - Reorder from 1/7. No change from v6. No change from v5. New change in v5. --- drivers/pci/pci.c | 2 +- drivers/pci/pci.h | 2 ++ drivers/pci/pcie/aspm.c | 4 +--- 3 files changed, 4 insertions(+), 4 deletions(-) linux-pcie-link-retrain-timeout.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4860,7 +4860,7 @@ static int pci_pm_reset(struct pci_dev * static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, int delay) { - int timeout = 1000; + int timeout = PCIE_LINK_RETRAIN_TIMEOUT_MS; bool ret; u16 lnk_status; Index: linux-macro/drivers/pci/pci.h =================================================================== --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -11,6 +11,8 @@ #define PCI_VSEC_ID_INTEL_TBT 0x1234 /* Thunderbolt */ +#define PCIE_LINK_RETRAIN_TIMEOUT_MS 1000 + extern const unsigned char pcie_link_speed[]; extern bool pci_early_dump; Index: linux-macro/drivers/pci/pcie/aspm.c =================================================================== --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -90,8 +90,6 @@ static const char *policy_str[] = { [POLICY_POWER_SUPERSAVE] = "powersupersave" }; -#define LINK_RETRAIN_TIMEOUT HZ - /* * The L1 PM substate capability is only implemented in function 0 in a * multi function device. @@ -213,7 +211,7 @@ static bool pcie_retrain_link(struct pci } /* Wait for link training end. Break out after waiting for timeout */ - end_jiffies = jiffies + LINK_RETRAIN_TIMEOUT; + end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { pcie_capability_read_word(parent, PCI_EXP_LNKSTA, ®16); if (!(reg16 & PCI_EXP_LNKSTA_LT)) From patchwork Sun Jun 11 17:19:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D1DC7EE37 for ; Sun, 11 Jun 2023 17:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233455AbjFKRT3 (ORCPT ); Sun, 11 Jun 2023 13:19:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233879AbjFKRT1 (ORCPT ); Sun, 11 Jun 2023 13:19:27 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1564A10CC; Sun, 11 Jun 2023 10:19:24 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 69A929200BF; Sun, 11 Jun 2023 19:19:23 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 637809200B3; Sun, 11 Jun 2023 18:19:23 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:23 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 03/14] PCI: Execute `quirk_enable_clear_retrain_link' earlier In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make `quirk_enable_clear_retrain_link' `pci_fixup_early' so that any later fixups can rely on `clear_retrain_link' to have been already initialised. Signed-off-by: Maciej W. Rozycki --- No change from v8. Changes from v7: - Reorder from 2/7. No change from v6. No change from v5. New change in v5. --- drivers/pci/quirks.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) linux-pcie-clear-retrain-link-early.diff Index: linux-macro/drivers/pci/quirks.c =================================================================== --- linux-macro.orig/drivers/pci/quirks.c +++ linux-macro/drivers/pci/quirks.c @@ -2407,9 +2407,9 @@ static void quirk_enable_clear_retrain_l dev->clear_retrain_link = 1; pci_info(dev, "Enable PCIe Retrain Link quirk\n"); } -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe110, quirk_enable_clear_retrain_link); -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe111, quirk_enable_clear_retrain_link); -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe130, quirk_enable_clear_retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe110, quirk_enable_clear_retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe111, quirk_enable_clear_retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe130, quirk_enable_clear_retrain_link); static void fixup_rev1_53c810(struct pci_dev *dev) { From patchwork Sun Jun 11 17:19:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 975B5C7EE37 for ; Sun, 11 Jun 2023 17:20:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233712AbjFKRU2 (ORCPT ); Sun, 11 Jun 2023 13:20:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233294AbjFKRUZ (ORCPT ); Sun, 11 Jun 2023 13:20:25 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 397CE1FE9; Sun, 11 Jun 2023 10:20:01 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id C29129200BF; Sun, 11 Jun 2023 19:19:27 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id C025E9200BB; Sun, 11 Jun 2023 18:19:27 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:27 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 04/14] PCI: Initialize `link_active_reporting' earlier In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Determine whether Data Link Layer Link Active Reporting is available ahead of calling any fixups so that the cached value can be used there and later on. Signed-off-by: Maciej W. Rozycki --- No change from v8. Changes from v7: - Reorder from 3/7. Changes from v6: - Regenerate against 6.3-rc5. New change in v6. --- drivers/pci/probe.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) linux-pcie-link-active-reporting-early.diff Index: linux-macro/drivers/pci/probe.c =================================================================== --- linux-macro.orig/drivers/pci/probe.c +++ linux-macro/drivers/pci/probe.c @@ -820,7 +820,6 @@ static void pci_set_bus_speed(struct pci pcie_capability_read_dword(bridge, PCI_EXP_LNKCAP, &linkcap); bus->max_bus_speed = pcie_link_speed[linkcap & PCI_EXP_LNKCAP_SLS]; - bridge->link_active_reporting = !!(linkcap & PCI_EXP_LNKCAP_DLLLARC); pcie_capability_read_word(bridge, PCI_EXP_LNKSTA, &linksta); pcie_update_link_speed(bus, linksta); @@ -1829,6 +1828,7 @@ int pci_setup_device(struct pci_dev *dev int err, pos = 0; struct pci_bus_region region; struct resource *res; + u32 linkcap; hdr_type = pci_hdr_type(dev); @@ -1876,6 +1876,10 @@ int pci_setup_device(struct pci_dev *dev /* "Unknown power state" */ dev->current_state = PCI_UNKNOWN; + /* Set it early to make it available to fixups, etc. */ + pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &linkcap); + dev->link_active_reporting = !!(linkcap & PCI_EXP_LNKCAP_DLLLARC); + /* Early fixups, before probing the BARs */ pci_fixup_device(pci_fixup_early, dev); From patchwork Sun Jun 11 17:19:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B413CC7EE45 for ; Sun, 11 Jun 2023 17:20:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233872AbjFKRUe (ORCPT ); Sun, 11 Jun 2023 13:20:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233464AbjFKRU3 (ORCPT ); Sun, 11 Jun 2023 13:20:29 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C3D9010E2; Sun, 11 Jun 2023 10:20:07 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 67DE69200C3; Sun, 11 Jun 2023 19:19:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 640209200BB; Sun, 11 Jun 2023 18:19:32 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:32 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 05/14] powerpc/eeh: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use `link_active_reporting' to determine whether Data Link Layer Link Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki --- NB this has been compile-tested only with a PPC64LE configuration. No change from v8. Changes from v7: - Reorder from 4/7. No change from v6. New change in v6. --- arch/powerpc/kernel/eeh_pe.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) linux-pcie-link-active-reporting-eeh.diff Index: linux-macro/arch/powerpc/kernel/eeh_pe.c =================================================================== --- linux-macro.orig/arch/powerpc/kernel/eeh_pe.c +++ linux-macro/arch/powerpc/kernel/eeh_pe.c @@ -671,9 +671,8 @@ static void eeh_bridge_check_link(struct eeh_ops->write_config(edev, cap + PCI_EXP_LNKCTL, 2, val); /* Check link */ - eeh_ops->read_config(edev, cap + PCI_EXP_LNKCAP, 4, &val); - if (!(val & PCI_EXP_LNKCAP_DLLLARC)) { - eeh_edev_dbg(edev, "No link reporting capability (0x%08x) \n", val); + if (!edev->pdev->link_active_reporting) { + eeh_edev_dbg(edev, "No link reporting capability\n"); msleep(1000); return; } From patchwork Sun Jun 11 17:19:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CCC1C77B7A for ; Sun, 11 Jun 2023 17:20:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229616AbjFKRUV (ORCPT ); Sun, 11 Jun 2023 13:20:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230515AbjFKRUU (ORCPT ); Sun, 11 Jun 2023 13:20:20 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 210661BDF; Sun, 11 Jun 2023 10:19:57 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 060279200C4; Sun, 11 Jun 2023 19:19:36 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id F3F579200C0; Sun, 11 Jun 2023 18:19:36 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:36 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 06/14] net/mlx5: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use `link_active_reporting' to determine whether Data Link Layer Link Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki --- NB this has been compile-tested only with PPC64LE and x86-64 configurations. No change from v8. Changes from v7: - Reorder from 5/7. Changes from v6: - Regenerate against 6.3-rc5. New change in v6. --- drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) linux-pcie-link-active-reporting-mlx5.diff Index: linux-macro/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c =================================================================== --- linux-macro.orig/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +++ linux-macro/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c @@ -307,7 +307,6 @@ static int mlx5_pci_link_toggle(struct m unsigned long timeout; struct pci_dev *sdev; int cap, err; - u32 reg32; /* Check that all functions under the pci bridge are PFs of * this device otherwise fail this function. @@ -346,11 +345,8 @@ static int mlx5_pci_link_toggle(struct m return err; /* Check link */ - err = pci_read_config_dword(bridge, cap + PCI_EXP_LNKCAP, ®32); - if (err) - return err; - if (!(reg32 & PCI_EXP_LNKCAP_DLLLARC)) { - mlx5_core_warn(dev, "No PCI link reporting capability (0x%08x)\n", reg32); + if (!bridge->link_active_reporting) { + mlx5_core_warn(dev, "No PCI link reporting capability\n"); msleep(1000); goto restore; } From patchwork Sun Jun 11 17:19:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE16CC83005 for ; Sun, 11 Jun 2023 17:20:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233718AbjFKRUe (ORCPT ); Sun, 11 Jun 2023 13:20:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233505AbjFKRU3 (ORCPT ); Sun, 11 Jun 2023 13:20:29 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9E47E10C1; Sun, 11 Jun 2023 10:20:06 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 267D59200BC; Sun, 11 Jun 2023 19:19:41 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 21FBD9200BB; Sun, 11 Jun 2023 18:19:41 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:41 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 07/14] PCI: Export `pcie_retrain_link' for use outside ASPM In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Export `pcie_retrain_link' for link retrain needs outside ASPM. There is no functional change at this point, but `struct pcie_link_state' is local to ASPM and not used within `pcie_retrain_link' other than to get at the associated PCI device, so change the operand and adjust the lone call site accordingly. Document the interface. Signed-off-by: Maciej W. Rozycki --- There's a missing full stop added in a comment in the course of the move, not worth mentioning in the change description IMHO and not worth its own change either. This comment will go away in a subsequent change anyway. New change in v9. --- drivers/pci/pci.c | 36 ++++++++++++++++++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/pcie/aspm.c | 32 +------------------------------- 3 files changed, 38 insertions(+), 31 deletions(-) linux-pcie-retrain-link-export.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4912,6 +4912,42 @@ bool pcie_wait_for_link(struct pci_dev * return pcie_wait_for_link_delay(pdev, active, 100); } +/** + * pcie_retrain_link - Request a link retrain and wait for it to complete + * @pdev: Device whose link to retrain. + * + * Return TRUE if successful, or FALSE if training has not completed + * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + */ +bool pcie_retrain_link(struct pci_dev *pdev) +{ + unsigned long end_jiffies; + u16 reg16; + + pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, ®16); + reg16 |= PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + if (pdev->clear_retrain_link) { + /* + * Due to an erratum in some devices the Retrain Link bit + * needs to be cleared again manually to allow the link + * training to succeed. + */ + reg16 &= ~PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + } + + /* Wait for link training end. Break out after waiting for timeout. */ + end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); + do { + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, ®16); + if (!(reg16 & PCI_EXP_LNKSTA_LT)) + break; + msleep(1); + } while (time_before(jiffies, end_jiffies)); + return !(reg16 & PCI_EXP_LNKSTA_LT); +} + /* * Find maximum D3cold delay required by all the devices on the bus. The * spec says 100 ms, but firmware can lower it and we allow drivers to Index: linux-macro/drivers/pci/pci.h =================================================================== --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -561,6 +561,7 @@ pci_ers_result_t pcie_do_recovery(struct pci_ers_result_t (*reset_subordinates)(struct pci_dev *pdev)); bool pcie_wait_for_link(struct pci_dev *pdev, bool active); +bool pcie_retrain_link(struct pci_dev *pdev); #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); Index: linux-macro/drivers/pci/pcie/aspm.c =================================================================== --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -191,36 +191,6 @@ static void pcie_clkpm_cap_init(struct p link->clkpm_disable = blacklist ? 1 : 0; } -static bool pcie_retrain_link(struct pcie_link_state *link) -{ - struct pci_dev *parent = link->pdev; - unsigned long end_jiffies; - u16 reg16; - - pcie_capability_read_word(parent, PCI_EXP_LNKCTL, ®16); - reg16 |= PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - if (parent->clear_retrain_link) { - /* - * Due to an erratum in some devices the Retrain Link bit - * needs to be cleared again manually to allow the link - * training to succeed. - */ - reg16 &= ~PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - } - - /* Wait for link training end. Break out after waiting for timeout */ - end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); - do { - pcie_capability_read_word(parent, PCI_EXP_LNKSTA, ®16); - if (!(reg16 & PCI_EXP_LNKSTA_LT)) - break; - msleep(1); - } while (time_before(jiffies, end_jiffies)); - return !(reg16 & PCI_EXP_LNKSTA_LT); -} - /* * pcie_aspm_configure_common_clock: check if the 2 ends of a link * could use common clock. If they are, configure them to use the @@ -287,7 +257,7 @@ static void pcie_aspm_configure_common_c reg16 &= ~PCI_EXP_LNKCTL_CCC; pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - if (pcie_retrain_link(link)) + if (pcie_retrain_link(link->pdev)) return; /* Training failed. Restore common clock configurations */ From patchwork Sun Jun 11 17:19:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81FA8C77B7A for ; Sun, 11 Jun 2023 17:20:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230515AbjFKRU3 (ORCPT ); Sun, 11 Jun 2023 13:20:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233466AbjFKRU2 (ORCPT ); Sun, 11 Jun 2023 13:20:28 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EF8F810C4; Sun, 11 Jun 2023 10:20:04 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 53DBD9200C5; Sun, 11 Jun 2023 19:19:45 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4D8169200C1; Sun, 11 Jun 2023 18:19:45 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:45 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 08/14] PCI: Use distinct local vars in `pcie_retrain_link' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use separate local variables to hold the respective values retrieved from the Link Control Register and the Link Status Register. Not only it improves readability, but it makes it possible for the compiler to detect actual uninitialised use should this code change in the future. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) linux-pcie-retrain-link-lnkctl-lnksta.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4922,30 +4922,31 @@ bool pcie_wait_for_link(struct pci_dev * bool pcie_retrain_link(struct pci_dev *pdev) { unsigned long end_jiffies; - u16 reg16; + u16 lnkctl; + u16 lnksta; - pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, ®16); - reg16 |= PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &lnkctl); + lnkctl |= PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); if (pdev->clear_retrain_link) { /* * Due to an erratum in some devices the Retrain Link bit * needs to be cleared again manually to allow the link * training to succeed. */ - reg16 &= ~PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + lnkctl &= ~PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } /* Wait for link training end. Break out after waiting for timeout. */ end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, ®16); - if (!(reg16 & PCI_EXP_LNKSTA_LT)) + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); + if (!(lnksta & PCI_EXP_LNKSTA_LT)) break; msleep(1); } while (time_before(jiffies, end_jiffies)); - return !(reg16 & PCI_EXP_LNKSTA_LT); + return !(lnksta & PCI_EXP_LNKSTA_LT); } /* From patchwork Sun Jun 11 17:19:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49A4CC7EE23 for ; Sun, 11 Jun 2023 17:20:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231952AbjFKRU4 (ORCPT ); Sun, 11 Jun 2023 13:20:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230233AbjFKRUz (ORCPT ); Sun, 11 Jun 2023 13:20:55 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7BCA110D0; Sun, 11 Jun 2023 10:20:29 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 90AD99200C1; Sun, 11 Jun 2023 19:19:49 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 8B5119200B3; Sun, 11 Jun 2023 18:19:49 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:49 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 09/14] PCI: Factor our waiting for link training end In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move code polling for the Link Training bit to clear into a function of its own. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) linux-pcie-wait-for-link-status.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4850,6 +4850,28 @@ static int pci_pm_reset(struct pci_dev * } /** + * pcie_wait_for_link_status - Wait for link training end + * @pdev: Device whose link to wait for. + * + * Return TRUE if successful, or FALSE if training has not completed + * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + */ +static bool pcie_wait_for_link_status(struct pci_dev *pdev) +{ + unsigned long end_jiffies; + u16 lnksta; + + end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); + do { + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); + if (!(lnksta & PCI_EXP_LNKSTA_LT)) + break; + msleep(1); + } while (time_before(jiffies, end_jiffies)); + return !(lnksta & PCI_EXP_LNKSTA_LT); +} + +/** * pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? @@ -4916,14 +4938,11 @@ bool pcie_wait_for_link(struct pci_dev * * pcie_retrain_link - Request a link retrain and wait for it to complete * @pdev: Device whose link to retrain. * - * Return TRUE if successful, or FALSE if training has not completed - * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + * Return TRUE if successful, or FALSE if training has not completed. */ bool pcie_retrain_link(struct pci_dev *pdev) { - unsigned long end_jiffies; u16 lnkctl; - u16 lnksta; pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &lnkctl); lnkctl |= PCI_EXP_LNKCTL_RL; @@ -4938,15 +4957,7 @@ bool pcie_retrain_link(struct pci_dev *p pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } - /* Wait for link training end. Break out after waiting for timeout. */ - end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); - do { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); - if (!(lnksta & PCI_EXP_LNKSTA_LT)) - break; - msleep(1); - } while (time_before(jiffies, end_jiffies)); - return !(lnksta & PCI_EXP_LNKSTA_LT); + return pcie_wait_for_link_status(pdev); } /* From patchwork Sun Jun 11 17:19:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57603C7EE37 for ; Sun, 11 Jun 2023 17:21:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233930AbjFKRVR (ORCPT ); Sun, 11 Jun 2023 13:21:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233888AbjFKRVN (ORCPT ); Sun, 11 Jun 2023 13:21:13 -0400 X-Greylist: delayed 92 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Sun, 11 Jun 2023 10:20:49 PDT Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E45F71700; Sun, 11 Jun 2023 10:20:48 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 4E8B29200C6; Sun, 11 Jun 2023 19:19:53 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4858E9200B3; Sun, 11 Jun 2023 18:19:53 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:53 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 10/14] PCI: Add support for polling DLLLA to `pcie_retrain_link' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Let the caller of `pcie_retrain_link' specify whether they want to use the LT bit or the DLLLA bit of the Link Status Register to determine if link training has completed. It is up to the caller to verify whether the use of the DLLLA bit, the implementation of which is optional, is valid for the device requested. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 28 ++++++++++++++++++++-------- drivers/pci/pci.h | 2 +- drivers/pci/pcie/aspm.c | 2 +- 3 files changed, 22 insertions(+), 10 deletions(-) linux-pcie-retrain-link-use-lt.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4850,25 +4850,32 @@ static int pci_pm_reset(struct pci_dev * } /** - * pcie_wait_for_link_status - Wait for link training end + * pcie_wait_for_link_status - Wait for link status change * @pdev: Device whose link to wait for. + * @use_lt: Use the LT bit if TRUE, or the DLLLA bit if FALSE. + * @active: Waiting for active or inactive? * - * Return TRUE if successful, or FALSE if training has not completed - * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + * Return TRUE if successful, or FALSE if status has not changed within + * PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. */ -static bool pcie_wait_for_link_status(struct pci_dev *pdev) +static bool pcie_wait_for_link_status(struct pci_dev *pdev, + bool use_lt, bool active) { + u16 lnksta_mask, lnksta_match; unsigned long end_jiffies; u16 lnksta; + lnksta_mask = use_lt ? PCI_EXP_LNKSTA_LT : PCI_EXP_LNKSTA_DLLLA; + lnksta_match = active ? lnksta_mask : 0; + end_jiffies = jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); - if (!(lnksta & PCI_EXP_LNKSTA_LT)) + if ((lnksta & lnksta_mask) == lnksta_match) break; msleep(1); } while (time_before(jiffies, end_jiffies)); - return !(lnksta & PCI_EXP_LNKSTA_LT); + return (lnksta & lnksta_mask) == lnksta_match; } /** @@ -4937,10 +4944,15 @@ bool pcie_wait_for_link(struct pci_dev * /** * pcie_retrain_link - Request a link retrain and wait for it to complete * @pdev: Device whose link to retrain. + * @use_lt: Use the LT bit if TRUE, or the DLLLA bit if FALSE, for status. + * + * Retrain completion status is retrieved from the Link Status Register + * according to @use_lt. It is not verified whether the use of the DLLLA + * bit is valid. * * Return TRUE if successful, or FALSE if training has not completed. */ -bool pcie_retrain_link(struct pci_dev *pdev) +bool pcie_retrain_link(struct pci_dev *pdev, bool use_lt) { u16 lnkctl; @@ -4957,7 +4969,7 @@ bool pcie_retrain_link(struct pci_dev *p pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } - return pcie_wait_for_link_status(pdev); + return pcie_wait_for_link_status(pdev, use_lt, !use_lt); } /* Index: linux-macro/drivers/pci/pci.h =================================================================== --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -561,7 +561,7 @@ pci_ers_result_t pcie_do_recovery(struct pci_ers_result_t (*reset_subordinates)(struct pci_dev *pdev)); bool pcie_wait_for_link(struct pci_dev *pdev, bool active); -bool pcie_retrain_link(struct pci_dev *pdev); +bool pcie_retrain_link(struct pci_dev *pdev, bool use_lt); #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); Index: linux-macro/drivers/pci/pcie/aspm.c =================================================================== --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -257,7 +257,7 @@ static void pcie_aspm_configure_common_c reg16 &= ~PCI_EXP_LNKCTL_CCC; pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - if (pcie_retrain_link(link->pdev)) + if (pcie_retrain_link(link->pdev, true)) return; /* Training failed. Restore common clock configurations */ From patchwork Sun Jun 11 17:19:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B473C7EE23 for ; Sun, 11 Jun 2023 17:20:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233254AbjFKRUt (ORCPT ); Sun, 11 Jun 2023 13:20:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230233AbjFKRUs (ORCPT ); Sun, 11 Jun 2023 13:20:48 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9B5ED199E; Sun, 11 Jun 2023 10:20:21 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id E73559200C7; Sun, 11 Jun 2023 19:19:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id E0F309200C0; Sun, 11 Jun 2023 18:19:57 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:57 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 11/14] PCI: Use `pcie_wait_for_link_status' in `pcie_wait_for_link_delay' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Remove a DLLLA status bit polling loop from `pcie_wait_for_link_delay' and call almost identical code in `pcie_wait_for_link_status' instead. This reduces the lower bound on the polling interval from 10ms to 1ms, possibly increasing the CPU load on the system in favour to reducing the wait time. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) linux-pcie-wait-for-link-delay-status.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4889,16 +4889,14 @@ static bool pcie_wait_for_link_status(st static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, int delay) { - int timeout = PCIE_LINK_RETRAIN_TIMEOUT_MS; bool ret; - u16 lnk_status; /* * Some controllers might not implement link active reporting. In this * case, we wait for 1000 ms + any delay requested by the caller. */ if (!pdev->link_active_reporting) { - msleep(timeout + delay); + msleep(PCIE_LINK_RETRAIN_TIMEOUT_MS + delay); return true; } @@ -4913,20 +4911,11 @@ static bool pcie_wait_for_link_delay(str */ if (active) msleep(20); - for (;;) { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); - ret = !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); - if (ret == active) - break; - if (timeout <= 0) - break; - msleep(10); - timeout -= 10; - } + ret = pcie_wait_for_link_status(pdev, false, active); if (active && ret) msleep(delay); - return ret == active; + return ret; } /** From patchwork Sun Jun 11 17:20:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76AC7C7EE45 for ; Sun, 11 Jun 2023 17:20:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233849AbjFKRU4 (ORCPT ); Sun, 11 Jun 2023 13:20:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230156AbjFKRUz (ORCPT ); Sun, 11 Jun 2023 13:20:55 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 21BFFE6F; Sun, 11 Jun 2023 10:20:30 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 529979200C9; Sun, 11 Jun 2023 19:20:02 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4E1479200C0; Sun, 11 Jun 2023 18:20:02 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:02 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 12/14] PCI: Provide stub failed link recovery for device probing and hot plug In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This now fails unconditionally and will be always optimised away, but provides for quirks to implement recovery for failed links detected in device probing and device hot plug events. Signed-off-by: Maciej W. Rozycki --- New change in v9, factored out from 7/7: - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain'. - Add stub implementation in "pci.h". --- drivers/pci/pci.c | 2 ++ drivers/pci/pci.h | 4 ++++ drivers/pci/probe.c | 2 ++ 3 files changed, 8 insertions(+) linux-pcie-failed-link-retrain.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4912,6 +4912,8 @@ static bool pcie_wait_for_link_delay(str if (active) msleep(20); ret = pcie_wait_for_link_status(pdev, false, active); + if (active && !ret) + ret = pcie_failed_link_retrain(pdev); if (active && ret) msleep(delay); Index: linux-macro/drivers/pci/pci.h =================================================================== --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -554,6 +554,10 @@ static inline int pci_dev_specific_disab return -ENOTTY; } #endif +static inline bool pcie_failed_link_retrain(struct pci_dev *dev) +{ + return false; +} /* PCI error reporting and recovery */ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, Index: linux-macro/drivers/pci/probe.c =================================================================== --- linux-macro.orig/drivers/pci/probe.c +++ linux-macro/drivers/pci/probe.c @@ -2549,6 +2549,8 @@ void pci_device_add(struct pci_dev *dev, dma_set_max_seg_size(&dev->dev, 65536); dma_set_seg_boundary(&dev->dev, 0xffffffff); + pcie_failed_link_retrain(dev); + /* Fix up broken headers */ pci_fixup_device(pci_fixup_header, dev); From patchwork Sun Jun 11 17:20:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE0A7C7EE37 for ; Sun, 11 Jun 2023 17:21:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233888AbjFKRVU (ORCPT ); Sun, 11 Jun 2023 13:21:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233992AbjFKRVT (ORCPT ); Sun, 11 Jun 2023 13:21:19 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id CCE1719AA; Sun, 11 Jun 2023 10:20:54 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 6DEA69200CB; Sun, 11 Jun 2023 19:20:06 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 66A4F9200CA; Sun, 11 Jun 2023 18:20:06 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:06 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 13/14] PCI: Add failed link recovery for device reset events In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Request failed link recovery with any upstream bridge where a device has not come back after reset within PCI_RESET_WAIT time. Reset the polling interval if recovery succeeded, otherwise continue as usual. Signed-off-by: Maciej W. Rozycki --- New change in v9, factored out from 7/7: - Remove duplicate succesful completion report previously added (not sure where it came from, possibly an unnoticed leftover from experiments). - Make the type of `retrain' variable `bool' rather than `int' and invert the logic used. - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain'. - Rename `pcie_upstream_link_retrain' to `pcie_parent_link_retrain'. Add documentation. --- drivers/pci/pci.c | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) linux-pcie-dev-wait-link-retrain.diff Index: linux-macro/drivers/pci/pci.c =================================================================== --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -1146,10 +1146,27 @@ void pci_resume_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_resume_one, NULL); } +/** + * pcie_parent_link_retrain - Check and retrain link we are downstream from + * @dev: PCI device to handle. + * + * Return TRUE if the link was retrained, FALSE otherwise. + */ +static bool pcie_parent_link_retrain(struct pci_dev *dev) +{ + struct pci_dev *bridge; + + bridge = pci_upstream_bridge(dev); + if (bridge) + return pcie_failed_link_retrain(bridge); + else + return false; +} + static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout) { + bool retrain = true; int delay = 1; - u32 id; /* * After reset, the device should not silently discard config @@ -1163,21 +1180,33 @@ static int pci_dev_wait(struct pci_dev * * Command register instead of Vendor ID so we don't have to * contend with the CRS SV value. */ - pci_read_config_dword(dev, PCI_COMMAND, &id); - while (PCI_POSSIBLE_ERROR(id)) { + for (;;) { + u32 id; + + pci_read_config_dword(dev, PCI_COMMAND, &id); + if (!PCI_POSSIBLE_ERROR(id)) + break; + if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", delay - 1, reset_type); return -ENOTTY; } - if (delay > PCI_RESET_WAIT) + if (delay > PCI_RESET_WAIT) { + if (retrain) { + retrain = false; + if (pcie_parent_link_retrain(dev)) { + delay = 1; + continue; + } + } pci_info(dev, "not ready %dms after %s; waiting\n", delay - 1, reset_type); + } msleep(delay); delay *= 2; - pci_read_config_dword(dev, PCI_COMMAND, &id); } if (delay > PCI_RESET_WAIT) From patchwork Sun Jun 11 17:20:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Maciej W. Rozycki" X-Patchwork-Id: 13275313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88C56C7EE37 for ; Sun, 11 Jun 2023 17:21:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233973AbjFKRVY (ORCPT ); Sun, 11 Jun 2023 13:21:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233952AbjFKRVU (ORCPT ); Sun, 11 Jun 2023 13:21:20 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 458D7E75; Sun, 11 Jun 2023 10:20:55 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 974869200CC; Sun, 11 Jun 2023 19:20:10 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 91D4E9200CA; Sun, 11 Jun 2023 18:20:10 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:10 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?utf-8?q?Pali_Roh?= =?utf-8?q?=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 14/14] PCI: Work around PCIe link training failures In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Attempt to handle cases such as with a downstream port of the ASMedia ASM2824 PCIe switch where link training never completes and the link continues switching between speeds indefinitely with the data link layer never reaching the active state. It has been observed with a downstream port of the ASMedia ASM2824 Gen 3 switch wired to the upstream port of the Pericom PI7C9X2G304 Gen 2 switch, using a Delock Riser Card PCI Express x1 > 2 x PCIe x1 device, P/N 41433, wired to a SiFive HiFive Unmatched board. In this setup the switches are supposed to negotiate the link speed of preferably 5.0GT/s, falling back to 2.5GT/s. Instead the link continues oscillating between the two speeds, at the rate of 34-35 times per second, with link training reported repeatedly active ~84% of the time. Forcibly limiting the target link speed to 2.5GT/s with the upstream ASM2824 device however makes the two switches communicate correctly. Removing the speed restriction afterwards makes the two devices switch to 5.0GT/s then. Make use of these observations then and detect the inability to train the link, by checking for the Data Link Layer Link Active status bit being off while the Link Bandwidth Management Status indicating that hardware has changed the link speed or width in an attempt to correct unreliable link operation. Restrict the speed to 2.5GT/s then with the Target Link Speed field, request a retrain and wait 200ms for the data link to go up. If this turns out successful, then lift the restriction, letting the devices negotiate a higher speed. Also check for a 2.5GT/s speed restriction the firmware may have already arranged and lift it too with ports of devices known to continue working afterwards, currently the ASM2824 only, that already report their data link being up. Signed-off-by: Maciej W. Rozycki Link: https://lore.kernel.org/r/alpine.DEB.2.21.2203022037020.56670@angie.orcam.me.uk/ Link: https://source.denx.de/u-boot/u-boot/-/commit/a398a51ccc68 --- Changes from v8: - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain', add a prototype in "pci.h", moving the stub implementation under !PCI_QUIRKS umbrella. - Move back to quirks.c, though as an internal API call rather than a regular quirk. - Adjust for PCIE_LINK_RETRAIN_TIMEOUT_MS expressed in milliseconds rather than jiffies. - Use a `pcie_retrain_link' call rather than retraining inline, and also use it in the restriction lift path, making it another possible failure point. No changes from v7. Changes from v6: - Regenerate against 6.3-rc5. - Shorten the lore.kernel.org archive link in the change description. Changes from v5: - Move from a quirk into PCI core and call at device probing, hot-plug, reset and resume. Keep the ASMedia part under CONFIG_PCI_QUIRKS. - Rely on `dev->link_active_reporting' rather than re-retrieving the capability. Changes from v4: - Remove inclusion no longer needed. - Make the quirk generic based on probing device features rather than specific to the ASM2824 part only; take the Retrain Link bit erratum into account. - Still lift the 2.5GT/s speed restriction with the ASM2824 only. - Increase retrain timeout from 200ms to 1s (PCIE_LINK_RETRAIN_TIMEOUT). - Remove retrain success notification. - Use PCIe helpers rather than generic PCI functions throughout. - Trim down and update the wording of the change description for the switch from an ASM2824-specific to a generic fixup. Changes from v3: - Remove the entry for the ASM2824. Changes from v2: - Regenerate for 5.17-rc2 for a merge conflict. - Replace BUG_ON for a missing PCI Express capability with WARN_ON and an early return. Changes from v1: - Regenerate for a merge conflict. --- drivers/pci/pci.h | 3 + drivers/pci/quirks.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) linux-pcie-asm2824-manual-retrain.diff Index: linux-macro/drivers/pci/pci.h =================================================================== --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -539,6 +539,7 @@ void pci_acs_init(struct pci_dev *dev); int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags); int pci_dev_specific_enable_acs(struct pci_dev *dev); int pci_dev_specific_disable_acs_redir(struct pci_dev *dev); +bool pcie_failed_link_retrain(struct pci_dev *dev); #else static inline int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags) @@ -553,11 +554,11 @@ static inline int pci_dev_specific_disab { return -ENOTTY; } -#endif static inline bool pcie_failed_link_retrain(struct pci_dev *dev) { return false; } +#endif /* PCI error reporting and recovery */ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, Index: linux-macro/drivers/pci/quirks.c =================================================================== --- linux-macro.orig/drivers/pci/quirks.c +++ linux-macro/drivers/pci/quirks.c @@ -33,6 +33,99 @@ #include #include "pci.h" +/* + * Retrain the link of a downstream PCIe port by hand if necessary. + * + * This is needed at least where a downstream port of the ASMedia ASM2824 + * Gen 3 switch is wired to the upstream port of the Pericom PI7C9X2G304 + * Gen 2 switch, and observed with the Delock Riser Card PCI Express x1 > + * 2 x PCIe x1 device, P/N 41433, plugged into the SiFive HiFive Unmatched + * board. + * + * In such a configuration the switches are supposed to negotiate the link + * speed of preferably 5.0GT/s, falling back to 2.5GT/s. However the link + * continues switching between the two speeds indefinitely and the data + * link layer never reaches the active state, with link training reported + * repeatedly active ~84% of the time. Forcing the target link speed to + * 2.5GT/s with the upstream ASM2824 device makes the two switches talk to + * each other correctly however. And more interestingly retraining with a + * higher target link speed afterwards lets the two successfully negotiate + * 5.0GT/s. + * + * With the ASM2824 we can rely on the otherwise optional Data Link Layer + * Link Active status bit and in the failed link training scenario it will + * be off along with the Link Bandwidth Management Status indicating that + * hardware has changed the link speed or width in an attempt to correct + * unreliable link operation. For a port that has been left unconnected + * both bits will be clear. So use this information to detect the problem + * rather than polling the Link Training bit and watching out for flips or + * at least the active status. + * + * Since the exact nature of the problem isn't known and in principle this + * could trigger where an ASM2824 device is downstream rather upstream, + * apply this erratum workaround to any downstream ports as long as they + * support Link Active reporting and have the Link Control 2 register. + * Restrict the speed to 2.5GT/s then with the Target Link Speed field, + * request a retrain and wait 200ms for the data link to go up. + * + * If this turns out successful and we know by the Vendor:Device ID it is + * safe to do so, then lift the restriction, letting the devices negotiate + * a higher speed. Also check for a similar 2.5GT/s speed restriction the + * firmware may have already arranged and lift it with ports that already + * report their data link being up. + * + * Return TRUE if the link has been successfully retrained, otherwise FALSE. + */ +bool pcie_failed_link_retrain(struct pci_dev *dev) +{ + static const struct pci_device_id ids[] = { + { PCI_VDEVICE(ASMEDIA, 0x2824) }, /* ASMedia ASM2824 */ + {} + }; + u16 lnksta, lnkctl2; + + if (!pci_is_pcie(dev) || !pcie_downstream_port(dev) || + !pcie_cap_has_lnkctl2(dev) || !dev->link_active_reporting) + return false; + + pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &lnkctl2); + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + if ((lnksta & (PCI_EXP_LNKSTA_LBMS | PCI_EXP_LNKSTA_DLLLA)) == + PCI_EXP_LNKSTA_LBMS) { + pci_info(dev, "broken device, retraining non-functional downstream link at 2.5GT/s\n"); + + lnkctl2 &= ~PCI_EXP_LNKCTL2_TLS; + lnkctl2 |= PCI_EXP_LNKCTL2_TLS_2_5GT; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, lnkctl2); + + if (!pcie_retrain_link(dev, false)) { + pci_info(dev, "retraining failed\n"); + return false; + } + + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + } + + if ((lnksta & PCI_EXP_LNKSTA_DLLLA) && + (lnkctl2 & PCI_EXP_LNKCTL2_TLS) == PCI_EXP_LNKCTL2_TLS_2_5GT && + pci_match_id(ids, dev)) { + u32 lnkcap; + + pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n"); + pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap); + lnkctl2 &= ~PCI_EXP_LNKCTL2_TLS; + lnkctl2 |= lnkcap & PCI_EXP_LNKCAP_SLS; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, lnkctl2); + + if (!pcie_retrain_link(dev, false)) { + pci_info(dev, "retraining failed\n"); + return false; + } + } + + return true; +} + static ktime_t fixup_debug_start(struct pci_dev *dev, void (*fn)(struct pci_dev *dev)) {