| Message ID | 20210826071557.29239-3-jeremy.linton@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | CM4 ACPI PCIe quirk | expand |
Hi Jeremy, sorry for the late reply, I've been on vacation. On Thu, 2021-08-26 at 02:15 -0500, Jeremy Linton wrote: [...] > +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, > + unsigned int devfn, int where) > +{ > + struct pci_config_window *cfg = bus->sysdata; > + void __iomem *base = cfg->win; > + int idx; > + u32 up; > + > + /* Accesses to the RC go right to the RC registers if slot==0 */ > + if (pci_is_root_bus(bus)) > + return PCI_SLOT(devfn) ? NULL : base + where; > + > + /* > + * Assure the link is up before sending requests downstream. This is done > + * to avoid sending transactions to EPs that don't exist. Link flap > + * conditions/etc make this race more probable. The resulting unrecoverable > + * SERRORs will result in the machine crashing. > + */ > + up = readl(base + PCIE_MISC_PCIE_STATUS); > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) > + return NULL; > + > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) > + return NULL; Couldn't this be integrated in the original brcm_pcie_map_conf()? IIUC there is nothing ACPI specific about it. It'd also make for less code duplication. Regards, Nicolas
Hi, On 8/30/21 3:36 AM, nicolas saenz julienne wrote: > Hi Jeremy, > sorry for the late reply, I've been on vacation. > > On Thu, 2021-08-26 at 02:15 -0500, Jeremy Linton wrote: > > [...] > >> +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, >> + unsigned int devfn, int where) >> +{ >> + struct pci_config_window *cfg = bus->sysdata; >> + void __iomem *base = cfg->win; >> + int idx; >> + u32 up; >> + >> + /* Accesses to the RC go right to the RC registers if slot==0 */ >> + if (pci_is_root_bus(bus)) >> + return PCI_SLOT(devfn) ? NULL : base + where; >> + >> + /* >> + * Assure the link is up before sending requests downstream. This is done >> + * to avoid sending transactions to EPs that don't exist. Link flap >> + * conditions/etc make this race more probable. The resulting unrecoverable >> + * SERRORs will result in the machine crashing. >> + */ >> + up = readl(base + PCIE_MISC_PCIE_STATUS); >> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) >> + return NULL; >> + >> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) >> + return NULL; > > Couldn't this be integrated in the original brcm_pcie_map_conf()? IIUC there is > nothing ACPI specific about it. It'd also make for less code duplication. That is where I started with this, but it wasn't the linkup check/etc which caused me to hoist it but the fact that if ACPI quirks are enabled they end up statically built into the kernel. While if this host bridge is enabled, it can end up being a module, and the resulting mess I created trying to satisfy the CONFIG variations. I'm not much of a fan of copy/paste programming, but that IMHO ended up being the cleanest here.
On 8/30/2021 9:23 AM, Jeremy Linton wrote: > Hi, > > On 8/30/21 3:36 AM, nicolas saenz julienne wrote: >> Hi Jeremy, >> sorry for the late reply, I've been on vacation. >> >> On Thu, 2021-08-26 at 02:15 -0500, Jeremy Linton wrote: >> >> [...] >> >>> +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, >>> + unsigned int devfn, int where) >>> +{ >>> + struct pci_config_window *cfg = bus->sysdata; >>> + void __iomem *base = cfg->win; >>> + int idx; >>> + u32 up; >>> + >>> + /* Accesses to the RC go right to the RC registers if slot==0 */ >>> + if (pci_is_root_bus(bus)) >>> + return PCI_SLOT(devfn) ? NULL : base + where; >>> + >>> + /* >>> + * Assure the link is up before sending requests downstream. >>> This is done >>> + * to avoid sending transactions to EPs that don't exist. Link flap >>> + * conditions/etc make this race more probable. The resulting >>> unrecoverable >>> + * SERRORs will result in the machine crashing. >>> + */ >>> + up = readl(base + PCIE_MISC_PCIE_STATUS); >>> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) >>> + return NULL; >>> + >>> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) >>> + return NULL; >> >> Couldn't this be integrated in the original brcm_pcie_map_conf()? IIUC >> there is >> nothing ACPI specific about it. It'd also make for less code duplication. > > That is where I started with this, but it wasn't the linkup check/etc > which caused me to hoist it but the fact that if ACPI quirks are enabled > they end up statically built into the kernel. While if this host bridge > is enabled, it can end up being a module, and the resulting mess I > created trying to satisfy the CONFIG variations. I'm not much of a fan > of copy/paste programming, but that IMHO ended up being the cleanest here. > Agreed, the open coding that is being done is reasonable IHMO, although we may have to update the link up code in both pcie-brcmstb.c and this file in the future if offsets/bits do change, nothing impossible though.
On Mon, 2021-08-30 at 09:27 -0700, Florian Fainelli wrote: > > On 8/30/2021 9:23 AM, Jeremy Linton wrote: > > Hi, > > > > On 8/30/21 3:36 AM, nicolas saenz julienne wrote: > > > Hi Jeremy, > > > sorry for the late reply, I've been on vacation. > > > > > > On Thu, 2021-08-26 at 02:15 -0500, Jeremy Linton wrote: > > > > > > [...] > > > > > > > +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, > > > > + unsigned int devfn, int where) > > > > +{ > > > > + struct pci_config_window *cfg = bus->sysdata; > > > > + void __iomem *base = cfg->win; > > > > + int idx; > > > > + u32 up; > > > > + > > > > + /* Accesses to the RC go right to the RC registers if slot==0 */ > > > > + if (pci_is_root_bus(bus)) > > > > + return PCI_SLOT(devfn) ? NULL : base + where; > > > > + > > > > + /* > > > > + * Assure the link is up before sending requests downstream. > > > > This is done > > > > + * to avoid sending transactions to EPs that don't exist. Link flap > > > > + * conditions/etc make this race more probable. The resulting > > > > unrecoverable > > > > + * SERRORs will result in the machine crashing. > > > > + */ > > > > + up = readl(base + PCIE_MISC_PCIE_STATUS); > > > > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) > > > > + return NULL; > > > > + > > > > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) > > > > + return NULL; > > > > > > Couldn't this be integrated in the original brcm_pcie_map_conf()? IIUC > > > there is > > > nothing ACPI specific about it. It'd also make for less code duplication. > > > > That is where I started with this, but it wasn't the linkup check/etc > > which caused me to hoist it but the fact that if ACPI quirks are enabled > > they end up statically built into the kernel. While if this host bridge > > is enabled, it can end up being a module, and the resulting mess I > > created trying to satisfy the CONFIG variations. I'm not much of a fan > > of copy/paste programming, but that IMHO ended up being the cleanest here. > > > > Agreed, the open coding that is being done is reasonable IHMO, although > we may have to update the link up code in both pcie-brcmstb.c and this > file in the future if offsets/bits do change, nothing impossible though. Fair enough. Acked-by: Nicolas Saenz Julienne <nsaenz@kernel.org> Regards, Nicolas
On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: > The Pi Firmware Task Force (PFTF: https://github.com/pftf) Compute > Module 4 (CM4: an embedded form factor RPi4) is an ACPI platform that > isn't ECAM compliant. Its config space is in two parts. One part is for > the root port registers and a second moveable window pointing at a > device's 4K config space. Thus it doesn't have an MCFG, and any MCFG > provided would be nonsense anyway. > > Instead, a custom pci_ecam_ops quirk is created. The custom ops override > the .init and .map_bus functions. The former to assure that cfg->win > points at a single mapping that contains the root port registers and the > device config window, as well as disabling MSIs due to lack of a > GICv2M. map_bus() then provides the address of either the standard > portion of the root port registers or to the device config window after > it has been moved. > > Additionally, some basic bus/device filtering exist to avoid sending > config transactions to invalid devices on the RP's primary or > secondary bus. A basic link check is also made to assure that > something is operational on the secondary side before probing the > remainder of the config space. If either of these constraints are > violated and a config operation is lost in the ether because an EP > doesn't respond an unrecoverable SERROR is raised. It's not "lost"; I assume the root port raises an error because it can't send a transaction over a link that is down. Is "SERROR" an ARM64 thing? My guess is the root port would raise an Unsupported Request error or similar, and the root complex turns that into a system-specific SERROR? > Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> > Acked-by: Florian Fainelli <f.fainelli@gmail.com> > Acked-by: Bjorn Helgaas <bhelgaas@google.com> > --- > drivers/pci/controller/Makefile | 1 + > drivers/pci/controller/pcie-brcmstb-acpi.c | 79 ++++++++++++++++++++++ > include/linux/pci-ecam.h | 1 + > 3 files changed, 81 insertions(+) > create mode 100644 drivers/pci/controller/pcie-brcmstb-acpi.c > > diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile > index aaf30b3dcc14..65aa6fd3ed89 100644 > --- a/drivers/pci/controller/Makefile > +++ b/drivers/pci/controller/Makefile > @@ -57,5 +57,6 @@ ifdef CONFIG_PCI_QUIRKS > obj-$(CONFIG_ARM64) += pci-thunder-ecam.o > obj-$(CONFIG_ARM64) += pci-thunder-pem.o > obj-$(CONFIG_ARM64) += pci-xgene.o > +obj-$(CONFIG_ARM64) += pcie-brcmstb-acpi.o > endif > endif > diff --git a/drivers/pci/controller/pcie-brcmstb-acpi.c b/drivers/pci/controller/pcie-brcmstb-acpi.c > new file mode 100644 > index 000000000000..528b2b3ffbd2 > --- /dev/null > +++ b/drivers/pci/controller/pcie-brcmstb-acpi.c > @@ -0,0 +1,79 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * ACPI quirks for Brcm2711 PCIe host controller > + * As used on the Raspberry Pi Compute Module 4 > + * > + * Copyright (C) 2021 Arm Ltd. > + */ > + > +#include <linux/io.h> > +#include <linux/pci.h> > +#include <linux/pci-ecam.h> > +#include "../pci.h" > +#include "pcie-brcmstb.h" > + > +static int brcm_acpi_init(struct pci_config_window *cfg) > +{ > + /* > + * This platform doesn't technically have anything that could be called > + * ECAM. Its config region has root port specific registers between > + * standard PCIe defined config registers. Thus the region setup by the > + * generic ECAM code needs to be adjusted. The HW can access bus 0-ff > + * but the footprint isn't a nice power of 2 (40k). For purposes of > + * mapping the config region we are just going to squash the standard > + * and nonstandard registers together rather than mapping them separately. Wrap this and comment below to fit in 80 columns. Nothing magic about 80 columns except for the fact that all the other code in drivers/pci fits in that width and consistency is helpful. > + */ > + iounmap(cfg->win); > + cfg->win = pci_remap_cfgspace(cfg->res.start, resource_size(&cfg->res)); > + if (!cfg->win) > + goto err_exit; > + > + /* MSI is nonstandard as well */ > + pci_no_msi(); This doesn't seem to fit in an MCFG quirk. > + return 0; > +err_exit: > + dev_err(cfg->parent, "PCI: Failed to remap config\n"); > + return -ENOMEM; > +} > + > +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, > + unsigned int devfn, int where) > +{ > + struct pci_config_window *cfg = bus->sysdata; > + void __iomem *base = cfg->win; > + int idx; > + u32 up; > + > + /* Accesses to the RC go right to the RC registers if slot==0 */ > + if (pci_is_root_bus(bus)) > + return PCI_SLOT(devfn) ? NULL : base + where; > + > + /* > + * Assure the link is up before sending requests downstream. This is done > + * to avoid sending transactions to EPs that don't exist. Link flap > + * conditions/etc make this race more probable. The resulting unrecoverable > + * SERRORs will result in the machine crashing. Is the crash because SERROR is fundamentally unrecoverable? Is there any control over what kind of system-specific error the PCIe errors are mapped to? I know there are other systems where PCIe errors always cause a system crash, but most platforms seem to be moving toward at least the theoretical ability to recover from I/O errors. > + */ > + up = readl(base + PCIE_MISC_PCIE_STATUS); > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) > + return NULL; > + > + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) > + return NULL; > + > + /* For devices, write to the config space index register */ > + idx = PCIE_ECAM_OFFSET(bus->number, devfn, 0); > + writel(idx, base + PCIE_EXT_CFG_INDEX); > + return base + PCIE_EXT_CFG_DATA + where; > +} > + > +const struct pci_ecam_ops bcm2711_pcie_ops = { > + .init = brcm_acpi_init, > + .bus_shift = 1, > + .pci_ops = { > + .map_bus = brcm_pcie_map_conf2, > + .read = pci_generic_config_read, > + .write = pci_generic_config_write, > + } > +}; > diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h > index adea5a4771cf..a5de0285bb7f 100644 > --- a/include/linux/pci-ecam.h > +++ b/include/linux/pci-ecam.h > @@ -87,6 +87,7 @@ extern const struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 * > extern const struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ > extern const struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ > extern const struct pci_ecam_ops tegra194_pcie_ops; /* Tegra194 PCIe */ > +extern const struct pci_ecam_ops bcm2711_pcie_ops; /* Bcm2711 PCIe */ > #endif > > #if IS_ENABLED(CONFIG_PCI_HOST_COMMON) > -- > 2.31.1 >
Hi, On 10/5/21 10:32 AM, Bjorn Helgaas wrote: > On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: >> The Pi Firmware Task Force (PFTF: https://github.com/pftf) Compute >> Module 4 (CM4: an embedded form factor RPi4) is an ACPI platform that >> isn't ECAM compliant. Its config space is in two parts. One part is for >> the root port registers and a second moveable window pointing at a >> device's 4K config space. Thus it doesn't have an MCFG, and any MCFG >> provided would be nonsense anyway. >> >> Instead, a custom pci_ecam_ops quirk is created. The custom ops override >> the .init and .map_bus functions. The former to assure that cfg->win >> points at a single mapping that contains the root port registers and the >> device config window, as well as disabling MSIs due to lack of a >> GICv2M. map_bus() then provides the address of either the standard >> portion of the root port registers or to the device config window after >> it has been moved. >> >> Additionally, some basic bus/device filtering exist to avoid sending >> config transactions to invalid devices on the RP's primary or >> secondary bus. A basic link check is also made to assure that >> something is operational on the secondary side before probing the >> remainder of the config space. If either of these constraints are >> violated and a config operation is lost in the ether because an EP >> doesn't respond an unrecoverable SERROR is raised. > > It's not "lost"; I assume the root port raises an error because it > can't send a transaction over a link that is down. The problem is AFAIK because the root port doesn't do that. > > Is "SERROR" an ARM64 thing? My guess is the root port would raise an > Unsupported Request error or similar, and the root complex turns that > into a system-specific SERROR? AFAIK, what is happening here the CPU core has an outstanding R/W request for which it never receives a response from the root port. So basically its an interconnect protocol violation that the CPU is complaining about rather than something PCIe specific. > >> Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> >> Acked-by: Florian Fainelli <f.fainelli@gmail.com> >> Acked-by: Bjorn Helgaas <bhelgaas@google.com> >> --- >> drivers/pci/controller/Makefile | 1 + >> drivers/pci/controller/pcie-brcmstb-acpi.c | 79 ++++++++++++++++++++++ >> include/linux/pci-ecam.h | 1 + >> 3 files changed, 81 insertions(+) >> create mode 100644 drivers/pci/controller/pcie-brcmstb-acpi.c >> >> diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile >> index aaf30b3dcc14..65aa6fd3ed89 100644 >> --- a/drivers/pci/controller/Makefile >> +++ b/drivers/pci/controller/Makefile >> @@ -57,5 +57,6 @@ ifdef CONFIG_PCI_QUIRKS >> obj-$(CONFIG_ARM64) += pci-thunder-ecam.o >> obj-$(CONFIG_ARM64) += pci-thunder-pem.o >> obj-$(CONFIG_ARM64) += pci-xgene.o >> +obj-$(CONFIG_ARM64) += pcie-brcmstb-acpi.o >> endif >> endif >> diff --git a/drivers/pci/controller/pcie-brcmstb-acpi.c b/drivers/pci/controller/pcie-brcmstb-acpi.c >> new file mode 100644 >> index 000000000000..528b2b3ffbd2 >> --- /dev/null >> +++ b/drivers/pci/controller/pcie-brcmstb-acpi.c >> @@ -0,0 +1,79 @@ >> +// SPDX-License-Identifier: GPL-2.0+ >> +/* >> + * ACPI quirks for Brcm2711 PCIe host controller >> + * As used on the Raspberry Pi Compute Module 4 >> + * >> + * Copyright (C) 2021 Arm Ltd. >> + */ >> + >> +#include <linux/io.h> >> +#include <linux/pci.h> >> +#include <linux/pci-ecam.h> >> +#include "../pci.h" >> +#include "pcie-brcmstb.h" >> + >> +static int brcm_acpi_init(struct pci_config_window *cfg) >> +{ >> + /* >> + * This platform doesn't technically have anything that could be called >> + * ECAM. Its config region has root port specific registers between >> + * standard PCIe defined config registers. Thus the region setup by the >> + * generic ECAM code needs to be adjusted. The HW can access bus 0-ff >> + * but the footprint isn't a nice power of 2 (40k). For purposes of >> + * mapping the config region we are just going to squash the standard >> + * and nonstandard registers together rather than mapping them separately. > > Wrap this and comment below to fit in 80 columns. Nothing magic about > 80 columns except for the fact that all the other code in drivers/pci > fits in that width and consistency is helpful. Hmm, I thought I had wrapped these, but probably at 78, guess I will rebase/repost. > >> + */ >> + iounmap(cfg->win); >> + cfg->win = pci_remap_cfgspace(cfg->res.start, resource_size(&cfg->res)); >> + if (!cfg->win) >> + goto err_exit; >> + >> + /* MSI is nonstandard as well */ >> + pci_no_msi(); > > This doesn't seem to fit in an MCFG quirk. "pcie quirk" > >> + return 0; >> +err_exit: >> + dev_err(cfg->parent, "PCI: Failed to remap config\n"); >> + return -ENOMEM; >> +} >> + >> +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, >> + unsigned int devfn, int where) >> +{ >> + struct pci_config_window *cfg = bus->sysdata; >> + void __iomem *base = cfg->win; >> + int idx; >> + u32 up; >> + >> + /* Accesses to the RC go right to the RC registers if slot==0 */ >> + if (pci_is_root_bus(bus)) >> + return PCI_SLOT(devfn) ? NULL : base + where; >> + >> + /* >> + * Assure the link is up before sending requests downstream. This is done >> + * to avoid sending transactions to EPs that don't exist. Link flap >> + * conditions/etc make this race more probable. The resulting unrecoverable >> + * SERRORs will result in the machine crashing. > > Is the crash because SERROR is fundamentally unrecoverable? Is there > any control over what kind of system-specific error the PCIe errors > are mapped to? Yes, that is basically where we are because the reason for the exception can't really be pinned down. There are some thoughts about ways to work around the problem, but they aren't pretty and probably need to be done at an exception level higher than what the kernel is operating at. Hence the SMC. > > I know there are other systems where PCIe errors always cause a system > crash, but most platforms seem to be moving toward at least the > theoretical ability to recover from I/O errors. Yes, that would be part of SBSA, which of course this platform isn't. > >> + */ >> + up = readl(base + PCIE_MISC_PCIE_STATUS); >> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) >> + return NULL; >> + >> + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) >> + return NULL; >> + >> + /* For devices, write to the config space index register */ >> + idx = PCIE_ECAM_OFFSET(bus->number, devfn, 0); >> + writel(idx, base + PCIE_EXT_CFG_INDEX); >> + return base + PCIE_EXT_CFG_DATA + where; >> +} >> + >> +const struct pci_ecam_ops bcm2711_pcie_ops = { >> + .init = brcm_acpi_init, >> + .bus_shift = 1, >> + .pci_ops = { >> + .map_bus = brcm_pcie_map_conf2, >> + .read = pci_generic_config_read, >> + .write = pci_generic_config_write, >> + } >> +}; >> diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h >> index adea5a4771cf..a5de0285bb7f 100644 >> --- a/include/linux/pci-ecam.h >> +++ b/include/linux/pci-ecam.h >> @@ -87,6 +87,7 @@ extern const struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 * >> extern const struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ >> extern const struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ >> extern const struct pci_ecam_ops tegra194_pcie_ops; /* Tegra194 PCIe */ >> +extern const struct pci_ecam_ops bcm2711_pcie_ops; /* Bcm2711 PCIe */ >> #endif >> >> #if IS_ENABLED(CONFIG_PCI_HOST_COMMON) >> -- >> 2.31.1 >>
Hello! On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: > Hi, > > On 10/5/21 10:32 AM, Bjorn Helgaas wrote: > > On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: > > > Additionally, some basic bus/device filtering exist to avoid sending > > > config transactions to invalid devices on the RP's primary or > > > secondary bus. A basic link check is also made to assure that > > > something is operational on the secondary side before probing the > > > remainder of the config space. If either of these constraints are > > > violated and a config operation is lost in the ether because an EP > > > doesn't respond an unrecoverable SERROR is raised. > > > > It's not "lost"; I assume the root port raises an error because it > > can't send a transaction over a link that is down. > > The problem is AFAIK because the root port doesn't do that. Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I guess contains also logic for Root Port) does not signal transaction failure for config requests? Or it is just your opinion? Because I'm dealing with similar issues and I'm trying to find a way how to detect if some PCIe IP signal transaction error via AXI SLVERR response OR it just does not send any response back. So if you know some way how to check which one it is, I would like to know it too. > > > > Is "SERROR" an ARM64 thing? My guess is the root port would raise an > > Unsupported Request error or similar, and the root complex turns that > > into a system-specific SERROR? Yes, SError is arm64 specific. It is asynchronous CPU interrupt and syndrome code then contains what happened. > AFAIK, what is happening here the CPU core has an outstanding R/W request > for which it never receives a response from the root port. So basically its > an interconnect protocol violation that the CPU is complaining about rather > than something PCIe specific. Could you describe (ideally in commit message) which SError is triggered? Normally if kernel receive SError interrupt it also puts into dmesg or oops message also syndrome code which describe what kind of error / event occurred. It could help also to other understand what is happening there.
Hi, On 10/5/21 2:43 PM, Pali Rohár wrote: > Hello! > > On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: >> Hi, >> >> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: >>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: >>>> Additionally, some basic bus/device filtering exist to avoid sending >>>> config transactions to invalid devices on the RP's primary or >>>> secondary bus. A basic link check is also made to assure that >>>> something is operational on the secondary side before probing the >>>> remainder of the config space. If either of these constraints are >>>> violated and a config operation is lost in the ether because an EP >>>> doesn't respond an unrecoverable SERROR is raised. >>> >>> It's not "lost"; I assume the root port raises an error because it >>> can't send a transaction over a link that is down. >> >> The problem is AFAIK because the root port doesn't do that. > > Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I > guess contains also logic for Root Port) does not signal transaction > failure for config requests? Or it is just your opinion? Because I'm > dealing with similar issues and I'm trying to find a way how to detect > if some PCIe IP signal transaction error via AXI SLVERR response OR it > just does not send any response back. So if you know some way how to > check which one it is, I would like to know it too. This is my _opinion_ based on what I've heard of some other IP integration issues, and what i've seen poking at this one from the perspective of a SW guy rather than a HW guy. So, basically worthless. But, you should consider that most of these cores/interconnects aren't aware of PCIe completion semantics so its the root ports responsibility to say, gracefully translate a non-posted write that doesn't have a completion for the interconnects its attached to, rather than tripping something generic like a SLVERR. Anyway, for this I would poke around the pile of exception registers, with your specific processors manual handy because a lot of them are implementation defined. >>> >>> Is "SERROR" an ARM64 thing? My guess is the root port would raise an >>> Unsupported Request error or similar, and the root complex turns that >>> into a system-specific SERROR? > > Yes, SError is arm64 specific. It is asynchronous CPU interrupt and > syndrome code then contains what happened. > >> AFAIK, what is happening here the CPU core has an outstanding R/W request >> for which it never receives a response from the root port. So basically its >> an interconnect protocol violation that the CPU is complaining about rather >> than something PCIe specific. > > Could you describe (ideally in commit message) which SError is > triggered? Normally if kernel receive SError interrupt it also puts into > dmesg or oops message also syndrome code which describe what kind of > error / event occurred. It could help also to other understand what is > happening there. >
On 10/5/2021 3:25 PM, Jeremy Linton wrote: > Hi, > > On 10/5/21 2:43 PM, Pali Rohár wrote: >> Hello! >> >> On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: >>> Hi, >>> >>> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: >>>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: >>>>> Additionally, some basic bus/device filtering exist to avoid sending >>>>> config transactions to invalid devices on the RP's primary or >>>>> secondary bus. A basic link check is also made to assure that >>>>> something is operational on the secondary side before probing the >>>>> remainder of the config space. If either of these constraints are >>>>> violated and a config operation is lost in the ether because an EP >>>>> doesn't respond an unrecoverable SERROR is raised. >>>> >>>> It's not "lost"; I assume the root port raises an error because it >>>> can't send a transaction over a link that is down. >>> >>> The problem is AFAIK because the root port doesn't do that. >> >> Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I >> guess contains also logic for Root Port) does not signal transaction >> failure for config requests? Or it is just your opinion? Because I'm >> dealing with similar issues and I'm trying to find a way how to detect >> if some PCIe IP signal transaction error via AXI SLVERR response OR it >> just does not send any response back. So if you know some way how to >> check which one it is, I would like to know it too. > > This is my _opinion_ based on what I've heard of some other IP > integration issues, and what i've seen poking at this one from the > perspective of a SW guy rather than a HW guy. So, basically worthless. > But, you should consider that most of these cores/interconnects aren't > aware of PCIe completion semantics so its the root ports responsibility > to say, gracefully translate a non-posted write that doesn't have a > completion for the interconnects its attached to, rather than tripping > something generic like a SLVERR. > > Anyway, for this I would poke around the pile of exception registers, > with your specific processors manual handy because a lot of them are > implementation defined. I should be able to get you an answer in the new few days whether configuration space requests also generate an error towards the ARM CPU, since memory space requests most definitively do.
On 10/5/21 7:07 PM, Florian Fainelli wrote: > > > On 10/5/2021 3:25 PM, Jeremy Linton wrote: >> Hi, >> >> On 10/5/21 2:43 PM, Pali Rohár wrote: >>> Hello! >>> >>> On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: >>>> Hi, >>>> >>>> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: >>>>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: >>>>>> Additionally, some basic bus/device filtering exist to avoid sending >>>>>> config transactions to invalid devices on the RP's primary or >>>>>> secondary bus. A basic link check is also made to assure that >>>>>> something is operational on the secondary side before probing the >>>>>> remainder of the config space. If either of these constraints are >>>>>> violated and a config operation is lost in the ether because an EP >>>>>> doesn't respond an unrecoverable SERROR is raised. >>>>> >>>>> It's not "lost"; I assume the root port raises an error because it >>>>> can't send a transaction over a link that is down. >>>> >>>> The problem is AFAIK because the root port doesn't do that. >>> >>> Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I >>> guess contains also logic for Root Port) does not signal transaction >>> failure for config requests? Or it is just your opinion? Because I'm >>> dealing with similar issues and I'm trying to find a way how to detect >>> if some PCIe IP signal transaction error via AXI SLVERR response OR it >>> just does not send any response back. So if you know some way how to >>> check which one it is, I would like to know it too. >> >> This is my _opinion_ based on what I've heard of some other IP >> integration issues, and what i've seen poking at this one from the >> perspective of a SW guy rather than a HW guy. So, basically worthless. >> But, you should consider that most of these cores/interconnects aren't >> aware of PCIe completion semantics so its the root ports >> responsibility to say, gracefully translate a non-posted write that >> doesn't have a completion for the interconnects its attached to, >> rather than tripping something generic like a SLVERR. >> >> Anyway, for this I would poke around the pile of exception registers, >> with your specific processors manual handy because a lot of them are >> implementation defined. > > I should be able to get you an answer in the new few days whether > configuration space requests also generate an error towards the ARM CPU, > since memory space requests most definitively do. Did not get an answer from the design team, but going through our bug tracker, there were evidences of configuration space accesses also generating external aborts: [ 8.988237] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff8009539004 [ 8.996539] Internal error: : 96000210 [#1] SMP [ 9.001107] Modules linked in: [ 9.004215] CPU: 2 PID: 6 Comm: kworker/u8:0 Not tainted 4.9.51-gstb-4.9 #1 [ 9.011216] Hardware name: BCM97278SV (DT) [ 9.015365] Workqueue: events_unbound async_run_entry_fn [ 9.020728] task: ffffffc00a4ab5c0 task.stack: ffffffc00a4e4000 [ 9.026698] PC is at pci_generic_config_read32+0x30/0xb0 [ 9.032053] LR is at pci_generic_config_read32+0x2c/0xb0 [ 9.037403] pc : [<ffffff8008394eb8>] lr : [<ffffff8008394eb4>] pstate: 800000c5 [ 9.044852] sp : ffffffc00a4e7ba0 [ 9.048197] x29: ffffffc00a4e7ba0 x28: ffffffc00a40caa8 [ 9.053574] x27: ffffffc00a40c878 x26: ffffffc00a40c820 [ 9.058949] x25: ffffff800935c77d x24: 0000000000000040 [ 9.064323] x23: ffffff80093d5ac0 x22: ffffffc00a4e7c66 [ 9.069698] x21: ffffffc00a4e7c24 x20: 0000000000000002 [ 9.075072] x19: 000000000000004c x18: ffffffffffffffff [ 9.080448] x17: fffeffffb7ffffff x16: fffffdffb6ffffff [ 9.085822] x15: 0000000000000000 x14: ffffffc009e8fdb8 [ 9.091196] x13: 0000000000000014 x12: 0000000000000000 [ 9.096571] x11: 0000000000000000 x10: 00000000000006d0 [ 9.101946] x9 : ffffffc00a4e4000 x8 : ffffffc00a4abcf0 [ 9.107322] x7 : 0000000000005ec7 x6 : 0000000000000005 [ 9.112696] x5 : ffffff8009530000 x4 : ffffff80087da530 [ 9.118073] x3 : ffffff8009539000 x2 : 000000000000004c [ 9.123448] x1 : 0000000000009004 x0 : ffffff8009539004 [ 9.128823] [ 9.130341] Process kworker/u8:0 (pid: 6, stack limit = 0xffffffc00a4e4020) [ 9.137346] Stack: (0xffffffc00a4e7ba0 to 0xffffffc00a4e8000) [ 9.143136] 7ba0: ffffffc00a4e7bd0 ffffff8008394be4 ffffff8009306000 0000000000000087 [ 9.151029] 7bc0: ffffffc009838000 ffffff800878a170 ffffffc00a4e7c30 ffffff800839b004 [ 9.158923] 7be0: ffffffc00a5bb000 0000000000000000 ffffff8009306000 ffffff80088f7048 [ 9.166814] 7c00: 0000000000000000 ffffffc00a40c800 ffffff80092d7440 000000000000004c [ 9.174706] 7c20: 0000000000000002 0000000000040933 ffffffc00a4e7c70 ffffff800839e570 [ 9.182598] 7c40: ffffffc00a5bb000 ffffff80093d5000 0000000000000000 ffffff80088f7048 [ 9.190490] 7c60: ffffffc00a5bb0a0 0000000000040933 ffffffc00a4e7c90 ffffff80083a1cc4 [ 9.198383] 7c80: ffffffc00a5bb0a0 ffffffc00a5bb000 ffffffc00a4e7cc0 ffffff800845c9cc [ 9.206275] 7ca0: ffffffc00a5bb0a0 ffffff80083a1ca0 0000000000000010 ffffffc00a4ab5c0 [ 9.214167] 7cc0: ffffffc00a4e7d00 ffffff800845cab4 ffffffc00a5bb0a0 0000000000000000 [ 9.222060] 7ce0: 0000000000000010 0000000000000000 ffffffc00a455100 ffffff800845ca80 [ 9.229953] 7d00: ffffffc00a4e7d30 ffffff800845cc84 ffffff80093d9828 ffffffc00a5bb0a0 [ 9.237845] 7d20: ffffffc009decd00 0000000000000000 ffffffc00a4e7d50 ffffff80080ba464 [ 9.245737] 7d40: ffffffc009decd20 ffffff80093a3000 ffffffc00a4e7d80 ffffff80080b0f28 [ 9.253628] 7d60: 0000000000000000 ffffffc00a454e40 ffffffc009decd20 0000000000000000 [ 9.261521] 7d80: ffffffc00a4e7dc0 ffffff80080b1130 ffffffc00a454e40 ffffffc00a40c800 [ 9.269413] 7da0: ffffffc00a454e70 ffffffc00a40c820 ffffff8009306000 ffffffc00a4e4000 [ 9.277305] 7dc0: ffffffc00a4e7e20 ffffff80080b7194 ffffffc00a494400 ffffffc00a4e4000 [ 9.285198] 7de0: ffffff80088afc00 ffffffc00a454e40 ffffff80080b10e8 0000000000000000 [ 9.293088] 7e00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.300981] 7e20: 0000000000000000 ffffff8008082900 ffffff80080b70a0 ffffffc00a494400 [ 9.308872] 7e40: 0000000000000000 0000000000000000 0000000000000000 705afddc1b1ccaa8 [ 9.316764] 7e60: 0000000000000000 ffffff80080bff90 ffffffc00a454e40 ffffffc000000000 [ 9.324656] 7e80: 0000000000000000 ffffffc00a4e7e88 ffffffc00a4e7e88 0000000000000000 [ 9.332547] 7ea0: 0000000000000000 ffffffc00a4e7ea8 ffffffc00a4e7ea8 0000000000040933 [ 9.340438] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.348329] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.356219] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.364111] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.372002] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.379894] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.387785] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.395675] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 9.403566] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000 [ 9.411458] 7fe0: 0000000000000000 0000000000000000 46525cd6105c6384 aa9421d6b6fa9074 [ 9.419343] Call trace: [ 9.421821] Exception stack(0xffffffc00a4e79b0 to 0xffffffc00a4e7ae0) [ 9.428301] 79a0: 000000000000004c 0000008000000000 [ 9.436193] 79c0: ffffffc00a4e7ba0 ffffff8008394eb8 00000000800000c5 ffffffc07fe8e440 [ 9.444086] 79e0: ffffffc00a4efd98 0000000000000007 ffffffc000000000 ffffff8009539004 [ 9.451977] 7a00: 0000000000000000 ffffffc009bfbe40 ffffff80092d7440 ffffff80092d7440 [ 9.459869] 7a20: 0000000000000002 ffffff80080c3ee8 ffffffc009bfbe40 ffffffc07fe64440 [ 9.467761] 7a40: 0000000200000000 ffffffc07fe8e440 ffffffc00a4e7a90 ffffff80080c3ee8 [ 9.475652] 7a60: 0000000000000001 0000000000040933 ffffff8009539004 0000000000009004 [ 9.483544] 7a80: 000000000000004c ffffff8009539000 ffffff80087da530 ffffff8009530000 [ 9.491435] 7aa0: 0000000000000005 0000000000005ec7 ffffffc00a4abcf0 ffffffc00a4e4000 [ 9.499327] 7ac0: 00000000000006d0 0000000000000000 0000000000000000 0000000000000014 [ 9.507223] [<ffffff8008394eb8>] pci_generic_config_read32+0x30/0xb0 [ 9.513623] [<ffffff8008394be4>] pci_bus_read_config_word+0x9c/0xc0 [ 9.519936] [<ffffff800839b004>] pci_raw_set_power_state+0x7c/0x248 [ 9.526250] [<ffffff800839e570>] pci_power_up+0x50/0x68 [ 9.531516] [<ffffff80083a1cc4>] pci_pm_resume_noirq+0x24/0xc0 [ 9.537395] [<ffffff800845c9cc>] dpm_run_callback+0x4c/0xc0 [ 9.543008] [<ffffff800845cab4>] device_resume_noirq+0x74/0x220 [ 9.548969] [<ffffff800845cc84>] async_resume_noirq+0x24/0x58 [ 9.554757] [<ffffff80080ba464>] async_run_entry_fn+0x3c/0x160 [ 9.560635] [<ffffff80080b0f28>] process_one_work+0x1d0/0x390 [ 9.566424] [<ffffff80080b1130>] worker_thread+0x48/0x4b0 [ 9.571863] [<ffffff80080b7194>] kthread+0xf4/0x108 [ 9.576782] [<ffffff8008082900>] ret_from_fork+0x10/0x50 [ 9.582136] Code: f9406005 f94008a3 d63f0060 b4000320 (b9400001) [ 9.588311] ---[ end trace efc83c99ae7412ee ]---
On Friday 22 October 2021 10:04:36 Florian Fainelli wrote: > On 10/5/21 7:07 PM, Florian Fainelli wrote: > > > > > > On 10/5/2021 3:25 PM, Jeremy Linton wrote: > >> Hi, > >> > >> On 10/5/21 2:43 PM, Pali Rohár wrote: > >>> Hello! > >>> > >>> On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: > >>>> Hi, > >>>> > >>>> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: > >>>>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: > >>>>>> Additionally, some basic bus/device filtering exist to avoid sending > >>>>>> config transactions to invalid devices on the RP's primary or > >>>>>> secondary bus. A basic link check is also made to assure that > >>>>>> something is operational on the secondary side before probing the > >>>>>> remainder of the config space. If either of these constraints are > >>>>>> violated and a config operation is lost in the ether because an EP > >>>>>> doesn't respond an unrecoverable SERROR is raised. > >>>>> > >>>>> It's not "lost"; I assume the root port raises an error because it > >>>>> can't send a transaction over a link that is down. > >>>> > >>>> The problem is AFAIK because the root port doesn't do that. > >>> > >>> Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I > >>> guess contains also logic for Root Port) does not signal transaction > >>> failure for config requests? Or it is just your opinion? Because I'm > >>> dealing with similar issues and I'm trying to find a way how to detect > >>> if some PCIe IP signal transaction error via AXI SLVERR response OR it > >>> just does not send any response back. So if you know some way how to > >>> check which one it is, I would like to know it too. > >> > >> This is my _opinion_ based on what I've heard of some other IP > >> integration issues, and what i've seen poking at this one from the > >> perspective of a SW guy rather than a HW guy. So, basically worthless. > >> But, you should consider that most of these cores/interconnects aren't > >> aware of PCIe completion semantics so its the root ports > >> responsibility to say, gracefully translate a non-posted write that > >> doesn't have a completion for the interconnects its attached to, > >> rather than tripping something generic like a SLVERR. > >> > >> Anyway, for this I would poke around the pile of exception registers, > >> with your specific processors manual handy because a lot of them are > >> implementation defined. > > > > I should be able to get you an answer in the new few days whether > > configuration space requests also generate an error towards the ARM CPU, > > since memory space requests most definitively do. > > Did not get an answer from the design team, but going through our bug > tracker, there were evidences of configuration space accesses also > generating external aborts: > > [ 8.988237] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff8009539004 > [ 9.026698] PC is at pci_generic_config_read32+0x30/0xb0 So this is error caused by reading from config space. Can you check if also writing to config space can trigger some crash? If yes, I would like to know if write would be also synchronous or rather asynchronous abort.
On 10/22/21 10:17 AM, Pali Rohár wrote: > On Friday 22 October 2021 10:04:36 Florian Fainelli wrote: >> On 10/5/21 7:07 PM, Florian Fainelli wrote: >>> >>> >>> On 10/5/2021 3:25 PM, Jeremy Linton wrote: >>>> Hi, >>>> >>>> On 10/5/21 2:43 PM, Pali Rohár wrote: >>>>> Hello! >>>>> >>>>> On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: >>>>>> Hi, >>>>>> >>>>>> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: >>>>>>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: >>>>>>>> Additionally, some basic bus/device filtering exist to avoid sending >>>>>>>> config transactions to invalid devices on the RP's primary or >>>>>>>> secondary bus. A basic link check is also made to assure that >>>>>>>> something is operational on the secondary side before probing the >>>>>>>> remainder of the config space. If either of these constraints are >>>>>>>> violated and a config operation is lost in the ether because an EP >>>>>>>> doesn't respond an unrecoverable SERROR is raised. >>>>>>> >>>>>>> It's not "lost"; I assume the root port raises an error because it >>>>>>> can't send a transaction over a link that is down. >>>>>> >>>>>> The problem is AFAIK because the root port doesn't do that. >>>>> >>>>> Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I >>>>> guess contains also logic for Root Port) does not signal transaction >>>>> failure for config requests? Or it is just your opinion? Because I'm >>>>> dealing with similar issues and I'm trying to find a way how to detect >>>>> if some PCIe IP signal transaction error via AXI SLVERR response OR it >>>>> just does not send any response back. So if you know some way how to >>>>> check which one it is, I would like to know it too. >>>> >>>> This is my _opinion_ based on what I've heard of some other IP >>>> integration issues, and what i've seen poking at this one from the >>>> perspective of a SW guy rather than a HW guy. So, basically worthless. >>>> But, you should consider that most of these cores/interconnects aren't >>>> aware of PCIe completion semantics so its the root ports >>>> responsibility to say, gracefully translate a non-posted write that >>>> doesn't have a completion for the interconnects its attached to, >>>> rather than tripping something generic like a SLVERR. >>>> >>>> Anyway, for this I would poke around the pile of exception registers, >>>> with your specific processors manual handy because a lot of them are >>>> implementation defined. >>> >>> I should be able to get you an answer in the new few days whether >>> configuration space requests also generate an error towards the ARM CPU, >>> since memory space requests most definitively do. >> >> Did not get an answer from the design team, but going through our bug >> tracker, there were evidences of configuration space accesses also >> generating external aborts: >> >> [ 8.988237] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff8009539004 >> [ 9.026698] PC is at pci_generic_config_read32+0x30/0xb0 > > So this is error caused by reading from config space. > > Can you check if also writing to config space can trigger some crash? If > yes, I would like to know if write would be also synchronous or rather > asynchronous abort. Yes it does and AFAICT it always shows up as a system error interrupt, here is an example: # setpci -d *:* latency_timer=40 [ 25.909644] SError Interrupt on CPU2, code 0xbf000002 -- SError [ 25.909647] CPU: 2 PID: 1676 Comm: setpci Not tainted 5.10.70-0.2pre-ge3872e15011b #2 [ 25.909649] Hardware name: BCM972165SV_V10 (DT) [ 25.909651] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--) [ 25.909652] pc : pci_user_write_config_byte+0x6c/0x78 [ 25.909654] lr : pci_user_write_config_byte+0x68/0x78 [ 25.909655] sp : ffffffc015853c20 [ 25.909656] x29: ffffffc015853c20 x28: ffffff8003053000 [ 25.909661] x27: 0000000000000000 x26: 0000000000000000 [ 25.909664] x25: 0000000000000001 x24: ffffff8004a23780 [ 25.909668] x23: ffffff80049aa000 x22: ffffffc015853d68 [ 25.909671] x21: 0000000000000040 x20: 000000000000000d [ 25.909674] x19: 000000000000000e x18: 0000000000000000 [ 25.909677] x17: 0000000000000000 x16: 0000000000000000 [ 25.909680] x15: 0000000000000000 x14: 0000000000000000 [ 25.909684] x13: 0000000000000000 x12: 0000000000000000 [ 25.909687] x11: 0000000000000000 x10: 0000000000000000 [ 25.909690] x9 : ffffffc010483214 x8 : 0000000000000000 [ 25.909693] x7 : ffffff800498df00 x6 : ffffff80049a8380 [ 25.909696] x5 : ffffffc015510000 x4 : ffffff80049a9800 [ 25.909699] x3 : 0000000000000000 x2 : 000000000000000d [ 25.909702] x1 : 0000000000000000 x0 : 0000000000000000 [ 25.909706] Kernel panic - not syncing: Asynchronous SError Interrupt [ 25.909708] CPU: 2 PID: 1676 Comm: setpci Not tainted 5.10.70-0.2pre-ge3872e15011b #2 [ 25.909710] Hardware name: BCM972165SV_V10 (DT) [ 25.909711] Call trace: [ 25.909712] dump_backtrace+0x0/0x1d0 [ 25.909713] show_stack+0x1c/0x24 [ 25.909714] dump_stack+0xd0/0x12c [ 25.909716] panic+0x128/0x308 [ 25.909717] nmi_panic+0x50/0x70 [ 25.909718] arm64_serror_panic+0x74/0x80 [ 25.909720] do_serror+0x28/0x60 [ 25.909721] el1_error+0x8c/0x10c [ 25.909722] pci_user_write_config_byte+0x6c/0x78 [ 25.909724] pci_write_config+0x7c/0x1a0 [ 25.909725] sysfs_kf_bin_write+0x64/0x84 [ 25.909727] kernfs_fop_write_iter+0xbc/0x170 [ 25.909728] new_sync_write+0x80/0xcc [ 25.909729] vfs_write+0xec/0x110 [ 25.909730] ksys_pwrite64+0x50/0x8c [ 25.909732] __arm64_sys_pwrite64+0x20/0x28 [ 25.909733] el0_svc_common.constprop.4+0x100/0x184 [ 25.909735] do_el0_svc+0x38/0x78 [ 25.909736] el0_svc+0x1c/0x28 [ 25.909737] el0_sync_handler+0x64/0x12c [ 25.909738] el0_sync+0x148/0x180 [ 25.909775] brcm-pcie 8b20000.pcie: Error: CFG Acc, 32bit, Write, Bus=1, Dev=0, Fun=0, Reg=0xc, lanes=01000000 [ 26.136082] brcm-pcie 8b20000.pcie: Type: TO=0 Abt=0 UnsupReq=0 AccTO=0 AccDsbld=1 Acc64bit=0 [ 26.144709] SMP: stopping secondary CPUs [ 26.144711] Kernel Offset: disabled [ 26.144712] CPU features: 0x0040002,24002004 [ 26.144713] Memory Limit: none
On Friday 22 October 2021 10:29:48 Florian Fainelli wrote: > On 10/22/21 10:17 AM, Pali Rohár wrote: > > On Friday 22 October 2021 10:04:36 Florian Fainelli wrote: > >> On 10/5/21 7:07 PM, Florian Fainelli wrote: > >>> > >>> > >>> On 10/5/2021 3:25 PM, Jeremy Linton wrote: > >>>> Hi, > >>>> > >>>> On 10/5/21 2:43 PM, Pali Rohár wrote: > >>>>> Hello! > >>>>> > >>>>> On Tuesday 05 October 2021 10:57:18 Jeremy Linton wrote: > >>>>>> Hi, > >>>>>> > >>>>>> On 10/5/21 10:32 AM, Bjorn Helgaas wrote: > >>>>>>> On Thu, Aug 26, 2021 at 02:15:55AM -0500, Jeremy Linton wrote: > >>>>>>>> Additionally, some basic bus/device filtering exist to avoid sending > >>>>>>>> config transactions to invalid devices on the RP's primary or > >>>>>>>> secondary bus. A basic link check is also made to assure that > >>>>>>>> something is operational on the secondary side before probing the > >>>>>>>> remainder of the config space. If either of these constraints are > >>>>>>>> violated and a config operation is lost in the ether because an EP > >>>>>>>> doesn't respond an unrecoverable SERROR is raised. > >>>>>>> > >>>>>>> It's not "lost"; I assume the root port raises an error because it > >>>>>>> can't send a transaction over a link that is down. > >>>>>> > >>>>>> The problem is AFAIK because the root port doesn't do that. > >>>>> > >>>>> Interesting! Does it mean that PCIe Root Complex / Host Bridge (which I > >>>>> guess contains also logic for Root Port) does not signal transaction > >>>>> failure for config requests? Or it is just your opinion? Because I'm > >>>>> dealing with similar issues and I'm trying to find a way how to detect > >>>>> if some PCIe IP signal transaction error via AXI SLVERR response OR it > >>>>> just does not send any response back. So if you know some way how to > >>>>> check which one it is, I would like to know it too. > >>>> > >>>> This is my _opinion_ based on what I've heard of some other IP > >>>> integration issues, and what i've seen poking at this one from the > >>>> perspective of a SW guy rather than a HW guy. So, basically worthless. > >>>> But, you should consider that most of these cores/interconnects aren't > >>>> aware of PCIe completion semantics so its the root ports > >>>> responsibility to say, gracefully translate a non-posted write that > >>>> doesn't have a completion for the interconnects its attached to, > >>>> rather than tripping something generic like a SLVERR. > >>>> > >>>> Anyway, for this I would poke around the pile of exception registers, > >>>> with your specific processors manual handy because a lot of them are > >>>> implementation defined. > >>> > >>> I should be able to get you an answer in the new few days whether > >>> configuration space requests also generate an error towards the ARM CPU, > >>> since memory space requests most definitively do. > >> > >> Did not get an answer from the design team, but going through our bug > >> tracker, there were evidences of configuration space accesses also > >> generating external aborts: > >> > >> [ 8.988237] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff8009539004 > >> [ 9.026698] PC is at pci_generic_config_read32+0x30/0xb0 > > > > So this is error caused by reading from config space. > > > > Can you check if also writing to config space can trigger some crash? If > > yes, I would like to know if write would be also synchronous or rather > > asynchronous abort. > > Yes it does and AFAICT it always shows up as a system error interrupt, > here is an example: > > # setpci -d *:* latency_timer=40 > [ 25.909644] SError Interrupt on CPU2, code 0xbf000002 -- SError > [ 25.909652] pc : pci_user_write_config_byte+0x6c/0x78 > [ 25.909706] Kernel panic - not syncing: Asynchronous SError Interrupt Ok! So writing to config space cause asynchronous abort. Looking at the codes and 0x96000210 on all ARMv8 should be Data Abort. 0xbf...... on ARMv8 is SError interrupt and other bits are CPU core specific. What CPU core do you have on this machine? I have just decoder for A53 core and on this core value 0xbf000002 means "SLVERR on external access". But I guess that it would mean also SLVERR for your CPU core. Because Exactly same behavior I'm seeing with PCIe controller on A3720 SoC which has A53 core. It looks like that PCIe controller translates PCIe CA and UR responses to AXI SLVERR responses which are delivered to CPU and kernel just see these fatal error interrupts. And same issue is not only for config requests but also for memory read / write commands. In my case PCIe controller really receives response (timeout does not occur) from PCIe core (which probably timeouts as it cannot send message when link is down) but instead of translating them to SLVOK with fabricated 0xffffffff response it sends to CPU that fatal SLVERR. I was told that the fix for this kind of issue is to "reconfigure" PCIe controller to never send SLVERR to CPU. And instead fabricate 0xffffffff SLVOK response. It should be configurable in PCIe wrapper or PCIe glue IP which do connection between CPU / AXI and PCIe core. I do not know if there is any way how to "ignores" these SLVERR responses from PCIe controller sent to CPU.
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile index aaf30b3dcc14..65aa6fd3ed89 100644 --- a/drivers/pci/controller/Makefile +++ b/drivers/pci/controller/Makefile @@ -57,5 +57,6 @@ ifdef CONFIG_PCI_QUIRKS obj-$(CONFIG_ARM64) += pci-thunder-ecam.o obj-$(CONFIG_ARM64) += pci-thunder-pem.o obj-$(CONFIG_ARM64) += pci-xgene.o +obj-$(CONFIG_ARM64) += pcie-brcmstb-acpi.o endif endif diff --git a/drivers/pci/controller/pcie-brcmstb-acpi.c b/drivers/pci/controller/pcie-brcmstb-acpi.c new file mode 100644 index 000000000000..528b2b3ffbd2 --- /dev/null +++ b/drivers/pci/controller/pcie-brcmstb-acpi.c @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * ACPI quirks for Brcm2711 PCIe host controller + * As used on the Raspberry Pi Compute Module 4 + * + * Copyright (C) 2021 Arm Ltd. + */ + +#include <linux/io.h> +#include <linux/pci.h> +#include <linux/pci-ecam.h> +#include "../pci.h" +#include "pcie-brcmstb.h" + +static int brcm_acpi_init(struct pci_config_window *cfg) +{ + /* + * This platform doesn't technically have anything that could be called + * ECAM. Its config region has root port specific registers between + * standard PCIe defined config registers. Thus the region setup by the + * generic ECAM code needs to be adjusted. The HW can access bus 0-ff + * but the footprint isn't a nice power of 2 (40k). For purposes of + * mapping the config region we are just going to squash the standard + * and nonstandard registers together rather than mapping them separately. + */ + iounmap(cfg->win); + cfg->win = pci_remap_cfgspace(cfg->res.start, resource_size(&cfg->res)); + if (!cfg->win) + goto err_exit; + + /* MSI is nonstandard as well */ + pci_no_msi(); + + return 0; +err_exit: + dev_err(cfg->parent, "PCI: Failed to remap config\n"); + return -ENOMEM; +} + +static void __iomem *brcm_pcie_map_conf2(struct pci_bus *bus, + unsigned int devfn, int where) +{ + struct pci_config_window *cfg = bus->sysdata; + void __iomem *base = cfg->win; + int idx; + u32 up; + + /* Accesses to the RC go right to the RC registers if slot==0 */ + if (pci_is_root_bus(bus)) + return PCI_SLOT(devfn) ? NULL : base + where; + + /* + * Assure the link is up before sending requests downstream. This is done + * to avoid sending transactions to EPs that don't exist. Link flap + * conditions/etc make this race more probable. The resulting unrecoverable + * SERRORs will result in the machine crashing. + */ + up = readl(base + PCIE_MISC_PCIE_STATUS); + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_DL_ACTIVE_MASK)) + return NULL; + + if (!(up & PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK)) + return NULL; + + /* For devices, write to the config space index register */ + idx = PCIE_ECAM_OFFSET(bus->number, devfn, 0); + writel(idx, base + PCIE_EXT_CFG_INDEX); + return base + PCIE_EXT_CFG_DATA + where; +} + +const struct pci_ecam_ops bcm2711_pcie_ops = { + .init = brcm_acpi_init, + .bus_shift = 1, + .pci_ops = { + .map_bus = brcm_pcie_map_conf2, + .read = pci_generic_config_read, + .write = pci_generic_config_write, + } +}; diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h index adea5a4771cf..a5de0285bb7f 100644 --- a/include/linux/pci-ecam.h +++ b/include/linux/pci-ecam.h @@ -87,6 +87,7 @@ extern const struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 * extern const struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ extern const struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ extern const struct pci_ecam_ops tegra194_pcie_ops; /* Tegra194 PCIe */ +extern const struct pci_ecam_ops bcm2711_pcie_ops; /* Bcm2711 PCIe */ #endif #if IS_ENABLED(CONFIG_PCI_HOST_COMMON)