| Message ID | 23031613.R8qN30TbTq@wuerfel (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
On Tue, Mar 29, 2016 at 6:52 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Tuesday 29 March 2016 06:32:29 Tim Harvey wrote: >> > >> > There is no upstream driver for this chip, so I don't know where to look >> > to find out if the driver tries to enable MSI. >> > >> > Is what you are saying that if you enable MSI support in the kernel, it >> > breaks legacy IRQs? >> >> Yes - any driver that does not support MSI will use legacy IRQ's and >> they never fire. >> >> The Ventana GW53xx and GW54xx boards have a Marvell PCIe GigE >> supported by the sky2 driver as eth1 which does support MSI and I >> verified it gets an MSI interrupt and does work (along ath9k devices >> with legacy interrupts that fail to work). >> >> root@ventana:~# cat /proc/interrupts | grep MSI >> 299: 0 0 PCI-MSI 0 Edge PCIe PME, aerdrv >> 308: 8726 0 PCI-MSI 9 Edge eth1 >> >> So it appears that MSI works for those drivers that use it, but does >> somehow cause legacy IRQ's to break. >> >> I sent you a GW5400 dev kit over a while back to use for through >> bridge testing on IMX6 that you should be able to repeat this with >> assuming you have a PCIe card with a driver that doesn't support MSI >> (or that you can tweak its driver to not support MSI). >> >> I think 31e98e0d24cd2537a63e06e235e050a06b175df7 "ARM: >> imx_v6_v7_defconfig: enable PCI_MSI" should be reverted as well until >> we figure this out. > > That doesn't sound like a helpful solution, multi_v7_defconfig for > instance will still be broken because it enables PCI_MSI, and so > will be all major distros. Arnd, True, but keep in mind that as far as I can tell this behavior has been around since MSI was added to the IMX6 (breakage of devices that use legacy irq's if MSI is enabled) which was in 3.16. > > What happens if we patch the pci-imx6 driver to not make use of > its MSI support even when that is enabled in the kernel? Does that > get both devices in your GW5xxx to work with legacy interrupts or > is one or both of them still broken? > > Arnd > > diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c > index eb5a2755a164..d7607b2695c6 100644 > --- a/drivers/pci/host/pci-imx6.c > +++ b/drivers/pci/host/pci-imx6.c > @@ -470,7 +470,7 @@ static void imx6_pcie_host_init(struct pcie_port *pp) > > imx6_pcie_establish_link(pp); > > - if (IS_ENABLED(CONFIG_PCI_MSI)) > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) > dw_pcie_msi_init(pp); > } > > @@ -490,7 +490,7 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp, > { > int ret; > > - if (IS_ENABLED(CONFIG_PCI_MSI)) { > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) { > pp->msi_irq = platform_get_irq_byname(pdev, "msi"); > if (pp->msi_irq <= 0) { > dev_err(&pdev->dev, "failed to get MSI irq\n"); > That is not enough - we would also need to disable a couple more in the designware core that imx6 uses, which is also used by several other SoC's. We should probably get some feedback from people with those SoC's regarding MSI breaking legacy irqs. PCI_MSI was enabled in imx_v6_v7_defconfig in 4.5 and enabled in multi_v7_defconfig back in 3.16. PCI MSI was first introduced for the IMX6 host controller in 3.16 as well. I verified that the same issue exists all the way back to 3.16. I don't know if its worse to disable PCI MSI for IMX6/designware all the way back to 3.16 or to disable it just back to 4.5 where imx_v6_v7 enabled it, or perhaps just figure out the actual issue and get that backported? Lucas, have you had any thoughts or time yet to think about why enabling MSI breaks legacy IRQs? Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tuesday 29 March 2016 07:29:34 Tim Harvey wrote: > On Tue, Mar 29, 2016 at 6:52 AM, Arnd Bergmann <arnd@arndb.de> wrote: > > On Tuesday 29 March 2016 06:32:29 Tim Harvey wrote: > >> I think 31e98e0d24cd2537a63e06e235e050a06b175df7 "ARM: > >> imx_v6_v7_defconfig: enable PCI_MSI" should be reverted as well until > >> we figure this out. > > > > That doesn't sound like a helpful solution, multi_v7_defconfig for > > instance will still be broken because it enables PCI_MSI, and so > > will be all major distros. > > Arnd, > > True, but keep in mind that as far as I can tell this behavior has > been around since MSI was added to the IMX6 (breakage of devices that > use legacy irq's if MSI is enabled) which was in 3.16. I see. This part wasn't clear to me. > > What happens if we patch the pci-imx6 driver to not make use of > > its MSI support even when that is enabled in the kernel? Does that > > get both devices in your GW5xxx to work with legacy interrupts or > > is one or both of them still broken? > > > > Arnd > > > > diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c > > index eb5a2755a164..d7607b2695c6 100644 > > --- a/drivers/pci/host/pci-imx6.c > > +++ b/drivers/pci/host/pci-imx6.c > > @@ -470,7 +470,7 @@ static void imx6_pcie_host_init(struct pcie_port *pp) > > > > imx6_pcie_establish_link(pp); > > > > - if (IS_ENABLED(CONFIG_PCI_MSI)) > > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) > > dw_pcie_msi_init(pp); > > } > > > > @@ -490,7 +490,7 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp, > > { > > int ret; > > > > - if (IS_ENABLED(CONFIG_PCI_MSI)) { > > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) { > > pp->msi_irq = platform_get_irq_byname(pdev, "msi"); > > if (pp->msi_irq <= 0) { > > dev_err(&pdev->dev, "failed to get MSI irq\n"); > > > > That is not enough - we would also need to disable a couple more in > the designware core that imx6 uses, which is also used by several > other SoC's. We should probably get some feedback from people with > those SoC's regarding MSI breaking legacy irqs. Good point. I really just meant this as an experiment, trying to figure out what causes it to break. I'd be surprised if the MSI support in the generic pcie-designware driver caused the same problem on the other SoCs. > PCI_MSI was enabled in imx_v6_v7_defconfig in 4.5 and enabled in > multi_v7_defconfig back in 3.16. PCI MSI was first introduced for the > IMX6 host controller in 3.16 as well. I verified that the same issue > exists all the way back to 3.16. Thanks for doing that research. > I don't know if its worse to disable PCI MSI for IMX6/designware all > the way back to 3.16 or to disable it just back to 4.5 where imx_v6_v7 > enabled it, or perhaps just figure out the actual issue and get that > backported? I'd like to see the problem understood better before we talk about backports. One thing I just noticed is that the same GPC interrupt line (GIC_SPI 120) is used for MSI and for IntD. Maybe there is something going on with sharing an interrupt line between a nested irqchip and a device? Could this be a bug in the generic IRQ handling code? Can you check which interrupt the broken device(s) on your machine are using? Is it always the 120 (IntD) line, or are the other three lines broken as well? I don't actually know how to look it up, but the 'lspci -t' output should let us reconstruct the swizzling manually. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 29, 2016 at 7:50 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Tuesday 29 March 2016 07:29:34 Tim Harvey wrote: >> On Tue, Mar 29, 2016 at 6:52 AM, Arnd Bergmann <arnd@arndb.de> wrote: >> > On Tuesday 29 March 2016 06:32:29 Tim Harvey wrote: > >> >> I think 31e98e0d24cd2537a63e06e235e050a06b175df7 "ARM: >> >> imx_v6_v7_defconfig: enable PCI_MSI" should be reverted as well until >> >> we figure this out. >> > >> > That doesn't sound like a helpful solution, multi_v7_defconfig for >> > instance will still be broken because it enables PCI_MSI, and so >> > will be all major distros. >> >> Arnd, >> >> True, but keep in mind that as far as I can tell this behavior has >> been around since MSI was added to the IMX6 (breakage of devices that >> use legacy irq's if MSI is enabled) which was in 3.16. > > I see. This part wasn't clear to me. > >> > What happens if we patch the pci-imx6 driver to not make use of >> > its MSI support even when that is enabled in the kernel? Does that >> > get both devices in your GW5xxx to work with legacy interrupts or >> > is one or both of them still broken? >> > >> > Arnd >> > >> > diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c >> > index eb5a2755a164..d7607b2695c6 100644 >> > --- a/drivers/pci/host/pci-imx6.c >> > +++ b/drivers/pci/host/pci-imx6.c >> > @@ -470,7 +470,7 @@ static void imx6_pcie_host_init(struct pcie_port *pp) >> > >> > imx6_pcie_establish_link(pp); >> > >> > - if (IS_ENABLED(CONFIG_PCI_MSI)) >> > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) >> > dw_pcie_msi_init(pp); >> > } >> > >> > @@ -490,7 +490,7 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp, >> > { >> > int ret; >> > >> > - if (IS_ENABLED(CONFIG_PCI_MSI)) { >> > + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) { >> > pp->msi_irq = platform_get_irq_byname(pdev, "msi"); >> > if (pp->msi_irq <= 0) { >> > dev_err(&pdev->dev, "failed to get MSI irq\n"); >> > >> >> That is not enough - we would also need to disable a couple more in >> the designware core that imx6 uses, which is also used by several >> other SoC's. We should probably get some feedback from people with >> those SoC's regarding MSI breaking legacy irqs. > > Good point. I really just meant this as an experiment, trying to > figure out what causes it to break. I'd be surprised if the > MSI support in the generic pcie-designware driver caused the same > problem on the other SoCs. > >> PCI_MSI was enabled in imx_v6_v7_defconfig in 4.5 and enabled in >> multi_v7_defconfig back in 3.16. PCI MSI was first introduced for the >> IMX6 host controller in 3.16 as well. I verified that the same issue >> exists all the way back to 3.16. > > Thanks for doing that research. > >> I don't know if its worse to disable PCI MSI for IMX6/designware all >> the way back to 3.16 or to disable it just back to 4.5 where imx_v6_v7 >> enabled it, or perhaps just figure out the actual issue and get that >> backported? > > I'd like to see the problem understood better before we talk about > backports. > > One thing I just noticed is that the same GPC interrupt line (GIC_SPI > 120) is used for MSI and for IntD. Maybe there is something going on with > sharing an interrupt line between a nested irqchip and a device? > > Could this be a bug in the generic IRQ handling code? Can you check > which interrupt the broken device(s) on your machine are using? Is > it always the 120 (IntD) line, or are the other three lines broken > as well? I don't actually know how to look it up, but the 'lspci -t' > output should let us reconstruct the swizzling manually. > > Arnd Arnd, Right, on the IMX the MSI interrupt is GIC-120 which is also the legacy INTD and I do see that if I happen to put a radio in a slot where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt does fire and the device works. Any other slot using GIC-123 (INTA), GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible that something in the designware core is masking out the legacy irqs. I would also think this was something IMX specific, but I really don't see any codepaths in pci-imx6.c that would cause that: a driver requesting a legacy PCI would get a GIC interrupt which is handled by the IMX6 gpc interrupt controller. Any dra7xxx, exynos, spear13xx, keystone, layerscape, hisi, qcom SoC users of designware PCIe core out there that can verify PCI MSI and legacy are both working at the same time? Lucas is the expert here and I believe he has the documentation for the designware core that Freescale doens't provide with the IMX6 documentation so hopefully he can provide some insight. He's the one that has authored all the MSI support and has been using it. I typically advise our users to 'not' enable MSI because architecturally you can spread 4 distinct legacy irq's across CPU's better than a single shared irq. Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tuesday 29 March 2016 08:10:08 Tim Harvey wrote: > Arnd, > > Right, on the IMX the MSI interrupt is GIC-120 which is also the > legacy INTD and I do see that if I happen to put a radio in a slot > where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt > does fire and the device works. Any other slot using GIC-123 (INTA), > GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible > that something in the designware core is masking out the legacy irqs. Interesting. I was actually expecting the opposite here, having the IRQs only work if they are not IntD. > I typically advise our users to 'not' enable MSI because > architecturally you can spread 4 distinct legacy irq's across CPU's > better than a single shared irq. That is a very good point, I never understood why we want to enable MSI support on any PCI host bridge that just forwards all MSIs to a single IRQ line. Originally MSI was meant as a performance feature, but there is nothing in this setup that makes things go faster, and several things that make it go slower. I would still hope that with disabling MSI support in just the i.MX driver (as in the trivial patch I suggested trying, or by reverting Lucas' d1dc9749a5b8 patch) will make things just work. If not, we may need to change the pcie-designware driver as well, so it doesn't try to enable MSI on its own. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/29/2016 05:10 PM, Tim Harvey wrote: > Arnd, > > Right, on the IMX the MSI interrupt is GIC-120 which is also the > legacy INTD and I do see that if I happen to put a radio in a slot > where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt > does fire and the device works. Any other slot using GIC-123 (INTA), > GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible > that something in the designware core is masking out the legacy irqs. > I would also think this was something IMX specific, but I really don't > see any codepaths in pci-imx6.c that would cause that: a driver > requesting a legacy PCI would get a GIC interrupt which is handled by > the IMX6 gpc interrupt controller. > > Any dra7xxx, exynos, spear13xx, keystone, layerscape, hisi, qcom SoC > users of designware PCIe core out there that can verify PCI MSI and > legacy are both working at the same time? > > Lucas is the expert here and I believe he has the documentation for > the designware core that Freescale doens't provide with the IMX6 > documentation so hopefully he can provide some insight. He's the one > that has authored all the MSI support and has been using it. > > I typically advise our users to 'not' enable MSI because > architecturally you can spread 4 distinct legacy irq's across CPU's > better than a single shared irq. Don't know if I'm facing similar problem, however devices connected in miniPCI slot behind a PCIe-to-PCI bridge (MSI is disabled) using INTA all is working ok, including shared IRQ. In case of INTB will not work, and the GIC irq quite often get stuck. -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 29, 2016 at 9:13 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: > On 03/29/2016 05:10 PM, Tim Harvey wrote: >> Arnd, >> >> Right, on the IMX the MSI interrupt is GIC-120 which is also the >> legacy INTD and I do see that if I happen to put a radio in a slot >> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt >> does fire and the device works. Any other slot using GIC-123 (INTA), >> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible >> that something in the designware core is masking out the legacy irqs. >> I would also think this was something IMX specific, but I really don't >> see any codepaths in pci-imx6.c that would cause that: a driver >> requesting a legacy PCI would get a GIC interrupt which is handled by >> the IMX6 gpc interrupt controller. >> >> Any dra7xxx, exynos, spear13xx, keystone, layerscape, hisi, qcom SoC >> users of designware PCIe core out there that can verify PCI MSI and >> legacy are both working at the same time? >> >> Lucas is the expert here and I believe he has the documentation for >> the designware core that Freescale doens't provide with the IMX6 >> documentation so hopefully he can provide some insight. He's the one >> that has authored all the MSI support and has been using it. >> >> I typically advise our users to 'not' enable MSI because >> architecturally you can spread 4 distinct legacy irq's across CPU's >> better than a single shared irq. > > Don't know if I'm facing similar problem, however devices connected in miniPCI slot behind > a PCIe-to-PCI bridge (MSI is disabled) using INTA all is working ok, including shared IRQ. > In case of INTB will not work, and the GIC irq quite often get stuck. > Roberto, What board/platform is this and what does /proc/interrupts look like? This sounds like what would happen if the downstream interrupts on the PCIe-to-PCI bridge are not mapped properly as was the case with a board I support (in which case I had to work out a bootloader fixup that placed a non-standard interrupt-map in the device-tree for the bridge). What bridge are you using? Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/29/2016 06:40 PM, Tim Harvey wrote: > On Tue, Mar 29, 2016 at 9:13 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: >> On 03/29/2016 05:10 PM, Tim Harvey wrote: >>> Arnd, >>> >>> Right, on the IMX the MSI interrupt is GIC-120 which is also the >>> legacy INTD and I do see that if I happen to put a radio in a slot >>> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt >>> does fire and the device works. Any other slot using GIC-123 (INTA), >>> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible >>> that something in the designware core is masking out the legacy irqs. >>> I would also think this was something IMX specific, but I really don't >>> see any codepaths in pci-imx6.c that would cause that: a driver >>> requesting a legacy PCI would get a GIC interrupt which is handled by >>> the IMX6 gpc interrupt controller. >>> >>> Any dra7xxx, exynos, spear13xx, keystone, layerscape, hisi, qcom SoC >>> users of designware PCIe core out there that can verify PCI MSI and >>> legacy are both working at the same time? >>> >>> Lucas is the expert here and I believe he has the documentation for >>> the designware core that Freescale doens't provide with the IMX6 >>> documentation so hopefully he can provide some insight. He's the one >>> that has authored all the MSI support and has been using it. >>> >>> I typically advise our users to 'not' enable MSI because >>> architecturally you can spread 4 distinct legacy irq's across CPU's >>> better than a single shared irq. >> Don't know if I'm facing similar problem, however devices connected in miniPCI slot behind >> a PCIe-to-PCI bridge (MSI is disabled) using INTA all is working ok, including shared IRQ. >> In case of INTB will not work, and the GIC irq quite often get stuck. >> > Roberto, > > What board/platform is this and what does /proc/interrupts look like? It's a custom board root@voneus-janas-imx6q:~# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 16: 936 637 2057 938 GIC 29 Edge twd 17: 0 0 0 0 GPC 55 Level i.MX Timer Tick 22: 247 0 0 0 GPC 26 Level 2020000.serial 34: 0 0 0 0 gpio-mxc 6 Edge Factory Reset Button 267: 0 0 0 0 GPC 49 Level imx_thermal 272: 0 0 0 0 GPC 19 Level rtc alarm 278: 0 0 0 0 GPC 2 Level sdma 281: 361 0 0 0 GIC 150 Level 2188000.ethernet 282: 0 0 0 0 GIC 151 Level 2188000.ethernet 283: 2882 0 0 0 GPC 25 Level mmc0 284: 95 0 0 0 GPC 37 Level 21a4000.i2c 290: 36546 0 0 0 GPC 123 Level PCIe PME, b4xxp 291: 2 0 0 0 GIC 137 Level 2101000.jr0 292: 0 0 0 0 GIC 138 Level 2102000.jr1 IPI0: 0 0 0 0 CPU wakeup interrupts IPI1: 0 0 0 0 Timer broadcast interrupts IPI2: 1642 1038 1626 1781 Rescheduling interrupts IPI3: 95 95 122 119 Function call interrupts IPI4: 3 0 2 0 Single function call interrupts IPI5: 0 0 0 0 CPU stop interrupts IPI6: 0 0 0 0 IRQ work interrupts IPI7: 0 0 0 0 completion interrupts Err: 0 > > This sounds like what would happen if the downstream interrupts on the > PCIe-to-PCI bridge are not mapped properly as was the case with a > board I support (in which case I had to work out a bootloader fixup > that placed a non-standard interrupt-map in the device-tree for the > bridge). What bridge are you using? PCIe-to-PCI bridge is a Ti XIO2001 where we are using INTA/B only wired 1:1 > > Tim > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 29, 2016 at 9:44 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: > On 03/29/2016 06:40 PM, Tim Harvey wrote: >> On Tue, Mar 29, 2016 at 9:13 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: >>> On 03/29/2016 05:10 PM, Tim Harvey wrote: >>>> Arnd, >>>> >>>> Right, on the IMX the MSI interrupt is GIC-120 which is also the >>>> legacy INTD and I do see that if I happen to put a radio in a slot >>>> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt >>>> does fire and the device works. Any other slot using GIC-123 (INTA), >>>> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible >>>> that something in the designware core is masking out the legacy irqs. >>>> I would also think this was something IMX specific, but I really don't >>>> see any codepaths in pci-imx6.c that would cause that: a driver >>>> requesting a legacy PCI would get a GIC interrupt which is handled by >>>> the IMX6 gpc interrupt controller. >>>> >>>> Any dra7xxx, exynos, spear13xx, keystone, layerscape, hisi, qcom SoC >>>> users of designware PCIe core out there that can verify PCI MSI and >>>> legacy are both working at the same time? >>>> >>>> Lucas is the expert here and I believe he has the documentation for >>>> the designware core that Freescale doens't provide with the IMX6 >>>> documentation so hopefully he can provide some insight. He's the one >>>> that has authored all the MSI support and has been using it. >>>> >>>> I typically advise our users to 'not' enable MSI because >>>> architecturally you can spread 4 distinct legacy irq's across CPU's >>>> better than a single shared irq. >>> Don't know if I'm facing similar problem, however devices connected in miniPCI slot behind >>> a PCIe-to-PCI bridge (MSI is disabled) using INTA all is working ok, including shared IRQ. >>> In case of INTB will not work, and the GIC irq quite often get stuck. >>> >> Roberto, >> >> What board/platform is this and what does /proc/interrupts look like? > > It's a custom board > > root@voneus-janas-imx6q:~# cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 > 16: 936 637 2057 938 GIC 29 Edge twd > 17: 0 0 0 0 GPC 55 Level i.MX Timer Tick > 22: 247 0 0 0 GPC 26 Level 2020000.serial > 34: 0 0 0 0 gpio-mxc 6 Edge Factory Reset Button > 267: 0 0 0 0 GPC 49 Level imx_thermal > 272: 0 0 0 0 GPC 19 Level rtc alarm > 278: 0 0 0 0 GPC 2 Level sdma > 281: 361 0 0 0 GIC 150 Level 2188000.ethernet > 282: 0 0 0 0 GIC 151 Level 2188000.ethernet > 283: 2882 0 0 0 GPC 25 Level mmc0 > 284: 95 0 0 0 GPC 37 Level 21a4000.i2c > 290: 36546 0 0 0 GPC 123 Level PCIe PME, b4xxp > 291: 2 0 0 0 GIC 137 Level 2101000.jr0 > 292: 0 0 0 0 GIC 138 Level 2102000.jr1 > IPI0: 0 0 0 0 CPU wakeup interrupts > IPI1: 0 0 0 0 Timer broadcast interrupts > IPI2: 1642 1038 1626 1781 Rescheduling interrupts > IPI3: 95 95 122 119 Function call interrupts > IPI4: 3 0 2 0 Single function call interrupts > IPI5: 0 0 0 0 CPU stop interrupts > IPI6: 0 0 0 0 IRQ work interrupts > IPI7: 0 0 0 0 completion interrupts > Err: 0 > > >> >> This sounds like what would happen if the downstream interrupts on the >> PCIe-to-PCI bridge are not mapped properly as was the case with a >> board I support (in which case I had to work out a bootloader fixup >> that placed a non-standard interrupt-map in the device-tree for the >> bridge). What bridge are you using? > > PCIe-to-PCI bridge is a Ti XIO2001 where we are using INTA/B only wired 1:1 > Roberto, That's right, we've talked about your bridge on IMX community. I don't see anything in your proc/interrupts other than GPC 123 - you probably only had one device populated when you did that. Put devices in all for slots then show me 'cat /proc/interrupts' as well as 'lspci -vv' (so that I can see what interrupt was given to pin1 and what interrupt that maps to on the IMX6). Check your XIO2001 routing and insure the following for proper IRQ mapping: Slot12: IDSEL A28: socket INTA = XIO2001 INTA Slot13: IDSEL A29: socket INTA = XIO2001 INTB Slot14: IDSEL A30: socket INTA = XIO2001 INTC Slot15: IDSEL A31: socket INTA = XIO2001 INTD The relationship between slot number of IDSEL is based on the PCI specification. The XIO2001 int mapping to socket mapping is based on Table 2 in the XIO2001 implementation guide. In my case what the hardware designer flipped the IDSEL mappings above such that slot12's idsel was hooked to A31 (so it was really slot15) etc, which created a non-standard mapping that required what ended up being a very time consuming and difficult to figure out software fixup (to say the least). Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 29, 2016 at 8:24 AM, Arnd Bergmann <arnd@arndb.de> wrote: > On Tuesday 29 March 2016 08:10:08 Tim Harvey wrote: >> Arnd, >> >> Right, on the IMX the MSI interrupt is GIC-120 which is also the >> legacy INTD and I do see that if I happen to put a radio in a slot >> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt >> does fire and the device works. Any other slot using GIC-123 (INTA), >> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible >> that something in the designware core is masking out the legacy irqs. > > Interesting. I was actually expecting the opposite here, having the > IRQs only work if they are not IntD. > > >> I typically advise our users to 'not' enable MSI because >> architecturally you can spread 4 distinct legacy irq's across CPU's >> better than a single shared irq. > > That is a very good point, I never understood why we want to enable > MSI support on any PCI host bridge that just forwards all MSIs > to a single IRQ line. Originally MSI was meant as a performance > feature, but there is nothing in this setup that makes things go > faster, and several things that make it go slower. I had a conversation once with Lucas about implementing the shared MSI interrupt in such a way that its smp affinity could be set to other CPU's to gain a performance benefit in certain multi-device cases. While this is technically possible it would involve creating a softirq glue between the different handlers but that would add overhead of a softirq plus potentially waking up another CPU to every IRQ which would end up adding some overhead to even the simple single-device case. Without any hard data it wasn't clear if this was worth it or if there was a clean way to provide this as build-time or run-time option. Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 29/03/16 16:24, Arnd Bergmann wrote: > On Tuesday 29 March 2016 08:10:08 Tim Harvey wrote: >> Arnd, >> >> Right, on the IMX the MSI interrupt is GIC-120 which is also the >> legacy INTD and I do see that if I happen to put a radio in a slot >> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt >> does fire and the device works. Any other slot using GIC-123 (INTA), >> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible >> that something in the designware core is masking out the legacy irqs. > > Interesting. I was actually expecting the opposite here, having the > IRQs only work if they are not IntD. > > >> I typically advise our users to 'not' enable MSI because >> architecturally you can spread 4 distinct legacy irq's across CPU's >> better than a single shared irq. > > That is a very good point, I never understood why we want to enable > MSI support on any PCI host bridge that just forwards all MSIs > to a single IRQ line. Originally MSI was meant as a performance > feature, but there is nothing in this setup that makes things go > faster, and several things that make it go slower. Feature-ticking exercise. "We support MSI", never mind if that negating the benefits of the mechanism and ending up with disastrous impacts on interrupt affinity, and a set of open questions regarding the effect of the MSI as a DMA fence. /me stops ranting for the day... M.
On Tuesday 29 March 2016 10:38:16 Tim Harvey wrote: > On Tue, Mar 29, 2016 at 8:24 AM, Arnd Bergmann <arnd@arndb.de> wrote: > > On Tuesday 29 March 2016 08:10:08 Tim Harvey wrote: > >> Arnd, > >> > >> Right, on the IMX the MSI interrupt is GIC-120 which is also the > >> legacy INTD and I do see that if I happen to put a radio in a slot > >> where due to swizzling its pin1 becomes INTD (GIC-120) the interrupt > >> does fire and the device works. Any other slot using GIC-123 (INTA), > >> GIC-122 (INTB), or GIC-121 (INTC) never fires so its very possible > >> that something in the designware core is masking out the legacy irqs. > > > > Interesting. I was actually expecting the opposite here, having the > > IRQs only work if they are not IntD. > > > > > >> I typically advise our users to 'not' enable MSI because > >> architecturally you can spread 4 distinct legacy irq's across CPU's > >> better than a single shared irq. > > > > That is a very good point, I never understood why we want to enable > > MSI support on any PCI host bridge that just forwards all MSIs > > to a single IRQ line. Originally MSI was meant as a performance > > feature, but there is nothing in this setup that makes things go > > faster, and several things that make it go slower. > > I had a conversation once with Lucas about implementing the shared MSI > interrupt in such a way that its smp affinity could be set to other > CPU's to gain a performance benefit in certain multi-device cases. > > While this is technically possible it would involve creating a softirq > glue between the different handlers but that would add overhead of a > softirq plus potentially waking up another CPU to every IRQ which > would end up adding some overhead to even the simple single-device > case. > > Without any hard data it wasn't clear if this was worth it or if there > was a clean way to provide this as build-time or run-time option. I think it's pretty clear that this would take things from 'somewhat silly' to 'completely bonkers' ;-) Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/29/2016 07:31 PM, Tim Harvey wrote: > On Tue, Mar 29, 2016 at 9:44 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: >>> >>> Roberto, >>> >>> What board/platform is this and what does /proc/interrupts look like? >> It's a custom board >> >> root@voneus-janas-imx6q:~# cat /proc/interrupts >> CPU0 CPU1 CPU2 CPU3 >> 16: 936 637 2057 938 GIC 29 Edge twd >> 17: 0 0 0 0 GPC 55 Level i.MX Timer Tick >> 22: 247 0 0 0 GPC 26 Level 2020000.serial >> 34: 0 0 0 0 gpio-mxc 6 Edge Factory Reset Button >> 267: 0 0 0 0 GPC 49 Level imx_thermal >> 272: 0 0 0 0 GPC 19 Level rtc alarm >> 278: 0 0 0 0 GPC 2 Level sdma >> 281: 361 0 0 0 GIC 150 Level 2188000.ethernet >> 282: 0 0 0 0 GIC 151 Level 2188000.ethernet >> 283: 2882 0 0 0 GPC 25 Level mmc0 >> 284: 95 0 0 0 GPC 37 Level 21a4000.i2c >> 290: 36546 0 0 0 GPC 123 Level PCIe PME, b4xxp >> 291: 2 0 0 0 GIC 137 Level 2101000.jr0 >> 292: 0 0 0 0 GIC 138 Level 2102000.jr1 >> IPI0: 0 0 0 0 CPU wakeup interrupts >> IPI1: 0 0 0 0 Timer broadcast interrupts >> IPI2: 1642 1038 1626 1781 Rescheduling interrupts >> IPI3: 95 95 122 119 Function call interrupts >> IPI4: 3 0 2 0 Single function call interrupts >> IPI5: 0 0 0 0 CPU stop interrupts >> IPI6: 0 0 0 0 IRQ work interrupts >> IPI7: 0 0 0 0 completion interrupts >> Err: 0 >> >> >>> This sounds like what would happen if the downstream interrupts on the >>> PCIe-to-PCI bridge are not mapped properly as was the case with a >>> board I support (in which case I had to work out a bootloader fixup >>> that placed a non-standard interrupt-map in the device-tree for the >>> bridge). What bridge are you using? >> PCIe-to-PCI bridge is a Ti XIO2001 where we are using INTA/B only wired 1:1 >> > Roberto, > > That's right, we've talked about your bridge on IMX community. > > I don't see anything in your proc/interrupts other than GPC 123 - you > probably only had one device populated when you did that. Yep! That was the case > Put devices > in all for slots then show me 'cat /proc/interrupts' as well as 'lspci > -vv' (so that I can see what interrupt was given to pin1 and what > interrupt that maps to on the IMX6). GPC 123 is serving J2 and J1, and looks ok root@voneus-janas-imx6q:~# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 16: 7409 25043 2869 71444 GIC 29 Edge twd 17: 0 0 0 0 GPC 55 Level i.MX Timer Tick 22: 526 0 0 0 GPC 26 Level 2020000.serial 34: 0 0 0 0 gpio-mxc 6 Edge Factory Reset Button 267: 0 0 0 0 GPC 49 Level imx_thermal 272: 0 0 0 0 GPC 19 Level rtc alarm 278: 0 0 0 0 GPC 2 Level sdma 281: 8808 0 0 0 GIC 150 Level 2188000.ethernet 282: 0 0 0 0 GIC 151 Level 2188000.ethernet 283: 2529 0 0 0 GPC 25 Level mmc0 284: 95 0 0 0 GPC 37 Level 21a4000.i2c 290: 2391578 0 0 0 GPC 123 Level PCIe PME, b4xxp, b4xxp 291: 2 0 0 0 GIC 137 Level 2101000.jr0 292: 0 0 0 0 GIC 138 Level 2102000.jr1 IPI0: 0 0 0 0 CPU wakeup interrupts IPI1: 0 0 0 0 Timer broadcast interrupts IPI2: 1122 3838 2051 9631 Rescheduling interrupts IPI3: 60 56 48 54 Function call interrupts IPI4: 2 1 2 3 Single function call interrupts IPI5: 0 0 0 0 CPU stop interrupts IPI6: 0 0 0 0 IRQ work interrupts IPI7: 0 0 0 0 completion interrupts Err: 0 root@voneus-janas-imx6q:~# lspci -vv -s 02: 02:00.0 ISDN controller: Cologne Chip Designs GmbH ISDN network Controller [HFC-4S] (rev 01) Subsystem: Cologne Chip Designs GmbH HFC-4S [OpenVox B200P / B400P] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 290 Region 0: I/O ports at 1000 [size=8] Region 1: Memory at 01100000 (32-bit, non-prefetchable) [size=4K] Capabilities: [40] Power Management version 2 Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: wcb4xxp Kernel modules: wcb4xxp 02:04.0 ISDN controller: Cologne Chip Designs GmbH ISDN network Controller [HFC-4S] (rev 01) Subsystem: Cologne Chip Designs GmbH HFC-4S [OpenVox B200P / B400P] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 290 Region 0: I/O ports at 1008 [size=8] Region 1: Memory at 01101000 (32-bit, non-prefetchable) [size=4K] Capabilities: [40] Power Management version 2 Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: wcb4xxp Kernel modules: wcb4xxp > > Check your XIO2001 routing and insure the following for proper IRQ mapping: > Slot12: IDSEL A28: socket INTA = XIO2001 INTA > Slot13: IDSEL A29: socket INTA = XIO2001 INTB > Slot14: IDSEL A30: socket INTA = XIO2001 INTC > Slot15: IDSEL A31: socket INTA = XIO2001 INTD After crosschecking with our hardware designer the PCB IRQ mapping is the following: J2 : IDSEL A16: => Device 0 : socket INTA = XIO2001 INTA J3 : IDSEL A18: => Device 2 : socket INTA = XIO2001 INTA* **(This should be INTC)* J11: IDSEL A20: => Device 4 : socket INTA = XIO2001 INTA The interrupt routing for J3 is wrong. The XIO2001 driver may expect Device 2 to interrupt on INTC - but it will interrupt on INTA. > > The relationship between slot number of IDSEL is based on the PCI > specification. The XIO2001 int mapping to socket mapping is based on > Table 2 in the XIO2001 implementation guide. In my case what the > hardware designer flipped the IDSEL mappings above such that slot12's > idsel was hooked to A31 (so it was really slot15) etc, which created a > non-standard mapping that required what ended up being a very time > consuming and difficult to figure out software fixup (to say the > least). Yep! I have read it > > Tim > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wednesday 30 March 2016 10:00:33 Roberto Fichera wrote: > > > > Check your XIO2001 routing and insure the following for proper IRQ mapping: > > Slot12: IDSEL A28: socket INTA = XIO2001 INTA > > Slot13: IDSEL A29: socket INTA = XIO2001 INTB > > Slot14: IDSEL A30: socket INTA = XIO2001 INTC > > Slot15: IDSEL A31: socket INTA = XIO2001 INTD > > After crosschecking with our hardware designer the PCB IRQ mapping is the following: > > J2 : IDSEL A16: => Device 0 : socket INTA = XIO2001 INTA > J3 : IDSEL A18: => Device 2 : socket INTA = XIO2001 INTA* **(This should be INTC)* > J11: IDSEL A20: => Device 4 : socket INTA = XIO2001 INTA > > The interrupt routing for J3 is wrong. The XIO2001 driver may expect Device 2 to interrupt on INTC - but it will > interrupt on INTA. What does your interrupt-map property look like then? Note that you have to override both map and map-mask in this case. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/30/2016 12:10 PM, Arnd Bergmann wrote: > On Wednesday 30 March 2016 10:00:33 Roberto Fichera wrote: >>> Check your XIO2001 routing and insure the following for proper IRQ mapping: >>> Slot12: IDSEL A28: socket INTA = XIO2001 INTA >>> Slot13: IDSEL A29: socket INTA = XIO2001 INTB >>> Slot14: IDSEL A30: socket INTA = XIO2001 INTC >>> Slot15: IDSEL A31: socket INTA = XIO2001 INTD >> After crosschecking with our hardware designer the PCB IRQ mapping is the following: >> >> J2 : IDSEL A16: => Device 0 : socket INTA = XIO2001 INTA >> J3 : IDSEL A18: => Device 2 : socket INTA = XIO2001 INTA* **(This should be INTC)* >> J11: IDSEL A20: => Device 4 : socket INTA = XIO2001 INTA >> >> The interrupt routing for J3 is wrong. The XIO2001 driver may expect Device 2 to interrupt on INTC - but it will >> interrupt on INTA. > What does your interrupt-map property look like then? Unfortunately it seems that J3 slot doesn't work anymore. Inserting a card there, PCIe link will not come up anymore. Likely I broke it. Looking at some spare logs I have, a card inserted in J3 will get another interrupt, was 291 however unfortunately I don't have an usefull lspci -vv output, sorry! Will check in it against another PCB when I can. > Note that you have to override both map and map-mask in this case. Can you please give more details where should I have a look? > > Arnd > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Mar 30, 2016 at 5:50 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: > On 03/30/2016 12:10 PM, Arnd Bergmann wrote: >> On Wednesday 30 March 2016 10:00:33 Roberto Fichera wrote: >>>> Check your XIO2001 routing and insure the following for proper IRQ mapping: >>>> Slot12: IDSEL A28: socket INTA = XIO2001 INTA >>>> Slot13: IDSEL A29: socket INTA = XIO2001 INTB >>>> Slot14: IDSEL A30: socket INTA = XIO2001 INTC >>>> Slot15: IDSEL A31: socket INTA = XIO2001 INTD >>> After crosschecking with our hardware designer the PCB IRQ mapping is the following: >>> >>> J2 : IDSEL A16: => Device 0 : socket INTA = XIO2001 INTA >>> J3 : IDSEL A18: => Device 2 : socket INTA = XIO2001 INTA* **(This should be INTC)* >>> J11: IDSEL A20: => Device 4 : socket INTA = XIO2001 INTA >>> >>> The interrupt routing for J3 is wrong. The XIO2001 driver may expect Device 2 to interrupt on INTC - but it will >>> interrupt on INTA. >> What does your interrupt-map property look like then? > > Unfortunately it seems that J3 slot doesn't work anymore. Inserting a card there, PCIe link will not come up anymore. > Likely I broke it. Looking at some spare logs I have, a card inserted in J3 will get another interrupt, was 291 however > unfortunately I don't have an usefull lspci -vv output, sorry! Will check in it against another PCB when I can. > >> Note that you have to override both map and map-mask in this case. > > Can you please give more details where should I have a look? > >> >> Arnd >> > Robert, The interrupt-map / interrupt-map-mask properties are the ones that dictate irq mapping. In most cases they are defined at the host controller only and the kernel will assume standard interrupt swizzling (aka barber pole'ing) through bridges and will rotate interrupts (swizzle) for each bridge you go through. It is only if the interrupts are 'not' mapped properly (as in your case, and as was mine on a specific add-in card) that you need to define interrupt-map / interrupt-map-mask for the actual bridge with the interrupt mapping issue. So in other words, you won't have an interrupt-map/mask for your TI XIO2001 'currently', but you need to add one to describe its non-standard mapping. If you look at imx6qdl.dtsi you'll see the interrupt-map/mask for standard mapping is: interrupt-map-mask = <0 0 0 0x7>; interrupt-map = <0 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>; This is because on the IMX6 INTA=GIC_123, INTB=GIC_122, INTC=GIC_121, INTD=GIC_120. The interrupt-map-mask is important because it decribes which bits of the 'unit interrupt specifier' each map pertains to. For the standard mapping above only the pin is important because this is the mapping for each slot so only the last three bits of the 'unit interrupt specifier' which has four cells is specified in the mask (0x7). In your case you will need to describe a map that depends not only on pin but on slot. The first 32bit cell in the 'unit interrupt specifier' defines the bus/domain/function (BDF) as: bus << 16 | dev << 11 | func <<8. This is also the same format for the first cell in the 'reg' property that describes each PCI device. If you are saying that your hardware did not swizzle the interrupts for the various slots then that means the above mapping needs to be applied to each slot the same. I You need to nest PCI devices as they appear on the bus, specifying reg properly. So, in your case you have a XIO2001 bridge hanging off of the IMX6 root complex. The root complex is at BDF 0:00.0, the XIO2001 is at BDF 1:00.0, and its sockets are at bus=2. So you will need to add the following to your device-tree (fixing pinctrl and reset-gpio per your board specifics): &pcie { pinctrl-names = "default"; pinctrl-0 = <&pinctrl_pcie>; reset-gpio = <&gpio1 29 GPIO_ACTIVE_LOW>; status = "okay"; /* 0:00.0 root complex */ pcie@0,0,0 { reg = <0 0 0 0 0>; /* bus=0, dev=0, func=0 */ /* 1:00.0 TI bridge with reversed IRQ mapping */ pcie@1,0,0 { device_type = "pci"; #address-cells = <3>; #size-cells = <2>; reg = <0x010000 0 0 0 0>; /* bus=1, dev=0, func=0 */ #interrupt-cells = <1>; interrupt-map-mask = <0xfff00 0 0 0x7>; /* match bus and device as well as pin */ interrupt-map = /* MAP for INTA/B/C/D in slot 2:00.00 - standard mapping */ <0x26000 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H <0x26000 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H <0x26000 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H <0x26000 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H /* MAP for INTA/B/C/D in slot 2:02.00 - wrong mapping */ <0x26800 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H <0x26800 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H <0x26800 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H <0x26800 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H /* MAP for INTA/B/C/D in slot 2:04.00 - standard mapping */ <0x27000 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H <0x27000 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H <0x27000 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H <0x27000 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H ... }; }; }; You will need to provide a full mapping which means you'll need to know how INTA/B/C/D are mapped to each slot. MiniPCIe only users INTA/B but afaik you have to specify all four. I 'think' what you are elluding to is that the hardware engineer didn't swizzle the slots at all so all slots are mapped with INTA->INTA, INTB->INTB, INTC->INTC, INTD->INTD. If this is the case then you just copy the above for all slots taking care to specify the first cell properly with b<<16 | d<<11. You may find http://devicetree.org/Device_Tree_Usage#Advanced_Interrupt_Mapping to be helpful as well. Regards, Tim -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/30/2016 03:38 PM, Tim Harvey wrote: > On Wed, Mar 30, 2016 at 5:50 AM, Roberto Fichera <kernel@tekno-soft.it> wrote: >> On 03/30/2016 12:10 PM, Arnd Bergmann wrote: >>> On Wednesday 30 March 2016 10:00:33 Roberto Fichera wrote: >>>>> Check your XIO2001 routing and insure the following for proper IRQ mapping: >>>>> Slot12: IDSEL A28: socket INTA = XIO2001 INTA >>>>> Slot13: IDSEL A29: socket INTA = XIO2001 INTB >>>>> Slot14: IDSEL A30: socket INTA = XIO2001 INTC >>>>> Slot15: IDSEL A31: socket INTA = XIO2001 INTD >>>> After crosschecking with our hardware designer the PCB IRQ mapping is the following: >>>> >>>> J2 : IDSEL A16: => Device 0 : socket INTA = XIO2001 INTA >>>> J3 : IDSEL A18: => Device 2 : socket INTA = XIO2001 INTA* **(This should be INTC)* >>>> J11: IDSEL A20: => Device 4 : socket INTA = XIO2001 INTA >>>> >>>> The interrupt routing for J3 is wrong. The XIO2001 driver may expect Device 2 to interrupt on INTC - but it will >>>> interrupt on INTA. >>> What does your interrupt-map property look like then? >> Unfortunately it seems that J3 slot doesn't work anymore. Inserting a card there, PCIe link will not come up anymore. >> Likely I broke it. Looking at some spare logs I have, a card inserted in J3 will get another interrupt, was 291 however >> unfortunately I don't have an usefull lspci -vv output, sorry! Will check in it against another PCB when I can. >> >>> Note that you have to override both map and map-mask in this case. >> Can you please give more details where should I have a look? >> >>> Arnd >>> > Robert, > > The interrupt-map / interrupt-map-mask properties are the ones that > dictate irq mapping. In most cases they are defined at the host > controller only and the kernel will assume standard interrupt > swizzling (aka barber pole'ing) through bridges and will rotate > interrupts (swizzle) for each bridge you go through. It is only if the > interrupts are 'not' mapped properly (as in your case, and as was mine > on a specific add-in card) that you need to define interrupt-map / > interrupt-map-mask for the actual bridge with the interrupt mapping > issue. So in other words, you won't have an interrupt-map/mask for > your TI XIO2001 'currently', but you need to add one to describe its > non-standard mapping. > > If you look at imx6qdl.dtsi you'll see the interrupt-map/mask for > standard mapping is: > > interrupt-map-mask = <0 0 0 0x7>; > interrupt-map = <0 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>, > <0 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>, > <0 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>, > <0 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>; > > This is because on the IMX6 INTA=GIC_123, INTB=GIC_122, INTC=GIC_121, > INTD=GIC_120. The interrupt-map-mask is important because it decribes > which bits of the 'unit interrupt specifier' each map pertains to. For > the standard mapping above only the pin is important because this is > the mapping for each slot so only the last three bits of the 'unit > interrupt specifier' which has four cells is specified in the mask > (0x7). > > In your case you will need to describe a map that depends not only on > pin but on slot. The first 32bit cell in the 'unit interrupt > specifier' defines the bus/domain/function (BDF) as: bus << 16 | dev > << 11 | func <<8. This is also the same format for the first cell in > the 'reg' property that describes each PCI device. > > If you are saying that your hardware did not swizzle the interrupts > for the various slots then that means the above mapping needs to be > applied to each slot the same. I > > You need to nest PCI devices as they appear on the bus, specifying reg > properly. So, in your case you have a XIO2001 bridge hanging off of > the IMX6 root complex. The root complex is at BDF 0:00.0, the XIO2001 > is at BDF 1:00.0, and its sockets are at bus=2. So you will need to > add the following to your device-tree (fixing pinctrl and reset-gpio > per your board specifics): > > &pcie { > pinctrl-names = "default"; > pinctrl-0 = <&pinctrl_pcie>; > reset-gpio = <&gpio1 29 GPIO_ACTIVE_LOW>; > status = "okay"; > > /* 0:00.0 root complex */ > pcie@0,0,0 { > reg = <0 0 0 0 0>; /* bus=0, dev=0, func=0 */ > > /* 1:00.0 TI bridge with reversed IRQ mapping */ > pcie@1,0,0 { > device_type = "pci"; > #address-cells = <3>; > #size-cells = <2>; > reg = <0x010000 0 0 0 0>; /* bus=1, dev=0, func=0 */ > #interrupt-cells = <1>; > interrupt-map-mask = <0xfff00 0 0 0x7>; /* > match bus and device as well as pin */ > interrupt-map = > /* MAP for INTA/B/C/D in slot 2:00.00 - > standard mapping */ > <0x26000 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H > <0x26000 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H > <0x26000 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H > <0x26000 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H > /* MAP for INTA/B/C/D in slot 2:02.00 - > wrong mapping */ > <0x26800 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H > <0x26800 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H > <0x26800 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H > <0x26800 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H > /* MAP for INTA/B/C/D in slot 2:04.00 - > standard mapping */ > <0x27000 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_H > <0x27000 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_H > <0x27000 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_H > <0x27000 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_H > ... > }; > }; > }; > > You will need to provide a full mapping which means you'll need to > know how INTA/B/C/D are mapped to each slot. MiniPCIe only users > INTA/B but afaik you have to specify all four. I 'think' what you are > elluding to is that the hardware engineer didn't swizzle the slots at > all so all slots are mapped with INTA->INTA, INTB->INTB, INTC->INTC, > INTD->INTD. If this is the case then you just copy the above for all > slots taking care to specify the first cell properly with b<<16 | > d<<11. > > You may find http://devicetree.org/Device_Tree_Usage#Advanced_Interrupt_Mapping > to be helpful as well. Tim, thanks for the detailed information. Will have a look soon. > > Regards, > > Tim > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c index eb5a2755a164..d7607b2695c6 100644 --- a/drivers/pci/host/pci-imx6.c +++ b/drivers/pci/host/pci-imx6.c @@ -470,7 +470,7 @@ static void imx6_pcie_host_init(struct pcie_port *pp) imx6_pcie_establish_link(pp); - if (IS_ENABLED(CONFIG_PCI_MSI)) + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) dw_pcie_msi_init(pp); } @@ -490,7 +490,7 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp, { int ret; - if (IS_ENABLED(CONFIG_PCI_MSI)) { + if (0 && IS_ENABLED(CONFIG_PCI_MSI)) { pp->msi_irq = platform_get_irq_byname(pdev, "msi"); if (pp->msi_irq <= 0) { dev_err(&pdev->dev, "failed to get MSI irq\n");