| Message ID | 20250128211559.1582598-4-Frank.Li@nxp.com (mailing list archive) |
|---|---|
| State | In Next, archived |
| Headers | show |
| Series | [1/5] arm64: dts: imx8-ss-hsio: fix indentation in pcie node | expand |
Hi Frank, Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > its. > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > --- > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > index 6b8470cb3461a..2cebeda43a52d 100644 > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > assigned-clock-parents = <0>, <0>, > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > + msi-map = <0x0 &its 0x10 0x1>, > + <0x100 &its 0x11 0x7>; Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. Either way, with this change PCIe on pcie0 is not working anymore, regardless of msi-map-mask. Without msi-map-mask: > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > r8169 0000:03:00.0: error -EINVAL: enable failure > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 With msi-map-mask: > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > r8169 0000:03:00.0: enabling device (0000 -> 0003) > r8169 0000:03:00.0: enabling Mem-Wr-Inval > r8169 0000:03:00.0: error -EIO: PCI read failed > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 Without msi-map/iommu-map: > r8169 0000:03:00.0: enabling device (0000 -> 0003) > r8169 0000:03:00.0: enabling Mem-Wr-Inval > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > r8169 0000:03:00.0 enp3s0: renamed from eth0 > r8169 0000:03:00.0: enabling bus mastering > r8169 0000:03:00.0 enp3s0: Link is Down pcie1 works as expected. But this is only a single PCIe device, rather than having a PCIe bridge. Any idea what's wrong here? Best regards, Alexander > + iommu-map = <0x000 &smmu 0x10 0x1>, > + <0x100 &smmu 0x11 0x7>; > + iommu-map-mask = <0x1ff>; > fsl,max-link-speed = <3>; > status = "disabled"; > }; > @@ -1640,6 +1646,14 @@ pcie1: pcie@4c380000 { > assigned-clock-parents = <0>, <0>, > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > + /* pcie1's Devid(BIT[7:6]) is 0x10, stream id(BIT[5:0]) is 0x18~0x1f */ > + msi-map = <0x0 &its 0x98 0x1>, > + <0x100 &its 0x99 0x7>; > + msi-map-mask = <0x1ff>; > + /* smmu have not Devid(BIT[7:6]) */ > + iommu-map = <0x000 &smmu 0x18 0x1>, > + <0x100 &smmu 0x19 0x7>; > + iommu-map-mask = <0x1ff>; > fsl,max-link-speed = <3>; > status = "disabled"; > }; >
On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > Hi Frank, > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > its. > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > --- > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > index 6b8470cb3461a..2cebeda43a52d 100644 > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > assigned-clock-parents = <0>, <0>, > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > + msi-map = <0x0 &its 0x10 0x1>, > > + <0x100 &its 0x11 0x7>; > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > Either way, with this change PCIe on pcie0 is not working anymore, > regardless of msi-map-mask. Yes, it should have msi-map-mask. During my test, I have not enable enetc so I have not found this problem. > > Without msi-map-mask: > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > r8169 0000:03:00.0: error -EINVAL: enable failure > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > With msi-map-mask: > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > r8169 0000:03:00.0: error -EIO: PCI read failed > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 Can you try remove iommu-map and keep msi-map? then remove msi-map and keep iommu-map to check which one cause this problem. > > Without msi-map/iommu-map: > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > r8169 0000:03:00.0: enabling bus mastering > > r8169 0000:03:00.0 enp3s0: Link is Down > > pcie1 works as expected. But this is only a single PCIe device, rather than > having a PCIe bridge. > Any idea what's wrong here? Can you help dump more information at for PCIe bridge case: imx_pcie_add_lut(), need rid and sid information. drivers/pci/controller/dwc/pci-imx6.c > > Best regards, > Alexander > > > + iommu-map = <0x000 &smmu 0x10 0x1>, > > + <0x100 &smmu 0x11 0x7>; > > + iommu-map-mask = <0x1ff>; > > fsl,max-link-speed = <3>; > > status = "disabled"; > > }; > > @@ -1640,6 +1646,14 @@ pcie1: pcie@4c380000 { > > assigned-clock-parents = <0>, <0>, > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > + /* pcie1's Devid(BIT[7:6]) is 0x10, stream id(BIT[5:0]) is 0x18~0x1f */ > > + msi-map = <0x0 &its 0x98 0x1>, > > + <0x100 &its 0x99 0x7>; > > + msi-map-mask = <0x1ff>; > > + /* smmu have not Devid(BIT[7:6]) */ > > + iommu-map = <0x000 &smmu 0x18 0x1>, > > + <0x100 &smmu 0x19 0x7>; > > + iommu-map-mask = <0x1ff>; > > fsl,max-link-speed = <3>; > > status = "disabled"; > > }; > > > > > -- > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany > Amtsgericht München, HRB 105018 > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > http://www.tq-group.com/ > >
On Wed, Feb 26, 2025 at 11:31:26AM -0500, Frank Li wrote: > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > Hi Frank, > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > its. > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > --- > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > 1 file changed, 14 insertions(+) > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > assigned-clock-parents = <0>, <0>, > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > + msi-map = <0x0 &its 0x10 0x1>, > > > + <0x100 &its 0x11 0x7>; > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > Either way, with this change PCIe on pcie0 is not working anymore, > > regardless of msi-map-mask. > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > so I have not found this problem. I check dts again. There are linux,pci-domain = <0>; It should work without msi-map-mask. I am not sure why 0000:03:00.0 ^^ bus number is 03 at your system. > > > > > Without msi-map-mask: > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > With msi-map-mask: > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > keep iommu-map to check which one cause this problem. > > > > > Without msi-map/iommu-map: > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > r8169 0000:03:00.0: enabling bus mastering > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > having a PCIe bridge. > > Any idea what's wrong here? > > Can you help dump more information at for PCIe bridge case: > > imx_pcie_add_lut(), need rid and sid information. > drivers/pci/controller/dwc/pci-imx6.c > > > > > Best regards, > > Alexander > > > > > + iommu-map = <0x000 &smmu 0x10 0x1>, > > > + <0x100 &smmu 0x11 0x7>; > > > + iommu-map-mask = <0x1ff>; > > > fsl,max-link-speed = <3>; > > > status = "disabled"; > > > }; > > > @@ -1640,6 +1646,14 @@ pcie1: pcie@4c380000 { > > > assigned-clock-parents = <0>, <0>, > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > + /* pcie1's Devid(BIT[7:6]) is 0x10, stream id(BIT[5:0]) is 0x18~0x1f */ > > > + msi-map = <0x0 &its 0x98 0x1>, > > > + <0x100 &its 0x99 0x7>; > > > + msi-map-mask = <0x1ff>; > > > + /* smmu have not Devid(BIT[7:6]) */ > > > + iommu-map = <0x000 &smmu 0x18 0x1>, > > > + <0x100 &smmu 0x19 0x7>; > > > + iommu-map-mask = <0x1ff>; > > > fsl,max-link-speed = <3>; > > > status = "disabled"; > > > }; > > > > > > > > > -- > > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany > > Amtsgericht München, HRB 105018 > > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > > http://www.tq-group.com/ > > > >
Hi Frank, Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > Hi Frank, > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > its. > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > --- > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > 1 file changed, 14 insertions(+) > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > assigned-clock-parents = <0>, <0>, > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > + msi-map = <0x0 &its 0x10 0x1>, > > > + <0x100 &its 0x11 0x7>; > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > Either way, with this change PCIe on pcie0 is not working anymore, > > regardless of msi-map-mask. > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > so I have not found this problem. Just to be clear: This is not about enetc. This works fine here. > > Without msi-map-mask: > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > With msi-map-mask: > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > keep iommu-map to check which one cause this problem. With only msi-map removed, but smmu enabled: > arm-smmu-v3 490d0000.iommu: event 0x10 received: > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > enp3s0: Link is Down With only iommu-map removed, both smmu enabled or disabled: > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > r8169 0000:03:00.0: enabling device (0000 -> 0003) > r8169 0000:03:00.0: enabling Mem-Wr-Inval > r8169 0000:03:00.0: error -EIO: PCI read failed > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 Only if smmu is disabled and msi-map is removed the driver probes successfully: > r8169 0000:03:00.0: enabling device (0000 -> 0003) > r8169 0000:03:00.0: enabling Mem-Wr-Inval > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > r8169 0000:03:00.0 enp3s0: renamed from eth0 > r8169 0000:03:00.0: enabling bus mastering > r8169 0000:03:00.0 enp3s0: Link is Down > > > > Without msi-map/iommu-map: > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > r8169 0000:03:00.0: enabling bus mastering > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > having a PCIe bridge. > > Any idea what's wrong here? > > Can you help dump more information at for PCIe bridge case: > > imx_pcie_add_lut(), need rid and sid information. > drivers/pci/controller/dwc/pci-imx6.c Just to be clear, without msi-map and iommu-map I get: > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 This function get called once for each device. Maybe the whole PCIe bus might help here, so I've put lspci output here as well. $ lspci 0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000 0000:01:00.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) 0000:02:01.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) 0000:02:02.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) 0000:03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c) 0000:04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c) 0001:00:00.0 PCI bridge: Philips Semiconductors Device 0000 0001:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 0002:00:00.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) 0002:00:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03) 0002:00:08.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) 0002:00:10.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) 0002:00:18.0 System peripheral: Philips Semiconductors Device ee02 (rev 04) 0003:01:00.0 System peripheral: Philips Semiconductors Device ee00 (rev 04) 0003:01:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03) $ lspci -t -[0000:00]---00.0-[01-ff]----00.0-[02-04]--+-01.0-[03]----00.0 \-02.0-[04]----00.0 -[0001:00]---00.0-[01-ff]----00.0 -[0002:00]-+-00.0 +-01.0 +-08.0 +-10.0 \-18.0 -[0003:01]-+-00.0 \-01.0 Thanks and best regards, Alexander
On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote: > Hi Frank, > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > > Hi Frank, > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > > its. > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > > --- > > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > > 1 file changed, 14 insertions(+) > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > > assigned-clock-parents = <0>, <0>, > > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > > + msi-map = <0x0 &its 0x10 0x1>, > > > > + <0x100 &its 0x11 0x7>; > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > > Either way, with this change PCIe on pcie0 is not working anymore, > > > regardless of msi-map-mask. > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > > so I have not found this problem. > > Just to be clear: This is not about enetc. This works fine here. > > > > Without msi-map-mask: > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > > > With msi-map-mask: > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > > keep iommu-map to check which one cause this problem. > > With only msi-map removed, but smmu enabled: > > arm-smmu-v3 490d0000.iommu: event 0x10 received: > > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > > enp3s0: Link is Down > > With only iommu-map removed, both smmu enabled or disabled: > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > r8169 0000:03:00.0: error -EIO: PCI read failed > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > Only if smmu is disabled and msi-map is removed the driver probes > successfully: > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > r8169 0000:03:00.0: enabling bus mastering > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > Without msi-map/iommu-map: > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > r8169 0000:03:00.0: enabling bus mastering > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > > having a PCIe bridge. > > > Any idea what's wrong here? > > > > Can you help dump more information at for PCIe bridge case: > > > > imx_pcie_add_lut(), need rid and sid information. > > drivers/pci/controller/dwc/pci-imx6.c > > Just to be clear, without msi-map and iommu-map I get: > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 Can you help dump register value PE0_LUT_CREQID offset 0x101 for your smmu-map or msi-map enable case 2nd test. change IMX95_PE0_LUT_MASK to 0x1ff Frank > > This function get called once for each device. > Maybe the whole PCIe bus might help here, so I've put lspci output here as well. > > $ lspci > 0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000 > 0000:01:00.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) > 0000:02:01.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) > 0000:02:02.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03) > 0000:03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c) > 0000:04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c) > 0001:00:00.0 PCI bridge: Philips Semiconductors Device 0000 > 0001:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection > 0002:00:00.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) > 0002:00:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03) > 0002:00:08.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) > 0002:00:10.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04) > 0002:00:18.0 System peripheral: Philips Semiconductors Device ee02 (rev 04) > 0003:01:00.0 System peripheral: Philips Semiconductors Device ee00 (rev 04) > 0003:01:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03) > > $ lspci -t > -[0000:00]---00.0-[01-ff]----00.0-[02-04]--+-01.0-[03]----00.0 > \-02.0-[04]----00.0 > -[0001:00]---00.0-[01-ff]----00.0 > -[0002:00]-+-00.0 > +-01.0 > +-08.0 > +-10.0 > \-18.0 > -[0003:01]-+-00.0 > \-01.0 > > Thanks and best regards, > Alexander > -- > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany > Amtsgericht München, HRB 105018 > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > http://www.tq-group.com/ > >
Hi Frank, Am Donnerstag, 27. Februar 2025, 17:39:47 CET schrieb Frank Li: > On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote: > > Hi Frank, > > > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > > > Hi Frank, > > > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > > > its. > > > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > > > --- > > > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > > > 1 file changed, 14 insertions(+) > > > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > > > assigned-clock-parents = <0>, <0>, > > > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > > > + msi-map = <0x0 &its 0x10 0x1>, > > > > > + <0x100 &its 0x11 0x7>; > > > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > > > Either way, with this change PCIe on pcie0 is not working anymore, > > > > regardless of msi-map-mask. > > > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > > > so I have not found this problem. > > > > Just to be clear: This is not about enetc. This works fine here. > > > > > > Without msi-map-mask: > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > > > > > With msi-map-mask: > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > > > keep iommu-map to check which one cause this problem. > > > > With only msi-map removed, but smmu enabled: > > > arm-smmu-v3 490d0000.iommu: event 0x10 received: > > > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > > > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > > > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > > > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > > > enp3s0: Link is Down > > > > With only iommu-map removed, both smmu enabled or disabled: > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > Only if smmu is disabled and msi-map is removed the driver probes > > successfully: > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > r8169 0000:03:00.0: enabling bus mastering > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > Without msi-map/iommu-map: > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > > > having a PCIe bridge. > > > > Any idea what's wrong here? > > > > > > Can you help dump more information at for PCIe bridge case: > > > > > > imx_pcie_add_lut(), need rid and sid information. > > > drivers/pci/controller/dwc/pci-imx6.c > > > > Just to be clear, without msi-map and iommu-map I get: > > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 > > Can you help dump register value PE0_LUT_CREQID offset 0x101 for your > smmu-map or msi-map enable case I am assuming you meant offset 0x101c, as stated in the RM. I added a dump directly before printing "PCI read failed" in r8169_main.c. Unfortunately this only returns 0 for both PCIe devices, so I'm wondering if this is correct. > 2nd test. > change IMX95_PE0_LUT_MASK to 0x1ff Unfortunately I do not notice any effect/difference. Best regards Alexander
On Fri, Feb 28, 2025 at 10:08:58AM +0100, Alexander Stein wrote: > Hi Frank, > > Am Donnerstag, 27. Februar 2025, 17:39:47 CET schrieb Frank Li: > > On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote: > > > Hi Frank, > > > > > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > > > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > > > > Hi Frank, > > > > > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > > > > its. > > > > > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > > > > --- > > > > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > > > > 1 file changed, 14 insertions(+) > > > > > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > > > > assigned-clock-parents = <0>, <0>, > > > > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > > > > + msi-map = <0x0 &its 0x10 0x1>, > > > > > > + <0x100 &its 0x11 0x7>; > > > > > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > > > > Either way, with this change PCIe on pcie0 is not working anymore, > > > > > regardless of msi-map-mask. > > > > > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > > > > so I have not found this problem. > > > > > > Just to be clear: This is not about enetc. This works fine here. > > > > > > > > Without msi-map-mask: > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > > > > > > > With msi-map-mask: > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > > > > keep iommu-map to check which one cause this problem. > > > > > > With only msi-map removed, but smmu enabled: > > > > arm-smmu-v3 490d0000.iommu: event 0x10 received: > > > > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > > > > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > > > > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > > > > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > > > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > > > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > > > > enp3s0: Link is Down > > > > > > With only iommu-map removed, both smmu enabled or disabled: > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > Only if smmu is disabled and msi-map is removed the driver probes > > > successfully: > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > r8169 0000:03:00.0: enabling bus mastering > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > > > > Without msi-map/iommu-map: > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > > > > having a PCIe bridge. > > > > > Any idea what's wrong here? > > > > > > > > Can you help dump more information at for PCIe bridge case: > > > > > > > > imx_pcie_add_lut(), need rid and sid information. > > > > drivers/pci/controller/dwc/pci-imx6.c > > > > > > Just to be clear, without msi-map and iommu-map I get: > > > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > > > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 > > > > Can you help dump register value PE0_LUT_CREQID offset 0x101 for your > > smmu-map or msi-map enable case > > I am assuming you meant offset 0x101c, as stated in the RM. > I added a dump directly before printing "PCI read failed" in r8169_main.c. Can you point me the code about where "error -EIO: PCI read failed"? I tested nvme devices worked at both PCIE0 and PCIE1. Which PCI switch do you use? Frank > Unfortunately this only returns 0 for both PCIe devices, so I'm wondering > if this is correct. > > > 2nd test. > > change IMX95_PE0_LUT_MASK to 0x1ff > > Unfortunately I do not notice any effect/difference. > > Best regards > Alexander > -- > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany > Amtsgericht München, HRB 105018 > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > http://www.tq-group.com/ > >
Hi Frank, Am Freitag, 28. Februar 2025, 16:32:50 CET schrieb Frank Li: > On Fri, Feb 28, 2025 at 10:08:58AM +0100, Alexander Stein wrote: > > Hi Frank, > > > > Am Donnerstag, 27. Februar 2025, 17:39:47 CET schrieb Frank Li: > > > On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote: > > > > Hi Frank, > > > > > > > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > > > > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > > > > > Hi Frank, > > > > > > > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > > > > > its. > > > > > > > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > > > > > --- > > > > > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > > > > > 1 file changed, 14 insertions(+) > > > > > > > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > > > > > assigned-clock-parents = <0>, <0>, > > > > > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > > > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > > > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > > > > > + msi-map = <0x0 &its 0x10 0x1>, > > > > > > > + <0x100 &its 0x11 0x7>; > > > > > > > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > > > > > Either way, with this change PCIe on pcie0 is not working anymore, > > > > > > regardless of msi-map-mask. > > > > > > > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > > > > > so I have not found this problem. > > > > > > > > Just to be clear: This is not about enetc. This works fine here. > > > > > > > > > > Without msi-map-mask: > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > > > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > > > > > > > > > With msi-map-mask: > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > > > > > keep iommu-map to check which one cause this problem. > > > > > > > > With only msi-map removed, but smmu enabled: > > > > > arm-smmu-v3 490d0000.iommu: event 0x10 received: > > > > > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > > > > > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > > > > > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > > > > > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > > > > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > > > > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > > > > > enp3s0: Link is Down > > > > > > > > With only iommu-map removed, both smmu enabled or disabled: > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > > > Only if smmu is disabled and msi-map is removed the driver probes > > > > successfully: > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > > > > > > > Without msi-map/iommu-map: > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > > > > > having a PCIe bridge. > > > > > > Any idea what's wrong here? > > > > > > > > > > Can you help dump more information at for PCIe bridge case: > > > > > > > > > > imx_pcie_add_lut(), need rid and sid information. > > > > > drivers/pci/controller/dwc/pci-imx6.c > > > > > > > > Just to be clear, without msi-map and iommu-map I get: > > > > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > > > > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 > > > > > > Can you help dump register value PE0_LUT_CREQID offset 0x101 for your > > > smmu-map or msi-map enable case > > > > I am assuming you meant offset 0x101c, as stated in the RM. > > I added a dump directly before printing "PCI read failed" in r8169_main.c. > > Can you point me the code about where "error -EIO: PCI read failed"? The error messages comes from [1]. As far as I can tell, this is the first read after pcim_iomap_table(). > I tested nvme devices worked at both PCIE0 and PCIE1. > > Which PCI switch do you use? This is a COTS MiniPCIe ethernet card [2]. If I'm reading correctly the bridge seems to be a P17C9X20 Thanks and best regards Alexander [1] https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/net/ethernet/realtek/r8169_main.c?h=next-20250228#n5454 [2] https://www.delock.com/produkt/95237/merkmale.html?f=s
On Fri, Feb 28, 2025 at 05:01:15PM +0100, Alexander Stein wrote: > Hi Frank, > > Am Freitag, 28. Februar 2025, 16:32:50 CET schrieb Frank Li: > > On Fri, Feb 28, 2025 at 10:08:58AM +0100, Alexander Stein wrote: > > > Hi Frank, > > > > > > Am Donnerstag, 27. Februar 2025, 17:39:47 CET schrieb Frank Li: > > > > On Thu, Feb 27, 2025 at 08:54:13AM +0100, Alexander Stein wrote: > > > > > Hi Frank, > > > > > > > > > > Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li: > > > > > > On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote: > > > > > > > Hi Frank, > > > > > > > > > > > > > > Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li: > > > > > > > > Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and > > > > > > > > its. > > > > > > > > > > > > > > > > Signed-off-by: Frank Li <Frank.Li@nxp.com> > > > > > > > > --- > > > > > > > > arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ > > > > > > > > 1 file changed, 14 insertions(+) > > > > > > > > > > > > > > > > diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > > index 6b8470cb3461a..2cebeda43a52d 100644 > > > > > > > > --- a/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > > +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi > > > > > > > > @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { > > > > > > > > assigned-clock-parents = <0>, <0>, > > > > > > > > <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; > > > > > > > > power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; > > > > > > > > + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ > > > > > > > > + msi-map = <0x0 &its 0x10 0x1>, > > > > > > > > + <0x100 &its 0x11 0x7>; > > > > > > > > > > > > > > Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1. > > > > > > > Either way, with this change PCIe on pcie0 is not working anymore, > > > > > > > regardless of msi-map-mask. > > > > > > > > > > > > Yes, it should have msi-map-mask. During my test, I have not enable enetc > > > > > > so I have not found this problem. > > > > > > > > > > Just to be clear: This is not about enetc. This works fine here. > > > > > > > > > > > > Without msi-map-mask: > > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > > OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null) > > > > > > > > r8169 0000:03:00.0: error -EINVAL: enable failure > > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -22 > > > > > > > > > > > > > > With msi-map-mask: > > > > > > > > OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > > > > > > > Can you try remove iommu-map and keep msi-map? then remove msi-map and > > > > > > keep iommu-map to check which one cause this problem. > > > > > > > > > > With only msi-map removed, but smmu enabled: > > > > > > arm-smmu-v3 490d0000.iommu: event 0x10 received: > > > > > > arm-smmu-v3 490d0000.iommu: 0x0000001100000010 > > > > > > arm-smmu-v3 490d0000.iommu: 0x0000020a00000000 > > > > > > arm-smmu-v3 490d0000.iommu: 0x000000009b0cc000 > > > > > > arm-smmu-v3 490d0000.iommu: 0x0000000000000000 > > > > > > arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0 > > > > > > arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0 > > > > > > enp3s0: Link is Down > > > > > > > > > > With only iommu-map removed, both smmu enabled or disabled: > > > > > > OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011 > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > r8169 0000:03:00.0: error -EIO: PCI read failed > > > > > > r8169 0000:03:00.0: probe with driver r8169 failed with error -5 > > > > > > > > > > Only if smmu is disabled and msi-map is removed the driver probes > > > > > successfully: > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160 > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > > > > > > > > > > Without msi-map/iommu-map: > > > > > > > > r8169 0000:03:00.0: enabling device (0000 -> 0003) > > > > > > > > r8169 0000:03:00.0: enabling Mem-Wr-Inval > > > > > > > > r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166 > > > > > > > > r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko] > > > > > > > > r8169 0000:03:00.0 enp3s0: renamed from eth0 > > > > > > > > r8169 0000:03:00.0: enabling bus mastering > > > > > > > > r8169 0000:03:00.0 enp3s0: Link is Down > > > > > > > > > > > > > > pcie1 works as expected. But this is only a single PCIe device, rather than > > > > > > > having a PCIe bridge. > > > > > > > Any idea what's wrong here? > > > > > > > > > > > > Can you help dump more information at for PCIe bridge case: > > > > > > > > > > > > imx_pcie_add_lut(), need rid and sid information. > > > > > > drivers/pci/controller/dwc/pci-imx6.c > > > > > > > > > > Just to be clear, without msi-map and iommu-map I get: > > > > > > imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18 > > > > > > imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19 > > > > > > > > Can you help dump register value PE0_LUT_CREQID offset 0x101 for your > > > > smmu-map or msi-map enable case > > > > > > I am assuming you meant offset 0x101c, as stated in the RM. > > > I added a dump directly before printing "PCI read failed" in r8169_main.c. > > > > Can you point me the code about where "error -EIO: PCI read failed"? > > The error messages comes from [1]. As far as I can tell, this is the first > read after pcim_iomap_table(). Strange, it is CPU read MMIO, should not go through iommu at all. > > > I tested nvme devices worked at both PCIE0 and PCIE1. > > > > Which PCI switch do you use? > > This is a COTS MiniPCIe ethernet card [2]. If I'm reading correctly > the bridge seems to be a P17C9X20 I order the same chipset one from amazon. Let me debug it after get it. Frank > > Thanks and best regards > Alexander > > [1] https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/net/ethernet/realtek/r8169_main.c?h=next-20250228#n5454 > [2] https://www.delock.com/produkt/95237/merkmale.html?f=s > -- > TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany > Amtsgericht München, HRB 105018 > Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider > http://www.tq-group.com/ > >
diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi index 6b8470cb3461a..2cebeda43a52d 100644 --- a/arch/arm64/boot/dts/freescale/imx95.dtsi +++ b/arch/arm64/boot/dts/freescale/imx95.dtsi @@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 { assigned-clock-parents = <0>, <0>, <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; + /* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */ + msi-map = <0x0 &its 0x10 0x1>, + <0x100 &its 0x11 0x7>; + iommu-map = <0x000 &smmu 0x10 0x1>, + <0x100 &smmu 0x11 0x7>; + iommu-map-mask = <0x1ff>; fsl,max-link-speed = <3>; status = "disabled"; }; @@ -1640,6 +1646,14 @@ pcie1: pcie@4c380000 { assigned-clock-parents = <0>, <0>, <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>; power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>; + /* pcie1's Devid(BIT[7:6]) is 0x10, stream id(BIT[5:0]) is 0x18~0x1f */ + msi-map = <0x0 &its 0x98 0x1>, + <0x100 &its 0x99 0x7>; + msi-map-mask = <0x1ff>; + /* smmu have not Devid(BIT[7:6]) */ + iommu-map = <0x000 &smmu 0x18 0x1>, + <0x100 &smmu 0x19 0x7>; + iommu-map-mask = <0x1ff>; fsl,max-link-speed = <3>; status = "disabled"; };
Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and its. Signed-off-by: Frank Li <Frank.Li@nxp.com> --- arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++ 1 file changed, 14 insertions(+)