diff mbox series

arm: Enlarge IO_SPACE_LIMIT needed for some SoC

Message ID 20210511021656.17719-1-ansuelsmth@gmail.com (mailing list archive)
State New, archived
Headers show
Series arm: Enlarge IO_SPACE_LIMIT needed for some SoC | expand

Commit Message

Ansuel Smith May 11, 2021, 2:16 a.m. UTC
Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
Each pci line I/O space require 0x100000 and the current limit in the
specific case of ipq8064 cause the malfunction of the second and the
third line that are commonly used for wifi cards.
Current IO space is set to support only 0x100000 space while for ipq806x
if all 3 pcie lines are connected, it's required 0x300000 of space.
Update the IO_SPACE_LIMIT to permit this larger space and update the
documentation for the Mapping PCI I/O space memory end to the new value
of ff0fffff.

Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
---
 Documentation/arm/memory.rst | 2 +-
 arch/arm/include/asm/io.h    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Comments

Matthew Wilcox May 11, 2021, 2:24 a.m. UTC | #1
On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.

Do you really?  I mean, yes, theoretically, I understand it, the
hardware supports 64kB of I/O port space per root port.  But I/O
port space is rather deprecated these days.  My laptop has precisely
two devices with I/O ports, one with 64 bytes and the other with 32
bytes.  Would you really suffer by allocating 16kB of I/O port
space to each root port?
Ansuel Smith May 11, 2021, 2:32 a.m. UTC | #2
On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> 
> Do you really?  I mean, yes, theoretically, I understand it, the
> hardware supports 64kB of I/O port space per root port.  But I/O
> port space is rather deprecated these days.  My laptop has precisely
> two devices with I/O ports, one with 64 bytes and the other with 32
> bytes.  Would you really suffer by allocating 16kB of I/O port
> space to each root port?

We were talking about this in the other wrong patch. I also think this
much space looks wrong. The current ipq806x dts have this space so it's
actually broken from a long time. The only reason pci worked before was
because the pci driver didn't actually check if the settings were right.
New kernel introduced more checks and this problem showed up. (to be
more precise, the pci port are commonly used by the ath10k wifi and the
second ath10k wifi fails to init because of this problem)
If you can give me any hint on how to check if the space can be reduced
I would be very happy to investigate it.
In the driver I notice that the max buffer is set to 2k, could be this a
hint?
Ard Biesheuvel May 11, 2021, 4:26 a.m. UTC | #3
On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> >
> > Do you really?  I mean, yes, theoretically, I understand it, the
> > hardware supports 64kB of I/O port space per root port.  But I/O
> > port space is rather deprecated these days.  My laptop has precisely
> > two devices with I/O ports, one with 64 bytes and the other with 32
> > bytes.  Would you really suffer by allocating 16kB of I/O port
> > space to each root port?
>
> We were talking about this in the other wrong patch. I also think this
> much space looks wrong. The current ipq806x dts have this space so it's
> actually broken from a long time. The only reason pci worked before was
> because the pci driver didn't actually check if the settings were right.
> New kernel introduced more checks and this problem showed up. (to be
> more precise, the pci port are commonly used by the ath10k wifi and the
> second ath10k wifi fails to init because of this problem)
> If you can give me any hint on how to check if the space can be reduced
> I would be very happy to investigate it.
> In the driver I notice that the max buffer is set to 2k, could be this a
> hint?
>

Could you share the output of lspci -vv from such a system?

I agree with Matthew that fiddling with the size of the I/O space
range probably papers over another problem, and with the odd
exception, no PCIe card used on ARM systems actually uses their I/O
BARs, even when they have them. (I used to carry a PCIe serial port
card to UEFI plugfests because that was the only thing that would stop
working if a system configured its I/O resource window incorrectly)
Ansuel Smith May 11, 2021, 12:15 p.m. UTC | #4
On Tue, May 11, 2021 at 06:26:28AM +0200, Ard Biesheuvel wrote:
> On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> >
> > On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> > >
> > > Do you really?  I mean, yes, theoretically, I understand it, the
> > > hardware supports 64kB of I/O port space per root port.  But I/O
> > > port space is rather deprecated these days.  My laptop has precisely
> > > two devices with I/O ports, one with 64 bytes and the other with 32
> > > bytes.  Would you really suffer by allocating 16kB of I/O port
> > > space to each root port?
> >
> > We were talking about this in the other wrong patch. I also think this
> > much space looks wrong. The current ipq806x dts have this space so it's
> > actually broken from a long time. The only reason pci worked before was
> > because the pci driver didn't actually check if the settings were right.
> > New kernel introduced more checks and this problem showed up. (to be
> > more precise, the pci port are commonly used by the ath10k wifi and the
> > second ath10k wifi fails to init because of this problem)
> > If you can give me any hint on how to check if the space can be reduced
> > I would be very happy to investigate it.
> > In the driver I notice that the max buffer is set to 2k, could be this a
> > hint?
> >
> 
> Could you share the output of lspci -vv from such a system?
> 
> I agree with Matthew that fiddling with the size of the I/O space
> range probably papers over another problem, and with the odd
> exception, no PCIe card used on ARM systems actually uses their I/O
> BARs, even when they have them. (I used to carry a PCIe serial port
> card to UEFI plugfests because that was the only thing that would stop
> working if a system configured its I/O resource window incorrectly)

Here is the output of lspci -vv

0000:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
        Device tree node: /sys/firmware/devicetree/base/soc/pci@1b500000/bridge@0,0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 44
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: 08000000-081fffff [size=2M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 000000004361bc88  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
        Subsystem: Qualcomm Atheros Device cafe
        Device tree node: /sys/firmware/devicetree/base/soc/pci@1b500000/bridge@0,0/wifi@1,0
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 53
        Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
                Address: 000000004361bc88  Data: 0001
                Masking: fffffffe  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [160 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                L1SubCtl2:
        Kernel driver in use: ath10k_pci

0001:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
        Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 46
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: 2e000000-2e1fffff [size=2M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000435e8688  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCap: CRSVisible-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
                RootCmd: CERptEn+ NFERptEn+ FERptEn+
                RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                         FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
                ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
        Kernel driver in use: pcieport

0001:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
        Subsystem: Qualcomm Atheros Device cafe
        Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0/wifi@1,0
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 54
        Region 0: Memory at 2e000000 (64-bit, non-prefetchable) [size=2M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
                Address: 00000000435e8688  Data: 0001
                Masking: fffffffe  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [160 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                L1SubCtl2:
        Kernel driver in use: ath10k_pci
Ard Biesheuvel May 11, 2021, 12:30 p.m. UTC | #5
On Tue, 11 May 2021 at 14:15, Ansuel Smith <ansuelsmth@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 06:26:28AM +0200, Ard Biesheuvel wrote:
> > On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > >
> > > On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > > > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> > > >
> > > > Do you really?  I mean, yes, theoretically, I understand it, the
> > > > hardware supports 64kB of I/O port space per root port.  But I/O
> > > > port space is rather deprecated these days.  My laptop has precisely
> > > > two devices with I/O ports, one with 64 bytes and the other with 32
> > > > bytes.  Would you really suffer by allocating 16kB of I/O port
> > > > space to each root port?
> > >
> > > We were talking about this in the other wrong patch. I also think this
> > > much space looks wrong. The current ipq806x dts have this space so it's
> > > actually broken from a long time. The only reason pci worked before was
> > > because the pci driver didn't actually check if the settings were right.
> > > New kernel introduced more checks and this problem showed up. (to be
> > > more precise, the pci port are commonly used by the ath10k wifi and the
> > > second ath10k wifi fails to init because of this problem)
> > > If you can give me any hint on how to check if the space can be reduced
> > > I would be very happy to investigate it.
> > > In the driver I notice that the max buffer is set to 2k, could be this a
> > > hint?
> > >
> >
> > Could you share the output of lspci -vv from such a system?
> >
> > I agree with Matthew that fiddling with the size of the I/O space
> > range probably papers over another problem, and with the odd
> > exception, no PCIe card used on ARM systems actually uses their I/O
> > BARs, even when they have them. (I used to carry a PCIe serial port
> > card to UEFI plugfests because that was the only thing that would stop
> > working if a system configured its I/O resource window incorrectly)
>
> Here is the output of lspci -vv
>
> 0000:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
>         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
>         I/O behind bridge: [disabled]
>         Memory behind bridge: 08000000-081fffff [size=2M]
>         Prefetchable memory behind bridge: [disabled]

So this a MMIO window to the endpoint

...

>
> 0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
>         Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]

... and the endpoint has a single *MMIO* BAR of size 2 MiB.

This has *nothing* to do with port I/O, which is what you are
modifying with your patch.

Did you check that the problem exists without the patch, and that the
patch makes it go away?



>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
>                 Address: 000000004361bc88  Data: 0001
>                 Masking: fffffffe  Pending: 00000000
>         Capabilities: [70] Express (v2) Endpoint, MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 5GT/s (ok), Width x1 (ok)
>                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
>                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
>                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
>                          FRS- TPHComp- ExtTPHComp-
>                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
>                          AtomicOpsCtl: ReqEn-
>                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
>                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
>                          Retimer- 2Retimers- CrosslinkRes: unsupported
>         Capabilities: [100 v2] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
>                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
>                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>                 HeaderLog: 00000000 00000000 00000000 00000000
>         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
>         Capabilities: [158 v1] Latency Tolerance Reporting
>                 Max snoop latency: 0ns
>                 Max no snoop latency: 0ns
>         Capabilities: [160 v1] L1 PM Substates
>                 L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
>                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>                 L1SubCtl2:
>         Kernel driver in use: ath10k_pci
>
> 0001:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
>         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 46
>         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
>         I/O behind bridge: [disabled]
>         Memory behind bridge: 2e000000-2e1fffff [size=2M]
>         Prefetchable memory behind bridge: [disabled]
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>         BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 00000000435e8688  Data: 0000
>         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>                         ExtTag- RBE+
>                 DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
>                         ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
>                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
>                         TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
>                 RootCap: CRSVisible-
>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
>                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
>                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
>                          FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
>                          AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
>                          AtomicOpsCtl: ReqEn- EgressBlck-
>                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
>                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
>                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
>                          Retimer- 2Retimers- CrosslinkRes: unsupported
>         Capabilities: [100 v1] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
>                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
>                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>                 HeaderLog: 00000000 00000000 00000000 00000000
>                 RootCmd: CERptEn+ NFERptEn+ FERptEn+
>                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
>                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
>                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
>         Kernel driver in use: pcieport
>
> 0001:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
>         Subsystem: Qualcomm Atheros Device cafe
>         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0/wifi@1,0
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 54
>         Region 0: Memory at 2e000000 (64-bit, non-prefetchable) [size=2M]
>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
>                 Address: 00000000435e8688  Data: 0001
>                 Masking: fffffffe  Pending: 00000000
>         Capabilities: [70] Express (v2) Endpoint, MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
>                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
>                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
>                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
>                          FRS- TPHComp- ExtTPHComp-
>                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
>                          AtomicOpsCtl: ReqEn-
>                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
>                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
>                          Retimer- 2Retimers- CrosslinkRes: unsupported
>         Capabilities: [100 v2] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
>                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
>                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
>                 HeaderLog: 00000000 00000000 00000000 00000000
>         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
>         Capabilities: [158 v1] Latency Tolerance Reporting
>                 Max snoop latency: 0ns
>                 Max no snoop latency: 0ns
>         Capabilities: [160 v1] L1 PM Substates
>                 L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
>                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>                 L1SubCtl2:
>         Kernel driver in use: ath10k_pci
Ansuel Smith May 11, 2021, 12:37 p.m. UTC | #6
On Tue, May 11, 2021 at 02:30:36PM +0200, Ard Biesheuvel wrote:
> On Tue, 11 May 2021 at 14:15, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> >
> > On Tue, May 11, 2021 at 06:26:28AM +0200, Ard Biesheuvel wrote:
> > > On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > > >
> > > > On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > > > > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > > > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> > > > >
> > > > > Do you really?  I mean, yes, theoretically, I understand it, the
> > > > > hardware supports 64kB of I/O port space per root port.  But I/O
> > > > > port space is rather deprecated these days.  My laptop has precisely
> > > > > two devices with I/O ports, one with 64 bytes and the other with 32
> > > > > bytes.  Would you really suffer by allocating 16kB of I/O port
> > > > > space to each root port?
> > > >
> > > > We were talking about this in the other wrong patch. I also think this
> > > > much space looks wrong. The current ipq806x dts have this space so it's
> > > > actually broken from a long time. The only reason pci worked before was
> > > > because the pci driver didn't actually check if the settings were right.
> > > > New kernel introduced more checks and this problem showed up. (to be
> > > > more precise, the pci port are commonly used by the ath10k wifi and the
> > > > second ath10k wifi fails to init because of this problem)
> > > > If you can give me any hint on how to check if the space can be reduced
> > > > I would be very happy to investigate it.
> > > > In the driver I notice that the max buffer is set to 2k, could be this a
> > > > hint?
> > > >
> > >
> > > Could you share the output of lspci -vv from such a system?
> > >
> > > I agree with Matthew that fiddling with the size of the I/O space
> > > range probably papers over another problem, and with the odd
> > > exception, no PCIe card used on ARM systems actually uses their I/O
> > > BARs, even when they have them. (I used to carry a PCIe serial port
> > > card to UEFI plugfests because that was the only thing that would stop
> > > working if a system configured its I/O resource window incorrectly)
> >
> > Here is the output of lspci -vv
> >
> > 0000:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
> >         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> >         I/O behind bridge: [disabled]
> >         Memory behind bridge: 08000000-081fffff [size=2M]
> >         Prefetchable memory behind bridge: [disabled]
> 
> So this a MMIO window to the endpoint
> 
> ...
> 
> >
> > 0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
> >         Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]
> 
> ... and the endpoint has a single *MMIO* BAR of size 2 MiB.
> 
> This has *nothing* to do with port I/O, which is what you are
> modifying with your patch.
> 
> Did you check that the problem exists without the patch, and that the
> patch makes it go away?
> 
>

Yes without the change to IO_SPACE_LIMIT, the ath10k driver fails to
init as it can't access the reg. Only the first pci wifi works but the
second one fails to init. By increasing the limit all comes back to
normal. What I really can't understand is if the big IO space set in the
ipq8064 dtsi was wrong from the start and the ath10k fails to init just
because is missconfigured. Any idea how to find the appropriate max IO space
for the pci? 

> 
> >         Capabilities: [40] Power Management version 3
> >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
> >                 Address: 000000004361bc88  Data: 0001
> >                 Masking: fffffffe  Pending: 00000000
> >         Capabilities: [70] Express (v2) Endpoint, MSI 00
> >                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
> >                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
> >                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                         MaxPayload 128 bytes, MaxReadReq 128 bytes
> >                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
> >                 LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
> >                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> >                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 5GT/s (ok), Width x1 (ok)
> >                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
> >                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
> >                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
> >                          FRS- TPHComp- ExtTPHComp-
> >                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
> >                          AtomicOpsCtl: ReqEn-
> >                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
> >                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> >                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> >                          Compliance De-emphasis: -6dB
> >                 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
> >                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
> >                          Retimer- 2Retimers- CrosslinkRes: unsupported
> >         Capabilities: [100 v2] Advanced Error Reporting
> >                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
> >                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> >                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> >                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >                 HeaderLog: 00000000 00000000 00000000 00000000
> >         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
> >         Capabilities: [158 v1] Latency Tolerance Reporting
> >                 Max snoop latency: 0ns
> >                 Max no snoop latency: 0ns
> >         Capabilities: [160 v1] L1 PM Substates
> >                 L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
> >                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> >                 L1SubCtl2:
> >         Kernel driver in use: ath10k_pci
> >
> > 0001:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
> >         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0, Cache Line Size: 64 bytes
> >         Interrupt: pin A routed to IRQ 46
> >         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> >         I/O behind bridge: [disabled]
> >         Memory behind bridge: 2e000000-2e1fffff [size=2M]
> >         Prefetchable memory behind bridge: [disabled]
> >         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> >         BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
> >                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >         Capabilities: [40] Power Management version 3
> >                 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> >                 Address: 00000000435e8688  Data: 0000
> >         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
> >                 DevCap: MaxPayload 128 bytes, PhantFunc 0
> >                         ExtTag- RBE+
> >                 DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
> >                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                         MaxPayload 128 bytes, MaxReadReq 128 bytes
> >                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> >                 LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
> >                         ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
> >                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
> >                         TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
> >                 RootCap: CRSVisible-
> >                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> >                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> >                 DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP+ LTR-
> >                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
> >                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
> >                          FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd-
> >                          AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> >                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
> >                          AtomicOpsCtl: ReqEn- EgressBlck-
> >                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
> >                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> >                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> >                          Compliance De-emphasis: -6dB
> >                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
> >                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
> >                          Retimer- 2Retimers- CrosslinkRes: unsupported
> >         Capabilities: [100 v1] Advanced Error Reporting
> >                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
> >                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> >                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> >                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >                 HeaderLog: 00000000 00000000 00000000 00000000
> >                 RootCmd: CERptEn+ NFERptEn+ FERptEn+
> >                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
> >                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
> >                 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
> >         Kernel driver in use: pcieport
> >
> > 0001:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
> >         Subsystem: Qualcomm Atheros Device cafe
> >         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0/wifi@1,0
> >         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0, Cache Line Size: 64 bytes
> >         Interrupt: pin A routed to IRQ 54
> >         Region 0: Memory at 2e000000 (64-bit, non-prefetchable) [size=2M]
> >         Capabilities: [40] Power Management version 3
> >                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >         Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit+
> >                 Address: 00000000435e8688  Data: 0001
> >                 Masking: fffffffe  Pending: 00000000
> >         Capabilities: [70] Express (v2) Endpoint, MSI 00
> >                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
> >                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
> >                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >                         MaxPayload 128 bytes, MaxReadReq 128 bytes
> >                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
> >                 LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
> >                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> >                 LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
> >                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
> >                          10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
> >                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
> >                          FRS- TPHComp- ExtTPHComp-
> >                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
> >                          AtomicOpsCtl: ReqEn-
> >                 LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
> >                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
> >                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> >                          Compliance De-emphasis: -6dB
> >                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
> >                          EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
> >                          Retimer- 2Retimers- CrosslinkRes: unsupported
> >         Capabilities: [100 v2] Advanced Error Reporting
> >                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> >                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> >                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
> >                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
> >                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
> >                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
> >                 HeaderLog: 00000000 00000000 00000000 00000000
> >         Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
> >         Capabilities: [158 v1] Latency Tolerance Reporting
> >                 Max snoop latency: 0ns
> >                 Max no snoop latency: 0ns
> >         Capabilities: [160 v1] L1 PM Substates
> >                 L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
> >                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> >                 L1SubCtl2:
> >         Kernel driver in use: ath10k_pci
Ard Biesheuvel May 11, 2021, 12:46 p.m. UTC | #7
On Tue, 11 May 2021 at 14:37, Ansuel Smith <ansuelsmth@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 02:30:36PM +0200, Ard Biesheuvel wrote:
> > On Tue, 11 May 2021 at 14:15, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > >
> > > On Tue, May 11, 2021 at 06:26:28AM +0200, Ard Biesheuvel wrote:
> > > > On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > > > >
> > > > > On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > > > > > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > > > > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> > > > > >
> > > > > > Do you really?  I mean, yes, theoretically, I understand it, the
> > > > > > hardware supports 64kB of I/O port space per root port.  But I/O
> > > > > > port space is rather deprecated these days.  My laptop has precisely
> > > > > > two devices with I/O ports, one with 64 bytes and the other with 32
> > > > > > bytes.  Would you really suffer by allocating 16kB of I/O port
> > > > > > space to each root port?
> > > > >
> > > > > We were talking about this in the other wrong patch. I also think this
> > > > > much space looks wrong. The current ipq806x dts have this space so it's
> > > > > actually broken from a long time. The only reason pci worked before was
> > > > > because the pci driver didn't actually check if the settings were right.
> > > > > New kernel introduced more checks and this problem showed up. (to be
> > > > > more precise, the pci port are commonly used by the ath10k wifi and the
> > > > > second ath10k wifi fails to init because of this problem)
> > > > > If you can give me any hint on how to check if the space can be reduced
> > > > > I would be very happy to investigate it.
> > > > > In the driver I notice that the max buffer is set to 2k, could be this a
> > > > > hint?
> > > > >
> > > >
> > > > Could you share the output of lspci -vv from such a system?
> > > >
> > > > I agree with Matthew that fiddling with the size of the I/O space
> > > > range probably papers over another problem, and with the odd
> > > > exception, no PCIe card used on ARM systems actually uses their I/O
> > > > BARs, even when they have them. (I used to carry a PCIe serial port
> > > > card to UEFI plugfests because that was the only thing that would stop
> > > > working if a system configured its I/O resource window incorrectly)
> > >
> > > Here is the output of lspci -vv
> > >
> > > 0000:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
> > >         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > >         I/O behind bridge: [disabled]
> > >         Memory behind bridge: 08000000-081fffff [size=2M]
> > >         Prefetchable memory behind bridge: [disabled]
> >
> > So this a MMIO window to the endpoint
> >
> > ...
> >
> > >
> > > 0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
> > >         Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]
> >
> > ... and the endpoint has a single *MMIO* BAR of size 2 MiB.
> >
> > This has *nothing* to do with port I/O, which is what you are
> > modifying with your patch.
> >
> > Did you check that the problem exists without the patch, and that the
> > patch makes it go away?
> >
> >
>
> Yes without the change to IO_SPACE_LIMIT, the ath10k driver fails to
> init as it can't access the reg. Only the first pci wifi works but the
> second one fails to init. By increasing the limit all comes back to
> normal. What I really can't understand is if the big IO space set in the
> ipq8064 dtsi was wrong from the start and the ath10k fails to init just
> because is missconfigured. Any idea how to find the appropriate max IO space
> for the pci?
>

OK, so the entire second host bridge fails to probe, because there is
no virtual address space left for the I/O window.

Just change the 'downstream I/O' window size in the DT to 64k
(0x10000) - I assume the current size (0x100000) is a typo anyway.
Ansuel Smith May 11, 2021, 12:51 p.m. UTC | #8
On Tue, May 11, 2021 at 02:46:49PM +0200, Ard Biesheuvel wrote:
> On Tue, 11 May 2021 at 14:37, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> >
> > On Tue, May 11, 2021 at 02:30:36PM +0200, Ard Biesheuvel wrote:
> > > On Tue, 11 May 2021 at 14:15, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > > >
> > > > On Tue, May 11, 2021 at 06:26:28AM +0200, Ard Biesheuvel wrote:
> > > > > On Tue, 11 May 2021 at 04:32, Ansuel Smith <ansuelsmth@gmail.com> wrote:
> > > > > >
> > > > > > On Tue, May 11, 2021 at 03:24:29AM +0100, Matthew Wilcox wrote:
> > > > > > > On Tue, May 11, 2021 at 04:16:54AM +0200, Ansuel Smith wrote:
> > > > > > > > Ipq8064 SoC requires larger IO_SPACE_LIMIT on second and third pci port.
> > > > > > >
> > > > > > > Do you really?  I mean, yes, theoretically, I understand it, the
> > > > > > > hardware supports 64kB of I/O port space per root port.  But I/O
> > > > > > > port space is rather deprecated these days.  My laptop has precisely
> > > > > > > two devices with I/O ports, one with 64 bytes and the other with 32
> > > > > > > bytes.  Would you really suffer by allocating 16kB of I/O port
> > > > > > > space to each root port?
> > > > > >
> > > > > > We were talking about this in the other wrong patch. I also think this
> > > > > > much space looks wrong. The current ipq806x dts have this space so it's
> > > > > > actually broken from a long time. The only reason pci worked before was
> > > > > > because the pci driver didn't actually check if the settings were right.
> > > > > > New kernel introduced more checks and this problem showed up. (to be
> > > > > > more precise, the pci port are commonly used by the ath10k wifi and the
> > > > > > second ath10k wifi fails to init because of this problem)
> > > > > > If you can give me any hint on how to check if the space can be reduced
> > > > > > I would be very happy to investigate it.
> > > > > > In the driver I notice that the max buffer is set to 2k, could be this a
> > > > > > hint?
> > > > > >
> > > > >
> > > > > Could you share the output of lspci -vv from such a system?
> > > > >
> > > > > I agree with Matthew that fiddling with the size of the I/O space
> > > > > range probably papers over another problem, and with the odd
> > > > > exception, no PCIe card used on ARM systems actually uses their I/O
> > > > > BARs, even when they have them. (I used to carry a PCIe serial port
> > > > > card to UEFI plugfests because that was the only thing that would stop
> > > > > working if a system configured its I/O resource window incorrectly)
> > > >
> > > > Here is the output of lspci -vv
> > > >
> > > > 0000:00:00.0 PCI bridge: Qualcomm Device 0101 (prog-if 00 [Normal decode])
> > > >         Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > >         I/O behind bridge: [disabled]
> > > >         Memory behind bridge: 08000000-081fffff [size=2M]
> > > >         Prefetchable memory behind bridge: [disabled]
> > >
> > > So this a MMIO window to the endpoint
> > >
> > > ...
> > >
> > > >
> > > > 0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
> > > >         Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]
> > >
> > > ... and the endpoint has a single *MMIO* BAR of size 2 MiB.
> > >
> > > This has *nothing* to do with port I/O, which is what you are
> > > modifying with your patch.
> > >
> > > Did you check that the problem exists without the patch, and that the
> > > patch makes it go away?
> > >
> > >
> >
> > Yes without the change to IO_SPACE_LIMIT, the ath10k driver fails to
> > init as it can't access the reg. Only the first pci wifi works but the
> > second one fails to init. By increasing the limit all comes back to
> > normal. What I really can't understand is if the big IO space set in the
> > ipq8064 dtsi was wrong from the start and the ath10k fails to init just
> > because is missconfigured. Any idea how to find the appropriate max IO space
> > for the pci?
> >
> 
> OK, so the entire second host bridge fails to probe, because there is
> no virtual address space left for the I/O window.
> 
> Just change the 'downstream I/O' window size in the DT to 64k
> (0x10000) - I assume the current size (0x100000) is a typo anyway.

Ok, so my fear were right... someone just typo the IO when the dtsi was
pushed and was wrong from all that times. Much easier and cleaner
solution.
Russell King (Oracle) May 11, 2021, 2:54 p.m. UTC | #9
On Tue, May 11, 2021 at 02:15:21PM +0200, Ansuel Smith wrote:
> 0000:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
>         Subsystem: Qualcomm Atheros Device cafe
>         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b500000/bridge@0,0/wifi@1,0
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 53
>         Region 0: Memory at 08000000 (64-bit, non-prefetchable) [size=2M]

It seems this Atheros device does not make any use of I/O, so this
device should be fine.

> 0001:01:00.0 Network controller: Qualcomm Atheros QCA9984 802.11ac Wave 2 Wireless Network Adapter
>         Subsystem: Qualcomm Atheros Device cafe
>         Device tree node: /sys/firmware/devicetree/base/soc/pci@1b700000/bridge@0,0/wifi@1,0
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 54
>         Region 0: Memory at 2e000000 (64-bit, non-prefetchable) [size=2M]

This device also seems to not make use of any I/O.

I don't see any reason probing of these two drivers should fail if
there is no or reduced I/O space.

Please can you share the kernel messages so we can see exactly what is
failing and how?
Russell King (Oracle) May 11, 2021, 2:55 p.m. UTC | #10
On Tue, May 11, 2021 at 02:51:15PM +0200, Ansuel Smith wrote:
> On Tue, May 11, 2021 at 02:46:49PM +0200, Ard Biesheuvel wrote:
> > OK, so the entire second host bridge fails to probe, because there is
> > no virtual address space left for the I/O window.
> > 
> > Just change the 'downstream I/O' window size in the DT to 64k
> > (0x10000) - I assume the current size (0x100000) is a typo anyway.
> 
> Ok, so my fear were right... someone just typo the IO when the dtsi was
> pushed and was wrong from all that times. Much easier and cleaner
> solution. 

Great news.
Linus Walleij May 17, 2021, 10:26 a.m. UTC | #11
On Tue, May 11, 2021 at 6:26 AM Ard Biesheuvel <ardb@kernel.org> wrote:

> I used to carry a PCIe serial port
> card to UEFI plugfests because that was the only thing that would stop
> working if a system configured its I/O resource window incorrectly

I've been looking for a thing like that for testing, I was actually
thinking that it would perhaps be a good idea to add a I/O-resource
requiring device to QEMU just to be able to test this kind of
thing for completeness.

Yours,
Linus Walleij
diff mbox series

Patch

diff --git a/Documentation/arm/memory.rst b/Documentation/arm/memory.rst
index 0cb1e2938823..938fe74c9020 100644
--- a/Documentation/arm/memory.rst
+++ b/Documentation/arm/memory.rst
@@ -53,7 +53,7 @@  ffc00000	ffc7ffff	Guard region
 ff800000	ffbfffff	Permanent, fixed read-only mapping of the
 				firmware provided DT blob
 
-fee00000	feffffff	Mapping of PCI I/O space. This is a static
+fee00000	ff0fffff	Mapping of PCI I/O space. This is a static
 				mapping within the vmalloc space.
 
 VMALLOC_START	VMALLOC_END-1	vmalloc() / ioremap() space.
diff --git a/arch/arm/include/asm/io.h b/arch/arm/include/asm/io.h
index fc748122f1e0..76f1c668df5e 100644
--- a/arch/arm/include/asm/io.h
+++ b/arch/arm/include/asm/io.h
@@ -197,7 +197,7 @@  void __iomem *pci_remap_cfgspace(resource_size_t res_cookie, size_t size);
 #ifdef CONFIG_NEED_MACH_IO_H
 #include <mach/io.h>
 #elif defined(CONFIG_PCI)
-#define IO_SPACE_LIMIT	((resource_size_t)0xfffff)
+#define IO_SPACE_LIMIT	((resource_size_t)0x2fffff)
 #define __io(a)		__typesafe_io(PCI_IO_VIRT_BASE + ((a) & IO_SPACE_LIMIT))
 #else
 #define __io(a)		__typesafe_io((a) & IO_SPACE_LIMIT)