Message ID | 1306882376.2866.68.camel@bwh-desktop (mailing list archive) |
---|---|
State | Superseded, archived |
Headers | show |
On 05/31/2011 03:52 PM, Ben Hutchings wrote: > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > bridge resources to get more big ranges when allocating space (again)'), > SFC9000-family network controllers in a Dell PE R905 are getting their > memory BARs disabled. > > These devices have: > BAR 0: I/O, 256 bytes > BAR 2: memory, 64-bit, 16 MB (for general registers) > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > can you send out lspci -tv too? Thanks Yinghai Lu -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2011-05-31 at 16:46 -0700, Yinghai Lu wrote: > On 05/31/2011 03:52 PM, Ben Hutchings wrote: > > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > > bridge resources to get more big ranges when allocating space (again)'), > > SFC9000-family network controllers in a Dell PE R905 are getting their > > memory BARs disabled. > > > > These devices have: > > BAR 0: I/O, 256 bytes > > BAR 2: memory, 64-bit, 16 MB (for general registers) > > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > > > > can you send out lspci -tv too? -+-[0000:20]-+-08.0-[0000:21]--+-00.0 Solarflare Communications SFC9020 [Solarstorm] | | \-00.1 Solarflare Communications SFC9020 [Solarstorm] | +-09.0-[0000:22]-- | +-0a.0-[0000:23]-- | +-0b.0-[0000:24]-- | \-0c.0-[0000:25]-- \-[0000:00]-+-01.0 Broadcom HT1100 HPX0 HT Host Bridge +-02.0-[0000:01]--+-0e.0 Broadcom HT1100 SATA Controller (PATA / IDE Mode) | \-0e.1 Broadcom HT1100 SATA Controller (PATA / IDE Mode) +-03.0-[0000:02]--+-0c.0 Broadcom HT1100 USB OHCI Controller | +-0c.1 Broadcom HT1100 USB OHCI Controller | +-0c.2 Broadcom HT1100 USB EHCI Controller | +-0d.0 Broadcom HT1100 USB OHCI Controller | +-0d.1 Broadcom HT1100 USB OHCI Controller | \-0d.2 Broadcom HT1100 USB EHCI Controller +-04.0-[0000:03-04]----00.0-[0000:04]----00.0 Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet +-05.0-[0000:05-06]----00.0-[0000:06]----00.0 Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet +-07.0 Broadcom HT1100 Legacy Device +-07.2 Broadcom HT1100 ISA-LPC Bridge +-08.0-[0000:0c]--+-00.0 Solarflare Communications SFL9021 [Solarstorm] | \-00.1 Solarflare Communications SFL9021 [Solarstorm] +-09.0-[0000:07-08]----00.0-[0000:08]----00.0 Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet +-0a.0-[0000:09-0a]----00.0-[0000:0a]----00.0 Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet +-0b.0-[0000:0b]----00.0 LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS +-0c.0-[0000:0d]-- +-0d.0 ATI Technologies Inc ES1000 +-18.0 Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration +-18.1 Advanced Micro Devices [AMD] Family 10h Processor Address Map +-18.2 Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller +-18.3 Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control +-18.4 Advanced Micro Devices [AMD] Family 10h Processor Link Control +-19.0 Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration +-19.1 Advanced Micro Devices [AMD] Family 10h Processor Address Map +-19.2 Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller +-19.3 Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control +-19.4 Advanced Micro Devices [AMD] Family 10h Processor Link Control +-1a.0 Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration +-1a.1 Advanced Micro Devices [AMD] Family 10h Processor Address Map +-1a.2 Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller +-1a.3 Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control +-1a.4 Advanced Micro Devices [AMD] Family 10h Processor Link Control +-1b.0 Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration +-1b.1 Advanced Micro Devices [AMD] Family 10h Processor Address Map +-1b.2 Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller +-1b.3 Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control \-1b.4 Advanced Micro Devices [AMD] Family 10h Processor Link Control
On 05/31/2011 03:52 PM, Ben Hutchings wrote: > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > bridge resources to get more big ranges when allocating space (again)'), > SFC9000-family network controllers in a Dell PE R905 are getting their > memory BARs disabled. > > These devices have: > BAR 0: I/O, 256 bytes > BAR 2: memory, 64-bit, 16 MB (for general registers) > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > > Here is a diff of 'lspci -vn' output before and after this commit: > > --- /home/bwh/tmp/lspci-good-init.log 2011-05-31 23:30:39.496353000 +0100 > +++ /home/bwh/tmp/lspci-bad-init.log 2011-05-31 23:22:02.507796000 +0100 > @@ -41,7 +41,7 @@ > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0 > I/O behind bridge: 0000b000-0000bfff > - Memory behind bridge: ec000000-eeffffff > + Prefetchable memory behind bridge: 00000000ec000000-00000000ec000000 > Capabilities: <access denied> > > 00:09.0 0604: 1166:0142 (rev a2) > @@ -270,8 +270,8 @@ > Subsystem: 1924:6102 > Flags: bus master, fast devsel, latency 0, IRQ 11 > I/O ports at b800 [size=256] > - Memory at ed000000 (64-bit, non-prefetchable) [size=16M] > - Memory at ecfe0000 (64-bit, non-prefetchable) [size=64K] > + Memory at <ignored> (64-bit, non-prefetchable) > + Memory at <ignored> (64-bit, non-prefetchable) > Expansion ROM at ec000000 [disabled] [size=128K] > Capabilities: <access denied> > > @@ -279,8 +279,8 @@ > Subsystem: 1924:6102 > Flags: bus master, fast devsel, latency 0, IRQ 11 > I/O ports at bc00 [size=256] > - Memory at ee000000 (64-bit, non-prefetchable) [size=16M] > - Memory at ecff0000 (64-bit, non-prefetchable) [size=64K] > + Memory at <ignored> (64-bit, non-prefetchable) > + Memory at <ignored> (64-bit, non-prefetchable) > Expansion ROM at ec020000 [disabled] [size=128K] > Capabilities: <access denied> > > @@ -288,7 +288,7 @@ > Flags: bus master, fast devsel, latency 0 > Bus: primary=20, secondary=21, subordinate=21, sec-latency=0 > I/O behind bridge: 00009000-00009fff > - Memory behind bridge: d5000000-d7ffffff > + Prefetchable memory behind bridge: 00000000d5000000-00000000d5000000 > Capabilities: <access denied> > > 20:09.0 0604: 1166:0142 (rev a2) > @@ -315,8 +315,8 @@ > Subsystem: 1924:6205 > Flags: bus master, fast devsel, latency 0, IRQ 5 > I/O ports at 9800 [size=256] > - Memory at d6000000 (64-bit, non-prefetchable) [size=16M] > - Memory at d5fe0000 (64-bit, non-prefetchable) [size=64K] > + Memory at <ignored> (64-bit, non-prefetchable) > + Memory at <ignored> (64-bit, non-prefetchable) > Expansion ROM at d5000000 [disabled] [size=128K] > Capabilities: <access denied> > > @@ -324,8 +324,8 @@ > Subsystem: 1924:6205 > Flags: bus master, fast devsel, latency 0, IRQ 5 > I/O ports at 9c00 [size=256] > - Memory at d7000000 (64-bit, non-prefetchable) [size=16M] > - Memory at d5ff0000 (64-bit, non-prefetchable) [size=64K] > + Memory at <ignored> (64-bit, non-prefetchable) > + Memory at <ignored> (64-bit, non-prefetchable) > Expansion ROM at d5020000 [disabled] [size=128K] > Capabilities: <access denied> > > --- END --- > > Below is a boot log of a kernel built from this commit, without the sfc > driver loaded. The devices in question are on buses 0c and 21. > > Ben. > > ... > PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1e]) > pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7] > pci_root PNP0A08:00: host bridge window [io 0xa000-0xffff] > pci_root PNP0A08:00: host bridge window [io 0x0d00-0x0fff] > pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff] > pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xf1ffffff] > pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff] > pci_root PNP0A08:00: host bridge window [mem 0xd8000000-0xdfffffff] > pci_root PNP0A08:00: host bridge window [mem 0xfed40000-0xfed44fff] ... > pci 0000:0c:00.0: [1924:0813] type 0 class 0x000200 > pci 0000:0c:00.0: reg 10: [io 0xb800-0xb8ff] > pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] > pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] > pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] > pci 0000:0c:00.0: PME# supported from D0 D3hot > pci 0000:0c:00.0: PME# disabled > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.1: [1924:0813] type 0 class 0x000200 > pci 0000:0c:00.1: reg 10: [io 0xbc00-0xbcff] > pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit] > pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit] > pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref] > pci 0000:0c:00.1: PME# supported from D0 D3hot > pci 0000:0c:00.1: PME# disabled > pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:00:08.0: PCI bridge to [bus 0c-0c] > pci 0000:00:08.0: bridge window [io 0xb000-0xbfff] > pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] > pci 0000:00:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) ... > ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 20-3e]) > pci_root PNP0A08:01: host bridge window [io 0x9000-0x9fff] > pci_root PNP0A08:01: host bridge window [mem 0xf2000000-0xf3ffffff] > pci_root PNP0A08:01: host bridge window [mem 0xd5000000-0xd7ffffff] ... > pci 0000:21:00.0: [1924:0803] type 0 class 0x000200 > pci 0000:21:00.0: reg 10: [io 0x9800-0x98ff] > pci 0000:21:00.0: reg 18: [mem 0xd6000000-0xd6ffffff 64bit] > pci 0000:21:00.0: reg 20: [mem 0xd5fe0000-0xd5feffff 64bit] > pci 0000:21:00.0: reg 30: [mem 0xd5000000-0xd501ffff pref] > pci 0000:21:00.0: PME# supported from D0 D3hot > pci 0000:21:00.0: PME# disabled > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.1: [1924:0803] type 0 class 0x000200 > pci 0000:21:00.1: reg 10: [io 0x9c00-0x9cff] > pci 0000:21:00.1: reg 18: [mem 0xd7000000-0xd7ffffff 64bit] > pci 0000:21:00.1: reg 20: [mem 0xd5ff0000-0xd5ffffff 64bit] > pci 0000:21:00.1: reg 30: [mem 0xd5000000-0xd501ffff pref] > pci 0000:21:00.1: PME# supported from D0 D3hot > pci 0000:21:00.1: PME# disabled > pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:20:08.0: PCI bridge to [bus 21-21] > pci 0000:20:08.0: bridge window [io 0x9000-0x9fff] > pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff] > pci 0000:20:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) .. > pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] > pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] your system is 4 sockets AMD quad cores system. two peer root bus: one to cpu 0, and one to cpu 3. 1. BIOS does assign same resource to func0 and func1. 2. BIOS does not assign resource to SR-IOV BAR... 3. BIOS does not preserve big enough allocation to peer root buses. then new code, try to assign resource to those unassigned or wrong assigned BARs, can not find enough resource for them. solution would be make SRIOV register BAR to be in good to have list will try to produce one patch for that. Thanks Yinghai > PCI: max bus depth: 2 pci_try_num: 3 > pci 0000:00:02.0: BAR 15: can't assign mem pref (size 0x100000) > pci 0000:00:08.0: BAR 15: can't assign mem pref (size 0x100000) > pci 0000:00:0d.0: BAR 6: assigned [mem 0xef000000-0xef01ffff pref] > pci 0000:01:0e.0: BAR 6: assigned [mem 0xef100000-0xef11ffff pref] > pci 0000:00:02.0: PCI bridge to [bus 01-01] > pci 0000:00:02.0: bridge window [io 0xf000-0xffff] > pci 0000:00:02.0: bridge window [mem 0xef100000-0xef1fffff] > pci 0000:00:02.0: bridge window [mem pref disabled] > pci 0000:00:03.0: PCI bridge to [bus 02-02] > pci 0000:00:03.0: bridge window [io 0xd000-0xefff] > pci 0000:00:03.0: bridge window [mem 0xef200000-0xef2fffff] > pci 0000:00:03.0: bridge window [mem pref disabled] > pci 0000:03:00.0: PCI bridge to [bus 04-04] > pci 0000:03:00.0: bridge window [io disabled] > pci 0000:03:00.0: bridge window [mem 0xe4000000-0xe5ffffff] > pci 0000:03:00.0: bridge window [mem pref disabled] > pci 0000:00:04.0: PCI bridge to [bus 03-04] > pci 0000:00:04.0: bridge window [io disabled] > pci 0000:00:04.0: bridge window [mem 0xe4000000-0xe5ffffff] > pci 0000:00:04.0: bridge window [mem pref disabled] > pci 0000:05:00.0: PCI bridge to [bus 06-06] > pci 0000:05:00.0: bridge window [io disabled] > pci 0000:05:00.0: bridge window [mem 0xe6000000-0xe7ffffff] > pci 0000:05:00.0: bridge window [mem pref disabled] > pci 0000:00:05.0: PCI bridge to [bus 05-06] > pci 0000:00:05.0: bridge window [io disabled] > pci 0000:00:05.0: bridge window [mem 0xe6000000-0xe7ffffff] > pci 0000:00:05.0: bridge window [mem pref disabled] > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: BAR 6: assigned [mem 0xec020000-0xec03ffff pref] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit] > pci 0000:0c:00.0: BAR 9: set to [mem 0xec040000-0xec82ffff 64bit] (PCI address [0xec040000-0xec82ffff]) > pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:0c:00.1: BAR 9: can't assign mem (size 0x7f0000) > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.0: BAR 7: assigned [mem 0xec830000-0xec92dfff 64bit] > pci 0000:0c:00.0: BAR 7: set to [mem 0xec830000-0xec92dfff 64bit] (PCI address [0xec830000-0xec92dfff]) > pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: BAR 7: assigned [mem 0xec92e000-0xeca2bfff 64bit] > pci 0000:0c:00.1: BAR 7: set to [mem 0xec92e000-0xeca2bfff 64bit] (PCI address [0xec92e000-0xeca2bfff]) > pci 0000:00:08.0: PCI bridge to [bus 0c-0c] > pci 0000:00:08.0: bridge window [io 0xb000-0xbfff] > pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] > pci 0000:00:08.0: bridge window [mem pref disabled] > pci 0000:07:00.0: PCI bridge to [bus 08-08] > pci 0000:07:00.0: bridge window [io disabled] > pci 0000:07:00.0: bridge window [mem 0xe8000000-0xe9ffffff] > pci 0000:07:00.0: bridge window [mem pref disabled] > pci 0000:00:09.0: PCI bridge to [bus 07-08] > pci 0000:00:09.0: bridge window [io disabled] > pci 0000:00:09.0: bridge window [mem 0xe8000000-0xe9ffffff] > pci 0000:00:09.0: bridge window [mem pref disabled] > pci 0000:09:00.0: PCI bridge to [bus 0a-0a] > pci 0000:09:00.0: bridge window [io disabled] > pci 0000:09:00.0: bridge window [mem 0xea000000-0xebffffff] > pci 0000:09:00.0: bridge window [mem pref disabled] > pci 0000:00:0a.0: PCI bridge to [bus 09-0a] > pci 0000:00:0a.0: bridge window [io disabled] > pci 0000:00:0a.0: bridge window [mem 0xea000000-0xebffffff] > pci 0000:00:0a.0: bridge window [mem pref disabled] > pci 0000:00:0b.0: PCI bridge to [bus 0b-0b] > pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff] > pci 0000:00:0b.0: bridge window [mem 0xef300000-0xef4fffff] > pci 0000:00:0b.0: bridge window [mem pref disabled] > pci 0000:00:0c.0: PCI bridge to [bus 0d-0d] > pci 0000:00:0c.0: bridge window [io disabled] > pci 0000:00:0c.0: bridge window [mem disabled] > pci 0000:00:0c.0: bridge window [mem pref disabled] > pci 0000:20:08.0: BAR 15: can't assign mem pref (size 0x100000) > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.1: BAR 6: assigned [mem 0xd5020000-0xd503ffff pref] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.0: BAR 9: assigned [mem 0xd5040000-0xd582ffff 64bit] > pci 0000:21:00.0: BAR 9: set to [mem 0xd5040000-0xd582ffff 64bit] (PCI address [0xd5040000-0xd582ffff]) > pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > pci 0000:21:00.1: BAR 9: can't assign mem (size 0x7f0000) > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.0: BAR 7: assigned [mem 0xd5830000-0xd592dfff 64bit] > pci 0000:21:00.0: BAR 7: set to [mem 0xd5830000-0xd592dfff 64bit] (PCI address [0xd5830000-0xd592dfff]) > pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:21:00.1: BAR 7: assigned [mem 0xd592e000-0xd5a2bfff 64bit] > pci 0000:21:00.1: BAR 7: set to [mem 0xd592e000-0xd5a2bfff 64bit] (PCI address [0xd592e000-0xd5a2bfff]) > pci 0000:20:08.0: PCI bridge to [bus 21-21] > pci 0000:20:08.0: bridge window [io 0x9000-0x9fff] > pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff] > pci 0000:20:08.0: bridge window [mem pref disabled] > pci 0000:20:09.0: PCI bridge to [bus 22-22] > pci 0000:20:09.0: bridge window [io disabled] > pci 0000:20:09.0: bridge window [mem disabled] > pci 0000:20:09.0: bridge window [mem pref disabled] > pci 0000:20:0a.0: PCI bridge to [bus 23-23] > pci 0000:20:0a.0: bridge window [io disabled] > pci 0000:20:0a.0: bridge window [mem disabled] > pci 0000:20:0a.0: bridge window [mem pref disabled] > pci 0000:20:0b.0: PCI bridge to [bus 24-24] > pci 0000:20:0b.0: bridge window [io disabled] > pci 0000:20:0b.0: bridge window [mem disabled] > pci 0000:20:0b.0: bridge window [mem pref disabled] > pci 0000:20:0c.0: PCI bridge to [bus 25-25] > pci 0000:20:0c.0: bridge window [io disabled] > pci 0000:20:0c.0: bridge window [mem disabled] > pci 0000:20:0c.0: bridge window [mem pref disabled] > PCI: No. 2 try to assign unassigned res > release child resource [mem 0xd5000000-0xd501ffff pref] > release child resource [mem 0xd5020000-0xd503ffff pref] > release child resource [mem 0xd5040000-0xd582ffff 64bit] > release child resource [mem 0xd5830000-0xd592dfff 64bit] > release child resource [mem 0xd592e000-0xd5a2bfff 64bit] > release child resource [mem 0xd5fe0000-0xd5feffff 64bit] > release child resource [mem 0xd5ff0000-0xd5ffffff 64bit] > release child resource [mem 0xd6000000-0xd6ffffff 64bit] > release child resource [mem 0xd7000000-0xd7ffffff 64bit] ... -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2011-05-31 at 17:51 -0700, Yinghai Lu wrote: > On 05/31/2011 03:52 PM, Ben Hutchings wrote: > > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > > bridge resources to get more big ranges when allocating space (again)'), > > SFC9000-family network controllers in a Dell PE R905 are getting their > > memory BARs disabled. > > > > These devices have: > > BAR 0: I/O, 256 bytes > > BAR 2: memory, 64-bit, 16 MB (for general registers) > > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) [...] > > pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] > > pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] > > your system is 4 sockets AMD quad cores system. > two peer root bus: one to cpu 0, and one to cpu 3. > > 1. BIOS does assign same resource to func0 and func1. Only for the expansion ROM BARs, which never need to be mapped at the same time. Previous kernel versions do fix this up, though. Would you like a log of the previous behaviour? > 2. BIOS does not assign resource to SR-IOV BAR... Right, sorry I forgot to mention the SR-IOV BAR. The current configuration for these devices has SR-IOV functionality disabled in the firmware, but unfortunately the PCIe core is hardwired to expose the capability. > 3. BIOS does not preserve big enough allocation to peer root buses. > > then new code, try to assign resource to those unassigned or wrong > assigned BARs, can not find enough resource for them. > > solution would be > make SRIOV register BAR to be in good to have list In this configuration it can be ignored completely, but I don't think there's any generic way to determine that. > will try to produce one patch for that. Thanks. Ben.
On Tue, May 31, 2011 at 6:51 PM, Yinghai Lu <yinghai@kernel.org> wrote: > On 05/31/2011 03:52 PM, Ben Hutchings wrote: >> Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update >> bridge resources to get more big ranges when allocating space (again)'), >> SFC9000-family network controllers in a Dell PE R905 are getting their >> memory BARs disabled. >> >> These devices have: >> BAR 0: I/O, 256 bytes >> BAR 2: memory, 64-bit, 16 MB (for general registers) >> BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) >> >> Here is a diff of 'lspci -vn' output before and after this commit: >> >> --- /home/bwh/tmp/lspci-good-init.log 2011-05-31 23:30:39.496353000 +0100 >> +++ /home/bwh/tmp/lspci-bad-init.log 2011-05-31 23:22:02.507796000 +0100 >> @@ -41,7 +41,7 @@ >> Flags: bus master, fast devsel, latency 0 >> Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0 >> I/O behind bridge: 0000b000-0000bfff >> - Memory behind bridge: ec000000-eeffffff >> + Prefetchable memory behind bridge: 00000000ec000000-00000000ec000000 >> Capabilities: <access denied> >> >> 00:09.0 0604: 1166:0142 (rev a2) >> @@ -270,8 +270,8 @@ >> Subsystem: 1924:6102 >> Flags: bus master, fast devsel, latency 0, IRQ 11 >> I/O ports at b800 [size=256] >> - Memory at ed000000 (64-bit, non-prefetchable) [size=16M] >> - Memory at ecfe0000 (64-bit, non-prefetchable) [size=64K] >> + Memory at <ignored> (64-bit, non-prefetchable) >> + Memory at <ignored> (64-bit, non-prefetchable) >> Expansion ROM at ec000000 [disabled] [size=128K] >> Capabilities: <access denied> >> >> @@ -279,8 +279,8 @@ >> Subsystem: 1924:6102 >> Flags: bus master, fast devsel, latency 0, IRQ 11 >> I/O ports at bc00 [size=256] >> - Memory at ee000000 (64-bit, non-prefetchable) [size=16M] >> - Memory at ecff0000 (64-bit, non-prefetchable) [size=64K] >> + Memory at <ignored> (64-bit, non-prefetchable) >> + Memory at <ignored> (64-bit, non-prefetchable) >> Expansion ROM at ec020000 [disabled] [size=128K] >> Capabilities: <access denied> >> >> @@ -288,7 +288,7 @@ >> Flags: bus master, fast devsel, latency 0 >> Bus: primary=20, secondary=21, subordinate=21, sec-latency=0 >> I/O behind bridge: 00009000-00009fff >> - Memory behind bridge: d5000000-d7ffffff >> + Prefetchable memory behind bridge: 00000000d5000000-00000000d5000000 >> Capabilities: <access denied> >> >> 20:09.0 0604: 1166:0142 (rev a2) >> @@ -315,8 +315,8 @@ >> Subsystem: 1924:6205 >> Flags: bus master, fast devsel, latency 0, IRQ 5 >> I/O ports at 9800 [size=256] >> - Memory at d6000000 (64-bit, non-prefetchable) [size=16M] >> - Memory at d5fe0000 (64-bit, non-prefetchable) [size=64K] >> + Memory at <ignored> (64-bit, non-prefetchable) >> + Memory at <ignored> (64-bit, non-prefetchable) >> Expansion ROM at d5000000 [disabled] [size=128K] >> Capabilities: <access denied> >> >> @@ -324,8 +324,8 @@ >> Subsystem: 1924:6205 >> Flags: bus master, fast devsel, latency 0, IRQ 5 >> I/O ports at 9c00 [size=256] >> - Memory at d7000000 (64-bit, non-prefetchable) [size=16M] >> - Memory at d5ff0000 (64-bit, non-prefetchable) [size=64K] >> + Memory at <ignored> (64-bit, non-prefetchable) >> + Memory at <ignored> (64-bit, non-prefetchable) >> Expansion ROM at d5020000 [disabled] [size=128K] >> Capabilities: <access denied> >> >> --- END --- >> >> Below is a boot log of a kernel built from this commit, without the sfc >> driver loaded. The devices in question are on buses 0c and 21. >> >> Ben. >> >> > ... >> PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug >> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1e]) >> pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7] >> pci_root PNP0A08:00: host bridge window [io 0xa000-0xffff] >> pci_root PNP0A08:00: host bridge window [io 0x0d00-0x0fff] >> pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff] >> pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xf1ffffff] >> pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff] >> pci_root PNP0A08:00: host bridge window [mem 0xd8000000-0xdfffffff] >> pci_root PNP0A08:00: host bridge window [mem 0xfed40000-0xfed44fff] > ... >> pci 0000:0c:00.0: [1924:0813] type 0 class 0x000200 >> pci 0000:0c:00.0: reg 10: [io 0xb800-0xb8ff] >> pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] >> pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] >> pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] >> pci 0000:0c:00.0: PME# supported from D0 D3hot >> pci 0000:0c:00.0: PME# disabled >> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] >> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] >> pci 0000:0c:00.1: [1924:0813] type 0 class 0x000200 >> pci 0000:0c:00.1: reg 10: [io 0xbc00-0xbcff] >> pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit] >> pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit] >> pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref] >> pci 0000:0c:00.1: PME# supported from D0 D3hot >> pci 0000:0c:00.1: PME# disabled >> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] >> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] >> pci 0000:00:08.0: PCI bridge to [bus 0c-0c] >> pci 0000:00:08.0: bridge window [io 0xb000-0xbfff] >> pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] >> pci 0000:00:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) > ... >> ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 20-3e]) >> pci_root PNP0A08:01: host bridge window [io 0x9000-0x9fff] >> pci_root PNP0A08:01: host bridge window [mem 0xf2000000-0xf3ffffff] >> pci_root PNP0A08:01: host bridge window [mem 0xd5000000-0xd7ffffff] > ... >> pci 0000:21:00.0: [1924:0803] type 0 class 0x000200 >> pci 0000:21:00.0: reg 10: [io 0x9800-0x98ff] >> pci 0000:21:00.0: reg 18: [mem 0xd6000000-0xd6ffffff 64bit] >> pci 0000:21:00.0: reg 20: [mem 0xd5fe0000-0xd5feffff 64bit] >> pci 0000:21:00.0: reg 30: [mem 0xd5000000-0xd501ffff pref] >> pci 0000:21:00.0: PME# supported from D0 D3hot >> pci 0000:21:00.0: PME# disabled >> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] >> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] >> pci 0000:21:00.1: [1924:0803] type 0 class 0x000200 >> pci 0000:21:00.1: reg 10: [io 0x9c00-0x9cff] >> pci 0000:21:00.1: reg 18: [mem 0xd7000000-0xd7ffffff 64bit] >> pci 0000:21:00.1: reg 20: [mem 0xd5ff0000-0xd5ffffff 64bit] >> pci 0000:21:00.1: reg 30: [mem 0xd5000000-0xd501ffff pref] >> pci 0000:21:00.1: PME# supported from D0 D3hot >> pci 0000:21:00.1: PME# disabled >> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] >> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] >> pci 0000:20:08.0: PCI bridge to [bus 21-21] >> pci 0000:20:08.0: bridge window [io 0x9000-0x9fff] >> pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff] >> pci 0000:20:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) > .. >> pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] >> pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] > > your system is 4 sockets AMD quad cores system. > two peer root bus: one to cpu 0, and one to cpu 3. > > 1. BIOS does assign same resource to func0 and func1. > 2. BIOS does not assign resource to SR-IOV BAR... > 3. BIOS does not preserve big enough allocation to peer root buses. > > then new code, try to assign resource to those unassigned or wrong assigned BARs, can not find enough resource for them. > > solution would be > make SRIOV register BAR to be in good to have list I don't think that's the right solution. Here's the path leading to 0c:00.0 and .1: pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff] pci 0000:00:08.0: PCI bridge to [bus 0c-0c] pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] (48MB) pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] (16MB) pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] (64KB) pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] (128KB) pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] (8KB) pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] (64KB) pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit] pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit] pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref] pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] Each function needs 16MB + 264KB. Both functions together will easily fit in the 48MB bridge window, even if we allocate separate ROM space and all the SR-IOV BARs. But we're doing something wrong when assigning the second SR-IOV BAR: pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit] This has a size of 0x7f0000 when it should only be 0x10000 (64KB). I don't think 0x7f0000 is even a legal size for a PCI BAR; it should be a power of two. Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2011-05-31 at 21:25 -0600, Bjorn Helgaas wrote: > On Tue, May 31, 2011 at 6:51 PM, Yinghai Lu <yinghai@kernel.org> wrote: [...] > > solution would be > > make SRIOV register BAR to be in good to have list > > I don't think that's the right solution. Here's the path leading to > 0c:00.0 and .1: > > pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff] > pci 0000:00:08.0: PCI bridge to [bus 0c-0c] > pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] (48MB) > pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] (16MB) > pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] (64KB) > pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] (128KB) > pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] (8KB) > pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] (64KB) > pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit] > pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit] > pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref] > pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] > pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] > > Each function needs 16MB + 264KB. Both functions together will easily > fit in the 48MB bridge window, even if we allocate separate ROM space > and all the SR-IOV BARs. > > But we're doing something wrong when assigning the second SR-IOV BAR: > > pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit] > > This has a size of 0x7f0000 when it should only be 0x10000 (64KB). I > don't think 0x7f0000 is even a legal size for a PCI BAR; it should be > a power of two. I would guess this is because we advertise support for up to 127 VFs per PF. (Which is the correct number when SR-IOV functionality is actually enabled in the firmware.) Ben.
On Tue, May 31, 2011 at 10:27 PM, Ben Hutchings <bhutchings@solarflare.com> wrote: > On Tue, 2011-05-31 at 21:25 -0600, Bjorn Helgaas wrote: >> On Tue, May 31, 2011 at 6:51 PM, Yinghai Lu <yinghai@kernel.org> wrote: > [...] >> > solution would be >> > make SRIOV register BAR to be in good to have list >> >> I don't think that's the right solution. Here's the path leading to >> 0c:00.0 and .1: >> >> pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff] >> pci 0000:00:08.0: PCI bridge to [bus 0c-0c] >> pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] (48MB) >> pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] (16MB) >> pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] (64KB) >> pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] (128KB) >> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] (8KB) >> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] (64KB) >> pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit] >> pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit] >> pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref] >> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit] >> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit] >> >> Each function needs 16MB + 264KB. Both functions together will easily >> fit in the 48MB bridge window, even if we allocate separate ROM space >> and all the SR-IOV BARs. >> >> But we're doing something wrong when assigning the second SR-IOV BAR: >> >> pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit] >> >> This has a size of 0x7f0000 when it should only be 0x10000 (64KB). I >> don't think 0x7f0000 is even a legal size for a PCI BAR; it should be >> a power of two. > > I would guess this is because we advertise support for up to 127 VFs per > PF. (Which is the correct number when SR-IOV functionality is actually > enabled in the firmware.) Oh, yep, I bet you're right. How about we mention the 127 VFs somewhere in the dmesg, so this makes a bit more sense to the non-experts like me? -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 2011-06-01 at 02:10 +0100, Ben Hutchings wrote: > On Tue, 2011-05-31 at 17:51 -0700, Yinghai Lu wrote: > > On 05/31/2011 03:52 PM, Ben Hutchings wrote: > > > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > > > bridge resources to get more big ranges when allocating space (again)'), > > > SFC9000-family network controllers in a Dell PE R905 are getting their > > > memory BARs disabled. > > > > > > These devices have: > > > BAR 0: I/O, 256 bytes > > > BAR 2: memory, 64-bit, 16 MB (for general registers) > > > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > [...] > > > pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] > > > pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] > > > > your system is 4 sockets AMD quad cores system. > > two peer root bus: one to cpu 0, and one to cpu 3. > > > > 1. BIOS does assign same resource to func0 and func1. > > Only for the expansion ROM BARs, which never need to be mapped at the > same time. Previous kernel versions do fix this up, though. Would you > like a log of the previous behaviour? > > > 2. BIOS does not assign resource to SR-IOV BAR... > > Right, sorry I forgot to mention the SR-IOV BAR. The current > configuration for these devices has SR-IOV functionality disabled in the > firmware, but unfortunately the PCIe core is hardwired to expose the > capability. [...] The SR-IOV capability can and should be initialised with INITIAL_VF=TOTAL_VF=0 when it's configured to be disabled. The non-zero initialisation was a bug in our firmware which we can fix. It seems to me that the fallback for VF BARs should not be to disable them, but to allocate a smaller multiple of the BAR size and to limit NUM_VF accordingly. Ben.
On Wed, Jun 01, 2011 at 02:10:56AM +0100, Ben Hutchings wrote: > On Tue, 2011-05-31 at 17:51 -0700, Yinghai Lu wrote: > > On 05/31/2011 03:52 PM, Ben Hutchings wrote: > > > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > > > bridge resources to get more big ranges when allocating space (again)'), > > > SFC9000-family network controllers in a Dell PE R905 are getting their > > > memory BARs disabled. > > > > > > These devices have: > > > BAR 0: I/O, 256 bytes > > > BAR 2: memory, 64-bit, 16 MB (for general registers) > > > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > [...] > > > pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] > > > pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] > > > > your system is 4 sockets AMD quad cores system. > > two peer root bus: one to cpu 0, and one to cpu 3. > > > > 1. BIOS does assign same resource to func0 and func1. > > Only for the expansion ROM BARs, which never need to be mapped at the > same time. Previous kernel versions do fix this up, though. Would you > like a log of the previous behaviour? > > > 2. BIOS does not assign resource to SR-IOV BAR... > > Right, sorry I forgot to mention the SR-IOV BAR. The current > configuration for these devices has SR-IOV functionality disabled in the > firmware, but unfortunately the PCIe core is hardwired to expose the > capability. Ben, Can you send me the output of lspci -vvv -s <your sriov capable device>? I wonder there might be something in there to indicate that the SRIOV capability of the device is not fully enabled?? RP -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2011-06-06 at 14:36 -0700, Ram Pai wrote: > On Wed, Jun 01, 2011 at 02:10:56AM +0100, Ben Hutchings wrote: > > On Tue, 2011-05-31 at 17:51 -0700, Yinghai Lu wrote: > > > On 05/31/2011 03:52 PM, Ben Hutchings wrote: > > > > Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update > > > > bridge resources to get more big ranges when allocating space (again)'), > > > > SFC9000-family network controllers in a Dell PE R905 are getting their > > > > memory BARs disabled. > > > > > > > > These devices have: > > > > BAR 0: I/O, 256 bytes > > > > BAR 2: memory, 64-bit, 16 MB (for general registers) > > > > BAR 4: memory, 64-bit, 64 KB (for MSI-X tables) > > [...] > > > > pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref] > > > > pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref] > > > > > > your system is 4 sockets AMD quad cores system. > > > two peer root bus: one to cpu 0, and one to cpu 3. > > > > > > 1. BIOS does assign same resource to func0 and func1. > > > > Only for the expansion ROM BARs, which never need to be mapped at the > > same time. Previous kernel versions do fix this up, though. Would you > > like a log of the previous behaviour? > > > > > 2. BIOS does not assign resource to SR-IOV BAR... > > > > Right, sorry I forgot to mention the SR-IOV BAR. The current > > configuration for these devices has SR-IOV functionality disabled in the > > firmware, but unfortunately the PCIe core is hardwired to expose the > > capability. > > Ben, > Can you send me the output of lspci -vvv -s <your sriov capable device>? > > I wonder there might be something in there to indicate that the SRIOV > capability of the device is not fully enabled?? > > RP Don't worry about that case; we have a firmware fix that sets the number of VFs to 0. But it would be good to have a fallback when SR-IOV *is* enabled and there is not enough address space for all VFs that the devicce supports. Ben.
--- /home/bwh/tmp/lspci-good-init.log 2011-05-31 23:30:39.496353000 +0100 +++ /home/bwh/tmp/lspci-bad-init.log 2011-05-31 23:22:02.507796000 +0100 @@ -41,7 +41,7 @@ Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0 I/O behind bridge: 0000b000-0000bfff - Memory behind bridge: ec000000-eeffffff + Prefetchable memory behind bridge: 00000000ec000000-00000000ec000000 Capabilities: <access denied> 00:09.0 0604: 1166:0142 (rev a2) @@ -270,8 +270,8 @@ Subsystem: 1924:6102 Flags: bus master, fast devsel, latency 0, IRQ 11 I/O ports at b800 [size=256] - Memory at ed000000 (64-bit, non-prefetchable) [size=16M] - Memory at ecfe0000 (64-bit, non-prefetchable) [size=64K] + Memory at <ignored> (64-bit, non-prefetchable) + Memory at <ignored> (64-bit, non-prefetchable) Expansion ROM at ec000000 [disabled] [size=128K] Capabilities: <access denied> @@ -279,8 +279,8 @@ Subsystem: 1924:6102 Flags: bus master, fast devsel, latency 0, IRQ 11 I/O ports at bc00 [size=256] - Memory at ee000000 (64-bit, non-prefetchable) [size=16M] - Memory at ecff0000 (64-bit, non-prefetchable) [size=64K] + Memory at <ignored> (64-bit, non-prefetchable) + Memory at <ignored> (64-bit, non-prefetchable) Expansion ROM at ec020000 [disabled] [size=128K] Capabilities: <access denied> @@ -288,7 +288,7 @@ Flags: bus master, fast devsel, latency 0 Bus: primary=20, secondary=21, subordinate=21, sec-latency=0 I/O behind bridge: 00009000-00009fff - Memory behind bridge: d5000000-d7ffffff + Prefetchable memory behind bridge: 00000000d5000000-00000000d5000000 Capabilities: <access denied> 20:09.0 0604: 1166:0142 (rev a2) @@ -315,8 +315,8 @@ Subsystem: 1924:6205 Flags: bus master, fast devsel, latency 0, IRQ 5 I/O ports at 9800 [size=256] - Memory at d6000000 (64-bit, non-prefetchable) [size=16M] - Memory at d5fe0000 (64-bit, non-prefetchable) [size=64K] + Memory at <ignored> (64-bit, non-prefetchable) + Memory at <ignored> (64-bit, non-prefetchable) Expansion ROM at d5000000 [disabled] [size=128K] Capabilities: <access denied> @@ -324,8 +324,8 @@ Subsystem: 1924:6205 Flags: bus master, fast devsel, latency 0, IRQ 5 I/O ports at 9c00 [size=256] - Memory at d7000000 (64-bit, non-prefetchable) [size=16M] - Memory at d5ff0000 (64-bit, non-prefetchable) [size=64K] + Memory at <ignored> (64-bit, non-prefetchable) + Memory at <ignored> (64-bit, non-prefetchable) Expansion ROM at d5020000 [disabled] [size=128K] Capabilities: <access denied> --- END ---