| Message ID | 3443aad0-8c3b-b97e-685a-96b0866827be@amd.com (mailing list archive) |
|---|---|
| State | New, archived |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
On 11/21/2017 08:34 AM, Christian König wrote: > Hi Boris, > > attached are two patches. > > The first one is a trivial fix for the infinite loop issue, it now > correctly aborts the fixup when it can't find address space for the > root window. > > The second is a workaround for your board. It simply checks if there > is exactly one Processor Function to apply this fix on. > > Both are based on linus current master branch. Please test if they fix > your issue. Yes, they do fix it but that's because the feature is disabled. Do you know what the actual problem was (on Xen)? Thanks. -boris > > Thanks for the help, > Christian. > > Am 20.11.2017 um 17:33 schrieb Boris Ostrovsky: >> On 11/20/2017 11:07 AM, Christian König wrote: >>> Am 20.11.2017 um 16:51 schrieb Boris Ostrovsky: >>>> (and then it breaks differently as a Xen guest --- we hung on the last >>>> pci_read_config_dword(), I haven't looked at this at all yet) >>> Hui? How does this fix applies to a Xen guest in the first place? >>> >>> Please provide the output of "lspci -nn" and explain further what is >>> your config with Xen. >>> >>> >> >> This is dom0. >> >> -bash-4.1# lspci -nn >> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD890 Northbridge only >> dual slot (2x16) PCI-e GFX Hydra part [1002:5a10] (rev 02) >> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device >> [1002:5a23] >> 00:0d.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI bridge >> (external gfx1 port B) [1002:5a1e] >> 00:11.0 SATA controller [0106]: ATI Technologies Inc SB700/SB800 SATA >> Controller [AHCI mode] [1002:4391] >> 00:12.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >> OHCI0 Controller [1002:4397] >> 00:12.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >> Controller [1002:4398] >> 00:12.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB EHCI >> Controller [1002:4396] >> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >> OHCI0 Controller [1002:4397] >> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >> Controller [1002:4398] >> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB EHCI >> Controller [1002:4396] >> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller >> [1002:4385] (rev 3d) >> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB700/SB800 LPC host >> controller [1002:439d] >> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge >> [1002:4384] >> 00:14.5 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >> OHCI2 Controller [1002:4399] >> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1600] >> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1601] >> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1602] >> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1603] >> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1604] >> 00:18.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1605] >> 00:19.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1600] >> 00:19.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1601] >> 00:19.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1602] >> 00:19.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1603] >> 00:19.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1604] >> 00:19.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >> [1022:1605] >> 01:04.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA >> G200eW WPCM450 [102b:0532] (rev 0a) >> 02:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >> Network Connection [8086:10c9] (rev 01) >> 02:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >> Network Connection [8086:10c9] (rev 01) >> -bash-4.1# >> >> >> -boris > >
Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: > On 11/21/2017 08:34 AM, Christian König wrote: >> Hi Boris, >> >> attached are two patches. >> >> The first one is a trivial fix for the infinite loop issue, it now >> correctly aborts the fixup when it can't find address space for the >> root window. >> >> The second is a workaround for your board. It simply checks if there >> is exactly one Processor Function to apply this fix on. >> >> Both are based on linus current master branch. Please test if they fix >> your issue. > > Yes, they do fix it but that's because the feature is disabled. > > Do you know what the actual problem was (on Xen)? I still haven't understood what you actually did with Xen. When you used PCI pass through with those devices then you have made a major configuration error. When the problem happened on dom0 then the explanation is most likely that some PCI device ended up in the configured space, but the routing was only setup correctly on one CPU socket. Regards, Christian. > > Thanks. > -boris > >> Thanks for the help, >> Christian. >> >> Am 20.11.2017 um 17:33 schrieb Boris Ostrovsky: >>> On 11/20/2017 11:07 AM, Christian König wrote: >>>> Am 20.11.2017 um 16:51 schrieb Boris Ostrovsky: >>>>> (and then it breaks differently as a Xen guest --- we hung on the last >>>>> pci_read_config_dword(), I haven't looked at this at all yet) >>>> Hui? How does this fix applies to a Xen guest in the first place? >>>> >>>> Please provide the output of "lspci -nn" and explain further what is >>>> your config with Xen. >>>> >>>> >>> This is dom0. >>> >>> -bash-4.1# lspci -nn >>> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD890 Northbridge only >>> dual slot (2x16) PCI-e GFX Hydra part [1002:5a10] (rev 02) >>> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device >>> [1002:5a23] >>> 00:0d.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI bridge >>> (external gfx1 port B) [1002:5a1e] >>> 00:11.0 SATA controller [0106]: ATI Technologies Inc SB700/SB800 SATA >>> Controller [AHCI mode] [1002:4391] >>> 00:12.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>> OHCI0 Controller [1002:4397] >>> 00:12.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>> Controller [1002:4398] >>> 00:12.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB EHCI >>> Controller [1002:4396] >>> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>> OHCI0 Controller [1002:4397] >>> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>> Controller [1002:4398] >>> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB EHCI >>> Controller [1002:4396] >>> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller >>> [1002:4385] (rev 3d) >>> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB700/SB800 LPC host >>> controller [1002:439d] >>> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge >>> [1002:4384] >>> 00:14.5 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>> OHCI2 Controller [1002:4399] >>> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1600] >>> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1601] >>> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1602] >>> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1603] >>> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1604] >>> 00:18.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1605] >>> 00:19.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1600] >>> 00:19.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1601] >>> 00:19.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1602] >>> 00:19.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1603] >>> 00:19.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1604] >>> 00:19.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>> [1022:1605] >>> 01:04.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA >>> G200eW WPCM450 [102b:0532] (rev 0a) >>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>> Network Connection [8086:10c9] (rev 01) >>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>> Network Connection [8086:10c9] (rev 01) >>> -bash-4.1# >>> >>> >>> -boris >>
On 11/22/2017 05:09 AM, Christian König wrote: > Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >> On 11/21/2017 08:34 AM, Christian König wrote: >>> Hi Boris, >>> >>> attached are two patches. >>> >>> The first one is a trivial fix for the infinite loop issue, it now >>> correctly aborts the fixup when it can't find address space for the >>> root window. >>> >>> The second is a workaround for your board. It simply checks if there >>> is exactly one Processor Function to apply this fix on. >>> >>> Both are based on linus current master branch. Please test if they fix >>> your issue. >> >> Yes, they do fix it but that's because the feature is disabled. >> >> Do you know what the actual problem was (on Xen)? > > I still haven't understood what you actually did with Xen. > > When you used PCI pass through with those devices then you have made a > major configuration error. > > When the problem happened on dom0 then the explanation is most likely > that some PCI device ended up in the configured space, but the routing > was only setup correctly on one CPU socket. The problem is that dom0 can be (and was in my case() booted with less than full physical memory and so the "rest" of the host memory is not necessarily reflected in iomem. Your patch then tried to configure that memory for MMIO and the system hang. And so my guess is that this patch will break dom0 on a single-socket system as well. -boris > > Regards, > Christian. > >> >> Thanks. >> -boris >> >>> Thanks for the help, >>> Christian. >>> >>> Am 20.11.2017 um 17:33 schrieb Boris Ostrovsky: >>>> On 11/20/2017 11:07 AM, Christian König wrote: >>>>> Am 20.11.2017 um 16:51 schrieb Boris Ostrovsky: >>>>>> (and then it breaks differently as a Xen guest --- we hung on the >>>>>> last >>>>>> pci_read_config_dword(), I haven't looked at this at all yet) >>>>> Hui? How does this fix applies to a Xen guest in the first place? >>>>> >>>>> Please provide the output of "lspci -nn" and explain further what is >>>>> your config with Xen. >>>>> >>>>> >>>> This is dom0. >>>> >>>> -bash-4.1# lspci -nn >>>> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD890 Northbridge >>>> only >>>> dual slot (2x16) PCI-e GFX Hydra part [1002:5a10] (rev 02) >>>> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device >>>> [1002:5a23] >>>> 00:0d.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI >>>> bridge >>>> (external gfx1 port B) [1002:5a1e] >>>> 00:11.0 SATA controller [0106]: ATI Technologies Inc SB700/SB800 SATA >>>> Controller [AHCI mode] [1002:4391] >>>> 00:12.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>> OHCI0 Controller [1002:4397] >>>> 00:12.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>> Controller [1002:4398] >>>> 00:12.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>> EHCI >>>> Controller [1002:4396] >>>> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>> OHCI0 Controller [1002:4397] >>>> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>> Controller [1002:4398] >>>> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>> EHCI >>>> Controller [1002:4396] >>>> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller >>>> [1002:4385] (rev 3d) >>>> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB700/SB800 LPC host >>>> controller [1002:439d] >>>> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI >>>> Bridge >>>> [1002:4384] >>>> 00:14.5 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>> OHCI2 Controller [1002:4399] >>>> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1600] >>>> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1601] >>>> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1602] >>>> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1603] >>>> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1604] >>>> 00:18.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1605] >>>> 00:19.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1600] >>>> 00:19.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1601] >>>> 00:19.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1602] >>>> 00:19.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1603] >>>> 00:19.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1604] >>>> 00:19.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>> [1022:1605] >>>> 01:04.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA >>>> G200eW WPCM450 [102b:0532] (rev 0a) >>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>> Network Connection [8086:10c9] (rev 01) >>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>> Network Connection [8086:10c9] (rev 01) >>>> -bash-4.1# >>>> >>>> >>>> -boris >>> >
Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: > On 11/22/2017 05:09 AM, Christian König wrote: >> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>> On 11/21/2017 08:34 AM, Christian König wrote: >>>> Hi Boris, >>>> >>>> attached are two patches. >>>> >>>> The first one is a trivial fix for the infinite loop issue, it now >>>> correctly aborts the fixup when it can't find address space for the >>>> root window. >>>> >>>> The second is a workaround for your board. It simply checks if there >>>> is exactly one Processor Function to apply this fix on. >>>> >>>> Both are based on linus current master branch. Please test if they fix >>>> your issue. >>> Yes, they do fix it but that's because the feature is disabled. >>> >>> Do you know what the actual problem was (on Xen)? >> I still haven't understood what you actually did with Xen. >> >> When you used PCI pass through with those devices then you have made a >> major configuration error. >> >> When the problem happened on dom0 then the explanation is most likely >> that some PCI device ended up in the configured space, but the routing >> was only setup correctly on one CPU socket. > The problem is that dom0 can be (and was in my case() booted with less > than full physical memory and so the "rest" of the host memory is not > necessarily reflected in iomem. Your patch then tried to configure that > memory for MMIO and the system hang. > > And so my guess is that this patch will break dom0 on a single-socket > system as well. Oh, thanks! I've thought about that possibility before, but wasn't able to find a system which actually does that. May I ask why the rest of the memory isn't reported to the OS? Sounds like I can't trust Linux resource management and probably need to read the DRAM config to figure things out after all. Thanks a lot for this information, Christian. > > -boris > >> Regards, >> Christian. >> >>> Thanks. >>> -boris >>> >>>> Thanks for the help, >>>> Christian. >>>> >>>> Am 20.11.2017 um 17:33 schrieb Boris Ostrovsky: >>>>> On 11/20/2017 11:07 AM, Christian König wrote: >>>>>> Am 20.11.2017 um 16:51 schrieb Boris Ostrovsky: >>>>>>> (and then it breaks differently as a Xen guest --- we hung on the >>>>>>> last >>>>>>> pci_read_config_dword(), I haven't looked at this at all yet) >>>>>> Hui? How does this fix applies to a Xen guest in the first place? >>>>>> >>>>>> Please provide the output of "lspci -nn" and explain further what is >>>>>> your config with Xen. >>>>>> >>>>>> >>>>> This is dom0. >>>>> >>>>> -bash-4.1# lspci -nn >>>>> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD890 Northbridge >>>>> only >>>>> dual slot (2x16) PCI-e GFX Hydra part [1002:5a10] (rev 02) >>>>> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device >>>>> [1002:5a23] >>>>> 00:0d.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI >>>>> bridge >>>>> (external gfx1 port B) [1002:5a1e] >>>>> 00:11.0 SATA controller [0106]: ATI Technologies Inc SB700/SB800 SATA >>>>> Controller [AHCI mode] [1002:4391] >>>>> 00:12.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI0 Controller [1002:4397] >>>>> 00:12.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>>> Controller [1002:4398] >>>>> 00:12.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> EHCI >>>>> Controller [1002:4396] >>>>> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI0 Controller [1002:4397] >>>>> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB700 USB OHCI1 >>>>> Controller [1002:4398] >>>>> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> EHCI >>>>> Controller [1002:4396] >>>>> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller >>>>> [1002:4385] (rev 3d) >>>>> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB700/SB800 LPC host >>>>> controller [1002:439d] >>>>> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI >>>>> Bridge >>>>> [1002:4384] >>>>> 00:14.5 USB Controller [0c03]: ATI Technologies Inc SB700/SB800 USB >>>>> OHCI2 Controller [1002:4399] >>>>> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1600] >>>>> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1601] >>>>> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1602] >>>>> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1603] >>>>> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1604] >>>>> 00:18.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1605] >>>>> 00:19.0 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1600] >>>>> 00:19.1 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1601] >>>>> 00:19.2 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1602] >>>>> 00:19.3 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1603] >>>>> 00:19.4 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1604] >>>>> 00:19.5 Host bridge [0600]: Advanced Micro Devices [AMD] Device >>>>> [1022:1605] >>>>> 01:04.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA >>>>> G200eW WPCM450 [102b:0532] (rev 0a) >>>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>>> Network Connection [8086:10c9] (rev 01) >>>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit >>>>> Network Connection [8086:10c9] (rev 01) >>>>> -bash-4.1# >>>>> >>>>> >>>>> -boris
On 11/22/2017 11:54 AM, Christian König wrote: > Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: >> On 11/22/2017 05:09 AM, Christian König wrote: >>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>>> On 11/21/2017 08:34 AM, Christian König wrote: >>>>> Hi Boris, >>>>> >>>>> attached are two patches. >>>>> >>>>> The first one is a trivial fix for the infinite loop issue, it now >>>>> correctly aborts the fixup when it can't find address space for the >>>>> root window. >>>>> >>>>> The second is a workaround for your board. It simply checks if there >>>>> is exactly one Processor Function to apply this fix on. >>>>> >>>>> Both are based on linus current master branch. Please test if they >>>>> fix >>>>> your issue. >>>> Yes, they do fix it but that's because the feature is disabled. >>>> >>>> Do you know what the actual problem was (on Xen)? >>> I still haven't understood what you actually did with Xen. >>> >>> When you used PCI pass through with those devices then you have made a >>> major configuration error. >>> >>> When the problem happened on dom0 then the explanation is most likely >>> that some PCI device ended up in the configured space, but the routing >>> was only setup correctly on one CPU socket. >> The problem is that dom0 can be (and was in my case() booted with less >> than full physical memory and so the "rest" of the host memory is not >> necessarily reflected in iomem. Your patch then tried to configure that >> memory for MMIO and the system hang. >> >> And so my guess is that this patch will break dom0 on a single-socket >> system as well. > > Oh, thanks! > > I've thought about that possibility before, but wasn't able to find a > system which actually does that. > > May I ask why the rest of the memory isn't reported to the OS? That memory doesn't belong to the OS (dom0), it is owned by the hypervisor. > > Sounds like I can't trust Linux resource management and probably need > to read the DRAM config to figure things out after all. My question is whether what you are trying to do should ever be done for a guest at all (any guest, not necessarily Xen). -boris
Am 22.11.2017 um 18:27 schrieb Boris Ostrovsky: > On 11/22/2017 11:54 AM, Christian König wrote: >> Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: >>> On 11/22/2017 05:09 AM, Christian König wrote: >>>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>>>> On 11/21/2017 08:34 AM, Christian König wrote: >>>>>> Hi Boris, >>>>>> >>>>>> attached are two patches. >>>>>> >>>>>> The first one is a trivial fix for the infinite loop issue, it now >>>>>> correctly aborts the fixup when it can't find address space for the >>>>>> root window. >>>>>> >>>>>> The second is a workaround for your board. It simply checks if there >>>>>> is exactly one Processor Function to apply this fix on. >>>>>> >>>>>> Both are based on linus current master branch. Please test if they >>>>>> fix >>>>>> your issue. >>>>> Yes, they do fix it but that's because the feature is disabled. >>>>> >>>>> Do you know what the actual problem was (on Xen)? >>>> I still haven't understood what you actually did with Xen. >>>> >>>> When you used PCI pass through with those devices then you have made a >>>> major configuration error. >>>> >>>> When the problem happened on dom0 then the explanation is most likely >>>> that some PCI device ended up in the configured space, but the routing >>>> was only setup correctly on one CPU socket. >>> The problem is that dom0 can be (and was in my case() booted with less >>> than full physical memory and so the "rest" of the host memory is not >>> necessarily reflected in iomem. Your patch then tried to configure that >>> memory for MMIO and the system hang. >>> >>> And so my guess is that this patch will break dom0 on a single-socket >>> system as well. >> Oh, thanks! >> >> I've thought about that possibility before, but wasn't able to find a >> system which actually does that. >> >> May I ask why the rest of the memory isn't reported to the OS? > That memory doesn't belong to the OS (dom0), it is owned by the hypervisor. > >> Sounds like I can't trust Linux resource management and probably need >> to read the DRAM config to figure things out after all. > > My question is whether what you are trying to do should ever be done for > a guest at all (any guest, not necessarily Xen). The issue is probably that I don't know enough about Xen: What exactly is dom0? My understanding was that dom0 is the hypervisor, but that seems to be incorrect. The issue is that under no circumstances *EVER* a virtualized guest should have access to the PCI devices marked as "Processor Function" on AMD platforms. Otherwise it is trivial to break out of the virtualization. When dom0 is something like the system domain with all hardware access then the approach seems legitimate, but then the hypervisor should report the stolen memory to the OS using the e820 table. When the hypervisor doesn't do that and the Linux kernel isn't aware that there is memory at a given location mapping PCI space there will obviously crash the hypervisor. Possible solutions as far as I can see are either disabling this feature when we detect that we are a Xen dom0, scanning the DRAM settings to update Linux resource handling or fixing Xen to report stolen memory to the dom0 OS as reserved. Opinions? Thanks, Christian. > > -boris >
On 11/23/2017 03:11 AM, Christian König wrote: > Am 22.11.2017 um 18:27 schrieb Boris Ostrovsky: >> On 11/22/2017 11:54 AM, Christian König wrote: >>> Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: >>>> On 11/22/2017 05:09 AM, Christian König wrote: >>>>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>>>>> On 11/21/2017 08:34 AM, Christian König wrote: >>>>>>> Hi Boris, >>>>>>> >>>>>>> attached are two patches. >>>>>>> >>>>>>> The first one is a trivial fix for the infinite loop issue, it now >>>>>>> correctly aborts the fixup when it can't find address space for the >>>>>>> root window. >>>>>>> >>>>>>> The second is a workaround for your board. It simply checks if there >>>>>>> is exactly one Processor Function to apply this fix on. >>>>>>> >>>>>>> Both are based on linus current master branch. Please test if they >>>>>>> fix >>>>>>> your issue. >>>>>> Yes, they do fix it but that's because the feature is disabled. >>>>>> >>>>>> Do you know what the actual problem was (on Xen)? >>>>> I still haven't understood what you actually did with Xen. >>>>> >>>>> When you used PCI pass through with those devices then you have made a >>>>> major configuration error. >>>>> >>>>> When the problem happened on dom0 then the explanation is most likely >>>>> that some PCI device ended up in the configured space, but the routing >>>>> was only setup correctly on one CPU socket. >>>> The problem is that dom0 can be (and was in my case() booted with less >>>> than full physical memory and so the "rest" of the host memory is not >>>> necessarily reflected in iomem. Your patch then tried to configure that >>>> memory for MMIO and the system hang. >>>> >>>> And so my guess is that this patch will break dom0 on a single-socket >>>> system as well. >>> Oh, thanks! >>> >>> I've thought about that possibility before, but wasn't able to find a >>> system which actually does that. >>> >>> May I ask why the rest of the memory isn't reported to the OS? >> That memory doesn't belong to the OS (dom0), it is owned by the >> hypervisor. >> >>> Sounds like I can't trust Linux resource management and probably need >>> to read the DRAM config to figure things out after all. >> >> My question is whether what you are trying to do should ever be done for >> a guest at all (any guest, not necessarily Xen). > > The issue is probably that I don't know enough about Xen: What exactly > is dom0? My understanding was that dom0 is the hypervisor, but that > seems to be incorrect. > > The issue is that under no circumstances *EVER* a virtualized guest > should have access to the PCI devices marked as "Processor Function" on > AMD platforms. Otherwise it is trivial to break out of the virtualization. > > When dom0 is something like the system domain with all hardware access > then the approach seems legitimate, but then the hypervisor should > report the stolen memory to the OS using the e820 table. > > When the hypervisor doesn't do that and the Linux kernel isn't aware > that there is memory at a given location mapping PCI space there will > obviously crash the hypervisor. > > Possible solutions as far as I can see are either disabling this feature > when we detect that we are a Xen dom0, scanning the DRAM settings to > update Linux resource handling or fixing Xen to report stolen memory to > the dom0 OS as reserved. > > Opinions? You are right, these functions are not exposed to a regular guest. I think for dom0 (which is a special Xen guest, with additional privileges) we may be able to add a reserved e820 region for host memory that is not assigned to dom0. Let me try it on Monday (I am out until then). -boris
On 11/23/2017 09:12 AM, Boris Ostrovsky wrote: > > > On 11/23/2017 03:11 AM, Christian König wrote: >> Am 22.11.2017 um 18:27 schrieb Boris Ostrovsky: >>> On 11/22/2017 11:54 AM, Christian König wrote: >>>> Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky: >>>>> On 11/22/2017 05:09 AM, Christian König wrote: >>>>>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky: >>>>>>> On 11/21/2017 08:34 AM, Christian König wrote: >>>>>>>> Hi Boris, >>>>>>>> >>>>>>>> attached are two patches. >>>>>>>> >>>>>>>> The first one is a trivial fix for the infinite loop issue, it now >>>>>>>> correctly aborts the fixup when it can't find address space for >>>>>>>> the >>>>>>>> root window. >>>>>>>> >>>>>>>> The second is a workaround for your board. It simply checks if >>>>>>>> there >>>>>>>> is exactly one Processor Function to apply this fix on. >>>>>>>> >>>>>>>> Both are based on linus current master branch. Please test if they >>>>>>>> fix >>>>>>>> your issue. >>>>>>> Yes, they do fix it but that's because the feature is disabled. >>>>>>> >>>>>>> Do you know what the actual problem was (on Xen)? >>>>>> I still haven't understood what you actually did with Xen. >>>>>> >>>>>> When you used PCI pass through with those devices then you have >>>>>> made a >>>>>> major configuration error. >>>>>> >>>>>> When the problem happened on dom0 then the explanation is most >>>>>> likely >>>>>> that some PCI device ended up in the configured space, but the >>>>>> routing >>>>>> was only setup correctly on one CPU socket. >>>>> The problem is that dom0 can be (and was in my case() booted with >>>>> less >>>>> than full physical memory and so the "rest" of the host memory is not >>>>> necessarily reflected in iomem. Your patch then tried to configure >>>>> that >>>>> memory for MMIO and the system hang. >>>>> >>>>> And so my guess is that this patch will break dom0 on a single-socket >>>>> system as well. >>>> Oh, thanks! >>>> >>>> I've thought about that possibility before, but wasn't able to find a >>>> system which actually does that. >>>> >>>> May I ask why the rest of the memory isn't reported to the OS? >>> That memory doesn't belong to the OS (dom0), it is owned by the >>> hypervisor. >>> >>>> Sounds like I can't trust Linux resource management and probably need >>>> to read the DRAM config to figure things out after all. >>> >>> My question is whether what you are trying to do should ever be done >>> for >>> a guest at all (any guest, not necessarily Xen). >> >> The issue is probably that I don't know enough about Xen: What >> exactly is dom0? My understanding was that dom0 is the hypervisor, >> but that seems to be incorrect. >> >> The issue is that under no circumstances *EVER* a virtualized guest >> should have access to the PCI devices marked as "Processor Function" >> on AMD platforms. Otherwise it is trivial to break out of the >> virtualization. >> >> When dom0 is something like the system domain with all hardware >> access then the approach seems legitimate, but then the hypervisor >> should report the stolen memory to the OS using the e820 table. >> >> When the hypervisor doesn't do that and the Linux kernel isn't aware >> that there is memory at a given location mapping PCI space there will >> obviously crash the hypervisor. >> >> Possible solutions as far as I can see are either disabling this >> feature when we detect that we are a Xen dom0, scanning the DRAM >> settings to update Linux resource handling or fixing Xen to report >> stolen memory to the dom0 OS as reserved. >> >> Opinions? > > You are right, these functions are not exposed to a regular guest. > > I think for dom0 (which is a special Xen guest, with additional > privileges) we may be able to add a reserved e820 region for host > memory that is not assigned to dom0. Let me try it on Monday (I am out > until then). One thing I realized while looking at solution for Xen dom0 is that this patch may cause problems for memory hotplug. What happens if new memory is added to the system and we have everything above current memory set to MMIO? -boris
From 2dc4461ba8ec1eb54a49e1e166de9a554556e572 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20K=C3=B6nig?= <christian.koenig@amd.com> Date: Tue, 21 Nov 2017 11:08:33 +0100 Subject: [PATCH 2/2] x86/PCI: only enable a 64bit BAR on single socket AMD Family 15h systems MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When we have a multi socket system each CPU core needs the same setup. Since this is tricky to do in the fixup code disable enabling a 64bit BAR on multi socket systems for now. Signed-off-by: Christian König <christian.koenig@amd.com> --- arch/x86/pci/fixup.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c index 5328e86f73eb..8f86060f5cf6 100644 --- a/arch/x86/pci/fixup.c +++ b/arch/x86/pci/fixup.c @@ -665,6 +665,16 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev) unsigned i; u32 base, limit, high; struct resource *res, *conflict; + struct pci_dev *other; + + /* Check that we are the only device of that type */ + other = pci_get_device(dev->vendor, dev->device, NULL); + if (other != dev || + (other = pci_get_device(dev->vendor, dev->device, other))) { + /* This is a multi socket system, don't touch it for now */ + pci_dev_put(other); + return; + } for (i = 0; i < 8; i++) { pci_read_config_dword(dev, AMD_141b_MMIO_BASE(i), &base); @@ -719,10 +729,10 @@ static void pci_amd_enable_64bit_bar(struct pci_dev *dev) pci_bus_add_resource(dev->bus, res, 0); } -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1401, pci_amd_enable_64bit_bar); -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar); -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1571, pci_amd_enable_64bit_bar); -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar); -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1401, pci_amd_enable_64bit_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1571, pci_amd_enable_64bit_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x15b1, pci_amd_enable_64bit_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1601, pci_amd_enable_64bit_bar); #endif -- 2.11.0