| Message ID | 1304956091.26106.39.camel@x201 (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 2011-05-09 17:48, Alex Williamson wrote: > On Mon, 2011-05-09 at 17:27 +0200, Jan Kiszka wrote: >> On 2011-05-09 16:55, Prasad Joshi wrote: >>> On Mon, May 9, 2011 at 12:14 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote: >>>> On 2011-05-05 17:17, Alex Williamson wrote: >>>>>> And what about the host? When does Linux release the legacy range? >>>>>> Always or only when a specific (!=vga/vesa) framebuffer driver is loaded? >>>>> >>>>> Well, that's where it'd be nice if the vga arbiter was actually in more >>>>> widespread use. It currently seems to be nothing more than a shared >>>>> mutex, but it would actually be useful if it included backends to do the >>>>> chipset vga routing changes. I think when I was testing this, I was >>>>> externally poking PCI bridge chipset to toggle the VGA_EN bit. >>>> >>>> Right, we had to drop the approach to pass through the secondary card >>>> for now, the arbiter was not switching properly. Haven't checked yet if >>>> VGA_EN was properly set, though the kernel code looks like it should >>>> take care of this. >>>> >>>> Even with handing out the primary adapter, we had only mixed success so >>>> far. The onboard adapter worked well (in VESA mode), but the NVIDIA is >>>> not displaying early boot messages at all. Maybe a vgabios issue. >>>> Windows was booting nevertheless - until we installed the NVIDIA >>>> drivers. Than it ran into a blue screen. >>>> >>>> BTW, what ATI adapter did you use precisely, and what did work, what not? >>> >>> Not hijacking the mail thread. Just wanted to provide some inputs. >> >> Much appreciated in fact! >> >>> >>> Few days back I had tried passing through the secondary graphics card. >>> I could pass-through two graphics cards to virtual machine. >>> >>> 02:00.0 VGA compatible controller: ATI Technologies Inc Redwood >>> [Radeon HD 5670] (prog-if 00 [VGA controller]) >>> Subsystem: PC Partner Limited Device e151 >>> Flags: bus master, fast devsel, latency 0, IRQ 87 >>> Memory at d0000000 (64-bit, prefetchable) [size=256M] >>> Memory at fe6e0000 (64-bit, non-prefetchable) [size=128K] >>> I/O ports at b000 [size=256] >>> Expansion ROM at fe6c0000 [disabled] [size=128K] >>> Capabilities: <access denied> >>> Kernel driver in use: radeon >>> Kernel modules: radeon >>> >>> 07:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS >>> 290] (rev a1) (prog-if 00 [VGA controller]) >>> Subsystem: nVidia Corporation Device 0492 >>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>> ParErr-Stepping- SERR+ FastB2B- DisINTx- >>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >>>> TAbort-<TAbort- <MAbort- >SERR- <PERR- INTx- >>> Latency: 0, Cache Line Size: 64 bytes >>> Interrupt: pin A routed to IRQ 24 >>> Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M] >>> Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M] >>> Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M] >>> Region 5: I/O ports at ec00 [size=128] >>> Expansion ROM at fe9e0000 [disabled] [size=128K] >>> Capabilities: <access denied> >>> Kernel driver in use: nouveau >>> Kernel modules: nouveau, nvidiafb >>> >>> Both of them are PCIe cards. I have one more ATI card and another >>> NVIDIA card which does not work. >> >> Interesting. That may rule out missing PCIe capabilities as source for >> the NVIDIA driver indisposition. >> >> Did you passed those cards each as primary to the guest, or was the >> guest seeing multiple adapters? I presume you only got output after >> early boot was completed, right? >> >> To avoid having to deal with legacy I/O forwarding, we started with a >> dual adapter setup in the hope to leave the primary guest adapter at >> know-to-work cirrus-vga. But already in a native setup with on-board >> primary + NVIDIA secondary, the NVIDIA Windows drivers refused to talk >> to its hardware in this constellation. >> >>> >>> One of the reason the pass-through did not work is because of the >>> limit on amount of pci configuration memory by SeaBIOS. SeaBIOS places >>> a hard limit of 256MB or so on the amount of PCI memory space. Thus, >>> for some of the VGA device that need more memory never worked for me. >>> >>> SeaBIOS allows this memory region to be extended to some value near >>> 512MB, but even then the range is not enough. >>> >>> Another problem with SeaBIOS which limits the amount of memory space >>> is: SeaBIOS allocates the BAR regions as they are encountered. As far >>> as I know, the BAR regions should be naturally aligned. Thus the >>> simple strategy of the SeaBIOS results in large fragmentation. >>> Therefore, even after increasing the PCI memory space to 512MB the BAR >>> regions were unallocated. >> >> That's an interesting trace! We'll check this here, but I bet it >> contributes to the problems. Our FX 3800 has 1G memory... > > Yes, qemu leaves far too little MMIO space to think about assigning > graphics cards. Both of my cards have 512MB and I hacked qemu to leave > a bigger gap via something like: > > diff --git a/hw/pc.c b/hw/pc.c > index 0ea6d10..a6376f8 100644 > --- a/hw/pc.c > +++ b/hw/pc.c > @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model) > } > } > > +#define PC_MAX_LOW_RAM 0xc0000000 > + > void pc_memory_init(ram_addr_t ram_size, > const char *kernel_filename, > const char *kernel_cmdline, > @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size, > int bios_size, isa_bios_size; > void *fw_cfg; > > - if (ram_size >= 0xe0000000 ) { > - above_4g_mem_size = ram_size - 0xe0000000; > - below_4g_mem_size = 0xe0000000; > + if (ram_size >= PC_MAX_LOW_RAM ) { > + above_4g_mem_size = ram_size - PC_MAX_LOW_RAM; > + below_4g_mem_size = PC_MAX_LOW_RAM; > } else { > below_4g_mem_size = ram_size; > } > > There's also a #define that needs to be changed in seabios config.h and > and acpi dsdt update, but I can't seem to find patches for those. Hmm, as this does not scale with the constantly growing memory sizes of GPUs, I guess this should would for us as well, even with 1G. The adapters likely only map a window to their on-board RAM. > Also > pay attention to the cpu_register_physical_memory calls in > i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO > range from you. I just commented them out: > > diff --git a/hw/piix_pci.c b/hw/piix_pci.c > index b5589b9..1327563 100644 > --- a/hw/piix_pci.c > +++ b/hw/piix_pci.c > @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d) > } > smram = d->dev.config[I440FX_SMRAM]; > if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) { > - cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); > + //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); > } else { > for(addr = 0xa0000; addr < 0xc0000; addr += 4096) { > - cpu_register_physical_memory(addr, 4096, > - d->isa_page_descs[(addr - 0xa0000) >> 12]); > + //cpu_register_physical_memory(addr, 4096, > + // d->isa_page_descs[(addr - 0xa0000) >> 12]); > } > } > } > > That's all the tricks I remember. Thanks, Yeah, we are already carrying half of the above in our tree (only the second disabling is actually needed, KVM does not support SMM). I started looking into fixing PAM/SMRAM mess, but it's not yet beautiful - partly because we urgently need slot management at core level. Jan
On 05/09/2011 06:48 PM, Alex Williamson wrote: > > That's an interesting trace! We'll check this here, but I bet it > > contributes to the problems. Our FX 3800 has 1G memory... > > Yes, qemu leaves far too little MMIO space to think about assigning > graphics cards. Both of my cards have 512MB and I hacked qemu to leave > a bigger gap via something like: > What about 64-bit BARs?
On 2011-05-11 13:25, Avi Kivity wrote: > On 05/09/2011 06:48 PM, Alex Williamson wrote: >>> That's an interesting trace! We'll check this here, but I bet it >>> contributes to the problems. Our FX 3800 has 1G memory... >> >> Yes, qemu leaves far too little MMIO space to think about assigning >> graphics cards. Both of my cards have 512MB and I hacked qemu to leave >> a bigger gap via something like: >> > > What about 64-bit BARs? Aren't they backward compatible? Or do you think some guest drivers may assume to find their 64-bit capable bars also registered as such and get upset when seeing them as 32-bit ones? Jan
On 05/11/2011 04:08 PM, Jan Kiszka wrote: > On 2011-05-11 13:25, Avi Kivity wrote: > > On 05/09/2011 06:48 PM, Alex Williamson wrote: > >>> That's an interesting trace! We'll check this here, but I bet it > >>> contributes to the problems. Our FX 3800 has 1G memory... > >> > >> Yes, qemu leaves far too little MMIO space to think about assigning > >> graphics cards. Both of my cards have 512MB and I hacked qemu to leave > >> a bigger gap via something like: > >> > > > > What about 64-bit BARs? > > Aren't they backward compatible? Or do you think some guest drivers may > assume to find their 64-bit capable bars also registered as such and get > upset when seeing them as 32-bit ones? > I mean, if you have a 1GB framebuffer, put it above 4GB and hope the OS/driver can handle it.
On 2011-05-11 15:26, Avi Kivity wrote: > On 05/11/2011 04:08 PM, Jan Kiszka wrote: >> On 2011-05-11 13:25, Avi Kivity wrote: >>> On 05/09/2011 06:48 PM, Alex Williamson wrote: >>>>> That's an interesting trace! We'll check this here, but I bet it >>>>> contributes to the problems. Our FX 3800 has 1G memory... >>>> >>>> Yes, qemu leaves far too little MMIO space to think about assigning >>>> graphics cards. Both of my cards have 512MB and I hacked qemu to leave >>>> a bigger gap via something like: >>>> >>> >>> What about 64-bit BARs? >> >> Aren't they backward compatible? Or do you think some guest drivers may >> assume to find their 64-bit capable bars also registered as such and get >> upset when seeing them as 32-bit ones? >> > > I mean, if you have a 1GB framebuffer, put it above 4GB and hope the > OS/driver can handle it. The question is if the drivers actually depend on this. At least the binary nvidia thing here on my notebook, it is obviously happy with below-4G-bars (and likely change the mapped window on demand): 01:00.0 VGA compatible controller: nVidia Corporation GT216 [Quadro FX 880M] (rev a2) (prog-if 00 [VGA controller]) Subsystem: Fujitsu Limited. Device 1584 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at cc000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at ce000000 (64-bit, prefetchable) [size=32M] I/O ports at 2000 [size=128] [virtual] Expansion ROM at cd000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nvidia Maybe the crashing Windows driver of the FX3800 has different requirements. Jan
On 05/11/2011 04:50 PM, Jan Kiszka wrote: > On 2011-05-11 15:26, Avi Kivity wrote: > > On 05/11/2011 04:08 PM, Jan Kiszka wrote: > >> On 2011-05-11 13:25, Avi Kivity wrote: > >>> On 05/09/2011 06:48 PM, Alex Williamson wrote: > >>>>> That's an interesting trace! We'll check this here, but I bet it > >>>>> contributes to the problems. Our FX 3800 has 1G memory... > >>>> > >>>> Yes, qemu leaves far too little MMIO space to think about assigning > >>>> graphics cards. Both of my cards have 512MB and I hacked qemu to leave > >>>> a bigger gap via something like: > >>>> > >>> > >>> What about 64-bit BARs? > >> > >> Aren't they backward compatible? Or do you think some guest drivers may > >> assume to find their 64-bit capable bars also registered as such and get > >> upset when seeing them as 32-bit ones? > >> > > > > I mean, if you have a 1GB framebuffer, put it above 4GB and hope the > > OS/driver can handle it. > > The question is if the drivers actually depend on this. At least the > binary nvidia thing here on my notebook, it is obviously happy with > below-4G-bars (and likely change the mapped window on demand): > > 01:00.0 VGA compatible controller: nVidia Corporation GT216 [Quadro FX 880M] (rev a2) (prog-if 00 [VGA controller]) > Subsystem: Fujitsu Limited. Device 1584 > Flags: bus master, fast devsel, latency 0, IRQ 16 > Memory at cc000000 (32-bit, non-prefetchable) [size=16M] > Memory at d0000000 (64-bit, prefetchable) [size=256M] > Memory at ce000000 (64-bit, prefetchable) [size=32M] > I/O ports at 2000 [size=128] > [virtual] Expansion ROM at cd000000 [disabled] [size=512K] > Capabilities: [60] Power Management version 3 > Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ > Capabilities: [78] Express Endpoint, MSI 00 > Capabilities: [b4] Vendor Specific Information: Len=14<?> > Capabilities: [100] Virtual Channel > Capabilities: [128] Power Budgeting<?> > Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024<?> > Kernel driver in use: nvidia > > Maybe the crashing Windows driver of the FX3800 has different > requirements. I doubt it. A 64-bit BAR would be configured as 32-bit on an older BIOS, no? I'd guess 64-bit BARs are only needed for large BARs.
On 2011-05-11 15:54, Avi Kivity wrote: > On 05/11/2011 04:50 PM, Jan Kiszka wrote: >> On 2011-05-11 15:26, Avi Kivity wrote: >>> On 05/11/2011 04:08 PM, Jan Kiszka wrote: >>>> On 2011-05-11 13:25, Avi Kivity wrote: >>>>> On 05/09/2011 06:48 PM, Alex Williamson wrote: >>>>>>> That's an interesting trace! We'll check this here, but I bet it >>>>>>> contributes to the problems. Our FX 3800 has 1G memory... >>>>>> >>>>>> Yes, qemu leaves far too little MMIO space to think about assigning >>>>>> graphics cards. Both of my cards have 512MB and I hacked qemu to leave >>>>>> a bigger gap via something like: >>>>>> >>>>> >>>>> What about 64-bit BARs? >>>> >>>> Aren't they backward compatible? Or do you think some guest drivers may >>>> assume to find their 64-bit capable bars also registered as such and get >>>> upset when seeing them as 32-bit ones? >>>> >>> >>> I mean, if you have a 1GB framebuffer, put it above 4GB and hope the >>> OS/driver can handle it. >> >> The question is if the drivers actually depend on this. At least the >> binary nvidia thing here on my notebook, it is obviously happy with >> below-4G-bars (and likely change the mapped window on demand): >> >> 01:00.0 VGA compatible controller: nVidia Corporation GT216 [Quadro FX 880M] (rev a2) (prog-if 00 [VGA controller]) >> Subsystem: Fujitsu Limited. Device 1584 >> Flags: bus master, fast devsel, latency 0, IRQ 16 >> Memory at cc000000 (32-bit, non-prefetchable) [size=16M] >> Memory at d0000000 (64-bit, prefetchable) [size=256M] >> Memory at ce000000 (64-bit, prefetchable) [size=32M] >> I/O ports at 2000 [size=128] >> [virtual] Expansion ROM at cd000000 [disabled] [size=512K] >> Capabilities: [60] Power Management version 3 >> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ >> Capabilities: [78] Express Endpoint, MSI 00 >> Capabilities: [b4] Vendor Specific Information: Len=14<?> >> Capabilities: [100] Virtual Channel >> Capabilities: [128] Power Budgeting<?> >> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024<?> >> Kernel driver in use: nvidia >> >> Maybe the crashing Windows driver of the FX3800 has different >> requirements. > > I doubt it. A 64-bit BAR would be configured as 32-bit on an older > BIOS, no? > > I'd guess 64-bit BARs are only needed for large BARs. > The BIOS can't configure the bars to 64 bit as it does not know which type of OS (32 or 64 bits) is going to pick them up. But maybe 64-bit Windows reconfigures the bars before it starts the driver. Would we support this? Jan
On 05/11/2011 05:06 PM, Jan Kiszka wrote: > > > > I doubt it. A 64-bit BAR would be configured as 32-bit on an older > > BIOS, no? > > > > I'd guess 64-bit BARs are only needed for large BARs. > > > > The BIOS can't configure the bars to 64 bit as it does not know which > type of OS (32 or 64 bits) is going to pick them up. If it's a really large BAR, it has no choice. BTW, a 32-bit OS can handle 64-bit BARs, all it needs is PAE or PSE-36. > But maybe 64-bit > Windows reconfigures the bars before it starts the driver. Would we > support this? Yes. qemu doesn't know if it's the BIOS reprogramming the BARs or the OS. Of course unmap+remap is not a heavily tested code path.
diff --git a/hw/pc.c b/hw/pc.c index 0ea6d10..a6376f8 100644 --- a/hw/pc.c +++ b/hw/pc.c @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model) } } +#define PC_MAX_LOW_RAM 0xc0000000 + void pc_memory_init(ram_addr_t ram_size, const char *kernel_filename, const char *kernel_cmdline, @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size, int bios_size, isa_bios_size; void *fw_cfg; - if (ram_size >= 0xe0000000 ) { - above_4g_mem_size = ram_size - 0xe0000000; - below_4g_mem_size = 0xe0000000; + if (ram_size >= PC_MAX_LOW_RAM ) { + above_4g_mem_size = ram_size - PC_MAX_LOW_RAM; + below_4g_mem_size = PC_MAX_LOW_RAM; } else { below_4g_mem_size = ram_size; } There's also a #define that needs to be changed in seabios config.h and and acpi dsdt update, but I can't seem to find patches for those. Also pay attention to the cpu_register_physical_memory calls in i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO range from you. I just commented them out: diff --git a/hw/piix_pci.c b/hw/piix_pci.c index b5589b9..1327563 100644 --- a/hw/piix_pci.c +++ b/hw/piix_pci.c @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d) } smram = d->dev.config[I440FX_SMRAM]; if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) { - cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); + //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); } else { for(addr = 0xa0000; addr < 0xc0000; addr += 4096) { - cpu_register_physical_memory(addr, 4096, - d->isa_page_descs[(addr - 0xa0000) >> 12]); + //cpu_register_physical_memory(addr, 4096, + // d->isa_page_descs[(addr - 0xa0000) >> 12]); } } }