| Message ID | 20120912155042.GD778@wantstofly.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi, > - Apply the patches at the end of this mail to kvm and SeaBIOS to > allow for more BAR space under 4G. (The relevant BARs on the > graphics cards _are_ 64 bit BARs, but kvm seemed to turn those > into 32 bit BARs in the guest.) Which qemu/seabios versions have you used? qemu-1.2 (+ bundled seabios) should handle that just fine without patching. There is no fixed I/O window any more, all memory space above lowmem is available for pci, i.e. if you give 2G to your guest everything above 0x80000000. And if there isn't enougth address space below 4G (if you assign lot of memory to your guest so qemu keeps only the 0xe0000000 - 0xffffffff window free) seabios should try to map 64bit bars above 4G. > - Apply the hacky patch at the end of this mail to SeaBIOS to > always skip initialising the Radeon's option ROMs, or the VM > would hang inside the Radeon option ROM if you boot the VM > without the default cirrus video. A better way to handle that would probably be to add an pci passthrough config option to not expose the rom to the guest. Any clue *why* the rom doesn't run? cheers, Gerd -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2012-09-13 07:55, Gerd Hoffmann wrote: > Hi, > >> - Apply the patches at the end of this mail to kvm and SeaBIOS to >> allow for more BAR space under 4G. (The relevant BARs on the >> graphics cards _are_ 64 bit BARs, but kvm seemed to turn those >> into 32 bit BARs in the guest.) > > Which qemu/seabios versions have you used? > > qemu-1.2 (+ bundled seabios) should handle that just fine without > patching. There is no fixed I/O window any more, all memory space above > lowmem is available for pci, i.e. if you give 2G to your guest > everything above 0x80000000. > > And if there isn't enougth address space below 4G (if you assign lot of > memory to your guest so qemu keeps only the 0xe0000000 - 0xffffffff > window free) seabios should try to map 64bit bars above 4G. > >> - Apply the hacky patch at the end of this mail to SeaBIOS to >> always skip initialising the Radeon's option ROMs, or the VM >> would hang inside the Radeon option ROM if you boot the VM >> without the default cirrus video. > > A better way to handle that would probably be to add an pci passthrough > config option to not expose the rom to the guest. -device pci-assign,option-rom=,... > > Any clue *why* the rom doesn't run? Maybe because we are not passing through the legacy VGA I/O ranges, maybe because the card is accessing one of the famous side channels to configure its mappings, and we do not virtualize them (as we usually do not know them). Jan
On Thu, Sep 13, 2012 at 07:55:00AM +0200, Gerd Hoffmann wrote: > Hi, Hi, > > - Apply the patches at the end of this mail to kvm and SeaBIOS to > > allow for more BAR space under 4G. (The relevant BARs on the > > graphics cards _are_ 64 bit BARs, but kvm seemed to turn those > > into 32 bit BARs in the guest.) > > Which qemu/seabios versions have you used? > > qemu-1.2 (+ bundled seabios) should handle that just fine without > patching. There is no fixed I/O window any more, all memory space above > lowmem is available for pci, i.e. if you give 2G to your guest > everything above 0x80000000. > > And if there isn't enougth address space below 4G (if you assign lot of > memory to your guest so qemu keeps only the 0xe0000000 - 0xffffffff > window free) seabios should try to map 64bit bars above 4G. This was some time ago, on (L)ubuntu 12.04, which has qemu-kvm 1.0 and seabios 0.6.2. We'll retry on a newer distro soon. > > - Apply the hacky patch at the end of this mail to SeaBIOS to > > always skip initialising the Radeon's option ROMs, or the VM > > would hang inside the Radeon option ROM if you boot the VM > > without the default cirrus video. > > A better way to handle that would probably be to add an pci passthrough > config option to not expose the rom to the guest. > > Any clue *why* the rom doesn't run? No idea, we didn't look into that -- this was just a one afternoon hacking session. thanks, Lennert -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2012-09-13 at 11:40 +0200, Lennert Buytenhek wrote: > On Thu, Sep 13, 2012 at 07:55:00AM +0200, Gerd Hoffmann wrote: > > > Hi, > > Hi, > > > > > - Apply the patches at the end of this mail to kvm and SeaBIOS to > > > allow for more BAR space under 4G. (The relevant BARs on the > > > graphics cards _are_ 64 bit BARs, but kvm seemed to turn those > > > into 32 bit BARs in the guest.) > > > > Which qemu/seabios versions have you used? > > > > qemu-1.2 (+ bundled seabios) should handle that just fine without > > patching. There is no fixed I/O window any more, all memory space above > > lowmem is available for pci, i.e. if you give 2G to your guest > > everything above 0x80000000. > > > > And if there isn't enougth address space below 4G (if you assign lot of > > memory to your guest so qemu keeps only the 0xe0000000 - 0xffffffff > > window free) seabios should try to map 64bit bars above 4G. > > This was some time ago, on (L)ubuntu 12.04, which has qemu-kvm 1.0 > and seabios 0.6.2. We'll retry on a newer distro soon. > > > > > - Apply the hacky patch at the end of this mail to SeaBIOS to > > > always skip initialising the Radeon's option ROMs, or the VM > > > would hang inside the Radeon option ROM if you boot the VM > > > without the default cirrus video. > > > > A better way to handle that would probably be to add an pci passthrough > > config option to not expose the rom to the guest. > > > > Any clue *why* the rom doesn't run? > > No idea, we didn't look into that -- this was just a one afternoon > hacking session. Thanks for the report. Spawned by your success, I tested a Radeon HD 5450 using VFIO based device assignment. I can get it to work on Windows XP, with no changes (from the version I'll post soon), but Win7 dies (still need to play around more with your suggestions of cpu type). For skipping the option rom, is it sufficient to not expose it (rombar=0) or does the guest OS driver need it as well? Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Sep 13, 2012 at 11:05:07AM -0600, Alex Williamson wrote: > > > > - Apply the hacky patch at the end of this mail to SeaBIOS to > > > > always skip initialising the Radeon's option ROMs, or the VM > > > > would hang inside the Radeon option ROM if you boot the VM > > > > without the default cirrus video. > > > > > > A better way to handle that would probably be to add an pci passthrough > > > config option to not expose the rom to the guest. > > > > > > Any clue *why* the rom doesn't run? > > > > No idea, we didn't look into that -- this was just a one afternoon > > hacking session. > > Thanks for the report. Spawned by your success, I tested a Radeon HD > 5450 using VFIO based device assignment. I can get it to work on > Windows XP, with no changes (from the version I'll post soon), Yay! > but Win7 dies (still need to play around more with your suggestions > of cpu type). ACK. That's a nasty one, don't ask how we found that out... Note that the bluescreen I described when the cpu type is wrong only actually happened for us if the AMD drivers were installed in the VM -- maybe you can try without AMD drivers to see if that makes it go away. If you still have a bluescreen without the AMD drivers installed, it's probably a different issue. > For skipping the option rom, is it sufficient to not expose it > (rombar=0) or does the guest OS driver need it as well? I don't actually know. Something to try out when we get round to testing this again, I suppose.. cheers, Lennert -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Lennert Buytenhek <buytenh <at> wantstofly.org> writes: > > Hi! > > Some time ago already, we got graphics card pci passthrough working > on a friend's computer, an ASRock Z68 Extreme4 Gen3 based PC with > an Intel i7-2600, running on a Linux 3.2 kernel, with two Gigabyte > GV-R677SL-1GD (AMD Radeon HD 6770) graphics cards in CrossFireX mode. > > To get things to work we had to: > > - Boot with intel_iommu=yes. This broke the onboard Marvell > 88SE9120 controller (an AHCI controller) with IOMMU related > kernel errors, and we had to move some drives around. > > - Blacklist the radeon audio driver so that the graphics cards > aren't in use on the host. > > - Apply the patches at the end of this mail to kvm and SeaBIOS to > allow for more BAR space under 4G. (The relevant BARs on the > graphics cards _are_ 64 bit BARs, but kvm seemed to turn those > into 32 bit BARs in the guest.) > > - Apply the hacky patch at the end of this mail to SeaBIOS to > always skip initialising the Radeon's option ROMs, or the VM > would hang inside the Radeon option ROM if you boot the VM > without the default cirrus video. > > With these patches, the Windows 7 guest seemed to work OK on basic > testing, allowing playing of all 3d games we tried, including in > CrossFireX mode. We ran into the following issues: > > - The AMD Catalyst graphics card drivers bluescreen the VM on win7 > boot if you boot with a CPU family/model/stepping that it does not > like and one or more Radeon cards passed through to the VM. > (Setting the guest family/model/stepping to be equal to the host's > CPU's fixes the bluescreen and allows booting to complete.) > > - AMD Catalyst 12.4 worked okay, but the 12.6 beta would bluescreen > the VM. > > - Playing video in Flash (e.g. youtube) and XBMC gives a green > rectangle where the video is supposed to be. This was fixable in > both cases by turning off hardware acceleration (in Flash) or > "dxva2" (in xbmc). A problem with these symptoms is noted in the > AMD Catalyst driver release notes, but the same problem does not > seem to appear when running Windows 7 directly on the host. > (Could this be related to skipping the option ROM?) > > We also tried passing through an NVidia graphics card (GTX 260) > instead of the Radeons, but that would crash the VM the couple of > times we tried, and we didn't look into this further. > > The setup (display on the right assigned to the Linux host, display > on the left assigned to the Windows guest): http://i.imgur.com/idtVH.jpg > > 3DMark running on Windows host: http://i.imgur.com/nvbQr.jpg > > 3DMark running on Windows as a VM: http://i.imgur.com/vuL7H.jpg > > (Running with only one graphics card gives about half the 3dmark > score, so CrossFireX does appear to be working in both cases.) > > cheers, > Lennert > > KVM patch for more <4G PCI BAR space: > > diff -ur qemu-kvm-1.0+noroms/hw/pc_piix.c ../qemu-kvm-1.0+noroms/hw/pc_piix.c > --- qemu-kvm-1.0+noroms/hw/pc_piix.c 2011-12-04 11:38:06.000000000 +0100 > +++ ../qemu-kvm-1.0+noroms/hw/pc_piix.c 2012-06-09 16:24:14.186327304 +0200 > @@ -112,9 +112,9 @@ > kvmclock_create(); > } > > - if (ram_size >= 0xe0000000 ) { > - above_4g_mem_size = ram_size - 0xe0000000; > - below_4g_mem_size = 0xe0000000; > + if (ram_size >= 0xc0000000 ) { > + above_4g_mem_size = ram_size - 0xc0000000; > + below_4g_mem_size = 0xc0000000; > } else { > above_4g_mem_size = 0; > below_4g_mem_size = ram_size; > > Corresponding SeaBIOS patch: > > diff --git a/src/acpi-dsdt.dsl b/src/acpi-dsdt.dsl > index 15299ee..31e7102 100644 > --- a/src/acpi-dsdt.dsl > +++ b/src/acpi-dsdt.dsl > @@ -170,10 +170,10 @@ DefinitionBlock ( > ,, , AddressRangeMemory, TypeStatic) > DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, > 0x00000000, // Address Space Granularity > - 0xE0000000, // Address Range Minimum > + 0xC0000000, // Address Range Minimum > 0xFEBFFFFF, // Address Range Maximum > 0x00000000, // Address Translation Offset > - 0x1EC00000, // Address Length > + 0x3EC00000, // Address Length > ,, PW32, AddressRangeMemory, TypeStatic) > }) > Name (CR64, ResourceTemplate () > diff --git a/src/config.h b/src/config.h > index 3a70867..b305d9a 100644 > --- a/src/config.h > +++ b/src/config.h > @@ -44,9 +44,9 @@ > #define BUILD_SMM_INIT_ADDR 0x38000 > #define BUILD_SMM_ADDR 0xa8000 > #define BUILD_SMM_SIZE 0x8000 > -#define BUILD_MAX_HIGHMEM 0xe0000000 > +#define BUILD_MAX_HIGHMEM 0xc0000000 > > -#define BUILD_PCIMEM_START 0xe0000000 > +#define BUILD_PCIMEM_START 0xc0000000 > #define BUILD_PCIMEM_END 0xfec00000 /* IOAPIC is mapped at */ > #define BUILD_PCIMEM64_START 0x8000000000ULL > #define BUILD_PCIMEM64_END 0x10000000000ULL > > SeaBIOS patch to always skip Radeon option ROM initialization: > > diff --git a/src/optionroms.c b/src/optionroms.c > index 00697b2..857956a 100644 > --- a/src/optionroms.c > +++ b/src/optionroms.c > @@ -440,6 +440,8 @@ vga_setup(void) > foreachpci(pci) { > if (!is_pci_vga(pci)) > continue; > + if (pci->vendor == 0x1002 && pci->device == 0x68ba) > + continue; > vgahook_setup(pci); > init_pcirom(pci, 1, NULL); > break; > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo <at> vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hi There, I just wanted to post a follow up to this to say I've also managed to get a reasonably stable system with a similar setup. Hardware: - asrock q77m vPro (with vt-d enabled) - core i7 3770 - sapphire radeon 7850 2gb Software inside guest - windows 8 64bit - standard/home edition - radeon catalyst drivers 12-10_vista_win7_win8_64_dd_cc_whql_net4 host OS: - ubuntu 12.10, booted with intel_iommu=yes - blacklisted radeon - stock kvm, libvirt, qemu - seabios recompiled to latest and including the below optionroms.c patch (modified for me device id), not sure if the patch has made any difference though. - sapphire rom downloaded from http://www.techpowerup.com/vgabios/ - libvirt / qemu config updated to include the rom in the PCI passthrough section: <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/etc/libvirt/roms/Sapphire.HD7850.2048.120306_1.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> What i've observed: * my original configuration used the stock seabios and without the rom file directive, this would mostly startup windows 8 fine from a cold boot of the host. restarting win8 would lead to the host locking up completely most of the time, nothing reported on the console for the kernel. * as above, but shutting down windows 8 and then restarting it would work for about the first three attempts, the fourth would cause the vm to goto 100% cpu, powering off the vm and restarting would lock the host After applying the seabios change and including the rom file the system appears to be much more stable. I have managed to shutdown and restart win8 a number of times without rebooting the host. Probably so far tried about 6 or 7 shutdown/starts in a row. again with the seabios and rom file changes, restarting windows8 seems to cause lockups. HTH, -vince -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff -ur qemu-kvm-1.0+noroms/hw/pc_piix.c ../qemu-kvm-1.0+noroms/hw/pc_piix.c --- qemu-kvm-1.0+noroms/hw/pc_piix.c 2011-12-04 11:38:06.000000000 +0100 +++ ../qemu-kvm-1.0+noroms/hw/pc_piix.c 2012-06-09 16:24:14.186327304 +0200 @@ -112,9 +112,9 @@ kvmclock_create(); } - if (ram_size >= 0xe0000000 ) { - above_4g_mem_size = ram_size - 0xe0000000; - below_4g_mem_size = 0xe0000000; + if (ram_size >= 0xc0000000 ) { + above_4g_mem_size = ram_size - 0xc0000000; + below_4g_mem_size = 0xc0000000; } else { above_4g_mem_size = 0; below_4g_mem_size = ram_size; Corresponding SeaBIOS patch: diff --git a/src/acpi-dsdt.dsl b/src/acpi-dsdt.dsl index 15299ee..31e7102 100644 --- a/src/acpi-dsdt.dsl +++ b/src/acpi-dsdt.dsl @@ -170,10 +170,10 @@ DefinitionBlock ( ,, , AddressRangeMemory, TypeStatic) DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, 0x00000000, // Address Space Granularity - 0xE0000000, // Address Range Minimum + 0xC0000000, // Address Range Minimum 0xFEBFFFFF, // Address Range Maximum 0x00000000, // Address Translation Offset - 0x1EC00000, // Address Length + 0x3EC00000, // Address Length ,, PW32, AddressRangeMemory, TypeStatic) }) Name (CR64, ResourceTemplate () diff --git a/src/config.h b/src/config.h index 3a70867..b305d9a 100644 --- a/src/config.h +++ b/src/config.h @@ -44,9 +44,9 @@ #define BUILD_SMM_INIT_ADDR 0x38000 #define BUILD_SMM_ADDR 0xa8000 #define BUILD_SMM_SIZE 0x8000 -#define BUILD_MAX_HIGHMEM 0xe0000000 +#define BUILD_MAX_HIGHMEM 0xc0000000 -#define BUILD_PCIMEM_START 0xe0000000 +#define BUILD_PCIMEM_START 0xc0000000 #define BUILD_PCIMEM_END 0xfec00000 /* IOAPIC is mapped at */ #define BUILD_PCIMEM64_START 0x8000000000ULL #define BUILD_PCIMEM64_END 0x10000000000ULL SeaBIOS patch to always skip Radeon option ROM initialization: diff --git a/src/optionroms.c b/src/optionroms.c index 00697b2..857956a 100644 --- a/src/optionroms.c +++ b/src/optionroms.c @@ -440,6 +440,8 @@ vga_setup(void) foreachpci(pci) { if (!is_pci_vga(pci)) continue; + if (pci->vendor == 0x1002 && pci->device == 0x68ba) + continue; vgahook_setup(pci); init_pcirom(pci, 1, NULL); break;