Patches for Nvidia GPU passthrough
diff mbox

Message ID a4bddcc46336a1076ed7189b264bf673@wetwa.re
State New, archived
Headers show

Commit Message

marcus@wetwa.re July 14, 2016, 8:19 p.m. UTC
Hello all,

I've been working on enabling passthrough for newer Nvidia cards and 
drivers (GTX 980 specifically) on Xen and I'd like to document my 
findings up to now and ask for assistance. I apologize if this is not 
the correct mailing list, but I thought xen-devel is more suitable since 
we are talking about code changes in Xen anyway.

Problem with Nvidia GPUs has been (for two years now) that drivers 
detect it being running inside a VM and refuse to work (Code 43 error) 
if the card is not a Quadro or other high-end non-consumer grade GPU 
(though few other things could cause Code 43 or BSOD also). Now, since 
KVM has supported Nvidia GPU passthrough for quite a while (and I've 
personally succeeded in passing through GTX 980 using KVM on both Win 7 
and Win 8.1 VM's), I decided to port those few patches from KVM to Xen.

------------------


#### Patch #1: Spoof Xen and Hypervisor signatures:

KVM has for a while supported hiding both the "KVMKVMKVMKVM" signature 
(with "-cpu kvm=off" flag) as well as the Viridian hypervisor signature 
("-cpu hv_vendor_id="..." flag). Currently there's no such functionality 
in Xen, so I patched it in quite similar way to what Alex Willimson did 
for KVM.

Attached is a patch for Xen 4.6.1 that spoofs Xen signature 
("XenVMMXenVMM" to "ZenZenZenZen") and Viridian signature ("Microsoft 
Hv" to "Wetware Labs") when "spoof_xen=1" and "spoof_viridian=1" are 
added to VM configuration file.

The signatures are currently hard-coded, and currently there's no way to 
modify them (beyond re-compiling Xen), since HVMLoader also uses a 
hard-coded string to detect Xen and there's no API (understandably) to 
change that signature in real-time.

This works with qemu-xen-traditional without any additional changes, but 
qemu-xen requires that SeaBIOS is patched as well: 
https://github.com/WetwareLabs/seabios/commit/ec102d72fc1d7b2e6c8e9607266dc9bd4a42bce0

With spoofing on, it was possible to use official binary drivers from 
NVidia (tested version 367.27) on Arch Linux VM (without spoofing the 
driver would fail with a error message such as "The NVIDIA GPU at 
PCI:0:5:0 is not supported by the 367.27 NVIDIA driver".  However this 
was not enough on Windows VM's, as the Code 43 would occur regardless of 
spoofing.


#### Patch #2: Disable NoSnoop.

Background information and the related patch for KVM is here: 
https://patchwork.kernel.org/patch/3019371/

The fix was quite simple for Xen: Just modify the initial PCIe DEVCTL 
capabilities to disable NoSnoop, and make the capability read-only. 
Double-checking with Linux VM, I can see that NoSnoop is disabled for 
all devices (with lspci -vvv), but this would not prevent Code 43 on 
Windows VM.

##### Patch #3: Set CPUID to Core2duo

There have been few reports where forcing CPUID to Core2duo on KVM (-cpu 
core2duo) would help alleviate Code 43 problems (and also increase 
compatibility with Windows 10 VMs), so I copied all CPUID registers from 
proven-to-be-working KVM configuration using libcpuid 
(https://github.com/anrieff/libcpuid) and applied them to Xen VM. LibXL 
is also patched (attached file) to allow hexadecimal input of CPUID (to 
make it easier to convert CPUID output from libcpuid).

cpuid = [
          '0:eax=0000000a,ebx=756e6547,ecx=6c65746e,edx=49656e69',
          '1:eax=000006fb,ebx=00000800,ecx=80202201,edx=0f8bfbff',
          '2:eax=00000001,ebx=00000000,ecx=00000000,edx=002c307d',
          '3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
          '4,0:eax=00000121,ebx=01c0003f,ecx=0000003f,edx=00000001',
          '4,1:eax=00000122,ebx=01c0003f,ecx=0000003f,edx=00000001',
          '4,2:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
          '4,3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
          '5:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
          '6:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
        '7,0:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
         
'0x80000000:eax=80000008,ebx=756e6547,ecx=6c65746e,edx=49656e69',
         
'0x80000001:eax=000006fb,ebx=00000000,ecx=00000001,edx=20100800',
         
'0x80000002:eax=65746e49,ebx=2952286c,ecx=726f4320,edx=4d542865',
         
'0x80000003:eax=44203229,ebx=43206f75,ecx=20205550,edx=54202020',
         
'0x80000004:eax=30303737,ebx=20402020,ecx=30342e32,edx=007a4847',
         
'0x80000005:eax=01ff01ff,ebx=01ff01ff,ecx=40020140,edx=40020140',
         
'0x80000006:eax=00000000,ebx=42004200,ecx=02008140,edx=00000000',
         
'0x80000007:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
         '0x80000008:eax=00003028,ebx=00000000,ecx=00000000,edx=00000000'
         ]

This makes the /proc/cpuinfo almost identical between KVM and Xen VMs 
running Linux. Only exceptions are flags "rep_good" (which is missing 
under Xen) and "eager_fpu" and "xsaveopt" (not seen under KVM), but as 
these are not explicitly set by CPUID but are Linux-specific flags, they 
shouldn't (?) matter on Windows VMs.

------------------

Anyway, even applying all of these patches would not alleviate Code 43. 
To be more specific, all NVidia drivers up to 364.72 would BSOD on boot 
(SYSTEM_SERVICE_EXCEPTION), and newer drivers (368.22+) would cause Code 
43. This happens on both Windows 7 Pro and 8.1 VMs. Result on qemu-xen 
and -traditional is identical. Dom0 is Qubes 3.1 (Linux 4.1.24), Xen 
4.6.1. Hardware: Intel i7-5820K, Asrock X99 WS motherboard, 32GB Corsair 
mem, EVGA GTX980.

I would love if some of you could try these patches with both newer and 
older NVidia cards. Also any suggestions, ideas and further patches 
would be greatly appreciated! :)

Thanks!

Best regards,
Marcus

Comments

Andrew Cooper July 14, 2016, 9:34 p.m. UTC | #1
On 14/07/2016 21:19, marcus@wetwa.re wrote:
> Hello all,
>
> I've been working on enabling passthrough for newer Nvidia cards and
> drivers (GTX 980 specifically) on Xen and I'd like to document my
> findings up to now and ask for assistance. I apologize if this is not
> the correct mailing list, but I thought xen-devel is more suitable
> since we are talking about code changes in Xen anyway.

This is definitely the right place, and I think a lot of users would be
very happy if this started working.

I think it is fine to accept patches which increase the flexibility of
configuration available to the VM administrator, although this is this
is inherently a cat-and-mouse game.

To answer your last question first, I suspect you will have a far easier
time getting things working by turning off all telltail signs of
virtualisation, getting things working, then slowly turning the options
back on to gain sensible performance back.

Off the top of my head, telltail signs include
* The hypervisor bit in cpuid
* The Xen cpuid leaves
* Other leaves, such as viridian
* Xen strings in the SMBIOS and ACPI tables
* The Xen platform PCI device

>
> Problem with Nvidia GPUs has been (for two years now) that drivers
> detect it being running inside a VM and refuse to work (Code 43 error)
> if the card is not a Quadro or other high-end non-consumer grade GPU
> (though few other things could cause Code 43 or BSOD also). Now, since
> KVM has supported Nvidia GPU passthrough for quite a while (and I've
> personally succeeded in passing through GTX 980 using KVM on both Win
> 7 and Win 8.1 VM's), I decided to port those few patches from KVM to Xen.

For the patches themselves, please read
http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches

Particularly, you need a Signed-off-by tag and for the patches to be
based on the staging branch.

>
> ------------------
>
>
> #### Patch #1: Spoof Xen and Hypervisor signatures:
>
> KVM has for a while supported hiding both the "KVMKVMKVMKVM" signature
> (with "-cpu kvm=off" flag) as well as the Viridian hypervisor
> signature ("-cpu hv_vendor_id="..." flag). Currently there's no such
> functionality in Xen, so I patched it in quite similar way to what
> Alex Willimson did for KVM.

Viridian can be turned off completely.  There is some libxc code which
seems to suggest that setting 0x40000000:eax=0 should be able to
restrict the Xen leaves, but I have never seen anyone actually try that.

>
> Attached is a patch for Xen 4.6.1 that spoofs Xen signature
> ("XenVMMXenVMM" to "ZenZenZenZen") and Viridian signature ("Microsoft
> Hv" to "Wetware Labs") when "spoof_xen=1" and "spoof_viridian=1" are
> added to VM configuration file.

Simply hacking these like this not ok upstream.  In particular, I don't
think we can reasonably take changes like those to the public header
files.  (Also, be careful with Zen - AMD's new range of processors is
called Zen/Zenith).

Wiring things up so a user can specify
0x40000000:ebx=0x5A6e655A,ecx=0x655A6e65,edx=0x6e655A6e would be ok though.

>
> The signatures are currently hard-coded, and currently there's no way
> to modify them (beyond re-compiling Xen), since HVMLoader also uses a
> hard-coded string to detect Xen and there's no API (understandably) to
> change that signature in real-time.
>
> This works with qemu-xen-traditional without any additional changes,
> but qemu-xen requires that SeaBIOS is patched as well:
> https://github.com/WetwareLabs/seabios/commit/ec102d72fc1d7b2e6c8e9607266dc9bd4a42bce0
>
> With spoofing on, it was possible to use official binary drivers from
> NVidia (tested version 367.27) on Arch Linux VM (without spoofing the
> driver would fail with a error message such as "The NVIDIA GPU at
> PCI:0:5:0 is not supported by the 367.27 NVIDIA driver".  However this
> was not enough on Windows VM's, as the Code 43 would occur regardless
> of spoofing.

That is very interesting to know.  I presume it will be the same for
other Linux distros?

>
>
> #### Patch #2: Disable NoSnoop.
>
> Background information and the related patch for KVM is here:
> https://patchwork.kernel.org/patch/3019371/
>
> The fix was quite simple for Xen: Just modify the initial PCIe DEVCTL
> capabilities to disable NoSnoop, and make the capability read-only.
> Double-checking with Linux VM, I can see that NoSnoop is disabled for
> all devices (with lspci -vvv), but this would not prevent Code 43 on
> Windows VM.

Is there any more information behind this?  In principle, having the
ability to prevent NoSnoop accesses is good, but it is definitely not as
clear cut as that.  Some IOMMUs have the ability to force a NoSnoop
access to be snooped, which is typically a better solution than playing
with guest-visible config space.  (Of course, this doesn't work if the
IOMMU doesn't have the force snoop feature.)

It seems curious that forcibly preventing NoSnoop fixes Code 43.  It
sounds like it is a fairly good hint that virtualisation is going on. 
NoSnoop transactions are certainly more efficient than snooped ones, and
the most common use with graphics cards is to put the GPU pagetables in
Write Combining memory for efficient updates.  I wonder whether the Code
43 is because the driver detects that NoSnoop accesses are not actually
working.

>
> ##### Patch #3: Set CPUID to Core2duo
>
> There have been few reports where forcing CPUID to Core2duo on KVM
> (-cpu core2duo) would help alleviate Code 43 problems (and also
> increase compatibility with Windows 10 VMs), so I copied all CPUID
> registers from proven-to-be-working KVM configuration using libcpuid
> (https://github.com/anrieff/libcpuid) and applied them to Xen VM.
> LibXL is also patched (attached file) to allow hexadecimal input of
> CPUID (to make it easier to convert CPUID output from libcpuid).

The hex patch looks like a straight improvement, and I would recommend
that you use it to follow the Xen patch submission process.  Pay
particular attention to tools/libxl/CODING_STYLE, as your current patch
somewhat inconsistent.

>
> cpuid = [
>          '0:eax=0000000a,ebx=756e6547,ecx=6c65746e,edx=49656e69',
>          '1:eax=000006fb,ebx=00000800,ecx=80202201,edx=0f8bfbff',
>          '2:eax=00000001,ebx=00000000,ecx=00000000,edx=002c307d',
>          '3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>          '4,0:eax=00000121,ebx=01c0003f,ecx=0000003f,edx=00000001',
>          '4,1:eax=00000122,ebx=01c0003f,ecx=0000003f,edx=00000001',
>          '4,2:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>          '4,3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>          '5:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>          '6:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>        '7,0:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>         '0x80000000:eax=80000008,ebx=756e6547,ecx=6c65746e,edx=49656e69',
>         '0x80000001:eax=000006fb,ebx=00000000,ecx=00000001,edx=20100800',
>         '0x80000002:eax=65746e49,ebx=2952286c,ecx=726f4320,edx=4d542865',
>         '0x80000003:eax=44203229,ebx=43206f75,ecx=20205550,edx=54202020',
>         '0x80000004:eax=30303737,ebx=20402020,ecx=30342e32,edx=007a4847',
>         '0x80000005:eax=01ff01ff,ebx=01ff01ff,ecx=40020140,edx=40020140',
>         '0x80000006:eax=00000000,ebx=42004200,ecx=02008140,edx=00000000',
>         '0x80000007:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000',
>         '0x80000008:eax=00003028,ebx=00000000,ecx=00000000,edx=00000000'
>         ]
>
> This makes the /proc/cpuinfo almost identical between KVM and Xen VMs
> running Linux. Only exceptions are flags "rep_good" (which is missing
> under Xen) and "eager_fpu" and "xsaveopt" (not seen under KVM), but as
> these are not explicitly set by CPUID but are Linux-specific flags,
> they shouldn't (?) matter on Windows VMs.

rep_good and eager_fpu are Linux synthetic flags.  rep_good comes from a
IA32_MISC_ENABLE setting, while eager_fpu is enabled by default in the
presence of xsaveopt.

If you upgrade to Xen 4.7 or later, you should find that xsaveopt starts
properly following your settings of the 0xd,1 leaf.  (I did a lot of
CPUID work in 4.7, and have a lot yet to go.)

~Andrew

Patch
diff mbox

diff -ur -x .cproject -x .project -x '*.swp' xen-4.6.1/tools/qemu-xen/hw/xen/xen_pt_config_init.c xen-4.6.1-new/tools/qemu-xen/hw/xen/xen_pt_config_init.c
--- xen-4.6.1/tools/qemu-xen/hw/xen/xen_pt_config_init.c	2016-01-06 18:42:43.000000000 +0200
+++ xen-4.6.1-new/tools/qemu-xen/hw/xen/xen_pt_config_init.c	2016-07-12 21:22:40.331000000 +0300
@@ -896,8 +896,8 @@ 
     {
         .offset     = PCI_EXP_DEVCTL,
         .size       = 2,
-        .init_val   = 0x2810,
-        .ro_mask    = 0x8400,
+        .init_val   = 0x2010,
+        .ro_mask    = 0x8C00,
         .emu_mask   = 0xFFFF,
         .init       = xen_pt_common_reg_init,
         .u.w.read   = xen_pt_word_reg_read,
diff -ur -x .cproject -x .project -x '*.swp' xen-4.6.1/tools/qemu-xen-traditional/hw/pass-through.c xen-4.6.1-new/tools/qemu-xen-traditional/hw/pass-through.c
--- xen-4.6.1/tools/qemu-xen-traditional/hw/pass-through.c	2016-01-04 17:35:42.000000000 +0200
+++ xen-4.6.1-new/tools/qemu-xen-traditional/hw/pass-through.c	2016-07-11 23:45:34.317000000 +0300
@@ -601,8 +601,8 @@ 
     {
         .offset     = PCI_EXP_DEVCTL,
         .size       = 2,
-        .init_val   = 0x2810,
-        .ro_mask    = 0x8400,
+        .init_val   = 0x2010,
+        .ro_mask    = 0x8C00,
         .emu_mask   = 0xFFFF,
         .init       = pt_common_reg_init,
         .u.w.read   = pt_word_reg_read,