From patchwork Thu Jul 14 20:19:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: marcus@wetwa.re X-Patchwork-Id: 9230579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3469C607D0 for ; Thu, 14 Jul 2016 20:23:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 208AE281B7 for ; Thu, 14 Jul 2016 20:23:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1562628308; Thu, 14 Jul 2016 20:23:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8F3E8281B7 for ; Thu, 14 Jul 2016 20:23:48 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bNn77-00042M-1y; Thu, 14 Jul 2016 20:20:05 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bNn76-00042G-7c for xen-devel@lists.xen.org; Thu, 14 Jul 2016 20:20:04 +0000 Received: from [85.158.137.68] by server-14.bemta-3.messagelabs.com id AF/90-07949-3F3F7875; Thu, 14 Jul 2016 20:20:03 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrEKsWRWlGSWpSXmKPExsXScNzyiu6nz+3 hBtNPsVgs+biYxYHR4+ju30wBjFGsmXlJ+RUJrBmT1i5gLbi8n7Hi3f0W5gbGd1sZuxi5OIQE FjJKTL4+HcxhFGhjlJi75wVrFyMnB4vANxaJaXfyQWwJAX2JrqUfmCE69jNK3D2xCMpZzCjxd u5vRpAqXgFBiZMzn7CA2MwCvhJrP51kgpikKnHt7Bkwm01AVOLt7Y9gNSIC0hLXPl8G6xUW0J bY97KbHcTmF5CSWDb1FDPETHOJLdsg5ksISEq8vHEcqJeDQ1RAVmLPVaMJjAKzkGyehWQzhC0 vsf3tHGYI20qie+NOqLiixJTuh+wQtqnEz45pbBC2t0THsz5WTDWWEnNm9UHZ/hIvJv1nxlRj LvF1wm0mmBs6FjUzLWDkXcWoUZxaVJZapGtoopdUlJmeUZKbmJmja2hgrJebWlycmJ6ak5hUr Jecn7uJERiXDECwg3HFds9DjJIcTEqivNMftocL8SXlp1RmJBZnxBeV5qQWH2KU4eBQkuB99g koJ1iUmp5akZaZA0wQMGkJDh4lEd52YJIQ4i0uSMwtzkyHSJ1iVJQS5/0I0icAksgozYNrgyW lS4yyUsK8jECHCPEUpBblZpagyr9iFOdgVBLmfQcyhSczrwRu+iugxUxAi63NwRaXJCKkpBoY m09yPph0WNPb4yrjqp9yuvk5rE2Tduo9WMurIZudKRN01sou8/sWEdeqbeuWWpz6muLLfD1nz ofUitvmdRcmGpWw7ir7LC1Zc+DhdP7N9+8piPcmbe/WLPrzXzZs/+lZkqwuCyPvtorqXGISD9 ZYaz3FcYpYqcxF5ZXsMUz1rBdf1aVv2X9UiaU4I9FQi7moOBEAiQeUBEUDAAA= X-Env-Sender: marcus@wetwa.re X-Msg-Ref: server-7.tower-31.messagelabs.com!1468527601!43125855!1 X-Originating-IP: [128.199.57.212] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.77; banners=-,-,- X-VirusChecked: Checked Received: (qmail 36089 invoked from network); 14 Jul 2016 20:20:02 -0000 Received: from oi.re (HELO tuo.re) (128.199.57.212) by server-7.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 14 Jul 2016 20:20:02 -0000 Received: from tuo.re (localhost [127.0.0.1]) by tuo.re (Postfix) with ESMTP id 48A2F423DC for ; Thu, 14 Jul 2016 20:20:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wetwa.re; h= user-agent:message-id:organization:subject:subject:to:from:from :date:date:content-type:content-type:mime-version; s=dkim; t= 1468527594; x=1469391595; bh=lMnHXiymtXGvbujlUZ1v2nPbx0fc7r2uUf5 WN+ShQZU=; b=OhbGOi51qzdc0EzfJiEjAwIKpz8KiOSxXrWpAAFk7FXxv4NlMKK cvU2/1phFK/CcnN81ngRfe82uKrakO7ZiBGSsdlK5gfWWyaw/otAraPBESvx5rWc CvvMHWIixCg4cTjab5xx1u9TBOeR5Kh7cR9Ygvv3ij07dY5Fr7dd84BIV1fchw3S JhODRLNOyEkZwG60kb7H9NUuIKidIE1UFJ7A+5xUoC2jLN6YRLF07vh2bZtGuqsy ym3TLo+aSoAnDJYCnotZFb+4CEk7hkl1lxlsXOKaHiXfHNz5EqlO6JECgZIjgDmx c3T1w/XPAeBbKfic4RB8/G37y8JGr8sV9qA== X-Virus-Scanned: Debian amavisd-new at wetwa.re Received: from tuo.re ([127.0.0.1]) by tuo.re (tuo.re [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bg976rB7DmXR for ; Thu, 14 Jul 2016 20:19:54 +0000 (UTC) Received: from tuo.re (localhost [127.0.0.1]) by tuo.re (Postfix) with ESMTPSA id E6D8242C8C for ; Thu, 14 Jul 2016 20:19:52 +0000 (UTC) MIME-Version: 1.0 Date: Thu, 14 Jul 2016 23:19:52 +0300 From: marcus@wetwa.re To: xen-devel@lists.xen.org Organization: Wetware Labs Message-ID: X-Sender: marcus@wetwa.re User-Agent: Roundcube Webmail Subject: [Xen-devel] Patches for Nvidia GPU passthrough X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Hello all, I've been working on enabling passthrough for newer Nvidia cards and drivers (GTX 980 specifically) on Xen and I'd like to document my findings up to now and ask for assistance. I apologize if this is not the correct mailing list, but I thought xen-devel is more suitable since we are talking about code changes in Xen anyway. Problem with Nvidia GPUs has been (for two years now) that drivers detect it being running inside a VM and refuse to work (Code 43 error) if the card is not a Quadro or other high-end non-consumer grade GPU (though few other things could cause Code 43 or BSOD also). Now, since KVM has supported Nvidia GPU passthrough for quite a while (and I've personally succeeded in passing through GTX 980 using KVM on both Win 7 and Win 8.1 VM's), I decided to port those few patches from KVM to Xen. ------------------ #### Patch #1: Spoof Xen and Hypervisor signatures: KVM has for a while supported hiding both the "KVMKVMKVMKVM" signature (with "-cpu kvm=off" flag) as well as the Viridian hypervisor signature ("-cpu hv_vendor_id="..." flag). Currently there's no such functionality in Xen, so I patched it in quite similar way to what Alex Willimson did for KVM. Attached is a patch for Xen 4.6.1 that spoofs Xen signature ("XenVMMXenVMM" to "ZenZenZenZen") and Viridian signature ("Microsoft Hv" to "Wetware Labs") when "spoof_xen=1" and "spoof_viridian=1" are added to VM configuration file. The signatures are currently hard-coded, and currently there's no way to modify them (beyond re-compiling Xen), since HVMLoader also uses a hard-coded string to detect Xen and there's no API (understandably) to change that signature in real-time. This works with qemu-xen-traditional without any additional changes, but qemu-xen requires that SeaBIOS is patched as well: https://github.com/WetwareLabs/seabios/commit/ec102d72fc1d7b2e6c8e9607266dc9bd4a42bce0 With spoofing on, it was possible to use official binary drivers from NVidia (tested version 367.27) on Arch Linux VM (without spoofing the driver would fail with a error message such as "The NVIDIA GPU at PCI:0:5:0 is not supported by the 367.27 NVIDIA driver". However this was not enough on Windows VM's, as the Code 43 would occur regardless of spoofing. #### Patch #2: Disable NoSnoop. Background information and the related patch for KVM is here: https://patchwork.kernel.org/patch/3019371/ The fix was quite simple for Xen: Just modify the initial PCIe DEVCTL capabilities to disable NoSnoop, and make the capability read-only. Double-checking with Linux VM, I can see that NoSnoop is disabled for all devices (with lspci -vvv), but this would not prevent Code 43 on Windows VM. ##### Patch #3: Set CPUID to Core2duo There have been few reports where forcing CPUID to Core2duo on KVM (-cpu core2duo) would help alleviate Code 43 problems (and also increase compatibility with Windows 10 VMs), so I copied all CPUID registers from proven-to-be-working KVM configuration using libcpuid (https://github.com/anrieff/libcpuid) and applied them to Xen VM. LibXL is also patched (attached file) to allow hexadecimal input of CPUID (to make it easier to convert CPUID output from libcpuid). cpuid = [ '0:eax=0000000a,ebx=756e6547,ecx=6c65746e,edx=49656e69', '1:eax=000006fb,ebx=00000800,ecx=80202201,edx=0f8bfbff', '2:eax=00000001,ebx=00000000,ecx=00000000,edx=002c307d', '3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '4,0:eax=00000121,ebx=01c0003f,ecx=0000003f,edx=00000001', '4,1:eax=00000122,ebx=01c0003f,ecx=0000003f,edx=00000001', '4,2:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '4,3:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '5:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '6:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '7,0:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '0x80000000:eax=80000008,ebx=756e6547,ecx=6c65746e,edx=49656e69', '0x80000001:eax=000006fb,ebx=00000000,ecx=00000001,edx=20100800', '0x80000002:eax=65746e49,ebx=2952286c,ecx=726f4320,edx=4d542865', '0x80000003:eax=44203229,ebx=43206f75,ecx=20205550,edx=54202020', '0x80000004:eax=30303737,ebx=20402020,ecx=30342e32,edx=007a4847', '0x80000005:eax=01ff01ff,ebx=01ff01ff,ecx=40020140,edx=40020140', '0x80000006:eax=00000000,ebx=42004200,ecx=02008140,edx=00000000', '0x80000007:eax=00000000,ebx=00000000,ecx=00000000,edx=00000000', '0x80000008:eax=00003028,ebx=00000000,ecx=00000000,edx=00000000' ] This makes the /proc/cpuinfo almost identical between KVM and Xen VMs running Linux. Only exceptions are flags "rep_good" (which is missing under Xen) and "eager_fpu" and "xsaveopt" (not seen under KVM), but as these are not explicitly set by CPUID but are Linux-specific flags, they shouldn't (?) matter on Windows VMs. ------------------ Anyway, even applying all of these patches would not alleviate Code 43. To be more specific, all NVidia drivers up to 364.72 would BSOD on boot (SYSTEM_SERVICE_EXCEPTION), and newer drivers (368.22+) would cause Code 43. This happens on both Windows 7 Pro and 8.1 VMs. Result on qemu-xen and -traditional is identical. Dom0 is Qubes 3.1 (Linux 4.1.24), Xen 4.6.1. Hardware: Intel i7-5820K, Asrock X99 WS motherboard, 32GB Corsair mem, EVGA GTX980. I would love if some of you could try these patches with both newer and older NVidia cards. Also any suggestions, ideas and further patches would be greatly appreciated! :) Thanks! Best regards, Marcus diff -ur -x .cproject -x .project -x '*.swp' xen-4.6.1/tools/qemu-xen/hw/xen/xen_pt_config_init.c xen-4.6.1-new/tools/qemu-xen/hw/xen/xen_pt_config_init.c --- xen-4.6.1/tools/qemu-xen/hw/xen/xen_pt_config_init.c 2016-01-06 18:42:43.000000000 +0200 +++ xen-4.6.1-new/tools/qemu-xen/hw/xen/xen_pt_config_init.c 2016-07-12 21:22:40.331000000 +0300 @@ -896,8 +896,8 @@ { .offset = PCI_EXP_DEVCTL, .size = 2, - .init_val = 0x2810, - .ro_mask = 0x8400, + .init_val = 0x2010, + .ro_mask = 0x8C00, .emu_mask = 0xFFFF, .init = xen_pt_common_reg_init, .u.w.read = xen_pt_word_reg_read, diff -ur -x .cproject -x .project -x '*.swp' xen-4.6.1/tools/qemu-xen-traditional/hw/pass-through.c xen-4.6.1-new/tools/qemu-xen-traditional/hw/pass-through.c --- xen-4.6.1/tools/qemu-xen-traditional/hw/pass-through.c 2016-01-04 17:35:42.000000000 +0200 +++ xen-4.6.1-new/tools/qemu-xen-traditional/hw/pass-through.c 2016-07-11 23:45:34.317000000 +0300 @@ -601,8 +601,8 @@ { .offset = PCI_EXP_DEVCTL, .size = 2, - .init_val = 0x2810, - .ro_mask = 0x8400, + .init_val = 0x2010, + .ro_mask = 0x8C00, .emu_mask = 0xFFFF, .init = pt_common_reg_init, .u.w.read = pt_word_reg_read,