From patchwork Fri Jan 20 23:51:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 9529721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 431BD60113 for ; Fri, 20 Jan 2017 23:51:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2198228484 for ; Fri, 20 Jan 2017 23:51:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 146F7286DD; Fri, 20 Jan 2017 23:51:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EA48728484 for ; Fri, 20 Jan 2017 23:51:29 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cUiv7-0003EM-MC; Fri, 20 Jan 2017 23:48:37 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cUiv6-0003EG-6h for xen-devel@lists.xen.org; Fri, 20 Jan 2017 23:48:36 +0000 Received: from [85.158.139.211] by server-15.bemta-5.messagelabs.com id 26/4F-06501-3D1A2885; Fri, 20 Jan 2017 23:48:35 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRWlGSWpSXmKPExsUyZ7p8oO6lhU0 RBh1XzSyWfFzM4sDocXT3b6YAxijWzLyk/IoE1oz5iyeyFKyQq9j/fCNzA+MPiS5GLg4hgYlM EnsOdbJCOL8ZJfqOdbBBOBsZJa4uWs0E4XQzSlw4cYmli5GTg01AT6L1/GdmEFtEwEii7fhEd pAiZoFtjBIvl85lA0kIC/hJvDtzhwnEZhFQlWj79QcszivgKdH1eDHYIAkBOYnzx38yQ9iGEq cfbmOcwMizgJFhFaN6cWpRWWqRrrFeUlFmekZJbmJmjq6hgalebmpxcWJ6ak5iUrFecn7uJka g9xmAYAfj3n9OhxglOZiURHl3f2yIEOJLyk+pzEgszogvKs1JLT7EKMPBoSTBu3dBU4SQYFFq empFWmYOMAxh0hIcPEoivJdB0rzFBYm5xZnpEKlTjIpS4ryHQBICIImM0jy4NljoX2KUlRLmZ QQ6RIinILUoN7MEVf4VozgHo5Iw72yQKTyZeSVw018BLWYCWmylXA+yuCQRISXVwGgh8mKe3c WIvAmHRXpMAtfItRcuCZS8vmFG98cGu8znbnWrvLg9n1g9c77Z0GYQrH3Yz2HlPQaF69F6zy6 f4duWbNrUfzz2yw8rEdefP15L7uI5e5jpFxPXSe0ynVCdvrebn0ycceC7kfCCXwq5MmsU5t7I mfl606OTIrVB6+JcBUN3TCja2qvEUpyRaKjFXFScCAD2i7C4eAIAAA== X-Env-Sender: joao.m.martins@oracle.com X-Msg-Ref: server-12.tower-206.messagelabs.com!1484956112!44728403!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 46461 invoked from network); 20 Jan 2017 23:48:34 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-12.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 20 Jan 2017 23:48:34 -0000 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v0KNmTH1000785 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 20 Jan 2017 23:48:30 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id v0KNmS4g027526 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 20 Jan 2017 23:48:29 GMT Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v0KNmSsN012243; Fri, 20 Jan 2017 23:48:28 GMT Received: from paddy.uk.oracle.com (/10.175.198.70) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 20 Jan 2017 15:48:27 -0800 From: Joao Martins To: Xen Development List Date: Fri, 20 Jan 2017 23:51:02 +0000 Message-Id: <1484956262-18692-1-git-send-email-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.1.4 X-Source-IP: userv0021.oracle.com [156.151.31.71] Cc: Andrew Cooper , Joao Martins , Boris Ostrovsky , Jan Beulich Subject: [Xen-devel] [PATCH] x86/hvm: do not set msr_tsc_adjust on hvm_set_guest_tsc_fixed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Commit 6e03363 ("x86: Implement TSC adjust feature for HVM guest") implemented TSC_ADJUST MSR for hvm guests. Though while booting an HVM guest the boot CPU would have a value set with delta_tsc - guest tsc while secondary CPUS would have 0. For example one can observe: $ xen-hvmctx 17 | grep tsc_adjust TSC_ADJUST: tsc_adjust ff9377dfef47fe66 TSC_ADJUST: tsc_adjust 0 TSC_ADJUST: tsc_adjust 0 TSC_ADJUST: tsc_adjust 0 Upcoming Linux 4.10 now validates whether this MSR is correct and adjusts them accordingly under the following conditions: values of < 0 (our case for CPU 0) or != 0 or values > 7FFFFFFF. In this conditions it will force set to 0 and for the CPUs that the value doesn't match all together. If this msr is not correct we would see messages such as: [Firmware Bug]: TSC ADJUST: CPU0: -30517044286984129 force to 0 And on HVM guests supporting TSC_ADJUST (requiring at least Haswell Intel) it won't boot. Our current vCPU 0 values are incorrect and according to Intel SDM which on section 17.15.3 states that "On RESET, the value of the IA32_TSC_ADJUST MSR is 0." hence we should set it 0 and be consistent across multiple vCPUs. Perhaps this MSR should be only changed by the guest which already happens through hvm_set_guest_tsc_adjust(..) routines (see below). After this patch guests running Linux 4.10 will see a valid IA32_TSC_ADJUST msr of value 0 for all CPUs and are able to boot. On the same section of the spec (17.15.3) it is also stated: "If an execution of WRMSR to the IA32_TIME_STAMP_COUNTER MSR adds (or subtracts) value X from the TSC, the logical processor also adds (or subtracts) value X from the IA32_TSC_ADJUST MSR." This suggests these MSRs values should only be changed through guest i.e. throught write intercept msrs. We keep IA32_TSC MSR logic such that writes so accomodate adjustments to TSC_ADJUST, hence no functional change in the msr_tsc_adjust for IA32_TSC msr. Though, we do that in a separate routine namely hvm_set_guest_tsc_msr instead of through hvm_set_guest_tsc(...). Signed-off-by: Joao Martins Reviewed-by: Jan Beulich --- Cc: Jan Beulich Cc: Andrew Cooper Cc: Boris Ostrovsky Even if this is not the right way to fix it, at least I wanted to raise the current issue with TSC_ADJUST. AFAICT this is the behaviour since Xen 4.3 or Xen 4.4 and this logic hasn't changed ever since. Probably because this MSR hasn't been really used/checked in guests up until now. Let me know you folks think. --- xen/arch/x86/hvm/hvm.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index 2ec0800..06a6007 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -387,13 +387,20 @@ void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc) } delta_tsc = guest_tsc - tsc; - v->arch.hvm_vcpu.msr_tsc_adjust += delta_tsc - - v->arch.hvm_vcpu.cache_tsc_offset; v->arch.hvm_vcpu.cache_tsc_offset = delta_tsc; hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset, at_tsc); } +static void hvm_set_guest_tsc_msr(struct vcpu *v, u64 guest_tsc) +{ + uint64_t tsc_offset = v->arch.hvm_vcpu.cache_tsc_offset; + + hvm_set_guest_tsc(v, guest_tsc); + v->arch.hvm_vcpu.msr_tsc_adjust += v->arch.hvm_vcpu.cache_tsc_offset + - tsc_offset; +} + void hvm_set_guest_tsc_adjust(struct vcpu *v, u64 tsc_adjust) { v->arch.hvm_vcpu.cache_tsc_offset += tsc_adjust @@ -3491,7 +3498,7 @@ int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content, break; case MSR_IA32_TSC: - hvm_set_guest_tsc(v, msr_content); + hvm_set_guest_tsc_msr(v, msr_content); break; case MSR_IA32_TSC_ADJUST: