From patchwork Tue Jan 24 11:22:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 9534949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C243D60434 for ; Tue, 24 Jan 2017 11:21:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B4E6426E16 for ; Tue, 24 Jan 2017 11:21:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A995126E3E; Tue, 24 Jan 2017 11:21:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3923526E16 for ; Tue, 24 Jan 2017 11:21:14 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cVz89-0001Ge-Cw; Tue, 24 Jan 2017 11:19:17 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cVz88-0001GQ-2L for xen-devel@lists.xen.org; Tue, 24 Jan 2017 11:19:16 +0000 Received: from [85.158.139.211] by server-2.bemta-5.messagelabs.com id 93/2C-27429-33837885; Tue, 24 Jan 2017 11:19:15 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrMLMWRWlGSWpSXmKPExsUyZ7p8oK6RRXu Ewdd7NhZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8b//5eYCzbIV3R/PMrWwPhSsouRk0NIYCKT xJEZXl2MXED2b0aJhR/Ws0M4GxklHt6ZwwjhNDJKXP71ng2khU1AT6L1/GdmEFtEwEii7fhEs A5mgQZGiZeXm5hAEsICgRI31swEsjk4WARUJTr68kDCvAIeEj/ufmIBsSUE5CTOH//JDGEbSn zeuJR5AiPPAkaGVYzqxalFZalFuoZ6SUWZ6RkluYmZObqGBqZ6uanFxYnpqTmJScV6yfm5mxi Bnq9nYGDcwdjU63yIUZKDSUmUN0iyPUKILyk/pTIjsTgjvqg0J7X4EKMMB4eSBK+WOVBOsCg1 PbUiLTMHGIIwaQkOHiURXhuQNG9xQWJucWY6ROoUo6KUOK8aSEIAJJFRmgfXBgv7S4yyUsK8j AwMDEI8BalFuZklqPKvGMU5GJWEeaVApvBk5pXATX8FtJgJaPEFZrDFJYkIKakGRuuLcbP+nN m45a5w8PKjx3l6/maknbO72bqIm60spCjs18z3X8LKhEJ8F4kdcAvfGRcosvDJ0pwn/AHnrqz +vbOR6coL8dw/R4yKfBdrN1cJzze+fkH2gH30n/D0LU2572tm7LSrvmp7ZHb6tatrAw5u7z3c y7xg28dpQqcVyysMmK4uDPl57IkSS3FGoqEWc1FxIgBEWJO3dgIAAA== X-Env-Sender: joao.m.martins@oracle.com X-Msg-Ref: server-7.tower-206.messagelabs.com!1485256753!81306399!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 18358 invoked from network); 24 Jan 2017 11:19:14 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-7.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 24 Jan 2017 11:19:14 -0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v0OBJAJl008704 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 24 Jan 2017 11:19:10 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v0OBJAYu019374 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 24 Jan 2017 11:19:10 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v0OBJAfm026662; Tue, 24 Jan 2017 11:19:10 GMT Received: from paddy.lan (/89.114.92.174) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 24 Jan 2017 03:19:09 -0800 From: Joao Martins To: Xen Development List Date: Tue, 24 Jan 2017 11:22:09 +0000 Message-Id: <1485256929-4868-1-git-send-email-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.1.4 X-Source-IP: userv0022.oracle.com [156.151.31.74] Cc: Andrew Cooper , Joao Martins , Jan Beulich Subject: [Xen-devel] [PATCH v2] x86/hvm: do not set msr_tsc_adjust on hvm_set_guest_tsc_fixed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Commit 6e03363 ("x86: Implement TSC adjust feature for HVM guest") implemented TSC_ADJUST MSR for hvm guests. Though while booting an HVM guest the boot CPU would have a value set with delta_tsc - guest tsc while secondary CPUS would have 0. For example one can observe: $ xen-hvmctx 17 | grep tsc_adjust TSC_ADJUST: tsc_adjust ff9377dfef47fe66 TSC_ADJUST: tsc_adjust 0 TSC_ADJUST: tsc_adjust 0 TSC_ADJUST: tsc_adjust 0 Upcoming Linux 4.10 now validates whether this MSR is correct and adjusts them accordingly under the following conditions: values of < 0 (our case for CPU 0) or != 0 or values > 7FFFFFFF. In this conditions it will force set to 0 and for the CPUs that the value doesn't match all together. If this msr is not correct we would see messages such as: [Firmware Bug]: TSC ADJUST: CPU0: -30517044286984129 force to 0 And on HVM guests supporting TSC_ADJUST (requiring at least Haswell Intel) it won't boot. Our current vCPU 0 values are incorrect and according to Intel SDM which on section "Time-Stamp Counter Adjustment" states that "On RESET, the value of the IA32_TSC_ADJUST MSR is 0." hence we should set it 0 and be consistent across multiple vCPUs. Perhaps this MSR should be only changed by the guest which already happens through hvm_set_guest_tsc_adjust(..) routines (see below). After this patch guests running Linux 4.10 will see a valid IA32_TSC_ADJUST msr of value 0 for all CPUs and are able to boot. On the same section of the spec ("Time-Stamp Counter Adjustment") it is also stated: "If an execution of WRMSR to the IA32_TIME_STAMP_COUNTER MSR adds (or subtracts) value X from the TSC, the logical processor also adds (or subtracts) value X from the IA32_TSC_ADJUST MSR. Unlike the TSC, the value of the IA32_TSC_ADJUST MSR changes only in response to WRMSR (either to the MSR itself, or to the IA32_TIME_STAMP_COUNTER MSR). Its value does not otherwise change as time elapses. Software seeking to adjust the TSC can do so by using WRMSR to write the same value to the IA32_TSC_ADJUST MSR on each logical processor." This suggests these MSRs values should only be changed through guest i.e. throught write intercept msrs. We keep IA32_TSC MSR logic such that writes accomodate adjustments to TSC_ADJUST, hence no functional change in the msr_tsc_adjust for IA32_TSC msr. Though, we do that in a separate routine namely hvm_set_guest_tsc_msr instead of through hvm_set_guest_tsc(...). Signed-off-by: Joao Martins Reviewed-by: Jan Beulich --- Since v1: * Change from section numbers to section titles * Add citation of Intel SDM mentioning TSC_ADJUST being changed only through WRMSR --- xen/arch/x86/hvm/hvm.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index 6ab60d2..e934aaa 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -387,13 +387,20 @@ void hvm_set_guest_tsc_fixed(struct vcpu *v, u64 guest_tsc, u64 at_tsc) } delta_tsc = guest_tsc - tsc; - v->arch.hvm_vcpu.msr_tsc_adjust += delta_tsc - - v->arch.hvm_vcpu.cache_tsc_offset; v->arch.hvm_vcpu.cache_tsc_offset = delta_tsc; hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset, at_tsc); } +static void hvm_set_guest_tsc_msr(struct vcpu *v, u64 guest_tsc) +{ + uint64_t tsc_offset = v->arch.hvm_vcpu.cache_tsc_offset; + + hvm_set_guest_tsc(v, guest_tsc); + v->arch.hvm_vcpu.msr_tsc_adjust += v->arch.hvm_vcpu.cache_tsc_offset + - tsc_offset; +} + void hvm_set_guest_tsc_adjust(struct vcpu *v, u64 tsc_adjust) { v->arch.hvm_vcpu.cache_tsc_offset += tsc_adjust @@ -3491,7 +3498,7 @@ int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content, break; case MSR_IA32_TSC: - hvm_set_guest_tsc(v, msr_content); + hvm_set_guest_tsc_msr(v, msr_content); break; case MSR_IA32_TSC_ADJUST: