From patchwork Mon Sep 2 18:27:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?RWR3aW4gVMO2csO2aw==?= X-Patchwork-Id: 11127005 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF9811398 for ; Mon, 2 Sep 2019 18:29:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA73121881 for ; Mon, 2 Sep 2019 18:29:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="VkI/uJhk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA73121881 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i4r2p-000105-3F; Mon, 02 Sep 2019 18:27:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i4r2n-0000zu-D2 for xen-devel@lists.xenproject.org; Mon, 02 Sep 2019 18:27:13 +0000 X-Inumbo-ID: 47d4c844-cdaf-11e9-b95f-bc764e2007e4 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 47d4c844-cdaf-11e9-b95f-bc764e2007e4; Mon, 02 Sep 2019 18:27:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1567448833; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Iw8lpmvWW06kgHF+Bh2AWLGQjfELVrhguAYDaK3ZqrU=; b=VkI/uJhk2wAFej8RYOXBxXdQIEOFNEQIJQRvrIs8FX7Xc21W8PuIyRLj qIl+voL5P3Qiwsbyi6ADRk1qXndfedJ5tDvvETZdehS7WS708GVfrxK5W rPnQVQpwh3HUpb25GO13eW8f51YgNzHJG7s3x4Wm3sWv5NRo41ofirOL3 8=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=edvin.torok@citrix.com; spf=Pass smtp.mailfrom=edvin.torok@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of edvin.torok@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="edvin.torok@citrix.com"; x-sender="edvin.torok@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of edvin.torok@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="edvin.torok@citrix.com"; x-sender="edvin.torok@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="edvin.torok@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: i9aMzh2f13U2vixr+H1V7v6iLZZvEKYkSkqco4q/va9Cl8pqTmBdFAYF4ocVJNZND2ZenZdzTL fP+nR9Q5EHYReyNrd42xPRwjoVNioJw2e1jc0JWASA6PBTx39EAKKH+7H9pDWN4H6U11jB1oVZ 51UpjhIg39sF/WeGd+mlt/I/x9bitkMHMSFpAavCa8N8lyagFDYW7SyPRGtVaJpBC6+LFcXBp+ fDSWCRFPju3E8ANnBHDBxwNToQ/OvhIBX6gf7q96FL0xbF3Dx9wm3ZZzmJhkcIP1MlhsyznhBU cpw= X-SBRS: 2.7 X-MesageID: 5025228 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.64,460,1559534400"; d="scan'208";a="5025228" From: =?utf-8?b?RWR3aW4gVMO2csO2aw==?= To: Date: Mon, 2 Sep 2019 19:27:07 +0100 Message-ID: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH 1/1] x86/arch: VM resume: avoid RDTSC emulation due to host clock drift X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , =?utf-8?b?RWR3aW4gVMO2csO2aw==?= , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" On a Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz the host frequency drifts: ``` (XEN) [ 6.607693] Detected 2600.004 MHz processor. (XEN) [ 2674.213081] dom1(hvm): mode=0,ofs=0xfffee6f70b7faa48,khz=2600018,inc=3 (XEN) [ 2674.213087] dom2(hvm): mode=0,ofs=0xfffee6fd499835c0,khz=2600018,inc=2 ``` The 2 domains were suspended prior to rebooting the host and applying a xen/microcode patch. After the reboot the frequency of the host was deemed to be slightly different, and therefore switching on RDTSC emulation for the Linux HVM guest, even though the difference was only 5 ppm. This CPU doesn't support TSC scaling. Therefore we should either measure the standard deviation of our calibration and have a range of acceptable frequencies as "same", or have a static tolerance value. The platform timer's clock frequency accuracy is: * IA-PC HPET Specification 1.0a sections 2.2 and 2.4.1: 500 ppm or better * ACPI PM timer, and PIT timer do not have defined accuracies * Intel 300 Series datasheet section 25.6: 24 MHz crystal 100 ppm or better * NTP FAQ section 3.3 Clock Quality: 11 ppm drift due to temperature * section 2.2.2 claims that PIT/ACPI PM timer share the same crystal as HPET and thus 500 ppm as an upper bound, "the real drift is usually smaller than 30ppm" For simplicity and determinism opted for a static tolerance value of 100 ppm here, such that the any error would be well within the error you would get with HPET/Linux's calibration. NTP can cope with a drift < 500 ppm. Most importantly this should stop Xen from claiming that the clock frequency on the same host is different across reboots. Specifications do not currently mandate an accuracy higher than 100 ppm, therefore OSes should already be able to cope with such drift on real hardware. Any improvements in accuracy from future specifications/motherboards wouldn't be applicable, because they would also come with newer CPUs that support TSC scaling. If the CPU does support TSC scaling Xen will of course still attempt to match the exact frequency value it thinks the guest had when it was suspended. See below for `if ( hvm_tsc_scaling_supported && !d->arch.vtsc )` (not visible in patch context). llabs() doesn't appear to be available when building xen, hence the 2 comparisons. After this patch when suspending a VM, and rebooting the host I get this output: ``` (XEN) [ 6.614703] Detected 2600.010 MHz processor. (XEN) [ 138.924342] TSC marked as reliable, warp = 0 (count=2) (XEN) [ 138.924346] dom1(hvm): mode=0,ofs=0xfffed01901016d18,khz=2600012,inc=2 ``` Signed-off-by: Edwin Török --- xen/arch/x86/time.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/time.c b/xen/arch/x86/time.c index 9a6ea8ffcb..a0b99f5fff 100644 --- a/xen/arch/x86/time.c +++ b/xen/arch/x86/time.c @@ -2171,6 +2171,12 @@ void tsc_get_info(struct domain *d, uint32_t *tsc_mode, *elapsed_nsec = 0; } +static inline int frequency_same_with_tolerance(int64_t khz1, int64_t khz2) +{ + int64_t ppm = (khz2 - khz1) * 1000000 / khz2; + return -100 < ppm && ppm < 100; +} + /* * This may be called as many as three times for a domain, once when the * hypervisor creates the domain, once when the toolstack creates the @@ -2207,7 +2213,7 @@ int tsc_set_info(struct domain *d, * d->arch.tsc_khz == cpu_khz. Thus no need to check incarnation. */ if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() && - (d->arch.tsc_khz == cpu_khz || + (frequency_same_with_tolerance(d->arch.tsc_khz, cpu_khz) || (is_hvm_domain(d) && hvm_get_tsc_scaling_ratio(d->arch.tsc_khz))) ) {