From patchwork Fri Jun 3 14:59:22 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 9153309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 25EE56072B for ; Fri, 3 Jun 2016 15:02:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15E9D26D17 for ; Fri, 3 Jun 2016 15:02:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A629282E8; Fri, 3 Jun 2016 15:02:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E1DFE26D17 for ; Fri, 3 Jun 2016 15:02:01 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1b8qZT-00076l-FE; Fri, 03 Jun 2016 14:59:35 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1b8qZR-00076e-OI for xen-devel@lists.xen.org; Fri, 03 Jun 2016 14:59:33 +0000 Received: from [85.158.139.211] by server-9.bemta-5.messagelabs.com id 07/81-03501-45B91575; Fri, 03 Jun 2016 14:59:32 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrCIsWRWlGSWpSXmKPExsXitHSDvW7I7MB wg+7zohZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bReb/YCuanVqz+OJW5gfFuaBcjJ4eEgL/E nC9LGEFsNgFdiUvTJjKD2LwCghInZz5hAbGZBXQkFuz+xAZhy0tsfzsHqsZB4vGZDewgNouAi sTR91vB5ogAzXm24BlU/SxGif1L40BsYQEXicU7u8HinAJaEldObGbqYuTiEBL4zCjx/MseNo iDjCVW3/8ItJgDKKEmMXd9PIgpIcAt8bfbfgIj/ywk181Cct0sJNctYGRexahRnFpUllqka2i il1SUmZ5RkpuYmaNraGCql5taXJyYnpqTmFSsl5yfu4kRGIIMQLCD8expz0OMkhxMSqK8588G hAvxJeWnVGYkFmfEF5XmpBYfYtTg4BDo+zxhEZMUS15+XqqSBO/kWYHhQoJFqempFWmZOcAog SmV4OBREuE9CJLmLS5IzC3OTIdInWJUlBLnNQZJCIAkMkrz4NpgkXmJUVZKmJcR6CghnoLUot zMElT5V4ziHIxKwrxTQabwZOaVwE1/BbSYCWhxwSN/kMUliQgpqQbGCOUlLBNe+H0rPGgwr0r Whd3+/v8VU3R9XcxvFm5UmXX24/5bzSa+ZzzqfnunHaoPzVuWscM6dc/7uqQVfeL7625FRv3P q2hInp+8qYmBp/xm/MoQ7d1Vk05Muvbnf8j6yMYUfRO31c/dlk+cWfHufcGay88fRyS/+2ac+ 2Dt2/jOqQ4PvK+UKrEUZyQaajEXFScCAKsfUmjHAgAA X-Env-Sender: prvs=955d361cc=Ian.Jackson@citrix.com X-Msg-Ref: server-14.tower-206.messagelabs.com!1464965970!6712003!1 X-Originating-IP: [66.165.176.63] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni42MyA9PiAzMDYwNDg=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 8.46; banners=-,-,- X-VirusChecked: Checked Received: (qmail 51782 invoked from network); 3 Jun 2016 14:59:31 -0000 Received: from smtp02.citrix.com (HELO SMTP02.CITRIX.COM) (66.165.176.63) by server-14.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 3 Jun 2016 14:59:31 -0000 X-IronPort-AV: E=Sophos;i="5.26,412,1459814400"; d="scan'208";a="365128018" From: Ian Jackson MIME-Version: 1.0 Message-ID: <22353.39754.180051.801720@mariner.uk.xensource.com> Date: Fri, 3 Jun 2016 15:59:22 +0100 To: Andrew Cooper In-Reply-To: <574EDF15.9030908@citrix.com> References: <1464714345-26571-1-git-send-email-andrew.cooper3@citrix.com> <22350.47106.304924.235480@mariner.uk.xensource.com> <574ECF7E.8080004@citrix.com> <22350.53670.593405.700877@mariner.uk.xensource.com> <574EDF15.9030908@citrix.com> X-Mailer: VM 8.1.0 under 23.4.1 (i486-pc-linux-gnu) X-DLP: MIA1 Cc: Wei Liu , Jan Beulich , Xen-devel Subject: [Xen-devel] [PATCH v2 for-4.7] docs: Feature Levelling feature document X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Andrew Cooper writes ("Re: [PATCH for-4.7] docs: Feature Levelling feature document"): > On 01/06/16 13:14, Ian Jackson wrote: > > IMO xl ought to have the moving parts necessary to allow an > > administrator to: 1. collect feature information from their hosts; > > 2. combine that information into the desired feature set to expose to > > guests; 3. specify the feature set in their host configuration; 4. do > > all of the above conveniently, without seddery. > > > > We should assume that the administrator has available tools like > > GNU parallel, ansible, or whatever. > > > > I don't want to design this now but I do want the feature levelling > > documentation to welcome suggestions for it, or at least not to seem > > to rule it out. > > 1) is currently available via the `xen-cpuid` binary introduced, > although I intended it more as a developer tool > > Combining is the awkward part, but in the common case, it is just a > bitwise AND of the bitmaps provided by `xen-cpuid`. Right. > 3) I don't know what you mean about their host configuration. Do you > mean guest configuration? No, I mean that 1. the admin should have the ability to write a default to be used for all guests, in one place 2. the admin should have the ability to write this information somewhere other than the domain config file (because domain config files are often generated by other tools) > All of this works in combination with the existing cpuid= guest > configuration. Great. Documentation on how to do it `by hand' would be nice but I don't think it's essential. Below: incremental diff as a "squash!" patch, followed by combined updated patch. Ian. From 6331d6673cd292e4b8b064b8eef36cb4ed80b72b Mon Sep 17 00:00:00 2001 From: Ian Jackson Date: Fri, 3 Jun 2016 15:43:36 +0100 Subject: [PATCH] squash! docs: Feature Levelling feature document --- v2: Better wording about xl Signed-off-by: Ian Jackson --- docs/features/feature-levelling.pandoc | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/features/feature-levelling.pandoc b/docs/features/feature-levelling.pandoc index 50bf099..e1f7231 100644 --- a/docs/features/feature-levelling.pandoc +++ b/docs/features/feature-levelling.pandoc @@ -42,10 +42,12 @@ It is unsafe to make any assumption about features remaining consistent across a host reboot. Xen recalculates all information from scratch each boot, and provides the information for the toolstack to consume. -N.B. `xl`, being inherently a single-host toolstack, doesn't make use of these -levelling improvements. These features are of interest to higher level -toolstacks such as `libvirt` or `XAPI`. - +`xl` currently has no facilities to help the user collect appropriate +feature information from relevant hosts and compute appropriate +feature specifications for use in host or domain configurations. +(`xl` being a single-host toolstack, it would in any case need +external support for accessing remote hosts eg via ssh, in the form of +automation software like GNU parallel or ansible.) # Technical details @@ -174,6 +176,9 @@ be of use. # Known issues / Areas for improvement +The feature querying and levelling functions should exposed in a +convenient-to-use way by `xl`. + Xen currently has no concept of per-{socket,core,thread} CPUID information. As a result, details such as APIC IDs, topology and cache information do not match real hardware, and do not match the documented expectations in the Intel -- 1.7.10.4 From debe87a91a0742d402a08baa7e572b2755da629f Mon Sep 17 00:00:00 2001 From: Andrew Cooper Date: Tue, 31 May 2016 18:05:45 +0100 Subject: [PATCH] docs: Feature Levelling feature document Signed-off-by: Andrew Cooper Signed-off-by: Ian Jackson --- v2: Better wording about xl --- docs/features/feature-levelling.pandoc | 216 ++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 docs/features/feature-levelling.pandoc diff --git a/docs/features/feature-levelling.pandoc b/docs/features/feature-levelling.pandoc new file mode 100644 index 0000000..e1f7231 --- /dev/null +++ b/docs/features/feature-levelling.pandoc @@ -0,0 +1,216 @@ +% Feature Levelling +% Draft 1 + +\clearpage + +# Basics + +---------------- ---------------------------------------------------- + Status: **Supported** + + Architecture: x86 + + Component: Hypervisor, toolstack, guest +---------------- ---------------------------------------------------- + + +# Overview + +On native hardware, a kernel will boot, detect features, typically optimise +certain codepaths based on the available features, and expect the features to +remain available until it shuts down. + +The same expectation exists for virtual machines, and it is up to the +hypervisor/toolstack to fulfil this expectation for the lifetime of the +virtual machine, including across migrate/suspend/resume. + + +# User details + +Many factors affect the featureset which a VM may use: + +* The CPU itself +* The BIOS/firmware/microcode version and settings +* The hypervisor version and command line settings +* Further restrictions the toolstack chooses to apply + +A firmware or software upgrade might reduce the available set of features +(e.g. Intel disabling TSX in a microcode update for certain Haswell/Broadwell +processors), as may editing the settings. + +It is unsafe to make any assumption about features remaining consistent across +a host reboot. Xen recalculates all information from scratch each boot, and +provides the information for the toolstack to consume. + +`xl` currently has no facilities to help the user collect appropriate +feature information from relevant hosts and compute appropriate +feature specifications for use in host or domain configurations. +(`xl` being a single-host toolstack, it would in any case need +external support for accessing remote hosts eg via ssh, in the form of +automation software like GNU parallel or ansible.) + +# Technical details + +The `CPUID` instruction is used by software to query for features. In the +virtualisation usecase, guest software should query Xen rather than hardware +directly. However, `CPUID` is an unprivileged instruction which doesn't +fault, complicating the task of hiding hardware features from guests. + +Important files: + +* Hypervisor + * `xen/arch/x86/cpu/*.c` + * `xen/arch/x86/cpuid.c` + * `xen/include/asm-x86/cpuid-autogen.h` + * `xen/include/public/arch-x86/cpufeatureset.h` + * `xen/tools/gen-cpuid.py` +* `libxc` + * `tools/libxc/xc_cpuid_x86.c` + +## Ability to control CPUID + +### HVM + +HVM guests (using `Intel VT-x` or `AMD SVM`) will unconditionally exit to Xen +on all `CPUID` instructions, allowing Xen full control over all information. + +### PV + +The `CPUID` instruction is unprivileged, so executing it in a PV guest will +not trap, leaving Xen no direct ability to control the information returned. + +### Xen Forced Emulation Prefix + +Xen-aware PV software can make use of the 'Forced Emulation Prefix' + +> `ud2a; .ascii 'xen'; cpuid` + +which Xen recognises as a deliberate attempt to get the fully-controlled +`CPUID` information rather than the hardware-reported information. This only +works with cooperative software. + +### Masking and Override MSRs + +AMD CPUs from the `K8` onwards support _Feature Override_ MSRs, which allow +direct control of the values returned for certain `CPUID` leaves. These MSRs +allow any result to be returned, including the ability to advertise features +which are not actually supported. + +Intel CPUs between `Nehalem` and `SandyBridge` have differing numbers of +_Feature Mask_ MSRs, which are a simple AND-mask applied to all `CPUID` +instructions requesting specific feature bitmap sets. The exact MSRs, and +which feature bitmap sets they affect are hardware specific. These MSRs allow +features to be hidden by clearing the appropriate bit in the mask, but does +not allow unsupported features to be advertised. + +### CPUID Faulting + +Intel CPUs from `IvyBridge` onwards have _CPUID Faulting_, which allows Xen to +cause `CPUID` instruction executed in PV guests to fault. This allows Xen +full control over all information, exactly like HVM guests. + +## Compile time + +As some features depend on other features, it is important that, when +disabling a certain feature, we disable all features which depend on it. This +allows runtime logic to be simplified, by being able to rely on testing only +the single appropriate feature, rather than the entire feature dependency +chain. + +To speed up runtime calculation of feature dependencies, the dependency chain +is calculated and flattened by `xen/tools/gen-cpuid.py` to create +`xen/include/asm-x86/cpuid-autogen.h` from +`xen/include/public/arch-x86/cpufeatureset.h`, allowing the runtime code to +disable all dependent features of a specific disabled feature in constant +time. + +## Host boot + +As Xen boots, it will enumerate the features it can see. This is stored as +the _raw\_featureset_. + +Errata checks and command line arguments are then taken into account to reduce +the _raw\_featureset_ into the _host\_featureset_, which is the set of +features Xen uses. On hardware with masking/override MSRs, the default MSR +values are picked from the _host\_featureset_. + +The _host\_featureset_ is then used to calculate the _pv\_featureset_ and +_hvm\_featureset_, which are the maximum featuresets Xen is willing to offer +to PV and HVM guests respectively. + +In addition, Xen will calculate how much control it has over non-cooperative +PV `CPUID` instructions, storing this information as _levelling\_caps_. + +## Domain creation + +The toolstack can query each of the calculated featureset via +`XEN_SYSCTL_get_cpu_featureset`, and query for the levelling caps via +`XEN_SYSCTL_get_cpu_levelling_caps`. + +These data should be used by the toolstack when choosing the eventual +featureset to offer to the guest. + +Once a featureset has been chosen, it is set (implicitly or explicitly) via +`XEN_DOMCTL_set_cpuid`. Xen will clamp the toolstacks choice to the +appropriate PV or HVM featureset. On hardware with masking/override MSRs, the +guest cpuid policy is reflected in the MSRs, which are context switched with +other vcpu state. + +# Limitations + +A guest which ignores the provided feature information and manually probes for +features will be able to find some of them. e.g. There is no way of forcibly +preventing a guest from using 1GB superpages if the hardware supports it. + +Some information simply cannot be hidden from guests. There is no way to +control certain behaviour such as the hardware MXCSR\_MASK or x87 FPU exception +behaviour. + + +# Testing + +Feature levelling is a very wide area, and used all over the hypervisor. +Please ask on xen-devel for help identifying more specific tests which could +be of use. + + +# Known issues / Areas for improvement + +The feature querying and levelling functions should exposed in a +convenient-to-use way by `xl`. + +Xen currently has no concept of per-{socket,core,thread} CPUID information. +As a result, details such as APIC IDs, topology and cache information do not +match real hardware, and do not match the documented expectations in the Intel +and AMD system manuals. + +The CPU feature flags are the only information which the toolstack has a +sensible interface for querying and levelling. Other information in the CPUID +policy is important and should be levelled (e.g. maxphysaddr). + +The CPUID policy is currently regenerated from scratch by the receiving side, +once memory and vcpu content has been restored. This means that the receiving +Xen cannot verify the memory/vcpu content against the CPUID policy, and can +end up running a guest which will subsequently crash. The CPUID policy should +be at the head of the migration stream. + +MSRs are another source of features for guests. There is no general provision +for controlling the available MSRs. E.g. 64bit versions of Windows notice +changes in IA32\_MISC\_ENABLE, and suffer a BSOD 0x109 (Critical Structure +Corruption) + + +# References + +[Intel Flexmigration](http://www.intel.co.uk/content/dam/www/public/us/en/documents/application-notes/virtualization-technology-flexmigration-application-note.pdf) + +[AMD Extended Migration Technology](http://developer.amd.com/wordpress/media/2012/10/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf) + + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-05-31 1 Xen 4.7 Document written +---------- -------- -------- -------------------------------------------