diff mbox series

[v3,12/19] x86/vmx: Introduce VMX_FEATURES_*

Message ID 20191119031240.7779-13-sean.j.christopherson@intel.com (mailing list archive)
State New, archived
Headers show
Series x86/cpu: Clean up handling of VMX features | expand

Commit Message

Sean Christopherson Nov. 19, 2019, 3:12 a.m. UTC
Add a VMX specific variant of X86_FEATURE_* flags, which will eventually
supplant the synthetic VMX flags defined in cpufeatures word 8.  Use the
Intel-defined layouts for the major VMX execution controls so that their
word entries can be directly populated from their respective MSRs, and
so that the VMX_FEATURE_* flags can be used to define the existing bit
definitions in asm/vmx.h, i.e. force developers to define a VMX_FEATURE
flag when adding support for a new hardware feature.

The majority of Intel's (and compatible CPU's) VMX capabilities are
enumerated via MSRs and not CPUID, i.e. querying /proc/cpuinfo doesn't
naturally provide any insight into the virtualization capabilities of
VMX enabled CPUs.  Commit e38e05a85828d ("x86: extended "flags" to show
virtualization HW feature in /proc/cpuinfo") attempted to address the
issue by synthesizing select VMX features into a Linux-defined word in
cpufeatures.

The synthetic cpufeatures approach has several flaws:

  - The set of synthesized VMX flags has become extremely stale with
    respect to the full set of VMX features, e.g. only one new flag
    (EPT A/D) has been added in the the decade since the introduction of
    the synthetic VMX features.  Failure to keep the VMX flags up to
    date is likely due to the lack of a mechanism that forces developers
    to consider whether or not a new feature is worth reporting.

  - The synthetic flags may incorrectly be misinterpreted as affecting
    kernel behavior, i.e. KVM, the kernel's sole consumer of VMX,
    completely ignores the synthetic flags.

  - New CPU vendors that support VMX have duplicated the hideous code
    that propagates VMX features from MSRs to cpufeatures.  Bringing the
    synthetic VMX flags up to date would exacerbate the copy+paste
    trainwreck.

Define separate VMX_FEATURE flags to set the stage for enumerating VMX
capabilities outside of the cpu_has() framework, and for adding
functional usage of VMX_FEATURE_* to help ensure the features reported
via /proc/cpuinfo is up to date with respect to kernel recognition of
VMX capabilities.

Note, the displayed names 'vnmi', 'tpr_shadow' and 'flexpriority' are
retained for backwards compatibility with the existing ABI.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 MAINTAINERS                        |  2 +-
 arch/x86/include/asm/processor.h   |  1 +
 arch/x86/include/asm/vmxfeatures.h | 81 ++++++++++++++++++++++++++++++
 3 files changed, 83 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/vmxfeatures.h

Comments

Borislav Petkov Nov. 21, 2019, 4:52 p.m. UTC | #1
On Mon, Nov 18, 2019 at 07:12:33PM -0800, Sean Christopherson wrote:
> Add a VMX specific variant of X86_FEATURE_* flags, which will eventually
> supplant the synthetic VMX flags defined in cpufeatures word 8.  Use the
> Intel-defined layouts for the major VMX execution controls so that their
> word entries can be directly populated from their respective MSRs, and
> so that the VMX_FEATURE_* flags can be used to define the existing bit
> definitions in asm/vmx.h, i.e. force developers to define a VMX_FEATURE
> flag when adding support for a new hardware feature.
> 
> The majority of Intel's (and compatible CPU's) VMX capabilities are
> enumerated via MSRs and not CPUID, i.e. querying /proc/cpuinfo doesn't
> naturally provide any insight into the virtualization capabilities of
> VMX enabled CPUs.  Commit e38e05a85828d ("x86: extended "flags" to show
> virtualization HW feature in /proc/cpuinfo") attempted to address the
> issue by synthesizing select VMX features into a Linux-defined word in
> cpufeatures.
> 
> The synthetic cpufeatures approach has several flaws:
> 
>   - The set of synthesized VMX flags has become extremely stale with
>     respect to the full set of VMX features, e.g. only one new flag
>     (EPT A/D) has been added in the the decade since the introduction of
>     the synthetic VMX features.  Failure to keep the VMX flags up to
>     date is likely due to the lack of a mechanism that forces developers
>     to consider whether or not a new feature is worth reporting.
> 
>   - The synthetic flags may incorrectly be misinterpreted as affecting
>     kernel behavior, i.e. KVM, the kernel's sole consumer of VMX,
>     completely ignores the synthetic flags.
> 
>   - New CPU vendors that support VMX have duplicated the hideous code
>     that propagates VMX features from MSRs to cpufeatures.  Bringing the
>     synthetic VMX flags up to date would exacerbate the copy+paste
>     trainwreck.
> 
> Define separate VMX_FEATURE flags to set the stage for enumerating VMX
> capabilities outside of the cpu_has() framework, and for adding
> functional usage of VMX_FEATURE_* to help ensure the features reported
> via /proc/cpuinfo is up to date with respect to kernel recognition of
> VMX capabilities.

That's all fine and good but who's going to use those feature bits?
Or are we reporting them just for the sake of it? Because if only
that, then it is not worth the effort. Sure, I don't mind extending
the framework so that you can use cpu_has() for VMX features but the
/proc/cpuinfo angle is not clear to me.

Especially since you're hiding most of them with the "" prepended in the
define comment.

> Note, the displayed names 'vnmi', 'tpr_shadow' and 'flexpriority' are
> retained for backwards compatibility with the existing ABI.
> 
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> ---
>  MAINTAINERS                        |  2 +-
>  arch/x86/include/asm/processor.h   |  1 +
>  arch/x86/include/asm/vmxfeatures.h | 81 ++++++++++++++++++++++++++++++
>  3 files changed, 83 insertions(+), 1 deletion(-)
>  create mode 100644 arch/x86/include/asm/vmxfeatures.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index df711965c377..6b736e78ee9e 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9009,7 +9009,7 @@ F:	arch/x86/include/uapi/asm/svm.h
>  F:	arch/x86/include/asm/kvm*
>  F:	arch/x86/include/asm/pvclock-abi.h
>  F:	arch/x86/include/asm/svm.h
> -F:	arch/x86/include/asm/vmx.h
> +F:	arch/x86/include/asm/vmx*.h
>  F:	arch/x86/kernel/kvm.c
>  F:	arch/x86/kernel/kvmclock.c
>  
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index b4e29d8b9e5a..772de8917430 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -25,6 +25,7 @@ struct vm86;
>  #include <asm/special_insns.h>
>  #include <asm/fpu/types.h>
>  #include <asm/unwind_hints.h>
> +#include <asm/vmxfeatures.h>
>  
>  #include <linux/personality.h>
>  #include <linux/cache.h>
> diff --git a/arch/x86/include/asm/vmxfeatures.h b/arch/x86/include/asm/vmxfeatures.h
> new file mode 100644
> index 000000000000..aea39b9f1587
> --- /dev/null
> +++ b/arch/x86/include/asm/vmxfeatures.h
> @@ -0,0 +1,81 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _ASM_X86_VMXFEATURES_H
> +#define _ASM_X86_VMXFEATURES_H
> +
> +/*
> + * Note: If the comment begins with a quoted string, that string is used
> + * in /proc/cpuinfo instead of the macro name.  If the string is "",
> + * this feature bit is not displayed in /proc/cpuinfo at all.
> + */
> +
> +/* Pin-Based VM-Execution Controls, EPT/VPID, APIC and VM-Functions, word 0 */
> +#define VMX_FEATURE_INTR_EXITING	( 0*32+  0) /* "" VM-Exit on vectored interrupts */
> +#define VMX_FEATURE_NMI_EXITING		( 0*32+  3) /* "" VM-Exit on NMIs */
> +#define VMX_FEATURE_VIRTUAL_NMIS	( 0*32+  5) /* "vnmi" NMI virtualization */
> +#define VMX_FEATURE_PREEMPTION_TIMER	( 0*32+  6) /* VMX Preemption Timer */

You really wanna have "preemption_timer" in /proc/cpuinfo? That should
at least say vmx-something, if it should be visible there at all.

> +#define VMX_FEATURE_POSTED_INTR		( 0*32+  7) /* Posted Interrupts */

Same here.

In general, the questions stand for all those feature bits which will be
visible in /proc/cpuinfo.

1. Which to show and why?

2. Who's going to use them?

3. If show and dumping them together with the other feature flags, have
their name be proper (vmx-prefixed etc).

> +/* EPT/VPID features, scattered to bits 16-23 */
> +#define VMX_FEATURE_INVVPID	        ( 0*32+ 16) /* INVVPID is supported */
> +#define VMX_FEATURE_EPT_EXECUTE_ONLY	( 0*32+ 17) /* "ept_x_only" EPT entries can be execute only */
> +#define VMX_FEATURE_EPT_AD      	( 0*32+ 18) /* EPT Accessed/Dirty bits */
> +#define VMX_FEATURE_EPT_1GB      	( 0*32+ 19) /* 1GB EPT pages */
			      ^^^^^^^^^^^^

There are some spaces that need to be converted to tabs here.

Thx.
Sean Christopherson Nov. 21, 2019, 9:50 p.m. UTC | #2
On Thu, Nov 21, 2019 at 05:52:58PM +0100, Borislav Petkov wrote:
> On Mon, Nov 18, 2019 at 07:12:33PM -0800, Sean Christopherson wrote:
> > Define separate VMX_FEATURE flags to set the stage for enumerating VMX
> > capabilities outside of the cpu_has() framework, and for adding
> > functional usage of VMX_FEATURE_* to help ensure the features reported
> > via /proc/cpuinfo is up to date with respect to kernel recognition of
> > VMX capabilities.
> 
> That's all fine and good but who's going to use those feature bits?
> Or are we reporting them just for the sake of it? Because if only
> that, then it is not worth the effort. Sure, I don't mind extending
> the framework so that you can use cpu_has() for VMX features but the
> /proc/cpuinfo angle is not clear to me.

I actually don't want to use cpu_has() for the VMX features, which is
why I put these in a separate array (one of the future patches).
 
The motivation is purely for /proc/cpuinfo.  Currently there is no sane
way for a user to query the capabilities of their platform.  The issue
comes up on a fairly regular basis, e.g. when trying to find a platform
to test a new feature or to debug an issue that has a hardware dependency.

Lack of reporting is especially annoying when the end user isn't familiar
with VMX, e.g. the format of the MSRs is non-standard, existence of some
MSRs is reported by bits in other MSRs, several features from KVM's point
of view are actually enumerated as 3+ separate features by hardware, etc...

Punting to a userspace utility isn't a viable option because all of the
capabilities are reported via MSRs.

As for why I want to keep these out of cpu_has()... VMX has a concept of
features being fixed "on", e.g. early CPUs don't allow disabling off CR3
interception.  A cpu_has() approach doesn't work well since it loses the
information regarding which bits are fixed-1.  KVM also has several module
params that can be used to disable use of features, i.e. we don't want
cpu_has() for VMX features because the KVM-specific variables need to be
the canonical reference.

> Especially since you're hiding most of them with the "" prepended in the
> define comment.
> 
> > +/* Pin-Based VM-Execution Controls, EPT/VPID, APIC and VM-Functions, word 0 */
> > +#define VMX_FEATURE_INTR_EXITING	( 0*32+  0) /* "" VM-Exit on vectored interrupts */
> > +#define VMX_FEATURE_NMI_EXITING		( 0*32+  3) /* "" VM-Exit on NMIs */
> > +#define VMX_FEATURE_VIRTUAL_NMIS	( 0*32+  5) /* "vnmi" NMI virtualization */
> > +#define VMX_FEATURE_PREEMPTION_TIMER	( 0*32+  6) /* VMX Preemption Timer */
> 
> You really wanna have "preemption_timer" in /proc/cpuinfo? That should
> at least say vmx-something, if it should be visible there at all.

Originally I wanted these to go in a completely separate line in
/proc/cpuinfo, e.g. "vmx flags".  That's what I submitted in v1, but then
found out it'd break the ABI due to handful of VMX features being
enumerated in "flags" via synthetic cpufeature bits.

Paolo suggested dropping the "vmx flags" approach as an easy solution, and
I obviously didn't update the names to prepend vmx_.  Preprending vmx_
would be somewhat annoying because changing the names of the synthetic
flags would also break the ABI...

Alternatively, what about adding "vmx flags" but keeping the existing
synthetic flags?  That'd mean having duplicates for tpr_shadow, vnmi, ept,
flexpriority, vpi and ept_ad.  On the plus side, we'd cap the pollution of
"flags" at those six features.
 
> > +#define VMX_FEATURE_POSTED_INTR		( 0*32+  7) /* Posted Interrupts */
> 
> Same here.
> 
> In general, the questions stand for all those feature bits which will be
> visible in /proc/cpuinfo.
> 
> 1. Which to show and why?
> 
> 2. Who's going to use them?
> 
> 3. If show and dumping them together with the other feature flags, have
> their name be proper (vmx-prefixed etc).
> 
> > +/* EPT/VPID features, scattered to bits 16-23 */
> > +#define VMX_FEATURE_INVVPID	        ( 0*32+ 16) /* INVVPID is supported */
> > +#define VMX_FEATURE_EPT_EXECUTE_ONLY	( 0*32+ 17) /* "ept_x_only" EPT entries can be execute only */
> > +#define VMX_FEATURE_EPT_AD      	( 0*32+ 18) /* EPT Accessed/Dirty bits */
> > +#define VMX_FEATURE_EPT_1GB      	( 0*32+ 19) /* 1GB EPT pages */
> 			      ^^^^^^^^^^^^
> 
> There are some spaces that need to be converted to tabs here.

Doh.
Borislav Petkov Nov. 22, 2019, 6:36 p.m. UTC | #3
On Thu, Nov 21, 2019 at 01:50:17PM -0800, Sean Christopherson wrote:
> I actually don't want to use cpu_has() for the VMX features, which is
> why I put these in a separate array (one of the future patches).
>  
> The motivation is purely for /proc/cpuinfo.  Currently there is no sane
> way for a user to query the capabilities of their platform.  The issue
> comes up on a fairly regular basis, e.g. when trying to find a platform
> to test a new feature or to debug an issue that has a hardware dependency.
> 
> Lack of reporting is especially annoying when the end user isn't familiar
> with VMX, e.g. the format of the MSRs is non-standard, existence of some
> MSRs is reported by bits in other MSRs, several features from KVM's point
> of view are actually enumerated as 3+ separate features by hardware, etc...
> 
> Punting to a userspace utility isn't a viable option because all of the
> capabilities are reported via MSRs.

Ok, this justification was missing in the commit message, please add the
gist of it so that it is clear why we're doing it. And yes, I agree that
having a single concentrated place for feature bits which you otherwise
have to painstakingly extract from a bunch of MSRs make sense.

> As for why I want to keep these out of cpu_has()... VMX has a concept of
> features being fixed "on", e.g. early CPUs don't allow disabling off CR3
> interception.  A cpu_has() approach doesn't work well since it loses the
> information regarding which bits are fixed-1.  KVM also has several module
> params that can be used to disable use of features, i.e. we don't want
> cpu_has() for VMX features because the KVM-specific variables need to be
> the canonical reference.

Well, you can use the cpu_has() machinery for stuff like that too - we
can clear bits there too: clear_cpu_cap() - and since clearing those
bits are only for /proc/cpuinfo reporting, it's not like anything would
break if that flag is gone. Just saying, in case you want to use the
machinery for that.

And that would avoid some of the duplication of having KVM-specific
variables *and* VMX_FEATURE_* flags, where latter are not really
toggleable but only for /proc/cpuinfo. Especially if you wanna enforce
"developers to define a VMX_FEATURE flag when adding support for a new
hardware feature."

> Alternatively, what about adding "vmx flags" but keeping the existing
> synthetic flags?  That'd mean having duplicates for tpr_shadow, vnmi, ept,
> flexpriority, vpi and ept_ad.  On the plus side, we'd cap the pollution of
> "flags" at those six features.

Yap, this is how you avoid ABI breakage. Makes sense to me.

Thx.
Sean Christopherson Nov. 22, 2019, 7:09 p.m. UTC | #4
On Fri, Nov 22, 2019 at 07:36:41PM +0100, Borislav Petkov wrote:
> On Thu, Nov 21, 2019 at 01:50:17PM -0800, Sean Christopherson wrote:
> > As for why I want to keep these out of cpu_has()... VMX has a concept of
> > features being fixed "on", e.g. early CPUs don't allow disabling off CR3
> > interception.  A cpu_has() approach doesn't work well since it loses the
> > information regarding which bits are fixed-1.  KVM also has several module
> > params that can be used to disable use of features, i.e. we don't want
> > cpu_has() for VMX features because the KVM-specific variables need to be
> > the canonical reference.
> 
> Well, you can use the cpu_has() machinery for stuff like that too - we
> can clear bits there too: clear_cpu_cap() - and since clearing those
> bits are only for /proc/cpuinfo reporting, it's not like anything would
> break if that flag is gone. Just saying, in case you want to use the
> machinery for that.

It doesn't fit the KVM use case very well.  There is an obnoxious amount
of legacy KVM code that exists only to support old processors (10+ years
old), but that we can't get rid of because people are still actively
running KVM on old hardware.  KVM provides module params so that we can
easily test those flows on modern hardware, e.g. for certain changes I'll
reload and retest KVM 2-3 times with different settings.

In theory we could do something like recompute VMX_FEATURE_* when KVM is
loaded, but that'd be a bit ugly and there are also tenative plans to move
the relevant module params under an ioctl() so that they can be toggled on
a per-VM basis to help automate testing, and IIRC for customers running
certain legacy workloads alongside normal VMs

> And that would avoid some of the duplication of having KVM-specific
> variables *and* VMX_FEATURE_* flags, where latter are not really
> toggleable but only for /proc/cpuinfo. Especially if you wanna enforce
> "developers to define a VMX_FEATURE flag when adding support for a new
> hardware feature."
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index df711965c377..6b736e78ee9e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9009,7 +9009,7 @@  F:	arch/x86/include/uapi/asm/svm.h
 F:	arch/x86/include/asm/kvm*
 F:	arch/x86/include/asm/pvclock-abi.h
 F:	arch/x86/include/asm/svm.h
-F:	arch/x86/include/asm/vmx.h
+F:	arch/x86/include/asm/vmx*.h
 F:	arch/x86/kernel/kvm.c
 F:	arch/x86/kernel/kvmclock.c
 
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index b4e29d8b9e5a..772de8917430 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -25,6 +25,7 @@  struct vm86;
 #include <asm/special_insns.h>
 #include <asm/fpu/types.h>
 #include <asm/unwind_hints.h>
+#include <asm/vmxfeatures.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
diff --git a/arch/x86/include/asm/vmxfeatures.h b/arch/x86/include/asm/vmxfeatures.h
new file mode 100644
index 000000000000..aea39b9f1587
--- /dev/null
+++ b/arch/x86/include/asm/vmxfeatures.h
@@ -0,0 +1,81 @@ 
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_VMXFEATURES_H
+#define _ASM_X86_VMXFEATURES_H
+
+/*
+ * Note: If the comment begins with a quoted string, that string is used
+ * in /proc/cpuinfo instead of the macro name.  If the string is "",
+ * this feature bit is not displayed in /proc/cpuinfo at all.
+ */
+
+/* Pin-Based VM-Execution Controls, EPT/VPID, APIC and VM-Functions, word 0 */
+#define VMX_FEATURE_INTR_EXITING	( 0*32+  0) /* "" VM-Exit on vectored interrupts */
+#define VMX_FEATURE_NMI_EXITING		( 0*32+  3) /* "" VM-Exit on NMIs */
+#define VMX_FEATURE_VIRTUAL_NMIS	( 0*32+  5) /* "vnmi" NMI virtualization */
+#define VMX_FEATURE_PREEMPTION_TIMER	( 0*32+  6) /* VMX Preemption Timer */
+#define VMX_FEATURE_POSTED_INTR		( 0*32+  7) /* Posted Interrupts */
+
+/* EPT/VPID features, scattered to bits 16-23 */
+#define VMX_FEATURE_INVVPID	        ( 0*32+ 16) /* INVVPID is supported */
+#define VMX_FEATURE_EPT_EXECUTE_ONLY	( 0*32+ 17) /* "ept_x_only" EPT entries can be execute only */
+#define VMX_FEATURE_EPT_AD      	( 0*32+ 18) /* EPT Accessed/Dirty bits */
+#define VMX_FEATURE_EPT_1GB      	( 0*32+ 19) /* 1GB EPT pages */
+
+/* Aggregated APIC features 24-27 */
+#define VMX_FEATURE_FLEXPRIORITY	( 0*32+ 24) /* TPR shadow + virt APIC */
+#define VMX_FEATURE_APICV	        ( 0*32+ 25) /* TPR shadow + APIC reg virt + virt intr delivery + posted interrupts */
+
+/* VM-Functions, shifted to bits 28-31 */
+#define VMX_FEATURE_EPTP_SWITCHING	( 0*32+ 28) /* EPTP switching (in guest) */
+
+/* Primary Processor-Based VM-Execution Controls, word 1 */
+#define VMX_FEATURE_VIRTUAL_INTR_PENDING ( 1*32+  2) /* "" VM-Exit if INTRs are unblocked in guest */
+#define VMX_FEATURE_TSC_OFFSETTING	( 1*32+  3) /* Offset hardware TSC when read in guest */
+#define VMX_FEATURE_HLT_EXITING		( 1*32+  7) /* "" VM-Exit on HLT */
+#define VMX_FEATURE_INVLPG_EXITING	( 1*32+  9) /* "" VM-Exit on INVLPG */
+#define VMX_FEATURE_MWAIT_EXITING	( 1*32+ 10) /* "" VM-Exit on MWAIT */
+#define VMX_FEATURE_RDPMC_EXITING	( 1*32+ 11) /* "" VM-Exit on RDPMC */
+#define VMX_FEATURE_RDTSC_EXITING	( 1*32+ 12) /* "" VM-Exit on RDTSC */
+#define VMX_FEATURE_CR3_LOAD_EXITING	( 1*32+ 15) /* "" VM-Exit on writes to CR3 */
+#define VMX_FEATURE_CR3_STORE_EXITING	( 1*32+ 16) /* "" VM-Exit on reads from CR3 */
+#define VMX_FEATURE_CR8_LOAD_EXITING	( 1*32+ 19) /* "" VM-Exit on writes to CR8 */
+#define VMX_FEATURE_CR8_STORE_EXITING	( 1*32+ 20) /* "" VM-Exit on reads from CR8 */
+#define VMX_FEATURE_VIRTUAL_TPR		( 1*32+ 21) /* "tpr_shadow" TPR virtualization */
+#define VMX_FEATURE_VIRTUAL_NMI_PENDING	( 1*32+ 22) /* "" VM-Exit if NMIs are unblocked in guest */
+#define VMX_FEATURE_MOV_DR_EXITING	( 1*32+ 23) /* "" VM-Exit on accesses to debug registers */
+#define VMX_FEATURE_UNCOND_IO_EXITING	( 1*32+ 24) /* "" VM-Exit on *all* IN{S} and OUT{S}*/
+#define VMX_FEATURE_USE_IO_BITMAPS	( 1*32+ 25) /* "" VM-Exit based on I/O port */
+#define VMX_FEATURE_MONITOR_TRAP_FLAG	( 1*32+ 27) /* "mtf" VMX single-step VM-Exits */
+#define VMX_FEATURE_USE_MSR_BITMAPS	( 1*32+ 28) /* "" VM-Exit based on MSR index */
+#define VMX_FEATURE_MONITOR_EXITING	( 1*32+ 29) /* "" VM-Exit on MONITOR (MWAIT's accomplice) */
+#define VMX_FEATURE_PAUSE_EXITING	( 1*32+ 30) /* "" VM-Exit on PAUSE (unconditionally) */
+#define VMX_FEATURE_SEC_CONTROLS	( 1*32+ 31) /* "" Enable Secondary VM-Execution Controls */
+
+/* Secondary Processor-Based VM-Execution Controls, word 2 */
+#define VMX_FEATURE_VIRT_APIC_ACCESSES	( 2*32+  0) /* Virtualize memory mapped APIC accesses */
+#define VMX_FEATURE_EPT			( 2*32+  1) /* Extended Page Tables, a.k.a. Two-Dimensional Paging */
+#define VMX_FEATURE_DESC_EXITING	( 2*32+  2) /* "" VM-Exit on {S,L}*DT instructions */
+#define VMX_FEATURE_RDTSCP		( 2*32+  3) /* "" Enable RDTSCP in guest */
+#define VMX_FEATURE_VIRTUAL_X2APIC	( 2*32+  4) /* "" Virtualize X2APIC for the guest */
+#define VMX_FEATURE_VPID		( 2*32+  5) /* Virtual Processor ID (TLB ASID modifier) */
+#define VMX_FEATURE_WBINVD_EXITING	( 2*32+  6) /* "" VM-Exit on WBINVD */
+#define VMX_FEATURE_UNRESTRICTED_GUEST	( 2*32+  7) /* Allow Big Real Mode and other "invalid" states */
+#define VMX_FEATURE_APIC_REGISTER_VIRT	( 2*32+  8) /* Hardware emulation of reads to the virtual-APIC */
+#define VMX_FEATURE_VIRT_INTR_DELIVERY	( 2*32+  9) /* Evaluation and delivery of pending virtual interrupts */
+#define VMX_FEATURE_PAUSE_LOOP_EXITING	( 2*32+ 10) /* "ple" Conditionally VM-Exit on PAUSE at CPL0 */
+#define VMX_FEATURE_RDRAND_EXITING	( 2*32+ 11) /* "" VM-Exit on RDRAND*/
+#define VMX_FEATURE_INVPCID		( 2*32+ 12) /* "" Enable INVPCID in guest */
+#define VMX_FEATURE_VMFUNC		( 2*32+ 13) /* "" Enable VM-Functions (leaf dependent) */
+#define VMX_FEATURE_SHADOW_VMCS		( 2*32+ 14) /* VMREAD/VMWRITE in guest can access shadow VMCS */
+#define VMX_FEATURE_ENCLS_EXITING	( 2*32+ 15) /* "" VM-Exit on ENCLS (leaf dependent) */
+#define VMX_FEATURE_RDSEED_EXITING	( 2*32+ 16) /* "" VM-Exit on RDSEED */
+#define VMX_FEATURE_PAGE_MOD_LOGGING	( 2*32+ 17) /* "pml" Log dirty pages into buffer */
+#define VMX_FEATURE_EPT_VIOLATION_VE	( 2*32+ 18) /* "" Conditionally reflect EPT violations as #VE exceptions */
+#define VMX_FEATURE_PT_CONCEAL_VMX	( 2*32+ 19) /* "" Suppress VMX indicators in Processor Trace */
+#define VMX_FEATURE_XSAVES		( 2*32+ 20) /* "" Enable XSAVES and XRSTORS in guest */
+#define VMX_FEATURE_MODE_BASED_EPT_EXEC	( 2*32+ 22) /* Enable separate EPT EXEC bits for supervisor vs. user */
+#define VMX_FEATURE_PT_USE_GPA		( 2*32+ 24) /* "" Processor Trace logs GPAs */
+#define VMX_FEATURE_TSC_SCALING		( 2*32+ 25) /* Scale hardware TSC when read in guest */
+#define VMX_FEATURE_ENCLV_EXITING	( 2*32+ 28) /* "" VM-Exit on ENCLV (leaf dependent) */
+
+#endif /* _ASM_X86_VMXFEATURES_H */