| Message ID | 1454455041-4647-1-git-send-email-boris.ostrovsky@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 02/02/2016 23:17, Boris Ostrovsky wrote: > Hypervisor may choose which features to emulate for HVMlite guests. > Guest will query the HVM CPUID leaf to find out what is available. > > Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> Roger also submitted a patch to do this. However, it is not appropriate, so was dropped. An HVMLite domain should assume there are no emulated devices. The very old legacy devices will never be implemented, and any others we care about possibly implementing in the future have APCI-based ways of indicating support. ~Andrew
On 02/02/2016 06:22 PM, Andrew Cooper wrote: > On 02/02/2016 23:17, Boris Ostrovsky wrote: >> Hypervisor may choose which features to emulate for HVMlite guests. >> Guest will query the HVM CPUID leaf to find out what is available. >> >> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> > Roger also submitted a patch to do this. However, it is not > appropriate, so was dropped. > > An HVMLite domain should assume there are no emulated devices. The very > old legacy devices will never be implemented, and any others we care > about possibly implementing in the future have APCI-based ways of > indicating support. OK, so I wasn't the first one to come up with this ;-) I think for now I mostly care about APIC and for that I can use HW CPUID bit (which I believe is cleared for HVMlite guests). The trouble is that I need to present Linux as having APIC (boot code doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC emulation private to Xen-related code. Which is doable. -boris
On 02/02/2016 23:30, Boris Ostrovsky wrote: > On 02/02/2016 06:22 PM, Andrew Cooper wrote: >> On 02/02/2016 23:17, Boris Ostrovsky wrote: >>> Hypervisor may choose which features to emulate for HVMlite guests. >>> Guest will query the HVM CPUID leaf to find out what is available. >>> >>> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> >> Roger also submitted a patch to do this. However, it is not >> appropriate, so was dropped. >> >> An HVMLite domain should assume there are no emulated devices. The very >> old legacy devices will never be implemented, and any others we care >> about possibly implementing in the future have APCI-based ways of >> indicating support. > > OK, so I wasn't the first one to come up with this ;-) > > I think for now I mostly care about APIC and for that I can use HW > CPUID bit (which I believe is cleared for HVMlite guests). The APIC bit in cpuid is magic and specified as a fast forward of the APICBASE_MSR enable bit. Therefore, the correct architectural behaviour is for this bit to be clear if the local APIC is disabled, or indeed not implemented. With my maintainers hat on, I will reject any attempt to introduce non-architectural behaviour; at the moment I am dealing with the stupidity that is the PV XSAVE interface, where broken bugfix piled on top of broken bugfix has resulted in a situation where many Linux PV guests crash if provided with architecturally correct behaviour of the OSXSAVE cpuid bit (yet another magic one). > The trouble is that I need to present Linux as having APIC (boot code > doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC > emulation private to Xen-related code. Which is doable. I see two choices. 1) Require that Linux DMLite guests require a Local APIC, and we allow that to be a configured option. Exposing APIC definitely makes sense longer term, because APICV hardware acceleration outperforms the hypercall-based method. 2) Find a way of telling the Linux boot path "trust me - here is an APIC driver - dont go looking under the hood". Possibly by registering a cpuid pvop which re-inserts the APIC bit, although this is liable to cause the boot code to then inspect the APICBASE_MSR, which will cause it to blow up slightly later on. ~Andrew
> From: Boris Ostrovsky > Sent: Wednesday, February 03, 2016 7:17 AM > To: jbeulich@suse.com; andrew.cooper3@citrix.com; keir@xen.org > Cc: Boris Ostrovsky; roger.pau@citrix.com; david.vrabel@citrix.com; > xen-devel@lists.xen.org > Subject: [Xen-devel] [PATCH] x86/hvm: Provide list of emulated features in HVM CPUID > leaf > > Hypervisor may choose which features to emulate for HVMlite guests. > Guest will query the HVM CPUID leaf to find out what is available. > > Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
El 3/2/16 a les 1:17, Andrew Cooper ha escrit: > On 02/02/2016 23:30, Boris Ostrovsky wrote: >> On 02/02/2016 06:22 PM, Andrew Cooper wrote: >>> On 02/02/2016 23:17, Boris Ostrovsky wrote: >>>> Hypervisor may choose which features to emulate for HVMlite guests. >>>> Guest will query the HVM CPUID leaf to find out what is available. >>>> >>>> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> >>> Roger also submitted a patch to do this. However, it is not >>> appropriate, so was dropped. >>> >>> An HVMLite domain should assume there are no emulated devices. The very >>> old legacy devices will never be implemented, and any others we care >>> about possibly implementing in the future have APCI-based ways of >>> indicating support. >> >> OK, so I wasn't the first one to come up with this ;-) Hehe, no I think I've posted the exact same patch a week ago: http://marc.info/?l=xen-devel&m=145339515912038 >> I think for now I mostly care about APIC and for that I can use HW >> CPUID bit (which I believe is cleared for HVMlite guests). > > The APIC bit in cpuid is magic and specified as a fast forward of the > APICBASE_MSR enable bit. > > Therefore, the correct architectural behaviour is for this bit to be > clear if the local APIC is disabled, or indeed not implemented. > > With my maintainers hat on, I will reject any attempt to introduce > non-architectural behaviour; at the moment I am dealing with the > stupidity that is the PV XSAVE interface, where broken bugfix piled on > top of broken bugfix has resulted in a situation where many Linux PV > guests crash if provided with architecturally correct behaviour of the > OSXSAVE cpuid bit (yet another magic one). > >> The trouble is that I need to present Linux as having APIC (boot code >> doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC >> emulation private to Xen-related code. Which is doable. I have to do the same for FreeBSD, I have to manually switch the APIC cpuid bit, or else FreeBSD refuses to do SMP initialization. IMHO, what we currently do (no APIC cpuid bit) is correct, and when a local APIC is available the bit will indeed be enabled. > I see two choices. > > 1) Require that Linux DMLite guests require a Local APIC, and we allow > that to be a configured option. Exposing APIC definitely makes sense > longer term, because APICV hardware acceleration outperforms the > hypercall-based method. This is what I aim to do long term, that is provide an emulated local APIC. The plan was to then also provide ACPI tables in order to notify the presence of the local and IO APICs (we are going to need both if we plan to do pci-passthrough of devices with PCI interrupts). Of course the APIC cpuid bit will also be enabled in this case. Roger.
On 02/03/2016 03:43 AM, Roger Pau Monné wrote: > El 3/2/16 a les 1:17, Andrew Cooper ha escrit: >> On 02/02/2016 23:30, Boris Ostrovsky wrote: >> >>> I think for now I mostly care about APIC and for that I can use HW >>> CPUID bit (which I believe is cleared for HVMlite guests). >> The APIC bit in cpuid is magic and specified as a fast forward of the >> APICBASE_MSR enable bit. >> >> Therefore, the correct architectural behaviour is for this bit to be >> clear if the local APIC is disabled, or indeed not implemented. >> >> With my maintainers hat on, I will reject any attempt to introduce >> non-architectural behaviour; at the moment I am dealing with the >> stupidity that is the PV XSAVE interface, where broken bugfix piled on >> top of broken bugfix has resulted in a situation where many Linux PV >> guests crash if provided with architecturally correct behaviour of the >> OSXSAVE cpuid bit (yet another magic one). >> >>> The trouble is that I need to present Linux as having APIC (boot code >>> doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC >>> emulation private to Xen-related code. Which is doable. > I have to do the same for FreeBSD, I have to manually switch the APIC > cpuid bit, How? In config file's 'cpuid' option? > or else FreeBSD refuses to do SMP initialization. IMHO, what > we currently do (no APIC cpuid bit) is correct, and when a local APIC is > available the bit will indeed be enabled. > >> I see two choices. >> >> 1) Require that Linux DMLite guests require a Local APIC, and we allow >> that to be a configured option. Exposing APIC definitely makes sense >> longer term, because APICV hardware acceleration outperforms the >> hypercall-based method. > This is what I aim to do long term, that is provide an emulated local > APIC. The plan was to then also provide ACPI tables in order to notify > the presence of the local and IO APICs (we are going to need both if we > plan to do pci-passthrough of devices with PCI interrupts). Of course > the APIC cpuid bit will also be enabled in this case. One might say that in Linux we have APIC even for PV guests --- we provide PV APIC ops. That's what I am using as justification for stating that the HVMlite guest has APIC to force-set X86_FEATURE_APIC bit. So this is somewhat similar to what Andrew is proposing in his option#2 (quoted below for convenience): 2) Find a way of telling the Linux boot path "trust me - here is an APIC driver - dont go looking under the hood". Possibly by registering a cpuid pvop which re-inserts the APIC bit, although this is liable to cause the boot code to then inspect the APICBASE_MSR, which will cause it to blow up slightly later on. -boris
On 03/02/16 14:30, Boris Ostrovsky wrote: > On 02/03/2016 03:43 AM, Roger Pau Monné wrote: >> El 3/2/16 a les 1:17, Andrew Cooper ha escrit: >>> On 02/02/2016 23:30, Boris Ostrovsky wrote: >>> >>>> I think for now I mostly care about APIC and for that I can use HW >>>> CPUID bit (which I believe is cleared for HVMlite guests). >>> The APIC bit in cpuid is magic and specified as a fast forward of the >>> APICBASE_MSR enable bit. >>> >>> Therefore, the correct architectural behaviour is for this bit to be >>> clear if the local APIC is disabled, or indeed not implemented. >>> >>> With my maintainers hat on, I will reject any attempt to introduce >>> non-architectural behaviour; at the moment I am dealing with the >>> stupidity that is the PV XSAVE interface, where broken bugfix piled on >>> top of broken bugfix has resulted in a situation where many Linux PV >>> guests crash if provided with architecturally correct behaviour of the >>> OSXSAVE cpuid bit (yet another magic one). >>> >>>> The trouble is that I need to present Linux as having APIC (boot code >>>> doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC >>>> emulation private to Xen-related code. Which is doable. >> I have to do the same for FreeBSD, I have to manually switch the APIC >> cpuid bit, > > How? In config file's 'cpuid' option? > >> or else FreeBSD refuses to do SMP initialization. IMHO, what >> we currently do (no APIC cpuid bit) is correct, and when a local APIC is >> available the bit will indeed be enabled. >> >>> I see two choices. >>> >>> 1) Require that Linux DMLite guests require a Local APIC, and we allow >>> that to be a configured option. Exposing APIC definitely makes sense >>> longer term, because APICV hardware acceleration outperforms the >>> hypercall-based method. >> This is what I aim to do long term, that is provide an emulated local >> APIC. The plan was to then also provide ACPI tables in order to notify >> the presence of the local and IO APICs (we are going to need both if we >> plan to do pci-passthrough of devices with PCI interrupts). Of course >> the APIC cpuid bit will also be enabled in this case. > > One might say that in Linux we have APIC even for PV guests > --- we provide PV APIC ops. That's what I am using as justification > for stating that the HVMlite guest has APIC to force-set > X86_FEATURE_APIC bit. So this is somewhat similar to what Andrew is > proposing in his option#2 (quoted below for convenience): > > 2) Find a way of telling the Linux boot path "trust me - here is > an APIC > driver - dont go looking under the hood". Possibly by registering a > cpuid pvop which re-inserts the APIC bit, although this is liable to > cause the boot code to then inspect the APICBASE_MSR, which will > cause > it to blow up slightly later on. PV guests currently have Xen's APIC leaked through despite not having access to an APIC. As with the XSAVE leakage, this has become an defacto part of the ABI despite being architecturally wrong. I expect PVOps Linux will blow up when run on older hardware which does lack a real APIC, or one which is disabled in the BIOS. ~Andrew
On 02/03/2016 09:37 AM, Andrew Cooper wrote: > On 03/02/16 14:30, Boris Ostrovsky wrote: >> >> One might say that in Linux we have APIC even for PV guests >> --- we provide PV APIC ops. That's what I am using as justification >> for stating that the HVMlite guest has APIC to force-set >> X86_FEATURE_APIC bit. So this is somewhat similar to what Andrew is >> proposing in his option#2 (quoted below for convenience): >> >> 2) Find a way of telling the Linux boot path "trust me - here is >> an APIC >> driver - dont go looking under the hood". Possibly by registering a >> cpuid pvop which re-inserts the APIC bit, although this is liable to >> cause the boot code to then inspect the APICBASE_MSR, which will >> cause >> it to blow up slightly later on. > PV guests currently have Xen's APIC leaked through despite not having > access to an APIC. What do you mean by "leaked through"? > As with the XSAVE leakage, this has become an > defacto part of the ABI despite being architecturally wrong. > > I expect PVOps Linux will blow up when run on older hardware which does > lack a real APIC, or one which is disabled in the BIOS. I didn't mean to say that we set X86_FEATURE_APIC for PV guests, it's only done for HVMlite. I don't think there is hardware with VT/SVM that doesn't have APIC. Besides, in Linux we have config XEN_PVHVM def_bool y depends on XEN && PCI && X86_LOCAL_APIC and HVMlite is considered PVHVM. -boris
El 3/2/16 a les 15:30, Boris Ostrovsky ha escrit: > On 02/03/2016 03:43 AM, Roger Pau Monné wrote: >> El 3/2/16 a les 1:17, Andrew Cooper ha escrit: >>> On 02/02/2016 23:30, Boris Ostrovsky wrote: >>> >>>> I think for now I mostly care about APIC and for that I can use HW >>>> CPUID bit (which I believe is cleared for HVMlite guests). >>> The APIC bit in cpuid is magic and specified as a fast forward of the >>> APICBASE_MSR enable bit. >>> >>> Therefore, the correct architectural behaviour is for this bit to be >>> clear if the local APIC is disabled, or indeed not implemented. >>> >>> With my maintainers hat on, I will reject any attempt to introduce >>> non-architectural behaviour; at the moment I am dealing with the >>> stupidity that is the PV XSAVE interface, where broken bugfix piled on >>> top of broken bugfix has resulted in a situation where many Linux PV >>> guests crash if provided with architecturally correct behaviour of the >>> OSXSAVE cpuid bit (yet another magic one). >>> >>>> The trouble is that I need to present Linux as having APIC (boot code >>>> doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC >>>> emulation private to Xen-related code. Which is doable. >> I have to do the same for FreeBSD, I have to manually switch the APIC >> cpuid bit, > > How? In config file's 'cpuid' option? Ah, no, I fix it inside FreeBSD. The FreeBSD kernel stores the result in a local variable, which i fixup when booting under HVMlite: cpu_feature |= CPUID_APIC; > >> or else FreeBSD refuses to do SMP initialization. IMHO, what >> we currently do (no APIC cpuid bit) is correct, and when a local APIC is >> available the bit will indeed be enabled. >> >>> I see two choices. >>> >>> 1) Require that Linux DMLite guests require a Local APIC, and we allow >>> that to be a configured option. Exposing APIC definitely makes sense >>> longer term, because APICV hardware acceleration outperforms the >>> hypercall-based method. >> This is what I aim to do long term, that is provide an emulated local >> APIC. The plan was to then also provide ACPI tables in order to notify >> the presence of the local and IO APICs (we are going to need both if we >> plan to do pci-passthrough of devices with PCI interrupts). Of course >> the APIC cpuid bit will also be enabled in this case. > > One might say that in Linux we have APIC even for PV guests --- we > provide PV APIC ops. That's what I am using as justification for stating > that the HVMlite guest has APIC to force-set X86_FEATURE_APIC bit. So > this is somewhat similar to what Andrew is proposing in his option#2 > (quoted below for convenience): > > 2) Find a way of telling the Linux boot path "trust me - here is an > APIC > driver - dont go looking under the hood". Possibly by registering a > cpuid pvop which re-inserts the APIC bit, although this is liable to > cause the boot code to then inspect the APICBASE_MSR, which will cause > it to blow up slightly later on. IMHO the APIC feature bit has a clear meaning: indicate the presence of a local APIC. A Xen PV APIC, or however we want to call it it's not a local APIC. I think OSes should fixup the CPUID feature bit if that's needed for them to work properly, but fixing it in Xen it's an error. Roger.
On 02/03/2016 10:05 AM, Roger Pau Monné wrote: > El 3/2/16 a les 15:30, Boris Ostrovsky ha escrit: >> On 02/03/2016 03:43 AM, Roger Pau Monné wrote: >>> El 3/2/16 a les 1:17, Andrew Cooper ha escrit: >>>> On 02/02/2016 23:30, Boris Ostrovsky wrote: >>>> >>>>> I think for now I mostly care about APIC and for that I can use HW >>>>> CPUID bit (which I believe is cleared for HVMlite guests). >>>> The APIC bit in cpuid is magic and specified as a fast forward of the >>>> APICBASE_MSR enable bit. >>>> >>>> Therefore, the correct architectural behaviour is for this bit to be >>>> clear if the local APIC is disabled, or indeed not implemented. >>>> >>>> With my maintainers hat on, I will reject any attempt to introduce >>>> non-architectural behaviour; at the moment I am dealing with the >>>> stupidity that is the PV XSAVE interface, where broken bugfix piled on >>>> top of broken bugfix has resulted in a situation where many Linux PV >>>> guests crash if provided with architecturally correct behaviour of the >>>> OSXSAVE cpuid bit (yet another magic one). >>>> >>>>> The trouble is that I need to present Linux as having APIC (boot code >>>>> doesn't feel good if !cpu_has_apic) so I'll need to keep no-APIC >>>>> emulation private to Xen-related code. Which is doable. >>> I have to do the same for FreeBSD, I have to manually switch the APIC >>> cpuid bit, >> How? In config file's 'cpuid' option? > Ah, no, I fix it inside FreeBSD. The FreeBSD kernel stores the result in > a local variable, which i fixup when booting under HVMlite: > > cpu_feature |= CPUID_APIC; > >>> or else FreeBSD refuses to do SMP initialization. IMHO, what >>> we currently do (no APIC cpuid bit) is correct, and when a local APIC is >>> available the bit will indeed be enabled. >>> >>>> I see two choices. >>>> >>>> 1) Require that Linux DMLite guests require a Local APIC, and we allow >>>> that to be a configured option. Exposing APIC definitely makes sense >>>> longer term, because APICV hardware acceleration outperforms the >>>> hypercall-based method. >>> This is what I aim to do long term, that is provide an emulated local >>> APIC. The plan was to then also provide ACPI tables in order to notify >>> the presence of the local and IO APICs (we are going to need both if we >>> plan to do pci-passthrough of devices with PCI interrupts). Of course >>> the APIC cpuid bit will also be enabled in this case. >> One might say that in Linux we have APIC even for PV guests --- we >> provide PV APIC ops. That's what I am using as justification for stating >> that the HVMlite guest has APIC to force-set X86_FEATURE_APIC bit. So >> this is somewhat similar to what Andrew is proposing in his option#2 >> (quoted below for convenience): >> >> 2) Find a way of telling the Linux boot path "trust me - here is an >> APIC >> driver - dont go looking under the hood". Possibly by registering a >> cpuid pvop which re-inserts the APIC bit, although this is liable to >> cause the boot code to then inspect the APICBASE_MSR, which will cause >> it to blow up slightly later on. > IMHO the APIC feature bit has a clear meaning: indicate the presence of > a local APIC. A Xen PV APIC, or however we want to call it it's not a > local APIC. I think OSes should fixup the CPUID feature bit if that's > needed for them to work properly, but fixing it in Xen it's an error. I have a feeling that I confused you (and possibly Andrew) when I said "we provide PV APIC ops". By "we" I meant Linux, not Xen. I.e. arch/x86/xen/apic.c. I do essentially the same thing in Linux as you do in FreeBSD by setting features bit with setup_force_cpu_cap(X86_FEATURE_APIC). http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00145.html -boris
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index dfeecca..8d63724 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -4585,6 +4585,10 @@ void hvm_hypervisor_cpuid_leaf(uint32_t sub_idx, /* Indicate presence of vcpu id and set it in ebx */ *eax |= XEN_HVM_CPUID_VCPU_ID_PRESENT; *ebx = current->vcpu_id; + + /* Indicate which features are emulated */ + *eax |= XEN_HVM_CPUID_EMU_FEATURES; + *ecx = current->domain->arch.emulation_flags; } } diff --git a/xen/include/public/arch-x86/cpuid.h b/xen/include/public/arch-x86/cpuid.h index d709340..a4f60c1 100644 --- a/xen/include/public/arch-x86/cpuid.h +++ b/xen/include/public/arch-x86/cpuid.h @@ -78,12 +78,18 @@ * HVM-specific features * EAX: Features * EBX: vcpu id (iff EAX has XEN_HVM_CPUID_VCPU_ID_PRESENT flag) + * ECX: Emulated features (iff EAX has XEN_HVM_CPUID_EMU_FEATURES flag) */ #define XEN_HVM_CPUID_APIC_ACCESS_VIRT (1u << 0) /* Virtualized APIC registers */ #define XEN_HVM_CPUID_X2APIC_VIRT (1u << 1) /* Virtualized x2APIC accesses */ /* Memory mapped from other domains has valid IOMMU entries */ #define XEN_HVM_CPUID_IOMMU_MAPPINGS (1u << 2) #define XEN_HVM_CPUID_VCPU_ID_PRESENT (1u << 3) /* vcpu id is present in EBX */ +/* + * Emulated features (enumerated in include/public/arch-x86/xen.h as + * XEN_X86_EMU_*) are present in ECX. + */ +#define XEN_HVM_CPUID_EMU_FEATURES (1u << 4) #define XEN_CPUID_MAX_NUM_LEAVES 4
Hypervisor may choose which features to emulate for HVMlite guests. Guest will query the HVM CPUID leaf to find out what is available. Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com> --- Another option to expose this is via features interface. The advantage of using cpuid is that we don't need to setup hypercall page to make the query (although at least for Linux and at least for now this does not buy us anything) I also don't particularly like the fact that we reference another file (even if it is a public header) for bits definition. Should we move those definitions here? xen/arch/x86/hvm/hvm.c | 4 ++++ xen/include/public/arch-x86/cpuid.h | 6 ++++++ 2 files changed, 10 insertions(+), 0 deletions(-)