| Message ID | cover.1622730232.git.viremana@linux.microsoft.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Hyper-V nested virt enlightenments for SVM | expand |
On 03/06/21 17:14, Vineeth Pillai wrote: > This patch series enables the nested virtualization enlightenments for > SVM. This is very similar to the enlightenments for VMX except for the > fact that there is no enlightened VMCS. For SVM, VMCB is already an > architectural in-memory data structure. > > Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework > based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) > https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/ > > The supported enlightenments are: > > Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate > only gva -> hpa entries. To flush entries derived from NPT, hyper-v > provided hypercalls (HvFlushGuestPhysicalAddressSpace or > HvFlushGuestPhysicalAddressList) should be used. > > Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does > not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must > invalidate the corresponding clean field after making changes to one of > the MSR bitmaps." > > Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls > (HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx, > HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow > operating systems to more efficiently manage the virtual TLB. The L1 > hypervisor can choose to allow its guest to use those hypercalls and > delegate the responsibility to handle them to the L0 hypervisor. This > requires the use of a partition assist page." > > L2 Windows boot time was measured with and without the patch. Time was > measured from power on to the login screen and was averaged over a > consecutive 5 trials: > Without the patch: 42 seconds > With the patch: 29 seconds > -- > > Changes from v4 > - Rebased on top of 5.13-rc1 and reworked based on the changes in the > patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) > > Changes from v3 > - Included definitions for software/hypervisor reserved fields in SVM > architectural data structures. > - Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce > the "ifdefs". This change applies only to SVM, VMX is not touched and > is not in the scope of this patch series. > > Changes from v2: > - Refactored the Remote TLB Flush logic into separate hyperv specific > source files (kvm_onhyperv.[ch]). > - Reverted the VMCB Clean bits macro changes as it is no longer needed. > > Changes from v1: > - Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv > to kvm_vcpu_arch and kvm_arch. > - Modify the VMCB clean mask runtime based on whether L1 hypervisor > is running on Hyper-V or not. > - Detect Hyper-V nested enlightenments based on > HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS. > - Address other minor review comments. > --- > > Vineeth Pillai (7): > hyperv: Detect Nested virtualization support for SVM > hyperv: SVM enlightened TLB flush support flag > KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx > KVM: SVM: Software reserved fields > KVM: SVM: hyper-v: Remote TLB flush for SVM > KVM: SVM: hyper-v: Enlightened MSR-Bitmap support > KVM: SVM: hyper-v: Direct Virtual Flush support > > arch/x86/include/asm/hyperv-tlfs.h | 9 ++ > arch/x86/include/asm/kvm_host.h | 9 ++ > arch/x86/include/asm/svm.h | 9 +- > arch/x86/include/uapi/asm/svm.h | 3 + > arch/x86/kernel/cpu/mshyperv.c | 10 ++- > arch/x86/kvm/Makefile | 9 ++ > arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++ > arch/x86/kvm/kvm_onhyperv.h | 32 +++++++ > arch/x86/kvm/svm/svm.c | 14 ++++ > arch/x86/kvm/svm/svm.h | 22 ++++- > arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++ > arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++ > arch/x86/kvm/vmx/vmx.c | 105 +---------------------- > arch/x86/kvm/vmx/vmx.h | 9 -- > arch/x86/kvm/x86.c | 9 ++ > 15 files changed, 384 insertions(+), 119 deletions(-) > create mode 100644 arch/x86/kvm/kvm_onhyperv.c > create mode 100644 arch/x86/kvm/kvm_onhyperv.h > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h > Queued, thanks. Paolo
On Thu, 2021-06-10 at 17:17 +0200, Paolo Bonzini wrote: > On 03/06/21 17:14, Vineeth Pillai wrote: > > This patch series enables the nested virtualization enlightenments for > > SVM. This is very similar to the enlightenments for VMX except for the > > fact that there is no enlightened VMCS. For SVM, VMCB is already an > > architectural in-memory data structure. > > > > Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework > > based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) > > https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/ > > > > The supported enlightenments are: > > > > Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate > > only gva -> hpa entries. To flush entries derived from NPT, hyper-v > > provided hypercalls (HvFlushGuestPhysicalAddressSpace or > > HvFlushGuestPhysicalAddressList) should be used. > > > > Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does > > not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must > > invalidate the corresponding clean field after making changes to one of > > the MSR bitmaps." > > > > Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls > > (HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx, > > HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow > > operating systems to more efficiently manage the virtual TLB. The L1 > > hypervisor can choose to allow its guest to use those hypercalls and > > delegate the responsibility to handle them to the L0 hypervisor. This > > requires the use of a partition assist page." > > > > L2 Windows boot time was measured with and without the patch. Time was > > measured from power on to the login screen and was averaged over a > > consecutive 5 trials: > > Without the patch: 42 seconds > > With the patch: 29 seconds > > -- > > > > Changes from v4 > > - Rebased on top of 5.13-rc1 and reworked based on the changes in the > > patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) > > > > Changes from v3 > > - Included definitions for software/hypervisor reserved fields in SVM > > architectural data structures. > > - Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce > > the "ifdefs". This change applies only to SVM, VMX is not touched and > > is not in the scope of this patch series. > > > > Changes from v2: > > - Refactored the Remote TLB Flush logic into separate hyperv specific > > source files (kvm_onhyperv.[ch]). > > - Reverted the VMCB Clean bits macro changes as it is no longer needed. > > > > Changes from v1: > > - Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv > > to kvm_vcpu_arch and kvm_arch. > > - Modify the VMCB clean mask runtime based on whether L1 hypervisor > > is running on Hyper-V or not. > > - Detect Hyper-V nested enlightenments based on > > HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS. > > - Address other minor review comments. > > --- > > > > Vineeth Pillai (7): > > hyperv: Detect Nested virtualization support for SVM > > hyperv: SVM enlightened TLB flush support flag > > KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx > > KVM: SVM: Software reserved fields > > KVM: SVM: hyper-v: Remote TLB flush for SVM > > KVM: SVM: hyper-v: Enlightened MSR-Bitmap support > > KVM: SVM: hyper-v: Direct Virtual Flush support > > > > arch/x86/include/asm/hyperv-tlfs.h | 9 ++ > > arch/x86/include/asm/kvm_host.h | 9 ++ > > arch/x86/include/asm/svm.h | 9 +- > > arch/x86/include/uapi/asm/svm.h | 3 + > > arch/x86/kernel/cpu/mshyperv.c | 10 ++- > > arch/x86/kvm/Makefile | 9 ++ > > arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++ > > arch/x86/kvm/kvm_onhyperv.h | 32 +++++++ > > arch/x86/kvm/svm/svm.c | 14 ++++ > > arch/x86/kvm/svm/svm.h | 22 ++++- > > arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++ > > arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++ > > arch/x86/kvm/vmx/vmx.c | 105 +---------------------- > > arch/x86/kvm/vmx/vmx.h | 9 -- > > arch/x86/kvm/x86.c | 9 ++ > > 15 files changed, 384 insertions(+), 119 deletions(-) > > create mode 100644 arch/x86/kvm/kvm_onhyperv.c > > create mode 100644 arch/x86/kvm/kvm_onhyperv.h > > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c > > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h > > > > Queued, thanks. > > Paolo > Hi! This patch series causes a build failure here: arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’: arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function) 7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; | ^~~~~~~~~~~~~~~~~~~ arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function) 7754 | hv_remote_flush_tlb_with_range; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Also this: arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’: arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function) 7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; | ^~~~~~~~~~~~~~~~~~~ arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function) 7754 | hv_remote_flush_tlb_with_range; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Best regards, Maxim Levitsky
Maxim Levitsky <mlevitsk@redhat.com> writes: > On Thu, 2021-06-10 at 17:17 +0200, Paolo Bonzini wrote: >> On 03/06/21 17:14, Vineeth Pillai wrote: >> > This patch series enables the nested virtualization enlightenments for >> > SVM. This is very similar to the enlightenments for VMX except for the >> > fact that there is no enlightened VMCS. For SVM, VMCB is already an >> > architectural in-memory data structure. >> > >> > Note: v5 is just a rebase on hyperv-next(5.13-rc1) and needed a rework >> > based on the patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) >> > https://lore.kernel.org/lkml/20210305183123.3978098-1-seanjc@google.com/ >> > >> > The supported enlightenments are: >> > >> > Enlightened TLB Flush: If this is enabled, ASID invalidations invalidate >> > only gva -> hpa entries. To flush entries derived from NPT, hyper-v >> > provided hypercalls (HvFlushGuestPhysicalAddressSpace or >> > HvFlushGuestPhysicalAddressList) should be used. >> > >> > Enlightened MSR bitmap(TLFS 16.5.3): "When enabled, L0 hypervisor does >> > not monitor the MSR bitmaps for changes. Instead, the L1 hypervisor must >> > invalidate the corresponding clean field after making changes to one of >> > the MSR bitmaps." >> > >> > Direct Virtual Flush(TLFS 16.8): The hypervisor exposes hypercalls >> > (HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx, >> > HvFlushVirtualAddressList, and HvFlushVirtualAddressListEx) that allow >> > operating systems to more efficiently manage the virtual TLB. The L1 >> > hypervisor can choose to allow its guest to use those hypercalls and >> > delegate the responsibility to handle them to the L0 hypervisor. This >> > requires the use of a partition assist page." >> > >> > L2 Windows boot time was measured with and without the patch. Time was >> > measured from power on to the login screen and was averaged over a >> > consecutive 5 trials: >> > Without the patch: 42 seconds >> > With the patch: 29 seconds >> > -- >> > >> > Changes from v4 >> > - Rebased on top of 5.13-rc1 and reworked based on the changes in the >> > patch series: (KVM: VMX: Clean up Hyper-V PV TLB flush) >> > >> > Changes from v3 >> > - Included definitions for software/hypervisor reserved fields in SVM >> > architectural data structures. >> > - Consolidated Hyper-V specific code into svm_onhyperv.[ch] to reduce >> > the "ifdefs". This change applies only to SVM, VMX is not touched and >> > is not in the scope of this patch series. >> > >> > Changes from v2: >> > - Refactored the Remote TLB Flush logic into separate hyperv specific >> > source files (kvm_onhyperv.[ch]). >> > - Reverted the VMCB Clean bits macro changes as it is no longer needed. >> > >> > Changes from v1: >> > - Move the remote TLB flush related fields from kvm_vcpu_hv and kvm_hv >> > to kvm_vcpu_arch and kvm_arch. >> > - Modify the VMCB clean mask runtime based on whether L1 hypervisor >> > is running on Hyper-V or not. >> > - Detect Hyper-V nested enlightenments based on >> > HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS. >> > - Address other minor review comments. >> > --- >> > >> > Vineeth Pillai (7): >> > hyperv: Detect Nested virtualization support for SVM >> > hyperv: SVM enlightened TLB flush support flag >> > KVM: x86: hyper-v: Move the remote TLB flush logic out of vmx >> > KVM: SVM: Software reserved fields >> > KVM: SVM: hyper-v: Remote TLB flush for SVM >> > KVM: SVM: hyper-v: Enlightened MSR-Bitmap support >> > KVM: SVM: hyper-v: Direct Virtual Flush support >> > >> > arch/x86/include/asm/hyperv-tlfs.h | 9 ++ >> > arch/x86/include/asm/kvm_host.h | 9 ++ >> > arch/x86/include/asm/svm.h | 9 +- >> > arch/x86/include/uapi/asm/svm.h | 3 + >> > arch/x86/kernel/cpu/mshyperv.c | 10 ++- >> > arch/x86/kvm/Makefile | 9 ++ >> > arch/x86/kvm/kvm_onhyperv.c | 93 +++++++++++++++++++++ >> > arch/x86/kvm/kvm_onhyperv.h | 32 +++++++ >> > arch/x86/kvm/svm/svm.c | 14 ++++ >> > arch/x86/kvm/svm/svm.h | 22 ++++- >> > arch/x86/kvm/svm/svm_onhyperv.c | 41 +++++++++ >> > arch/x86/kvm/svm/svm_onhyperv.h | 129 +++++++++++++++++++++++++++++ >> > arch/x86/kvm/vmx/vmx.c | 105 +---------------------- >> > arch/x86/kvm/vmx/vmx.h | 9 -- >> > arch/x86/kvm/x86.c | 9 ++ >> > 15 files changed, 384 insertions(+), 119 deletions(-) >> > create mode 100644 arch/x86/kvm/kvm_onhyperv.c >> > create mode 100644 arch/x86/kvm/kvm_onhyperv.h >> > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.c >> > create mode 100644 arch/x86/kvm/svm/svm_onhyperv.h >> > >> >> Queued, thanks. >> >> Paolo >> > > Hi! > > This patch series causes a build failure here: > > arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’: > arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function) > 7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; > | ^~~~~~~~~~~~~~~~~~~ > arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in > arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function) > 7754 | hv_remote_flush_tlb_with_range; > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > Also this: > > arch/x86/kvm/vmx/vmx.c: In function ‘hardware_setup’: > arch/x86/kvm/vmx/vmx.c:7752:34: error: ‘hv_remote_flush_tlb’ undeclared (first use in this function) > 7752 | vmx_x86_ops.tlb_remote_flush = hv_remote_flush_tlb; > | ^~~~~~~~~~~~~~~~~~~ > arch/x86/kvm/vmx/vmx.c:7752:34: note: each undeclared identifier is reported only once for each function it appears in > arch/x86/kvm/vmx/vmx.c:7754:5: error: ‘hv_remote_flush_tlb_with_range’ undeclared (first use in this function) > 7754 | hv_remote_flush_tlb_with_range; > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Yea, reported already: https://lore.kernel.org/kvm/683fa50765b29f203cb4b0953542dc43226a7a2f.camel@redhat.com/T/#mf732ba9567923d800329f896761e4ee104475894 > > Best regards, > Maxim Levitsky >
On 11/06/21 11:44, Vitaly Kuznetsov wrote: > Yea, reported already: > > https://lore.kernel.org/kvm/683fa50765b29f203cb4b0953542dc43226a7a2f.camel@redhat.com/T/#mf732ba9567923d800329f896761e4ee104475894 And that's how I learnt that 1) IS_ENABLED != #ifdef for modules 2) CONFIG_HYPERV=m is possible. Sorry Vineeth for clobbering your perfectly fine patch! Paolo
Paolo Bonzini <pbonzini@redhat.com> writes:
> CONFIG_HYPERV=m is possible.
We've stubmled upon this multiple times already. Initially, the whole
Hyper-V support code was a module (what's now in drivers/hv/) but then
some core functionallity was moved out to arch/x86/ but we didn't add a
new config back then. Still suffering :-)
Ideally, we would want to have
CONFIG_HYPERV_GUEST=y/n for what's in arch/x86/ (just like CONFIG_KVM_GUEST)
CONFIG_HYPERV_VMBUS=y/n/m for what's in drivers/hv