| Message ID | 20210210164033.607612-1-vkuznets@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | i386: KVM: expand Hyper-V features early and provide simple 'hv-default=on' option | expand |
On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: > Changes since v3: > - Make 'hv-default' override 'hv-*' options which were already set > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' > behave the same way. > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to > support the above mentioned changes. > - Expand qtest to check the above mentioned improvements. > > Original description: > > Upper layer tools like libvirt want to figure out which Hyper-V features are > supported by the underlying stack (QEMU/KVM) but currently they are unable to > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has > no effect on e.g. QMP's > > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} > > command as we parse Hyper-V features after creating KVM vCPUs and not at > feature expansion time. To support the use-case we first need to make > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing > vCPU version can't be used that early. This is what KVM part does. With > that done, we can make early Hyper-V feature expansion (this series). > > In addition, provide a simple 'hv-default' option which enables (and > requires from KVM) all currently supported Hyper-V enlightenments. > Unlike 'hv-passthrough' mode, this is going to be migratable. How is it going to be migratable if the semantics vary depending on the host kernel KVM reporting features, because different kernels will expose different features ? Regards, Daniel
On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote: > On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: > > Changes since v3: > > - Make 'hv-default' override 'hv-*' options which were already set > > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' > > behave the same way. > > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid > > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. > > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch > > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. > > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to > > support the above mentioned changes. > > - Expand qtest to check the above mentioned improvements. > > > > Original description: > > > > Upper layer tools like libvirt want to figure out which Hyper-V features are > > supported by the underlying stack (QEMU/KVM) but currently they are unable to > > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has > > no effect on e.g. QMP's > > > > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} > > > > command as we parse Hyper-V features after creating KVM vCPUs and not at > > feature expansion time. To support the use-case we first need to make > > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing > > vCPU version can't be used that early. This is what KVM part does. With > > that done, we can make early Hyper-V feature expansion (this series). > > > > In addition, provide a simple 'hv-default' option which enables (and > > requires from KVM) all currently supported Hyper-V enlightenments. > > Unlike 'hv-passthrough' mode, this is going to be migratable. > > How is it going to be migratable if the semantics vary depending on > the host kernel KVM reporting features, because different kernels > will expose different features ? "all currently supported" in this context means "all features supported when the machine type was added", not "all features supported by the host kernel".
Eduardo Habkost <ehabkost@redhat.com> writes: > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote: >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: >> > Changes since v3: >> > - Make 'hv-default' override 'hv-*' options which were already set >> > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' >> > behave the same way. >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid >> > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch >> > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to >> > support the above mentioned changes. >> > - Expand qtest to check the above mentioned improvements. >> > >> > Original description: >> > >> > Upper layer tools like libvirt want to figure out which Hyper-V features are >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has >> > no effect on e.g. QMP's >> > >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} >> > >> > command as we parse Hyper-V features after creating KVM vCPUs and not at >> > feature expansion time. To support the use-case we first need to make >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing >> > vCPU version can't be used that early. This is what KVM part does. With >> > that done, we can make early Hyper-V feature expansion (this series). >> > >> > In addition, provide a simple 'hv-default' option which enables (and >> > requires from KVM) all currently supported Hyper-V enlightenments. >> > Unlike 'hv-passthrough' mode, this is going to be migratable. >> >> How is it going to be migratable if the semantics vary depending on >> the host kernel KVM reporting features, because different kernels >> will expose different features ? > > "all currently supported" in this context means "all features > supported when the machine type was added", not "all features > supported by the host kernel". Yes, exactly. 'hv-passthrough' enables 'everything supported by the host' and this is not migratable. 'hv-default' requires a certain set of features (depending on the machine type) so the VM won't start if the host lacks something.
On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote: > Eduardo Habkost <ehabkost@redhat.com> writes: > > > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote: > >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: > >> > Changes since v3: > >> > - Make 'hv-default' override 'hv-*' options which were already set > >> > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' > >> > behave the same way. > >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid > >> > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. > >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch > >> > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. > >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to > >> > support the above mentioned changes. > >> > - Expand qtest to check the above mentioned improvements. > >> > > >> > Original description: > >> > > >> > Upper layer tools like libvirt want to figure out which Hyper-V features are > >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to > >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has > >> > no effect on e.g. QMP's > >> > > >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} > >> > > >> > command as we parse Hyper-V features after creating KVM vCPUs and not at > >> > feature expansion time. To support the use-case we first need to make > >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing > >> > vCPU version can't be used that early. This is what KVM part does. With > >> > that done, we can make early Hyper-V feature expansion (this series). > >> > > >> > In addition, provide a simple 'hv-default' option which enables (and > >> > requires from KVM) all currently supported Hyper-V enlightenments. > >> > Unlike 'hv-passthrough' mode, this is going to be migratable. > >> > >> How is it going to be migratable if the semantics vary depending on > >> the host kernel KVM reporting features, because different kernels > >> will expose different features ? > > > > "all currently supported" in this context means "all features > > supported when the machine type was added", not "all features > > supported by the host kernel". > > Yes, exactly. > > 'hv-passthrough' enables 'everything supported by the host' and this is > not migratable. > > 'hv-default' requires a certain set of features (depending on the > machine type) so the VM won't start if the host lacks something. Ok, so I presume HV features will only be added to hv-default when we know they are available in the oldest kernel we are targetting ? Upsteam is more conservative in this respect than downstreams, the latter can guarantee much more modern kernels. Regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> writes: > On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote: >> Eduardo Habkost <ehabkost@redhat.com> writes: >> >> > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote: >> >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: >> >> > Changes since v3: >> >> > - Make 'hv-default' override 'hv-*' options which were already set >> >> > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' >> >> > behave the same way. >> >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid >> >> > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. >> >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch >> >> > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. >> >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to >> >> > support the above mentioned changes. >> >> > - Expand qtest to check the above mentioned improvements. >> >> > >> >> > Original description: >> >> > >> >> > Upper layer tools like libvirt want to figure out which Hyper-V features are >> >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to >> >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has >> >> > no effect on e.g. QMP's >> >> > >> >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} >> >> > >> >> > command as we parse Hyper-V features after creating KVM vCPUs and not at >> >> > feature expansion time. To support the use-case we first need to make >> >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing >> >> > vCPU version can't be used that early. This is what KVM part does. With >> >> > that done, we can make early Hyper-V feature expansion (this series). >> >> > >> >> > In addition, provide a simple 'hv-default' option which enables (and >> >> > requires from KVM) all currently supported Hyper-V enlightenments. >> >> > Unlike 'hv-passthrough' mode, this is going to be migratable. >> >> >> >> How is it going to be migratable if the semantics vary depending on >> >> the host kernel KVM reporting features, because different kernels >> >> will expose different features ? >> > >> > "all currently supported" in this context means "all features >> > supported when the machine type was added", not "all features >> > supported by the host kernel". >> >> Yes, exactly. >> >> 'hv-passthrough' enables 'everything supported by the host' and this is >> not migratable. >> >> 'hv-default' requires a certain set of features (depending on the >> machine type) so the VM won't start if the host lacks something. > > Ok, so I presume HV features will only be added to hv-default when we > know they are available in the oldest kernel we are targetting ? Upsteam > is more conservative in this respect than downstreams, the latter can > guarantee much more modern kernels. > Yes, it is kind of an open question when a feature gets 'promoted' to 'hv-default'. Currently, the latest feature we include is 'HYPERV_FEAT_STIMER_DIRECT' which dates back to Linux 5.0. It is also possible to use something like 'hv-default,hv-stimer-direct=off,...' when running on an older kernel (and this is still migratable).
On Thu, Feb 11, 2021 at 10:34:15AM +0100, Vitaly Kuznetsov wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > > On Thu, Feb 11, 2021 at 09:30:53AM +0100, Vitaly Kuznetsov wrote: > >> Eduardo Habkost <ehabkost@redhat.com> writes: > >> > >> > On Wed, Feb 10, 2021 at 04:56:06PM +0000, Daniel P. Berrangé wrote: > >> >> On Wed, Feb 10, 2021 at 05:40:12PM +0100, Vitaly Kuznetsov wrote: > >> >> > Changes since v3: > >> >> > - Make 'hv-default' override 'hv-*' options which were already set > >> >> > (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' > >> >> > behave the same way. > >> >> > - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid > >> >> > enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. > >> >> > - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch > >> >> > to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. > >> >> > - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to > >> >> > support the above mentioned changes. > >> >> > - Expand qtest to check the above mentioned improvements. > >> >> > > >> >> > Original description: > >> >> > > >> >> > Upper layer tools like libvirt want to figure out which Hyper-V features are > >> >> > supported by the underlying stack (QEMU/KVM) but currently they are unable to > >> >> > do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has > >> >> > no effect on e.g. QMP's > >> >> > > >> >> > query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} > >> >> > > >> >> > command as we parse Hyper-V features after creating KVM vCPUs and not at > >> >> > feature expansion time. To support the use-case we first need to make > >> >> > KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing > >> >> > vCPU version can't be used that early. This is what KVM part does. With > >> >> > that done, we can make early Hyper-V feature expansion (this series). > >> >> > > >> >> > In addition, provide a simple 'hv-default' option which enables (and > >> >> > requires from KVM) all currently supported Hyper-V enlightenments. > >> >> > Unlike 'hv-passthrough' mode, this is going to be migratable. > >> >> > >> >> How is it going to be migratable if the semantics vary depending on > >> >> the host kernel KVM reporting features, because different kernels > >> >> will expose different features ? > >> > > >> > "all currently supported" in this context means "all features > >> > supported when the machine type was added", not "all features > >> > supported by the host kernel". > >> > >> Yes, exactly. > >> > >> 'hv-passthrough' enables 'everything supported by the host' and this is > >> not migratable. > >> > >> 'hv-default' requires a certain set of features (depending on the > >> machine type) so the VM won't start if the host lacks something. > > > > Ok, so I presume HV features will only be added to hv-default when we > > know they are available in the oldest kernel we are targetting ? Upsteam > > is more conservative in this respect than downstreams, the latter can > > guarantee much more modern kernels. > > > > Yes, it is kind of an open question when a feature gets 'promoted' to > 'hv-default'. Currently, the latest feature we include is > 'HYPERV_FEAT_STIMER_DIRECT' which dates back to Linux 5.0. It is also > possible to use something like Upstream we have a clear set of targetted OS platforms: https://qemu.readthedocs.io/en/latest/system/build-platforms.html This will inform what our minimum possible kernel version will be, which should influence when we can promote something. Downstream's of course have their own min kernel model, so they can do things differently if they have their own machine types. > 'hv-default,hv-stimer-direct=off,...' > > when running on an older kernel (and this is still migratable). At that point the mgmt app needs to know exactly what features the host supports, and if they know that, there's no real need to use hv-default in the first place. IOW, I think we need to strive to ensure "hv-default" is always usable on supported platforms without needing to know about turning off things. Regards, Daniel
Changes since v3: - Make 'hv-default' override 'hv-*' options which were already set (e.g. 'hv-feature=on,hv-default' case) [Igor]. Make 'hv-passthrough' behave the same way. - Add "i386: be more picky about implicit 'hv-evmcs' enablement" patch to avoid enabling 'hv-evmcs' with hv-default/hv-passthrough when guest CPU lacks VMX. - Add "i386: support 'hv-passthrough,hv-feature=off' on the command line" patch to make 'hv-passthrough' semantics match the newly introduced 'hv-default'. - Add "i386: track explicit 'hv-*' features enablement/disablement" patch to support the above mentioned changes. - Expand qtest to check the above mentioned improvements. Original description: Upper layer tools like libvirt want to figure out which Hyper-V features are supported by the underlying stack (QEMU/KVM) but currently they are unable to do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has no effect on e.g. QMP's query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}} command as we parse Hyper-V features after creating KVM vCPUs and not at feature expansion time. To support the use-case we first need to make KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing vCPU version can't be used that early. This is what KVM part does. With that done, we can make early Hyper-V feature expansion (this series). In addition, provide a simple 'hv-default' option which enables (and requires from KVM) all currently supported Hyper-V enlightenments. Unlike 'hv-passthrough' mode, this is going to be migratable. Vitaly Kuznetsov (21): i386: keep hyperv_vendor string up-to-date i386: invert hyperv_spinlock_attempts setting logic with hv_passthrough i386: always fill Hyper-V CPUID feature leaves from X86CPU data i386: stop using env->features[] for filling Hyper-V CPUIDs i386: introduce hyperv_feature_supported() i386: introduce hv_cpuid_get_host() i386: drop FEAT_HYPERV feature leaves i386: introduce hv_cpuid_cache i386: split hyperv_handle_properties() into hyperv_expand_features()/hyperv_fill_cpuids() i386: move eVMCS enablement to hyperv_init_vcpu() i386: switch hyperv_expand_features() to using error_setg() i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one i386: use global kvm_state in hyperv_enabled() check i386: expand Hyper-V features during CPU feature expansion time i386: track explicit 'hv-*' features enablement/disablement i386: support 'hv-passthrough,hv-feature=off' on the command line i386: be more picky about implicit 'hv-evmcs' enablement i386: introduce kvm_hv_evmcs_available() i386: provide simple 'hv-default=on' option qtest/hyperv: Introduce a simple hyper-v test MAINTAINERS | 1 + docs/hyperv.txt | 16 +- target/i386/cpu.c | 430 ++++++++++++++++++++--------- target/i386/cpu.h | 11 +- target/i386/kvm/kvm-stub.c | 10 + target/i386/kvm/kvm.c | 535 +++++++++++++++++++++---------------- target/i386/kvm/kvm_i386.h | 2 + tests/qtest/hyperv-test.c | 312 +++++++++++++++++++++ tests/qtest/meson.build | 3 +- 9 files changed, 950 insertions(+), 370 deletions(-) create mode 100644 tests/qtest/hyperv-test.c