| Message ID | 1309820954-8629-1-git-send-email-levinsasha928@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 07/05/2011 02:09 AM, Sasha Levin wrote: > Add a module parameter 'check_inuse' to allow disabling the check of whether > virtualization has already been enabled on the given cpu. > > This is needed to deal with broken BIOS which set the SVM/VMX bit by default. Please split the vmx and svm parts. What machine was this? Did you try a BIOS update/complaint to manufacturer/etc? Let's try the proper channels before workarounds.
On Tue, 2011-07-05 at 11:09 +0300, Avi Kivity wrote: > On 07/05/2011 02:09 AM, Sasha Levin wrote: > > Add a module parameter 'check_inuse' to allow disabling the check of whether > > virtualization has already been enabled on the given cpu. > > > > This is needed to deal with broken BIOS which set the SVM/VMX bit by default. > > Please split the vmx and svm parts. > > What machine was this? Did you try a BIOS update/complaint to > manufacturer/etc? Let's try the proper channels before workarounds. > I have no information about the machine. It was the result of trying to debug an issue reported on IRC where we found that the SVM flag is being set after boot even before the kvm module is loaded.
On Tue, Jul 05, 2011 at 11:14:43AM +0300, Sasha Levin wrote: > I have no information about the machine. It was the result of trying > to debug an issue reported on IRC where we found that the SVM flag is > being set after boot even before the kvm module is loaded. Have you ruled out that any other third-party hypervisor module was loaded before KVM? The virtual-box module for example? Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2011-07-05 at 11:11 +0200, Joerg Roedel wrote: > On Tue, Jul 05, 2011 at 11:14:43AM +0300, Sasha Levin wrote: > > I have no information about the machine. It was the result of trying > > to debug an issue reported on IRC where we found that the SVM flag is > > being set after boot even before the kvm module is loaded. > > Have you ruled out that any other third-party hypervisor module was > loaded before KVM? The virtual-box module for example? No other hypervisor was installed and lsmod output was clean.
On 07/05/2011 12:32 PM, Sasha Levin wrote: > On Tue, 2011-07-05 at 11:11 +0200, Joerg Roedel wrote: > > On Tue, Jul 05, 2011 at 11:14:43AM +0300, Sasha Levin wrote: > > > I have no information about the machine. It was the result of trying > > > to debug an issue reported on IRC where we found that the SVM flag is > > > being set after boot even before the kvm module is loaded. > > > > Have you ruled out that any other third-party hypervisor module was > > loaded before KVM? The virtual-box module for example? > > No other hypervisor was installed and lsmod output was clean. This needs to be understood further.
On Tue, 2011-07-05 at 12:37 +0300, Avi Kivity wrote: > On 07/05/2011 12:32 PM, Sasha Levin wrote: > > On Tue, 2011-07-05 at 11:11 +0200, Joerg Roedel wrote: > > > On Tue, Jul 05, 2011 at 11:14:43AM +0300, Sasha Levin wrote: > > > > I have no information about the machine. It was the result of trying > > > > to debug an issue reported on IRC where we found that the SVM flag is > > > > being set after boot even before the kvm module is loaded. > > > > > > Have you ruled out that any other third-party hypervisor module was > > > loaded before KVM? The virtual-box module for example? > > > > No other hypervisor was installed and lsmod output was clean. > > This needs to be understood further. > Please note that I don't have access to the hardware in question, this was done over IRC. Here are the steps taken in debugging this issue: 1. Looking at the dmesg ( http://pastebin.com/eM7bDY8r ) we saw that when trying to load the kvm module, the following error shows up: 'kvm: enabling virtualization on CPU0 failed'. 2. We went through the lsmod output (unfortunately I don't have the link as it's gone from my IRC buffer) and didn't see any modules belonging to other hypervisors. 3. At that point, looking at the code - we figured that a set SVM flag is the possible culprit since it's the only code path which fails loading the module with that error message without printing anything else. 4. Installed msr-tools and injected the msr module so that we could read msr values from userspace. 5. Ran 'rdmsr 0xc0000080' to read the extended feature register. The output had bit 12 set - which means that SVM bit was enabled. 6. Ran 'wrmsr 0xc0000080 0xd01' which disabled the SVM bit. 7. kvm module loaded ok.
On 07/05/2011 12:56 PM, Sasha Levin wrote: > Please note that I don't have access to the hardware in question, this > was done over IRC. > I understand that. Can you get in contact with the reporter again? > Here are the steps taken in debugging this issue: > > 1. Looking at the dmesg ( http://pastebin.com/eM7bDY8r ) we saw that > when trying to load the kvm module, the following error shows up: 'kvm: > enabling virtualization on CPU0 failed'. > > 2. We went through the lsmod output (unfortunately I don't have the link > as it's gone from my IRC buffer) and didn't see any modules belonging to > other hypervisors. > > 3. At that point, looking at the code - we figured that a set SVM flag > is the possible culprit since it's the only code path which fails > loading the module with that error message without printing anything > else. > > 4. Installed msr-tools and injected the msr module so that we could read > msr values from userspace. > > 5. Ran 'rdmsr 0xc0000080' to read the extended feature register. The > output had bit 12 set - which means that SVM bit was enabled. > > 6. Ran 'wrmsr 0xc0000080 0xd01' which disabled the SVM bit. > > 7. kvm module loaded ok. My questions are: - was a BIOS update attempted? at least VMware uses the same check as kvm, and probably virtualbox as well, so this problem should have been seen before. - was the vendor contacted? Not that I think we'll see a lot of good from that. - was this after a reset or cold boot? - maybe a stealth rootkit is involved?
On Tue, 2011-07-05 at 13:37 +0300, Avi Kivity wrote: > On 07/05/2011 12:56 PM, Sasha Levin wrote: > > Please note that I don't have access to the hardware in question, this > > was done over IRC. > > > > I understand that. Can you get in contact with the reporter again? > Hopefully, If he comes back on IRC (or reads these mails :) ). > > Here are the steps taken in debugging this issue: > > > > 1. Looking at the dmesg ( http://pastebin.com/eM7bDY8r ) we saw that > > when trying to load the kvm module, the following error shows up: 'kvm: > > enabling virtualization on CPU0 failed'. > > > > 2. We went through the lsmod output (unfortunately I don't have the link > > as it's gone from my IRC buffer) and didn't see any modules belonging to > > other hypervisors. > > > > 3. At that point, looking at the code - we figured that a set SVM flag > > is the possible culprit since it's the only code path which fails > > loading the module with that error message without printing anything > > else. > > > > 4. Installed msr-tools and injected the msr module so that we could read > > msr values from userspace. > > > > 5. Ran 'rdmsr 0xc0000080' to read the extended feature register. The > > output had bit 12 set - which means that SVM bit was enabled. > > > > 6. Ran 'wrmsr 0xc0000080 0xd01' which disabled the SVM bit. > > > > 7. kvm module loaded ok. > > My questions are: > > - was a BIOS update attempted? at least VMware uses the same check as > kvm, and probably virtualbox as well, so this problem should have been > seen before. We didn't update the BIOS. virtualbox was installed previously and didn't work properly either - thats why he tried kvm afaik. We made sure to remove virtualbox properly and did a reset afterwards. After removal, no virtualbox modules were loaded at any point. > - was the vendor contacted? Not that I think we'll see a lot of good > from that. Nope. > - was this after a reset or cold boot? This was a reset, we didn't try a cold boot. > - maybe a stealth rootkit is involved? > A rootkit that messed up the MSRs or runs a hidden guest sounds like a possibility too. Alexander Graf suggested it's a simple case of a BIOS vendor not implementing specs properly as he has seen a similar case of BIOS only allowing to start virtualization on the first CPU.
On 07/05/2011 02:07 PM, Sasha Levin wrote: > > > > - was a BIOS update attempted? at least VMware uses the same check as > > kvm, and probably virtualbox as well, so this problem should have been > > seen before. > > We didn't update the BIOS. > > virtualbox was installed previously and didn't work properly either - > thats why he tried kvm afaik. That's a good data point. > > - was this after a reset or cold boot? > > This was a reset, we didn't try a cold boot. Unlikely to help, since it was a preexisting problem. > > - maybe a stealth rootkit is involved? > > > > A rootkit that messed up the MSRs or runs a hidden guest sounds like a > possibility too. > > Alexander Graf suggested it's a simple case of a BIOS vendor not > implementing specs properly as he has seen a similar case of BIOS only > allowing to start virtualization on the first CPU. I agree. But let's try a BIOS update first, if possible. If that fails, please repost the patches, but for svm only, since we haven't seen these issues on Intel.
diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c index 475d1c9..5ca76e3 100644 --- a/arch/x86/kvm/svm.c +++ b/arch/x86/kvm/svm.c @@ -183,6 +183,9 @@ module_param(npt, int, S_IRUGO); static int nested = 1; module_param(nested, int, S_IRUGO); +static bool check_inuse = 1; +module_param(check_inuse, bool, S_IRUGO); + static void svm_flush_tlb(struct kvm_vcpu *vcpu); static void svm_complete_interrupts(struct vcpu_svm *svm); @@ -587,7 +590,7 @@ static int svm_hardware_enable(void *garbage) int me = raw_smp_processor_id(); rdmsrl(MSR_EFER, efer); - if (efer & EFER_SVME) + if (check_inuse && (efer & EFER_SVME)) return -EBUSY; if (!has_svm()) { diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index f5b49c7..3046b07 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -74,6 +74,9 @@ module_param(vmm_exclusive, bool, S_IRUGO); static int __read_mostly yield_on_hlt = 1; module_param(yield_on_hlt, bool, S_IRUGO); +static bool check_inuse = 1; +module_param(check_inuse, bool, S_IRUGO); + /* * If nested=1, nested virtualization is supported, i.e., guests may use * VMX and be a hypervisor for its own guests. If nested=0, guests may not @@ -2230,7 +2233,7 @@ static int hardware_enable(void *garbage) u64 phys_addr = __pa(per_cpu(vmxarea, cpu)); u64 old, test_bits; - if (read_cr4() & X86_CR4_VMXE) + if (check_inuse && (read_cr4() & X86_CR4_VMXE)) return -EBUSY; INIT_LIST_HEAD(&per_cpu(loaded_vmcss_on_cpu, cpu));
Add a module parameter 'check_inuse' to allow disabling the check of whether virtualization has already been enabled on the given cpu. This is needed to deal with broken BIOS which set the SVM/VMX bit by default. Cc: Avi Kivity <avi@redhat.com> Cc: Marcelo Tosatti <mtosatti@redhat.com> Suggested-by: Alexander Graf <agraf@suse.de> Signed-off-by: Sasha Levin <levinsasha928@gmail.com> --- arch/x86/kvm/svm.c | 5 ++++- arch/x86/kvm/vmx.c | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-)