diff mbox

[Help] Windows2012 as Guest 64+cores on KVM Halts

Message ID 33183CC9F5247A488A2544077AF19020DA1BD1CA@DGGEMA505-MBX.china.huawei.com (mailing list archive)
State New, archived
Headers show

Commit Message

Gonglei (Arei) Feb. 16, 2017, 1:31 a.m. UTC 
Hi,

> 

> On Sat, 2017-02-11 at 10:39 -0500, Paolo Bonzini wrote:

> > >

> > >

> > > >

> > > > On 10/02/2017 10:31, Gonglei (Arei) wrote:

> > > > >

> > > > > But We tested the same cases on Xen platform and VMware, and

> > > > > the guest booted successfully.

> > > >

> > > > Were these two also tested with enlightenments enabled?  TCG

> > > > surely isn't.

> > >

> > > About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from the

> > > below QEMU

> > > Command line, I thought the hyper-V enabled then. Sorry about that.

> > >

> > > But for Xen, we set 'viridian=1' which be thought the Hyper-V is

> > > enabled.

> > >

> > > For VMWare we also enabled the Hyper-V enlightenments.

> If I'm not mistaken, even Hyper-V server doesn't allow specify more

> than 64 vCPUs for Generation 1 VMs.


Normally yes, but I found the explanation from Microsoft document about it:

Maximum Supported Virtual Processors

On Windows operating systems versions through Windows Server 2008 R2, 
reporting the HV#1 hypervisor interface limits the Windows virtual machine 
to a maximum of 64 VPs, regardless of what is reported via CPUID.40000005.EAX.
Starting with Windows Server 2012 and Windows 8, if CPUID.40000005.EAX 
contains a value of -1, Windows assumes that the hypervisor imposes no specific
limit to the number of VPs. In this case, Windows Server 2012 guest VMs may
use more than 64 VPs, up to the maximum supported number of processors 
applicable to the specific Windows version being used.

Link: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/tlfs

"Requirements for Implementing the Microsoft Hypervisor Interface"

And the below patch works for me, I can support max 255 vcpus for WS2012
with hyper-v enlightenments.


> In any case, if you are only interested in hv_relaxed, you can drop it

> off for WS2012 as long as you have cpu hypervisor flag

> (CPUID.1:ECX [bit 31]=1) turned on.

> 

hy_relaxed is just a example of enabling hyperv-v enlightenments.

Thanks,
-Gonglei

Comments

Vadim Rozenfeld Feb. 16, 2017, 6:26 a.m. UTC | #1 
On Thu, 2017-02-16 at 01:31 +0000, Gonglei (Arei) wrote:
> Hi,
> 
> > 
> > 
> > On Sat, 2017-02-11 at 10:39 -0500, Paolo Bonzini wrote:
> > > 
> > > > 
> > > > 
> > > > 
> > > > > 
> > > > > 
> > > > > On 10/02/2017 10:31, Gonglei (Arei) wrote:
> > > > > > 
> > > > > > 
> > > > > > But We tested the same cases on Xen platform and VMware,
> > > > > > and
> > > > > > the guest booted successfully.
> > > > > 
> > > > > Were these two also tested with enlightenments enabled?  TCG
> > > > > surely isn't.
> > > > 
> > > > About TCG, I just remove ' accel=kvm,' and 'hy_releaxed' from
> > > > the
> > > > below QEMU
> > > > Command line, I thought the hyper-V enabled then. Sorry about
> > > > that.
> > > > 
> > > > But for Xen, we set 'viridian=1' which be thought the Hyper-V
> > > > is
> > > > enabled.
> > > > 
> > > > For VMWare we also enabled the Hyper-V enlightenments.
> > If I'm not mistaken, even Hyper-V server doesn't allow specify more
> > than 64 vCPUs for Generation 1 VMs.
> 
> Normally yes, but I found the explanation from Microsoft document
> about it:
> 
> Maximum Supported Virtual Processors
> 
> On Windows operating systems versions through Windows Server 2008
> R2, 
> reporting the HV#1 hypervisor interface limits the Windows virtual
> machine 
> to a maximum of 64 VPs, regardless of what is reported via
> CPUID.40000005.EAX.
> Starting with Windows Server 2012 and Windows 8, if
> CPUID.40000005.EAX 
> contains a value of -1, Windows assumes that the hypervisor imposes
> no specific
> limit to the number of VPs. In this case, Windows Server 2012 guest
> VMs may
> use more than 64 VPs, up to the maximum supported number of
> processors 
> applicable to the specific Windows version being used.
> 
> Link: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-wind
> ows/reference/tlfs
> 
> "Requirements for Implementing the Microsoft Hypervisor Interface"
> 
> And the below patch works for me, I can support max 255 vcpus for
> WS2012
> with hyper-v enlightenments.
> 
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index 27fd050..efe3cbc 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
> 
>          c = &cpuid_data.entries[cpuid_i++];
>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
> -        c->eax = 0x40;
> +        c->eax = -1;
>          c->ebx = 0x40;
> 
>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;
> 

Nice.
I tried the following patch some time ago. Unfortunately it didn't work
for me for some reason:

@@ -772,8 +773,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
-        c->ebx = 0x40;
+        c->eax = 0x000000f0;//0x40;
+        c->ebx = 0x00000200;//0x40;
+        c->ecx = 0x00000648;

I used the same numbers as provided by WS2016 for both Gen1 and Gen2
VMs.

> > 
> > > > 
> > In any case, if you are only interested in hv_relaxed, you can drop
> > it
> > off for WS2012 as long as you have cpu hypervisor flag
> > (CPUID.1:ECX [bit 31]=1) turned on.
> > 
> hy_relaxed is just a example of enabling hyperv-v enlightenments.
> 
> Thanks,
> -Gonglei
Paolo Bonzini Feb. 17, 2017, 11:21 a.m. UTC | #2 
On 16/02/2017 02:31, Gonglei (Arei) wrote:
> And the below patch works for me, I can support max 255 vcpus for WS2012
> with hyper-v enlightenments.
> 
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index 27fd050..efe3cbc 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
> 
>          c = &cpuid_data.entries[cpuid_i++];
>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
> -        c->eax = 0x40;
> +        c->eax = -1;
>          c->ebx = 0x40;
> 
>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

This needs to depend on the machine type, but apart from that I think
you should submit the patch for 2.9.

Thanks,

Paolo
Gonglei (Arei) Feb. 20, 2017, 9:19 a.m. UTC | #3 
Hi Paolo,

> 

> 

> On 16/02/2017 02:31, Gonglei (Arei) wrote:

> > And the below patch works for me, I can support max 255 vcpus for WS2012

> > with hyper-v enlightenments.

> >

> > diff --git a/target/i386/kvm.c b/target/i386/kvm.c

> > index 27fd050..efe3cbc 100644

> > --- a/target/i386/kvm.c

> > +++ b/target/i386/kvm.c

> > @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)

> >

> >          c = &cpuid_data.entries[cpuid_i++];

> >          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;

> > -        c->eax = 0x40;

> > +        c->eax = -1;

> >          c->ebx = 0x40;

> >

> >          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

> 

> This needs to depend on the machine type, but apart from that I think


I don't know why. Because the negative effects for this change don't exist
on current QEMU IIUC, and we don't have compatible problems for live migration.

> you should submit the patch for 2.9.

> 


Thanks,
-Gonglei
Paolo Bonzini Feb. 20, 2017, 11:06 a.m. UTC | #4 
On 20/02/2017 10:19, Gonglei (Arei) wrote:
> Hi Paolo,
> 
>>
>>
>> On 16/02/2017 02:31, Gonglei (Arei) wrote:
>>> And the below patch works for me, I can support max 255 vcpus for WS2012
>>> with hyper-v enlightenments.
>>>
>>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
>>> index 27fd050..efe3cbc 100644
>>> --- a/target/i386/kvm.c
>>> +++ b/target/i386/kvm.c
>>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>>
>>>          c = &cpuid_data.entries[cpuid_i++];
>>>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
>>> -        c->eax = 0x40;
>>> +        c->eax = -1;
>>>          c->ebx = 0x40;
>>>
>>>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;
>>
>> This needs to depend on the machine type, but apart from that I think
> 
> I don't know why. Because the negative effects for this change don't exist
> on current QEMU IIUC, and we don't have compatible problems for live migration.

CPUID should never change with the same machine type and command line.

Paolo

>> you should submit the patch for 2.9.
>>
> 
> Thanks,
> -Gonglei
>
Gonglei (Arei) Feb. 20, 2017, 11:54 a.m. UTC | #5 
> 

> On 20/02/2017 10:19, Gonglei (Arei) wrote:

> > Hi Paolo,

> >

> >>

> >>

> >> On 16/02/2017 02:31, Gonglei (Arei) wrote:

> >>> And the below patch works for me, I can support max 255 vcpus for

> WS2012

> >>> with hyper-v enlightenments.

> >>>

> >>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c

> >>> index 27fd050..efe3cbc 100644

> >>> --- a/target/i386/kvm.c

> >>> +++ b/target/i386/kvm.c

> >>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)

> >>>

> >>>          c = &cpuid_data.entries[cpuid_i++];

> >>>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;

> >>> -        c->eax = 0x40;

> >>> +        c->eax = -1;

> >>>          c->ebx = 0x40;

> >>>

> >>>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

> >>

> >> This needs to depend on the machine type, but apart from that I think

> >

> > I don't know why. Because the negative effects for this change don't exist

> > on current QEMU IIUC, and we don't have compatible problems for live

> migration.

> 

> CPUID should never change with the same machine type and command line.

> 

OK, then how do we add compat code for the CPUID change? It's not a visible property.

Thanks,
-Gonglei

> Paolo

> 

> >> you should submit the patch for 2.9.

> >>

> >

> > Thanks,

> > -Gonglei

> >
Paolo Bonzini Feb. 20, 2017, 12:57 p.m. UTC | #6 
On 20/02/2017 12:54, Gonglei (Arei) wrote:
>>
>> On 20/02/2017 10:19, Gonglei (Arei) wrote:
>>> Hi Paolo,
>>>
>>>>
>>>>
>>>> On 16/02/2017 02:31, Gonglei (Arei) wrote:
>>>>> And the below patch works for me, I can support max 255 vcpus for
>> WS2012
>>>>> with hyper-v enlightenments.
>>>>>
>>>>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
>>>>> index 27fd050..efe3cbc 100644
>>>>> --- a/target/i386/kvm.c
>>>>> +++ b/target/i386/kvm.c
>>>>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>>>>
>>>>>          c = &cpuid_data.entries[cpuid_i++];
>>>>>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
>>>>> -        c->eax = 0x40;
>>>>> +        c->eax = -1;
>>>>>          c->ebx = 0x40;
>>>>>
>>>>>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;
>>>>
>>>> This needs to depend on the machine type, but apart from that I think
>>>
>>> I don't know why. Because the negative effects for this change don't exist
>>> on current QEMU IIUC, and we don't have compatible problems for live
>> migration.
>>
>> CPUID should never change with the same machine type and command line.
>>
> OK, then how do we add compat code for the CPUID change? It's not a visible property.

You can add a new property, something like hv-cpuid-limits-eax.

Paolo
Gonglei (Arei) Feb. 20, 2017, 1:18 p.m. UTC | #7 
> 

> 

> On 20/02/2017 12:54, Gonglei (Arei) wrote:

> >>

> >> On 20/02/2017 10:19, Gonglei (Arei) wrote:

> >>> Hi Paolo,

> >>>

> >>>>

> >>>>

> >>>> On 16/02/2017 02:31, Gonglei (Arei) wrote:

> >>>>> And the below patch works for me, I can support max 255 vcpus for

> >> WS2012

> >>>>> with hyper-v enlightenments.

> >>>>>

> >>>>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c

> >>>>> index 27fd050..efe3cbc 100644

> >>>>> --- a/target/i386/kvm.c

> >>>>> +++ b/target/i386/kvm.c

> >>>>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)

> >>>>>

> >>>>>          c = &cpuid_data.entries[cpuid_i++];

> >>>>>          c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;

> >>>>> -        c->eax = 0x40;

> >>>>> +        c->eax = -1;

> >>>>>          c->ebx = 0x40;

> >>>>>

> >>>>>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;

> >>>>

> >>>> This needs to depend on the machine type, but apart from that I think

> >>>

> >>> I don't know why. Because the negative effects for this change don't exist

> >>> on current QEMU IIUC, and we don't have compatible problems for live

> >> migration.

> >>

> >> CPUID should never change with the same machine type and command line.

> >>

> > OK, then how do we add compat code for the CPUID change? It's not a visible

> property.

> 

> You can add a new property, something like hv-cpuid-limits-eax.

> 

Sounds good. Let me try.  :)

Thanks,
-Gonglei
Vadim Rozenfeld Sept. 5, 2017, 6:33 a.m. UTC | #8 
On 21/02/17 00:18, Gonglei (Arei) wrote:
>>
>> On 20/02/2017 12:54, Gonglei (Arei) wrote:
>>>> On 20/02/2017 10:19, Gonglei (Arei) wrote:
>>>>> Hi Paolo,
>>>>>
>>>>>>
>>>>>> On 16/02/2017 02:31, Gonglei (Arei) wrote:
>>>>>>> And the below patch works for me, I can support max 255 vcpus for
>>>> WS2012
>>>>>>> with hyper-v enlightenments.
>>>>>>>
>>>>>>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
>>>>>>> index 27fd050..efe3cbc 100644
>>>>>>> --- a/target/i386/kvm.c
>>>>>>> +++ b/target/i386/kvm.c
>>>>>>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>>>>>>
>>>>>>>           c = &cpuid_data.entries[cpuid_i++];
>>>>>>>           c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
>>>>>>> -        c->eax = 0x40;
>>>>>>> +        c->eax = -1;
>>>>>>>           c->ebx = 0x40;
>>>>>>>
>>>>>>>           kvm_base = KVM_CPUID_SIGNATURE_NEXT;
>>>>>> This needs to depend on the machine type, but apart from that I think
>>>>> I don't know why. Because the negative effects for this change don't exist
>>>>> on current QEMU IIUC, and we don't have compatible problems for live
>>>> migration.
>>>>
>>>> CPUID should never change with the same machine type and command line.
>>>>
>>> OK, then how do we add compat code for the CPUID change? It's not a visible
>> property.
>>
>> You can add a new property, something like hv-cpuid-limits-eax.
>>
> Sounds good. Let me try.  :)
>
> Thanks,
> -Gonglei
Hi Gonglei,
Any update on this issue?

We've found that SVVP on WS2016  is passing successfully with the above 
change
and more than 64 vCPUs defined.

Vadim.
Gonglei (Arei) Sept. 5, 2017, 7:51 a.m. UTC | #9 
> -----Original Message-----
> From: Vadim Rozenfeld [mailto:vrozenfe@redhat.com]
> Sent: Tuesday, September 05, 2017 2:33 PM
> To: Gonglei (Arei); Paolo Bonzini
> Cc: Hangaohuai; Yan Vugenfirer; qemu-devel@nongnu.org;
> kvm@vger.kernel.org; Radim Krcmar
> Subject: Re: [Qemu-devel] [Help] Windows2012 as Guest 64+cores on KVM
> Halts
> 
> 
> 
> On 21/02/17 00:18, Gonglei (Arei) wrote:
> >>
> >> On 20/02/2017 12:54, Gonglei (Arei) wrote:
> >>>> On 20/02/2017 10:19, Gonglei (Arei) wrote:
> >>>>> Hi Paolo,
> >>>>>
> >>>>>>
> >>>>>> On 16/02/2017 02:31, Gonglei (Arei) wrote:
> >>>>>>> And the below patch works for me, I can support max 255 vcpus for
> >>>> WS2012
> >>>>>>> with hyper-v enlightenments.
> >>>>>>>
> >>>>>>> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> >>>>>>> index 27fd050..efe3cbc 100644
> >>>>>>> --- a/target/i386/kvm.c
> >>>>>>> +++ b/target/i386/kvm.c
> >>>>>>> @@ -772,7 +772,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >>>>>>>
> >>>>>>>           c = &cpuid_data.entries[cpuid_i++];
> >>>>>>>           c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
> >>>>>>> -        c->eax = 0x40;
> >>>>>>> +        c->eax = -1;
> >>>>>>>           c->ebx = 0x40;
> >>>>>>>
> >>>>>>>           kvm_base = KVM_CPUID_SIGNATURE_NEXT;
> >>>>>> This needs to depend on the machine type, but apart from that I think
> >>>>> I don't know why. Because the negative effects for this change don't
> exist
> >>>>> on current QEMU IIUC, and we don't have compatible problems for live
> >>>> migration.
> >>>>
> >>>> CPUID should never change with the same machine type and command
> line.
> >>>>
> >>> OK, then how do we add compat code for the CPUID change? It's not a
> visible
> >> property.
> >>
> >> You can add a new property, something like hv-cpuid-limits-eax.
> >>
> > Sounds good. Let me try.  :)
> >
> > Thanks,
> > -Gonglei
> Hi Gonglei,
> Any update on this issue?
> 
> We've found that SVVP on WS2016  is passing successfully with the above
> change
> and more than 64 vCPUs defined.
> 
> Vadim.

Thanks for reminding me :)

I'll send this patch this evening, and welcome to test it then.


Regards,
-Gonglei
diff mbox

Patch

diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 27fd050..efe3cbc 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -772,7 +772,7 @@  int kvm_arch_init_vcpu(CPUState *cs)

         c = &cpuid_data.entries[cpuid_i++];
         c->function = HYPERV_CPUID_IMPLEMENT_LIMITS;
-        c->eax = 0x40;
+        c->eax = -1;
         c->ebx = 0x40;

         kvm_base = KVM_CPUID_SIGNATURE_NEXT;