| Message ID | 1310214320-12488-1-git-send-email-levinsasha928@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Sat, Jul 9, 2011 at 3:25 PM, Sasha Levin <levinsasha928@gmail.com> wrote: > The patch raises the hard limit of VCPU count to 1024. > > This will allow developers to easily work on scalability > and will allow users to test high VCPU setups easily without > patching the kernel. > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS > now returns the recommended VCPU limit (which is still 64) - this > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS > returns the hard limit which is now 1024. > > Cc: Avi Kivity <avi@redhat.com> > Cc: Ingo Molnar <mingo@elte.hu> > Cc: Marcelo Tosatti <mtosatti@redhat.com> > Cc: Pekka Enberg <penberg@kernel.org> Acked-by: Pekka Enbeg <penberg@kernel.org> > Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi> > Signed-off-by: Sasha Levin <levinsasha928@gmail.com> -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, 2011-07-09 at 15:25 +0300, Sasha Levin wrote: > The patch raises the hard limit of VCPU count to 1024. > > This will allow developers to easily work on scalability > and will allow users to test high VCPU setups easily without > patching the kernel. > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS > now returns the recommended VCPU limit (which is still 64) - this > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS > returns the hard limit which is now 1024. > > Cc: Avi Kivity <avi@redhat.com> > Cc: Ingo Molnar <mingo@elte.hu> > Cc: Marcelo Tosatti <mtosatti@redhat.com> > Cc: Pekka Enberg <penberg@kernel.org> > Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi> > Signed-off-by: Sasha Levin <levinsasha928@gmail.com> > --- Ping? > Documentation/virtual/kvm/api.txt | 11 +++++++++-- > arch/x86/include/asm/kvm_host.h | 3 ++- > arch/x86/kvm/x86.c | 3 +++ > include/linux/kvm.h | 3 ++- > 4 files changed, 16 insertions(+), 4 deletions(-) > > diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt > index 42542eb..84883a0 100644 > --- a/Documentation/virtual/kvm/api.txt > +++ b/Documentation/virtual/kvm/api.txt > @@ -175,10 +175,17 @@ Parameters: vcpu id (apic id on x86) > Returns: vcpu fd on success, -1 on error > > This API adds a vcpu to a virtual machine. The vcpu id is a small integer > -in the range [0, max_vcpus). You can use KVM_CAP_NR_VCPUS of the > -KVM_CHECK_EXTENSION ioctl() to determine the value for max_vcpus at run-time. > +in the range [0, max_vcpus). > + > +The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of > +the KVM_CHECK_EXTENSION ioctl() at run-time. > +The maximum possible value for max_vcpus can be retrieved using the > +KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time. > + > If the KVM_CAP_NR_VCPUS does not exist, you should assume that max_vcpus is 4 > cpus max. > +If the KVM_CAP_MAX_VCPUS does not exist, you should assume that max_vcpus is > +same as the value returned from KVM_CAP_NR_VCPUS. > > 4.8 KVM_GET_DIRTY_LOG (vm ioctl) > > diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h > index d2ac8e2..7db3e66 100644 > --- a/arch/x86/include/asm/kvm_host.h > +++ b/arch/x86/include/asm/kvm_host.h > @@ -26,7 +26,8 @@ > #include <asm/mtrr.h> > #include <asm/msr-index.h> > > -#define KVM_MAX_VCPUS 64 > +#define KVM_MAX_VCPUS 1024 > +#define KVM_SOFT_MAX_VCPUS 64 > #define KVM_MEMORY_SLOTS 32 > /* memory slots that does not exposed to userspace */ > #define KVM_PRIVATE_MEM_SLOTS 4 > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > index 77c9d86..9766f46 100644 > --- a/arch/x86/kvm/x86.c > +++ b/arch/x86/kvm/x86.c > @@ -2006,6 +2006,9 @@ int kvm_dev_ioctl_check_extension(long ext) > r = !kvm_x86_ops->cpu_has_accelerated_tpr(); > break; > case KVM_CAP_NR_VCPUS: > + r = KVM_SOFT_MAX_VCPUS; > + break; > + case KVM_CAP_MAX_VCPUS: > r = KVM_MAX_VCPUS; > break; > case KVM_CAP_NR_MEMSLOTS: > diff --git a/include/linux/kvm.h b/include/linux/kvm.h > index 55ef181..077c193 100644 > --- a/include/linux/kvm.h > +++ b/include/linux/kvm.h > @@ -457,7 +457,7 @@ struct kvm_ppc_pvinfo { > #define KVM_CAP_VAPIC 6 > #define KVM_CAP_EXT_CPUID 7 > #define KVM_CAP_CLOCKSOURCE 8 > -#define KVM_CAP_NR_VCPUS 9 /* returns max vcpus per vm */ > +#define KVM_CAP_NR_VCPUS 9 /* returns recommended max vcpus per vm */ > #define KVM_CAP_NR_MEMSLOTS 10 /* returns max memory slots per vm */ > #define KVM_CAP_PIT 11 > #define KVM_CAP_NOP_IO_DELAY 12 > @@ -544,6 +544,7 @@ struct kvm_ppc_pvinfo { > #define KVM_CAP_TSC_CONTROL 60 > #define KVM_CAP_GET_TSC_KHZ 61 > #define KVM_CAP_PPC_BOOKE_SREGS 62 > +#define KVM_CAP_MAX_VCPUS 63 /* returns max vcpus per vm */ > > #ifdef KVM_CAP_IRQ_ROUTING >
On 07/09/2011 03:25 PM, Sasha Levin wrote: > The patch raises the hard limit of VCPU count to 1024. > > This will allow developers to easily work on scalability > and will allow users to test high VCPU setups easily without > patching the kernel. > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS > now returns the recommended VCPU limit (which is still 64) - this > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS > returns the hard limit which is now 1024. > Can 1024 vcpus even work without interrupt remapping? Looks like the patch will break coalesced mmio: static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev, gpa_t addr, int len) { struct kvm_coalesced_mmio_zone *zone; struct kvm_coalesced_mmio_ring *ring; unsigned avail; int i; /* Are we able to batch it ? */ /* last is the first free entry * check if we don't meet the first used entry * there is always one unused entry in the buffer */ ring = dev->kvm->coalesced_mmio_ring; avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX; if (avail < KVM_MAX_VCPUS) { /* full */ return 0; }
On Wed, 2011-07-13 at 16:30 +0300, Avi Kivity wrote: > On 07/09/2011 03:25 PM, Sasha Levin wrote: > > The patch raises the hard limit of VCPU count to 1024. > > > > This will allow developers to easily work on scalability > > and will allow users to test high VCPU setups easily without > > patching the kernel. > > > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS > > now returns the recommended VCPU limit (which is still 64) - this > > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS > > returns the hard limit which is now 1024. > > > > Can 1024 vcpus even work without interrupt remapping? > I'm not sure. I've successfully tried it with 255 vcpus. > Looks like the patch will break coalesced mmio: > > static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev, > gpa_t addr, int len) > { > struct kvm_coalesced_mmio_zone *zone; > struct kvm_coalesced_mmio_ring *ring; > unsigned avail; > int i; > > /* Are we able to batch it ? */ > > /* last is the first free entry > * check if we don't meet the first used entry > * there is always one unused entry in the buffer > */ > ring = dev->kvm->coalesced_mmio_ring; > avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX; > if (avail < KVM_MAX_VCPUS) { > /* full */ > return 0; > } > > I don't quite understand what KVM_MAX_VCPUS has to do with that if (). Shouldn't it check whether theres more than one buffer between first and last? What role does KVM_MAX_VCPUS play there?
On 07/13/2011 11:00 PM, Sasha Levin wrote: > On Wed, 2011-07-13 at 16:30 +0300, Avi Kivity wrote: > > On 07/09/2011 03:25 PM, Sasha Levin wrote: > > > The patch raises the hard limit of VCPU count to 1024. > > > > > > This will allow developers to easily work on scalability > > > and will allow users to test high VCPU setups easily without > > > patching the kernel. > > > > > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS > > > now returns the recommended VCPU limit (which is still 64) - this > > > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS > > > returns the hard limit which is now 1024. > > > > > > > Can 1024 vcpus even work without interrupt remapping? > > > > I'm not sure. I've successfully tried it with 255 vcpus. > Even 255 is problematic. One APIC ID is consumed by the IO-APIC, and ID 255 is reserved for broadcast IIRC. So at most 254 vcpus can be addressed. > > Looks like the patch will break coalesced mmio: > > > > static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev, > > gpa_t addr, int len) > > { > > struct kvm_coalesced_mmio_zone *zone; > > struct kvm_coalesced_mmio_ring *ring; > > unsigned avail; > > int i; > > > > /* Are we able to batch it ? */ > > > > /* last is the first free entry > > * check if we don't meet the first used entry > > * there is always one unused entry in the buffer > > */ > > ring = dev->kvm->coalesced_mmio_ring; > > avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX; > > if (avail< KVM_MAX_VCPUS) { > > /* full */ > > return 0; > > } > > > > > > I don't quite understand what KVM_MAX_VCPUS has to do with that if (). > Shouldn't it check whether theres more than one buffer between first and > last? What role does KVM_MAX_VCPUS play there? At most KVM_MAX_VCPUS can be writing simultaneously. Since we're checking outside the spinlock, we have to consider the worst case.
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt index 42542eb..84883a0 100644 --- a/Documentation/virtual/kvm/api.txt +++ b/Documentation/virtual/kvm/api.txt @@ -175,10 +175,17 @@ Parameters: vcpu id (apic id on x86) Returns: vcpu fd on success, -1 on error This API adds a vcpu to a virtual machine. The vcpu id is a small integer -in the range [0, max_vcpus). You can use KVM_CAP_NR_VCPUS of the -KVM_CHECK_EXTENSION ioctl() to determine the value for max_vcpus at run-time. +in the range [0, max_vcpus). + +The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of +the KVM_CHECK_EXTENSION ioctl() at run-time. +The maximum possible value for max_vcpus can be retrieved using the +KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time. + If the KVM_CAP_NR_VCPUS does not exist, you should assume that max_vcpus is 4 cpus max. +If the KVM_CAP_MAX_VCPUS does not exist, you should assume that max_vcpus is +same as the value returned from KVM_CAP_NR_VCPUS. 4.8 KVM_GET_DIRTY_LOG (vm ioctl) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index d2ac8e2..7db3e66 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -26,7 +26,8 @@ #include <asm/mtrr.h> #include <asm/msr-index.h> -#define KVM_MAX_VCPUS 64 +#define KVM_MAX_VCPUS 1024 +#define KVM_SOFT_MAX_VCPUS 64 #define KVM_MEMORY_SLOTS 32 /* memory slots that does not exposed to userspace */ #define KVM_PRIVATE_MEM_SLOTS 4 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 77c9d86..9766f46 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2006,6 +2006,9 @@ int kvm_dev_ioctl_check_extension(long ext) r = !kvm_x86_ops->cpu_has_accelerated_tpr(); break; case KVM_CAP_NR_VCPUS: + r = KVM_SOFT_MAX_VCPUS; + break; + case KVM_CAP_MAX_VCPUS: r = KVM_MAX_VCPUS; break; case KVM_CAP_NR_MEMSLOTS: diff --git a/include/linux/kvm.h b/include/linux/kvm.h index 55ef181..077c193 100644 --- a/include/linux/kvm.h +++ b/include/linux/kvm.h @@ -457,7 +457,7 @@ struct kvm_ppc_pvinfo { #define KVM_CAP_VAPIC 6 #define KVM_CAP_EXT_CPUID 7 #define KVM_CAP_CLOCKSOURCE 8 -#define KVM_CAP_NR_VCPUS 9 /* returns max vcpus per vm */ +#define KVM_CAP_NR_VCPUS 9 /* returns recommended max vcpus per vm */ #define KVM_CAP_NR_MEMSLOTS 10 /* returns max memory slots per vm */ #define KVM_CAP_PIT 11 #define KVM_CAP_NOP_IO_DELAY 12 @@ -544,6 +544,7 @@ struct kvm_ppc_pvinfo { #define KVM_CAP_TSC_CONTROL 60 #define KVM_CAP_GET_TSC_KHZ 61 #define KVM_CAP_PPC_BOOKE_SREGS 62 +#define KVM_CAP_MAX_VCPUS 63 /* returns max vcpus per vm */ #ifdef KVM_CAP_IRQ_ROUTING
The patch raises the hard limit of VCPU count to 1024. This will allow developers to easily work on scalability and will allow users to test high VCPU setups easily without patching the kernel. To prevent possible issues with current setups, KVM_CAP_NR_VCPUS now returns the recommended VCPU limit (which is still 64) - this should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS returns the hard limit which is now 1024. Cc: Avi Kivity <avi@redhat.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Marcelo Tosatti <mtosatti@redhat.com> Cc: Pekka Enberg <penberg@kernel.org> Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi> Signed-off-by: Sasha Levin <levinsasha928@gmail.com> --- Documentation/virtual/kvm/api.txt | 11 +++++++++-- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/x86.c | 3 +++ include/linux/kvm.h | 3 ++- 4 files changed, 16 insertions(+), 4 deletions(-)