| Message ID | 20200608115714.1139735-1-clg@kaod.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | KVM: PPC: Book3S HV: increase KVMPPC_NR_LPIDS on POWER8 and POWER9 | expand |
On 6/8/20 1:57 PM, Cédric Le Goater wrote: > POWER8 and POWER9 have 12-bit LPIDs. Change LPID_RSVD to support up to > (4096 - 2) guests on these processors. POWER7 is kept the same with a > limitation of (1024 - 2), but it might be time to drop KVM support for > POWER7. > > Tested with 2048 guests * 4 vCPUs on a witherspoon system with 512G > RAM and a bit of swap. For the record, it is possible to run 4094 guests * 4 vCPUs on a POWER9 system with 1TB. It takes ~5m to boot them all. CONFIG_NR_IRQS needs to be increased to support 4094 * 4 escalation interrupts. Cheers, C. > > Signed-off-by: Cédric Le Goater <clg@kaod.org> > --- > arch/powerpc/include/asm/reg.h | 3 ++- > arch/powerpc/kvm/book3s_64_mmu_hv.c | 8 ++++++-- > 2 files changed, 8 insertions(+), 3 deletions(-) > > diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h > index 88e6c78100d9..b70bbfb0ea3c 100644 > --- a/arch/powerpc/include/asm/reg.h > +++ b/arch/powerpc/include/asm/reg.h > @@ -473,7 +473,8 @@ > #ifndef SPRN_LPID > #define SPRN_LPID 0x13F /* Logical Partition Identifier */ > #endif > -#define LPID_RSVD 0x3ff /* Reserved LPID for partn switching */ > +#define LPID_RSVD_POWER7 0x3ff /* Reserved LPID for partn switching */ > +#define LPID_RSVD 0xfff /* Reserved LPID for partn switching */ > #define SPRN_HMER 0x150 /* Hypervisor maintenance exception reg */ > #define HMER_DEBUG_TRIG (1ul << (63 - 17)) /* Debug trigger */ > #define SPRN_HMEER 0x151 /* Hyp maintenance exception enable reg */ > diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c > index 18aed9775a3c..23035ab2ec50 100644 > --- a/arch/powerpc/kvm/book3s_64_mmu_hv.c > +++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c > @@ -260,11 +260,15 @@ int kvmppc_mmu_hv_init(void) > if (!mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE)) > return -EINVAL; > > - /* POWER7 has 10-bit LPIDs (12-bit in POWER8) */ > host_lpid = 0; > if (cpu_has_feature(CPU_FTR_HVMODE)) > host_lpid = mfspr(SPRN_LPID); > - rsvd_lpid = LPID_RSVD; > + > + /* POWER8 and above have 12-bit LPIDs (10-bit in POWER7) */ > + if (cpu_has_feature(CPU_FTR_ARCH_207S)) > + rsvd_lpid = LPID_RSVD; > + else > + rsvd_lpid = LPID_RSVD_POWER7; > > kvmppc_init_lpid(rsvd_lpid + 1); > >
On Mon, Jun 08, 2020 at 01:57:14PM +0200, Cédric Le Goater wrote: > POWER8 and POWER9 have 12-bit LPIDs. Change LPID_RSVD to support up to > (4096 - 2) guests on these processors. POWER7 is kept the same with a > limitation of (1024 - 2), but it might be time to drop KVM support for > POWER7. > > Tested with 2048 guests * 4 vCPUs on a witherspoon system with 512G > RAM and a bit of swap. > > Signed-off-by: Cédric Le Goater <clg@kaod.org> Thanks, patch applied to my kvm-ppc-next branch. Paul.
On 7/23/20 8:20 AM, Paul Mackerras wrote: > On Mon, Jun 08, 2020 at 01:57:14PM +0200, Cédric Le Goater wrote: >> POWER8 and POWER9 have 12-bit LPIDs. Change LPID_RSVD to support up to >> (4096 - 2) guests on these processors. POWER7 is kept the same with a >> limitation of (1024 - 2), but it might be time to drop KVM support for >> POWER7. >> >> Tested with 2048 guests * 4 vCPUs on a witherspoon system with 512G >> RAM and a bit of swap. >> >> Signed-off-by: Cédric Le Goater <clg@kaod.org> > > Thanks, patch applied to my kvm-ppc-next branch. We have pushed the limits further on a 1TB system and reached the limit of 4094 guests with 16 vCPUs. With more vCPUs, the system starts to check-stop. We believe that the pages used by the interrupt controller for the backing store of the XIVE internal tables (END and NVT) allocated with GFP_KERNEL are reclaimable. I am thinking of changing the allocation flags with : __GFP_NORETRY | __GFP_NOWARN | __GFP_NOMEMALLOC because XIVE should be able to fail gracefully if the system is low on mem. Is that correct ? Thanks, C.
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h index 88e6c78100d9..b70bbfb0ea3c 100644 --- a/arch/powerpc/include/asm/reg.h +++ b/arch/powerpc/include/asm/reg.h @@ -473,7 +473,8 @@ #ifndef SPRN_LPID #define SPRN_LPID 0x13F /* Logical Partition Identifier */ #endif -#define LPID_RSVD 0x3ff /* Reserved LPID for partn switching */ +#define LPID_RSVD_POWER7 0x3ff /* Reserved LPID for partn switching */ +#define LPID_RSVD 0xfff /* Reserved LPID for partn switching */ #define SPRN_HMER 0x150 /* Hypervisor maintenance exception reg */ #define HMER_DEBUG_TRIG (1ul << (63 - 17)) /* Debug trigger */ #define SPRN_HMEER 0x151 /* Hyp maintenance exception enable reg */ diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c index 18aed9775a3c..23035ab2ec50 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_hv.c +++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c @@ -260,11 +260,15 @@ int kvmppc_mmu_hv_init(void) if (!mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE)) return -EINVAL; - /* POWER7 has 10-bit LPIDs (12-bit in POWER8) */ host_lpid = 0; if (cpu_has_feature(CPU_FTR_HVMODE)) host_lpid = mfspr(SPRN_LPID); - rsvd_lpid = LPID_RSVD; + + /* POWER8 and above have 12-bit LPIDs (10-bit in POWER7) */ + if (cpu_has_feature(CPU_FTR_ARCH_207S)) + rsvd_lpid = LPID_RSVD; + else + rsvd_lpid = LPID_RSVD_POWER7; kvmppc_init_lpid(rsvd_lpid + 1);
POWER8 and POWER9 have 12-bit LPIDs. Change LPID_RSVD to support up to (4096 - 2) guests on these processors. POWER7 is kept the same with a limitation of (1024 - 2), but it might be time to drop KVM support for POWER7. Tested with 2048 guests * 4 vCPUs on a witherspoon system with 512G RAM and a bit of swap. Signed-off-by: Cédric Le Goater <clg@kaod.org> --- arch/powerpc/include/asm/reg.h | 3 ++- arch/powerpc/kvm/book3s_64_mmu_hv.c | 8 ++++++-- 2 files changed, 8 insertions(+), 3 deletions(-)