Message ID | 20180925175844.20277-7-vkuznets@redhat.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | x86/kvm/nVMX: optimize MMU switch between L1 and L2 | expand |
On Tue, Sep 25, 2018 at 07:58:41PM +0200, Vitaly Kuznetsov wrote: > In preparation to MMU reconfiguration avoidance we need a space to > cache source data. As this partially intersects with kvm_mmu_page_role, > create 64bit sized union kvm_mmu_role holding both base and extended data. > No functional change. > > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> > --- One nit below, other than that... Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com> > diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c > index e59e5f49c8c2..bb1ef0f68f8e 100644 > --- a/arch/x86/kvm/mmu.c > +++ b/arch/x86/kvm/mmu.c > @@ -2359,7 +2359,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, > int collisions = 0; > LIST_HEAD(invalid_list); > > - role = vcpu->arch.mmu->base_role; > + role = vcpu->arch.mmu->mmu_role.base; > role.level = level; > role.direct = direct; > if (role.direct) > @@ -4407,7 +4407,8 @@ static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu, > void > reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context) > { > - bool uses_nx = context->nx || context->base_role.smep_andnot_wp; > + bool uses_nx = context->nx || > + context->mmu_role.base.smep_andnot_wp; > struct rsvd_bits_validate *shadow_zero_check; > int i; > > @@ -4726,7 +4727,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) > { > struct kvm_mmu *context = vcpu->arch.mmu; > > - context->base_role.word = mmu_base_role_mask.word & > + context->mmu_role.base.word = mmu_base_role_mask.word & > kvm_calc_tdp_mmu_root_page_role(vcpu).word; > context->page_fault = tdp_page_fault; > context->sync_page = nonpaging_sync_page; > @@ -4807,7 +4808,7 @@ void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu) > else > paging32_init_context(vcpu, context); > > - context->base_role.word = mmu_base_role_mask.word & > + context->mmu_role.base.word = mmu_base_role_mask.word & > kvm_calc_shadow_mmu_root_page_role(vcpu).word; > reset_shadow_zero_bits_mask(vcpu, context); > } > @@ -4816,7 +4817,7 @@ EXPORT_SYMBOL_GPL(kvm_init_shadow_mmu); > static union kvm_mmu_page_role > kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty) > { > - union kvm_mmu_page_role role = vcpu->arch.mmu->base_role; > + union kvm_mmu_page_role role = vcpu->arch.mmu->mmu_role.base; > > role.level = PT64_ROOT_4LEVEL; > role.direct = false; > @@ -4846,7 +4847,8 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, > context->update_pte = ept_update_pte; > context->root_level = PT64_ROOT_4LEVEL; > context->direct_map = false; > - context->base_role.word = root_page_role.word & mmu_base_role_mask.word; > + context->mmu_role.base.word = > + root_page_role.word & mmu_base_role_mask.word; > context->get_pdptr = kvm_pdptr_read; > > update_permission_bitmask(vcpu, context, true); > @@ -5161,10 +5163,13 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, > > local_flush = true; > while (npte--) { > + unsigned int base_role = Nit: should this be a u32 to match mmu_role.base.word? > + vcpu->arch.mmu->mmu_role.base.word; > + > entry = *spte; > mmu_page_zap_pte(vcpu->kvm, sp, spte); > if (gentry && > - !((sp->role.word ^ vcpu->arch.mmu->base_role.word) > + !((sp->role.word ^ base_role) > & mmu_base_role_mask.word) && rmap_can_add(vcpu)) > mmu_pte_write_new_pte(vcpu, sp, spte, &gentry); > if (need_remote_flush(entry, *spte)) > @@ -5861,6 +5866,16 @@ int kvm_mmu_module_init(void) > { > int ret = -ENOMEM; > > + /* > + * MMU roles use union aliasing which is, generally speaking, an > + * undefined behavior. However, we supposedly know how compilers behave > + * and the current status quo is unlikely to change. Guardians below are > + * supposed to let us know if the assumption becomes false. > + */ > + BUILD_BUG_ON(sizeof(union kvm_mmu_page_role) != sizeof(u32)); > + BUILD_BUG_ON(sizeof(union kvm_mmu_extended_role) != sizeof(u32)); > + BUILD_BUG_ON(sizeof(union kvm_mmu_role) != sizeof(u64)); > + > kvm_mmu_reset_all_pte_masks(); > > pte_list_desc_cache = kmem_cache_create("pte_list_desc", > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > index 93ff08136fc1..c56a80c15c4f 100644 > --- a/arch/x86/kvm/vmx.c > +++ b/arch/x86/kvm/vmx.c > @@ -9321,7 +9321,7 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, > > kvm_mmu_unload(vcpu); > mmu->ept_ad = accessed_dirty; > - mmu->base_role.ad_disabled = !accessed_dirty; > + mmu->mmu_role.base.ad_disabled = !accessed_dirty; > vmcs12->ept_pointer = address; > /* > * TODO: Check what's the correct approach in case > -- > 2.17.1 >
Sean Christopherson <sean.j.christopherson@intel.com> writes: > On Tue, Sep 25, 2018 at 07:58:41PM +0200, Vitaly Kuznetsov wrote: >> In preparation to MMU reconfiguration avoidance we need a space to >> cache source data. As this partially intersects with kvm_mmu_page_role, >> create 64bit sized union kvm_mmu_role holding both base and extended data. >> No functional change. >> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> >> --- > > One nit below, other than that... > > Reviewed-by: Sean Christopherson <sean.j.christopherson@intel.com> > >> diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c >> index e59e5f49c8c2..bb1ef0f68f8e 100644 >> --- a/arch/x86/kvm/mmu.c >> +++ b/arch/x86/kvm/mmu.c >> @@ -2359,7 +2359,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, >> int collisions = 0; >> LIST_HEAD(invalid_list); >> >> - role = vcpu->arch.mmu->base_role; >> + role = vcpu->arch.mmu->mmu_role.base; >> role.level = level; >> role.direct = direct; >> if (role.direct) >> @@ -4407,7 +4407,8 @@ static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu, >> void >> reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context) >> { >> - bool uses_nx = context->nx || context->base_role.smep_andnot_wp; >> + bool uses_nx = context->nx || >> + context->mmu_role.base.smep_andnot_wp; >> struct rsvd_bits_validate *shadow_zero_check; >> int i; >> >> @@ -4726,7 +4727,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) >> { >> struct kvm_mmu *context = vcpu->arch.mmu; >> >> - context->base_role.word = mmu_base_role_mask.word & >> + context->mmu_role.base.word = mmu_base_role_mask.word & >> kvm_calc_tdp_mmu_root_page_role(vcpu).word; >> context->page_fault = tdp_page_fault; >> context->sync_page = nonpaging_sync_page; >> @@ -4807,7 +4808,7 @@ void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu) >> else >> paging32_init_context(vcpu, context); >> >> - context->base_role.word = mmu_base_role_mask.word & >> + context->mmu_role.base.word = mmu_base_role_mask.word & >> kvm_calc_shadow_mmu_root_page_role(vcpu).word; >> reset_shadow_zero_bits_mask(vcpu, context); >> } >> @@ -4816,7 +4817,7 @@ EXPORT_SYMBOL_GPL(kvm_init_shadow_mmu); >> static union kvm_mmu_page_role >> kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty) >> { >> - union kvm_mmu_page_role role = vcpu->arch.mmu->base_role; >> + union kvm_mmu_page_role role = vcpu->arch.mmu->mmu_role.base; >> >> role.level = PT64_ROOT_4LEVEL; >> role.direct = false; >> @@ -4846,7 +4847,8 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, >> context->update_pte = ept_update_pte; >> context->root_level = PT64_ROOT_4LEVEL; >> context->direct_map = false; >> - context->base_role.word = root_page_role.word & mmu_base_role_mask.word; >> + context->mmu_role.base.word = >> + root_page_role.word & mmu_base_role_mask.word; >> context->get_pdptr = kvm_pdptr_read; >> >> update_permission_bitmask(vcpu, context, true); >> @@ -5161,10 +5163,13 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, >> >> local_flush = true; >> while (npte--) { >> + unsigned int base_role = > > Nit: should this be a u32 to match mmu_role.base.word? > Of course, will do in v3. Thanks! >> + vcpu->arch.mmu->mmu_role.base.word; >> + >> entry = *spte; >> mmu_page_zap_pte(vcpu->kvm, sp, spte); >> if (gentry && >> - !((sp->role.word ^ vcpu->arch.mmu->base_role.word) >> + !((sp->role.word ^ base_role) >> & mmu_base_role_mask.word) && rmap_can_add(vcpu)) >> mmu_pte_write_new_pte(vcpu, sp, spte, &gentry); >> if (need_remote_flush(entry, *spte)) >> @@ -5861,6 +5866,16 @@ int kvm_mmu_module_init(void) >> { >> int ret = -ENOMEM; >> >> + /* >> + * MMU roles use union aliasing which is, generally speaking, an >> + * undefined behavior. However, we supposedly know how compilers behave >> + * and the current status quo is unlikely to change. Guardians below are >> + * supposed to let us know if the assumption becomes false. >> + */ >> + BUILD_BUG_ON(sizeof(union kvm_mmu_page_role) != sizeof(u32)); >> + BUILD_BUG_ON(sizeof(union kvm_mmu_extended_role) != sizeof(u32)); >> + BUILD_BUG_ON(sizeof(union kvm_mmu_role) != sizeof(u64)); >> + >> kvm_mmu_reset_all_pte_masks(); >> >> pte_list_desc_cache = kmem_cache_create("pte_list_desc", >> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c >> index 93ff08136fc1..c56a80c15c4f 100644 >> --- a/arch/x86/kvm/vmx.c >> +++ b/arch/x86/kvm/vmx.c >> @@ -9321,7 +9321,7 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, >> >> kvm_mmu_unload(vcpu); >> mmu->ept_ad = accessed_dirty; >> - mmu->base_role.ad_disabled = !accessed_dirty; >> + mmu->mmu_role.base.ad_disabled = !accessed_dirty; >> vmcs12->ept_pointer = address; >> /* >> * TODO: Check what's the correct approach in case >> -- >> 2.17.1 >>
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index a64da200aed1..1821b0215230 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -247,7 +247,7 @@ struct kvm_mmu_memory_cache { * @nxe, @cr0_wp, @smep_andnot_wp and @smap_andnot_wp. */ union kvm_mmu_page_role { - unsigned word; + u32 word; struct { unsigned level:4; unsigned cr4_pae:1; @@ -273,6 +273,18 @@ union kvm_mmu_page_role { }; }; +union kvm_mmu_extended_role { + u32 word; +}; + +union kvm_mmu_role { + u64 as_u64; + struct { + union kvm_mmu_page_role base; + union kvm_mmu_extended_role ext; + }; +}; + struct kvm_rmap_head { unsigned long val; }; @@ -360,7 +372,7 @@ struct kvm_mmu { void (*update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, const void *pte); hpa_t root_hpa; - union kvm_mmu_page_role base_role; + union kvm_mmu_role mmu_role; u8 root_level; u8 shadow_root_level; u8 ept_ad; diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index e59e5f49c8c2..bb1ef0f68f8e 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -2359,7 +2359,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, int collisions = 0; LIST_HEAD(invalid_list); - role = vcpu->arch.mmu->base_role; + role = vcpu->arch.mmu->mmu_role.base; role.level = level; role.direct = direct; if (role.direct) @@ -4407,7 +4407,8 @@ static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu, void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, struct kvm_mmu *context) { - bool uses_nx = context->nx || context->base_role.smep_andnot_wp; + bool uses_nx = context->nx || + context->mmu_role.base.smep_andnot_wp; struct rsvd_bits_validate *shadow_zero_check; int i; @@ -4726,7 +4727,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) { struct kvm_mmu *context = vcpu->arch.mmu; - context->base_role.word = mmu_base_role_mask.word & + context->mmu_role.base.word = mmu_base_role_mask.word & kvm_calc_tdp_mmu_root_page_role(vcpu).word; context->page_fault = tdp_page_fault; context->sync_page = nonpaging_sync_page; @@ -4807,7 +4808,7 @@ void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu) else paging32_init_context(vcpu, context); - context->base_role.word = mmu_base_role_mask.word & + context->mmu_role.base.word = mmu_base_role_mask.word & kvm_calc_shadow_mmu_root_page_role(vcpu).word; reset_shadow_zero_bits_mask(vcpu, context); } @@ -4816,7 +4817,7 @@ EXPORT_SYMBOL_GPL(kvm_init_shadow_mmu); static union kvm_mmu_page_role kvm_calc_shadow_ept_root_page_role(struct kvm_vcpu *vcpu, bool accessed_dirty) { - union kvm_mmu_page_role role = vcpu->arch.mmu->base_role; + union kvm_mmu_page_role role = vcpu->arch.mmu->mmu_role.base; role.level = PT64_ROOT_4LEVEL; role.direct = false; @@ -4846,7 +4847,8 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, context->update_pte = ept_update_pte; context->root_level = PT64_ROOT_4LEVEL; context->direct_map = false; - context->base_role.word = root_page_role.word & mmu_base_role_mask.word; + context->mmu_role.base.word = + root_page_role.word & mmu_base_role_mask.word; context->get_pdptr = kvm_pdptr_read; update_permission_bitmask(vcpu, context, true); @@ -5161,10 +5163,13 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, local_flush = true; while (npte--) { + unsigned int base_role = + vcpu->arch.mmu->mmu_role.base.word; + entry = *spte; mmu_page_zap_pte(vcpu->kvm, sp, spte); if (gentry && - !((sp->role.word ^ vcpu->arch.mmu->base_role.word) + !((sp->role.word ^ base_role) & mmu_base_role_mask.word) && rmap_can_add(vcpu)) mmu_pte_write_new_pte(vcpu, sp, spte, &gentry); if (need_remote_flush(entry, *spte)) @@ -5861,6 +5866,16 @@ int kvm_mmu_module_init(void) { int ret = -ENOMEM; + /* + * MMU roles use union aliasing which is, generally speaking, an + * undefined behavior. However, we supposedly know how compilers behave + * and the current status quo is unlikely to change. Guardians below are + * supposed to let us know if the assumption becomes false. + */ + BUILD_BUG_ON(sizeof(union kvm_mmu_page_role) != sizeof(u32)); + BUILD_BUG_ON(sizeof(union kvm_mmu_extended_role) != sizeof(u32)); + BUILD_BUG_ON(sizeof(union kvm_mmu_role) != sizeof(u64)); + kvm_mmu_reset_all_pte_masks(); pte_list_desc_cache = kmem_cache_create("pte_list_desc", diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 93ff08136fc1..c56a80c15c4f 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -9321,7 +9321,7 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, kvm_mmu_unload(vcpu); mmu->ept_ad = accessed_dirty; - mmu->base_role.ad_disabled = !accessed_dirty; + mmu->mmu_role.base.ad_disabled = !accessed_dirty; vmcs12->ept_pointer = address; /* * TODO: Check what's the correct approach in case
In preparation to MMU reconfiguration avoidance we need a space to cache source data. As this partially intersects with kvm_mmu_page_role, create 64bit sized union kvm_mmu_role holding both base and extended data. No functional change. Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com> --- Changes since v1: - Rename: kvm_mmu_scache -> kvm_mmu_extended_role mmu_role.scache -> mmu_role.ext mmu_role.base_role -> mmu_role.base [Sean Christopherson] - Add BUILD_BUG_ONs checking unions sizes. [Sean Christopherson] - Use explicit u32 for kvm_mmu_page_role/kvm_mmu_extended_role.word and u64 for kvm_mmu_role.as_u64 --- arch/x86/include/asm/kvm_host.h | 16 ++++++++++++++-- arch/x86/kvm/mmu.c | 29 ++++++++++++++++++++++------- arch/x86/kvm/vmx.c | 2 +- 3 files changed, 37 insertions(+), 10 deletions(-)