| Message ID | 20210107001935.3732070-1-bgardon@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v3,1/2] KVM: x86/mmu: Ensure TDP MMU roots are freed after yield | expand |
On 07.01.2021 01:19, Ben Gardon wrote: > Many TDP MMU functions which need to perform some action on all TDP MMU > roots hold a reference on that root so that they can safely drop the MMU > lock in order to yield to other threads. However, when releasing the > reference on the root, there is a bug: the root will not be freed even > if its reference count (root_count) is reduced to 0. > > To simplify acquiring and releasing references on TDP MMU root pages, and > to ensure that these roots are properly freed, move the get/put operations > into another TDP MMU root iterator macro. > > Moving the get/put operations into an iterator macro also helps > simplify control flow when a root does need to be freed. Note that using > the list_for_each_entry_safe macro would not have been appropriate in > this situation because it could keep a pointer to the next root across > an MMU lock release + reacquire, during which time that root could be > freed. > > Reported-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> > Suggested-by: Paolo Bonzini <pbonzini@redhat.com> > Fixes: faaf05b00aec ("kvm: x86/mmu: Support zapping SPTEs in the TDP MMU") > Fixes: 063afacd8730 ("kvm: x86/mmu: Support invalidate range MMU notifier for TDP MMU") > Fixes: a6a0b05da9f3 ("kvm: x86/mmu: Support dirty logging for the TDP MMU") > Fixes: 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU") > Signed-off-by: Ben Gardon <bgardon@google.com> Tested this version, too, and it works fine, so: Tested-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> Thanks, Maciej
In the future, please document the changes in each revision, e.g. in a cover letter or in the ignored part of the diff. On Wed, Jan 06, 2021, Ben Gardon wrote: > Many TDP MMU functions which need to perform some action on all TDP MMU > roots hold a reference on that root so that they can safely drop the MMU > lock in order to yield to other threads. However, when releasing the > reference on the root, there is a bug: the root will not be freed even > if its reference count (root_count) is reduced to 0. > > To simplify acquiring and releasing references on TDP MMU root pages, and > to ensure that these roots are properly freed, move the get/put operations > into another TDP MMU root iterator macro. > > Moving the get/put operations into an iterator macro also helps > simplify control flow when a root does need to be freed. Note that using > the list_for_each_entry_safe macro would not have been appropriate in > this situation because it could keep a pointer to the next root across > an MMU lock release + reacquire, during which time that root could be > freed. > > Reported-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> > Suggested-by: Paolo Bonzini <pbonzini@redhat.com> > Fixes: faaf05b00aec ("kvm: x86/mmu: Support zapping SPTEs in the TDP MMU") > Fixes: 063afacd8730 ("kvm: x86/mmu: Support invalidate range MMU notifier for TDP MMU") > Fixes: a6a0b05da9f3 ("kvm: x86/mmu: Support dirty logging for the TDP MMU") > Fixes: 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU") > Signed-off-by: Ben Gardon <bgardon@google.com> Reviewed-by: Sean Christopherson <seanjc@google.com> > --- > arch/x86/kvm/mmu/tdp_mmu.c | 104 +++++++++++++++++-------------------- > 1 file changed, 48 insertions(+), 56 deletions(-) > > diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c > index 75db27fda8f3..d4191ed193cd 100644 > --- a/arch/x86/kvm/mmu/tdp_mmu.c > +++ b/arch/x86/kvm/mmu/tdp_mmu.c > @@ -44,7 +44,48 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) > WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); > } > > -#define for_each_tdp_mmu_root(_kvm, _root) \ > +static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) > +{ > + if (kvm_mmu_put_root(kvm, root)) > + kvm_tdp_mmu_free_root(kvm, root); > +} > + > +static inline bool tdp_mmu_next_root_valid(struct kvm *kvm, > + struct kvm_mmu_page *root) > +{ > + lockdep_assert_held(&kvm->mmu_lock); > + > + if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link)) > + return false; > + > + kvm_mmu_get_root(kvm, root); > + return true; > + > +} > + > +static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, > + struct kvm_mmu_page *root) > +{ > + struct kvm_mmu_page *next_root; > + > + next_root = list_next_entry(root, link); > + tdp_mmu_put_root(kvm, root); > + return next_root; > +} > + > +/* > + * Note: this iterator gets and puts references to the roots it iterates over. Maybe refer to it as "the yield_safe() variant" instead of "this" so that the comment makes sense with minimal context? > + * This makes it safe to release the MMU lock and yield within the loop, but > + * if exiting the loop early, the caller must drop the reference to the most > + * recent root. (Unless keeping a live reference is desirable.) > + */ Rather than encourage manually dropping the reference, what adding about a scary warning about not exiting the loop early? At this point, it seems unlikely that we'll end up with a legitimate use case for exiting yield_safe() early. And if we do, I think it'd be better to provide a macro to do the bookeeping instead of open coding it in the caller. And maybe throw a blurb into the changelog about that so future developers understand that that scary warning isn't set in stone? /* * The yield_safe() variant of the TDP root iterator gets and puts references to * the roots it iterates over. This makes it safe to release the MMU lock and * yield within the loop, but the caller MUST NOT exit the loop early. */
On 07/01/21 18:40, Sean Christopherson wrote: > /* > * The yield_safe() variant of the TDP root iterator gets and puts references to > * the roots it iterates over. This makes it safe to release the MMU lock and > * yield within the loop, but the caller MUST NOT exit the loop early. > */ > Nicer, thanks! Paolo
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 75db27fda8f3..d4191ed193cd 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -44,7 +44,48 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); } -#define for_each_tdp_mmu_root(_kvm, _root) \ +static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) +{ + if (kvm_mmu_put_root(kvm, root)) + kvm_tdp_mmu_free_root(kvm, root); +} + +static inline bool tdp_mmu_next_root_valid(struct kvm *kvm, + struct kvm_mmu_page *root) +{ + lockdep_assert_held(&kvm->mmu_lock); + + if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link)) + return false; + + kvm_mmu_get_root(kvm, root); + return true; + +} + +static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, + struct kvm_mmu_page *root) +{ + struct kvm_mmu_page *next_root; + + next_root = list_next_entry(root, link); + tdp_mmu_put_root(kvm, root); + return next_root; +} + +/* + * Note: this iterator gets and puts references to the roots it iterates over. + * This makes it safe to release the MMU lock and yield within the loop, but + * if exiting the loop early, the caller must drop the reference to the most + * recent root. (Unless keeping a live reference is desirable.) + */ +#define for_each_tdp_mmu_root_yield_safe(_kvm, _root) \ + for (_root = list_first_entry(&_kvm->arch.tdp_mmu_roots, \ + typeof(*_root), link); \ + tdp_mmu_next_root_valid(_kvm, _root); \ + _root = tdp_mmu_next_root(_kvm, _root)) + +#define for_each_tdp_mmu_root(_kvm, _root) \ list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link) bool is_tdp_mmu_root(struct kvm *kvm, hpa_t hpa) @@ -447,18 +488,9 @@ bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end) struct kvm_mmu_page *root; bool flush = false; - for_each_tdp_mmu_root(kvm, root) { - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - + for_each_tdp_mmu_root_yield_safe(kvm, root) flush |= zap_gfn_range(kvm, root, start, end, true); - kvm_mmu_put_root(kvm, root); - } - return flush; } @@ -619,13 +651,7 @@ static int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm, unsigned long start, int ret = 0; int as_id; - for_each_tdp_mmu_root(kvm, root) { - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - + for_each_tdp_mmu_root_yield_safe(kvm, root) { as_id = kvm_mmu_page_as_id(root); slots = __kvm_memslots(kvm, as_id); kvm_for_each_memslot(memslot, slots) { @@ -647,8 +673,6 @@ static int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm, unsigned long start, ret |= handler(kvm, memslot, root, gfn_start, gfn_end, data); } - - kvm_mmu_put_root(kvm, root); } return ret; @@ -838,21 +862,13 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, struct kvm_memory_slot *slot, int root_as_id; bool spte_set = false; - for_each_tdp_mmu_root(kvm, root) { + for_each_tdp_mmu_root_yield_safe(kvm, root) { root_as_id = kvm_mmu_page_as_id(root); if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= wrprot_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages, min_level); - - kvm_mmu_put_root(kvm, root); } return spte_set; @@ -906,21 +922,13 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, struct kvm_memory_slot *slot) int root_as_id; bool spte_set = false; - for_each_tdp_mmu_root(kvm, root) { + for_each_tdp_mmu_root_yield_safe(kvm, root) { root_as_id = kvm_mmu_page_as_id(root); if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= clear_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - kvm_mmu_put_root(kvm, root); } return spte_set; @@ -1029,21 +1037,13 @@ bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot) int root_as_id; bool spte_set = false; - for_each_tdp_mmu_root(kvm, root) { + for_each_tdp_mmu_root_yield_safe(kvm, root) { root_as_id = kvm_mmu_page_as_id(root); if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= set_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - kvm_mmu_put_root(kvm, root); } return spte_set; } @@ -1089,21 +1089,13 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, struct kvm_mmu_page *root; int root_as_id; - for_each_tdp_mmu_root(kvm, root) { + for_each_tdp_mmu_root_yield_safe(kvm, root) { root_as_id = kvm_mmu_page_as_id(root); if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - zap_collapsible_spte_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - kvm_mmu_put_root(kvm, root); } }
Many TDP MMU functions which need to perform some action on all TDP MMU roots hold a reference on that root so that they can safely drop the MMU lock in order to yield to other threads. However, when releasing the reference on the root, there is a bug: the root will not be freed even if its reference count (root_count) is reduced to 0. To simplify acquiring and releasing references on TDP MMU root pages, and to ensure that these roots are properly freed, move the get/put operations into another TDP MMU root iterator macro. Moving the get/put operations into an iterator macro also helps simplify control flow when a root does need to be freed. Note that using the list_for_each_entry_safe macro would not have been appropriate in this situation because it could keep a pointer to the next root across an MMU lock release + reacquire, during which time that root could be freed. Reported-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Fixes: faaf05b00aec ("kvm: x86/mmu: Support zapping SPTEs in the TDP MMU") Fixes: 063afacd8730 ("kvm: x86/mmu: Support invalidate range MMU notifier for TDP MMU") Fixes: a6a0b05da9f3 ("kvm: x86/mmu: Support dirty logging for the TDP MMU") Fixes: 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU") Signed-off-by: Ben Gardon <bgardon@google.com> --- arch/x86/kvm/mmu/tdp_mmu.c | 104 +++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 56 deletions(-)