| Message ID | 20210527125134.2116404-2-qperret@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | KVM: arm64: Reduce hyp_vmemmap overhead | expand |
On Thu, 27 May 2021 13:51:28 +0100, Quentin Perret <qperret@google.com> wrote: > > The hyp_page refcount helpers currently rely on the hyp_pool lock for > serialization. However, this means the refcounts can't be changed from > the buddy allocator core as it already holds the lock, which means pages > have to go through odd transient states. > > For example, when a page is freed, its refcount is set to 0, and the > lock is transiently released before the page can be attached to a free > list in the buddy tree. This is currently harmless as the allocator > checks the list node of each page to see if it is available for > allocation or not, but it means the page refcount can't be trusted to > represent the state of the page even if the pool lock is held. > > In order to fix this, remove the pool locking from the refcount helpers, > and move all the logic to the buddy allocator. This will simplify the > removal of the list node from struct hyp_page in a later patch. Is there any chance some documentation could be added so that we have a record of what the locking boundaries are? Something along the line of what we have in arch/arm64/kvm/vgic/vgic.c, for example. Thanks, M.
On Tuesday 01 Jun 2021 at 13:02:00 (+0100), Marc Zyngier wrote: > On Thu, 27 May 2021 13:51:28 +0100, > Quentin Perret <qperret@google.com> wrote: > > > > The hyp_page refcount helpers currently rely on the hyp_pool lock for > > serialization. However, this means the refcounts can't be changed from > > the buddy allocator core as it already holds the lock, which means pages > > have to go through odd transient states. > > > > For example, when a page is freed, its refcount is set to 0, and the > > lock is transiently released before the page can be attached to a free > > list in the buddy tree. This is currently harmless as the allocator > > checks the list node of each page to see if it is available for > > allocation or not, but it means the page refcount can't be trusted to > > represent the state of the page even if the pool lock is held. > > > > In order to fix this, remove the pool locking from the refcount helpers, > > and move all the logic to the buddy allocator. This will simplify the > > removal of the list node from struct hyp_page in a later patch. > > Is there any chance some documentation could be added so that we have > a record of what the locking boundaries are? Something along the line > of what we have in arch/arm64/kvm/vgic/vgic.c, for example. Sounds like a good idea, I'll go write something. Cheers, Quentin
diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h index 18a4494337bd..aada4d97de49 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/gfp.h +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -24,37 +24,20 @@ struct hyp_pool { static inline void hyp_page_ref_inc(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); p->refcount++; - hyp_spin_unlock(&pool->lock); } static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - int ret; - - hyp_spin_lock(&pool->lock); p->refcount--; - ret = (p->refcount == 0); - hyp_spin_unlock(&pool->lock); - - return ret; + return (p->refcount == 0); } static inline void hyp_set_page_refcounted(struct hyp_page *p) { - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - if (p->refcount) { - hyp_spin_unlock(&pool->lock); + if (p->refcount) BUG(); - } p->refcount = 1; - hyp_spin_unlock(&pool->lock); } /* Allocation */ diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index 237e03bf0cb1..04573bf35441 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -93,15 +93,6 @@ static void __hyp_attach_page(struct hyp_pool *pool, list_add_tail(&p->node, &pool->free_area[order]); } -static void hyp_attach_page(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - __hyp_attach_page(pool, p); - hyp_spin_unlock(&pool->lock); -} - static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, struct hyp_page *p, unsigned int order) @@ -128,16 +119,22 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, void hyp_put_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); if (hyp_page_ref_dec_and_test(p)) - hyp_attach_page(p); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); } void hyp_get_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); hyp_page_ref_inc(p); + hyp_spin_unlock(&pool->lock); } void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) @@ -159,8 +156,8 @@ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) p = list_first_entry(&pool->free_area[i], struct hyp_page, node); p = __hyp_extract_page(pool, p, order); - hyp_spin_unlock(&pool->lock); hyp_set_page_refcounted(p); + hyp_spin_unlock(&pool->lock); return hyp_page_to_virt(p); }
The hyp_page refcount helpers currently rely on the hyp_pool lock for serialization. However, this means the refcounts can't be changed from the buddy allocator core as it already holds the lock, which means pages have to go through odd transient states. For example, when a page is freed, its refcount is set to 0, and the lock is transiently released before the page can be attached to a free list in the buddy tree. This is currently harmless as the allocator checks the list node of each page to see if it is available for allocation or not, but it means the page refcount can't be trusted to represent the state of the page even if the pool lock is held. In order to fix this, remove the pool locking from the refcount helpers, and move all the logic to the buddy allocator. This will simplify the removal of the list node from struct hyp_page in a later patch. Signed-off-by: Quentin Perret <qperret@google.com> --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 21 ++------------------- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 19 ++++++++----------- 2 files changed, 10 insertions(+), 30 deletions(-)