diff mbox series

[for-next,6/6] RDMA/efa: Do not delay freeing of DMA pages

Message ID 20200114085706.82229-7-galpress@amazon.com (mailing list archive)
State Changes Requested
Headers show
Series EFA updates 2020-01-14 | expand

Commit Message

Gal Pressman Jan. 14, 2020, 8:57 a.m. UTC 
When destroying a DMA mmapped object, there is no need to delay the
pages freeing to dealloc_ucontext as the kernel itself will keep
reference count for these pages.

Remove the special handling of DMA pages and call free_pages_exact on
destroy_qp/cq.

Reviewed-by: Firas JahJah <firasj@amazon.com>
Reviewed-by: Yossi Leybovich <sleybo@amazon.com>
Signed-off-by: Gal Pressman <galpress@amazon.com>
---
 drivers/infiniband/hw/efa/efa_verbs.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

Comments

Jason Gunthorpe Jan. 15, 2020, 7:51 p.m. UTC | #1 
On Tue, Jan 14, 2020 at 10:57:06AM +0200, Gal Pressman wrote:
> When destroying a DMA mmapped object, there is no need to delay the
> pages freeing to dealloc_ucontext as the kernel itself will keep
> reference count for these pages.

Why does the commit message talk about dealloc_ucontext but doesn't
change dealloc_ucontext?

> +	free_pages_exact(cq->cpu_addr, cq->size);
> 	rdma_user_mmap_entry_remove(cq->mmap_entry);

This is out of order, the pages can't be freed until the entry is
removed.

There is also a bug in rdma_user_mmap_entry_remove(),
entry->driver_removed needs to be set while holding the xa_lock or
this is not the required fence.

Jason
Gal Pressman Jan. 16, 2020, 8:26 a.m. UTC | #2 
On 15/01/2020 21:51, Jason Gunthorpe wrote:
> On Tue, Jan 14, 2020 at 10:57:06AM +0200, Gal Pressman wrote:
>> When destroying a DMA mmapped object, there is no need to delay the
>> pages freeing to dealloc_ucontext as the kernel itself will keep
>> reference count for these pages.
> 
> Why does the commit message talk about dealloc_ucontext but doesn't
> change dealloc_ucontext?

The commit message is wrong :\, we should not delay the free_pages_exact to the
mmap_free callback. We can "put" them on destroy flow and the pages will be
freed by the last consumer (either unmap or destroy).

> 
>> +	free_pages_exact(cq->cpu_addr, cq->size);
>> 	rdma_user_mmap_entry_remove(cq->mmap_entry);
> 
> This is out of order, the pages can't be freed until the entry is
> removed.

Right, I think the order is correct except rdma_user_mmap_entry_remove should be
moved to the beginning of the function.

> 
> There is also a bug in rdma_user_mmap_entry_remove(),
> entry->driver_removed needs to be set while holding the xa_lock or
> this is not the required fence.

I see you sent a patch, I'll take a look.
Thanks for the review.
diff mbox series

Patch

diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
index 9a6cff718c49..fd2fc60d569f 100644
--- a/drivers/infiniband/hw/efa/efa_verbs.c
+++ b/drivers/infiniband/hw/efa/efa_verbs.c
@@ -413,6 +413,7 @@  int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
 			  &qp->rq_dma_addr);
 		dma_unmap_single(&dev->pdev->dev, qp->rq_dma_addr, qp->rq_size,
 				 DMA_TO_DEVICE);
+		free_pages_exact(qp->rq_cpu_addr, qp->rq_size);
 	}
 
 	efa_qp_user_mmap_entries_remove(qp);
@@ -723,9 +724,7 @@  struct ib_qp *efa_create_qp(struct ib_pd *ibpd,
 	if (qp->rq_size) {
 		dma_unmap_single(&dev->pdev->dev, qp->rq_dma_addr, qp->rq_size,
 				 DMA_TO_DEVICE);
-
-		if (!qp->rq_mmap_entry)
-			free_pages_exact(qp->rq_cpu_addr, qp->rq_size);
+		free_pages_exact(qp->rq_cpu_addr, qp->rq_size);
 	}
 err_free_qp:
 	kfree(qp);
@@ -848,6 +847,7 @@  void efa_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
 	efa_destroy_cq_idx(dev, cq->cq_idx);
 	dma_unmap_single(&dev->pdev->dev, cq->dma_addr, cq->size,
 			 DMA_FROM_DEVICE);
+	free_pages_exact(cq->cpu_addr, cq->size);
 	rdma_user_mmap_entry_remove(cq->mmap_entry);
 }
 
@@ -987,8 +987,7 @@  int efa_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 err_free_mapped:
 	dma_unmap_single(&dev->pdev->dev, cq->dma_addr, cq->size,
 			 DMA_FROM_DEVICE);
-	if (!cq->mmap_entry)
-		free_pages_exact(cq->cpu_addr, cq->size);
+	free_pages_exact(cq->cpu_addr, cq->size);
 
 err_out:
 	atomic64_inc(&dev->stats.sw_stats.create_cq_err);
@@ -1549,10 +1548,6 @@  void efa_mmap_free(struct rdma_user_mmap_entry *rdma_entry)
 {
 	struct efa_user_mmap_entry *entry = to_emmap(rdma_entry);
 
-	/* DMA mapping is already gone, now free the pages */
-	if (entry->mmap_flag == EFA_MMAP_DMA_PAGE)
-		free_pages_exact(phys_to_virt(entry->address),
-				 entry->rdma_entry.npages * PAGE_SIZE);
 	kfree(entry);
 }