| Message ID | 20230907014711.3869840-1-rkannoth@marvell.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net,v2] octeontx2-pf: Fix page pool cache index corruption. | expand |
On 2023-09-07 07:17:11 [+0530], Ratheesh Kannoth wrote: > The access to page pool `cache' array and the `count' variable > is not locked. Page pool cache access is fine as long as there > is only one consumer per pool. > > octeontx2 driver fills in rx buffers from page pool in NAPI context. > If system is stressed and could not allocate buffers, refiiling work > will be delegated to a delayed workqueue. This means that there are > two cosumers to the page pool cache. > > Either workqueue or IRQ/NAPI can be run on other CPU. This will lead > to lock less access, hence corruption of cache pool indexes. > > To fix this issue, NAPI is rescheduled from workqueue context to refill > rx buffers. > > Fixes: b2e3406a38f0 ("octeontx2-pf: Add support for page pool") > Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com> Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c > index 8511906cb4e2..997fedac3a98 100644 > --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c > +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c > static void otx2_pool_refill_task(struct work_struct *work) > { > struct otx2_cq_queue *cq; > - struct otx2_pool *rbpool; > struct refill_work *wrk; > - int qidx, free_ptrs = 0; > struct otx2_nic *pfvf; > - dma_addr_t bufptr; > + int qidx; > > wrk = container_of(work, struct refill_work, pool_refill_work.work); > pfvf = wrk->pf; > qidx = wrk - pfvf->refill_wrk; > cq = &pfvf->qset.cq[qidx]; … > cq->refill_task_sched = false; > + > + local_bh_disable(); > + napi_schedule(wrk->napi); > + local_bh_enable(); This is a nitpick since I haven't look how it works exactly: Is it possible that the wrk->napi pointer gets overwritten by otx2_napi_handler() since you cleared cq->refill_task_sched() earlier? > } > > int otx2_config_nix_queues(struct otx2_nic *pfvf) > diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c > index e369baf11530..b778ed366f81 100644 > --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c > +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c > @@ -561,9 +565,24 @@ int otx2_napi_handler(struct napi_struct *napi, int budget) > otx2_config_irq_coalescing(pfvf, i); > } > > - /* Re-enable interrupts */ > - otx2_write64(pfvf, NIX_LF_CINTX_ENA_W1S(cq_poll->cint_idx), > - BIT_ULL(0)); > + if (unlikely(!filled_cnt)) { > + struct refill_work *work; > + struct delayed_work *dwork; > + > + work = &pfvf->refill_wrk[cq->cq_idx]; > + dwork = &work->pool_refill_work; > + /* Schedule a task if no other task is running */ > + if (!cq->refill_task_sched) { > + work->napi = napi; > + cq->refill_task_sched = true; > + schedule_delayed_work(dwork, > + msecs_to_jiffies(100)); > + } > + } else { > + /* Re-enable interrupts */ > + otx2_write64(pfvf, NIX_LF_CINTX_ENA_W1S(cq_poll->cint_idx), > + BIT_ULL(0)); > + } > } > return workdone; > } Sebastian
> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > Subject: [EXT] Re: [PATCH net v2] octeontx2-pf: Fix page pool cache index > corruption. > > cq->refill_task_sched = false; > > + > > + local_bh_disable(); > > + napi_schedule(wrk->napi); > > + local_bh_enable(); > > This is a nitpick since I haven't look how it works exactly: Is it possible that the > wrk->napi pointer gets overwritten by > otx2_napi_handler() since you cleared cq->refill_task_sched() earlier? I don’t see any issue here. As NAPI and workqueue execution is serialized (as interrupt is disabled when Workqueue is scheduled). I don’t think we can move "refill_task_sched = false" after local_bh_enable(). But we can move refill_task_sched = false as below . but I don’t see a need. + + local_bh_disable(); + napi_schedule(wrk->napi); + cq->refill_task_sched = false; + local_bh_enable();
On 2023-09-07 08:15:58 [+0000], Ratheesh Kannoth wrote: > > From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > > Subject: [EXT] Re: [PATCH net v2] octeontx2-pf: Fix page pool cache index > > corruption. > > > cq->refill_task_sched = false; > > > + > > > + local_bh_disable(); > > > + napi_schedule(wrk->napi); > > > + local_bh_enable(); > > > > This is a nitpick since I haven't look how it works exactly: Is it possible that the > > wrk->napi pointer gets overwritten by > > otx2_napi_handler() since you cleared cq->refill_task_sched() earlier? > I don’t see any issue here. As NAPI and workqueue execution is serialized (as interrupt is disabled when > Workqueue is scheduled). I don’t think we can move "refill_task_sched = false" after local_bh_enable(). > But we can move refill_task_sched = false as below . but I don’t see a need. Right, there might be no issue I was just asking. I don't know how the cq <-> NAPI mapping is working. If you say that there is no race/issue, okay ;) > + > + local_bh_disable(); > + napi_schedule(wrk->napi); > + cq->refill_task_sched = false; > + local_bh_enable(); Sebastian
On Thu, Sep 07, 2023 at 07:17:11AM +0530, Ratheesh Kannoth wrote: > The access to page pool `cache' array and the `count' variable > is not locked. Page pool cache access is fine as long as there > is only one consumer per pool. > > octeontx2 driver fills in rx buffers from page pool in NAPI context. > If system is stressed and could not allocate buffers, refiiling work > will be delegated to a delayed workqueue. This means that there are > two cosumers to the page pool cache. > > Either workqueue or IRQ/NAPI can be run on other CPU. This will lead > to lock less access, hence corruption of cache pool indexes. > > To fix this issue, NAPI is rescheduled from workqueue context to refill > rx buffers. > > Fixes: b2e3406a38f0 ("octeontx2-pf: Add support for page pool") > Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com> > > --- > ChangeLogs > v1 -> v2: Added local_bh_disable() to be precise on napi scheduling. > > v0 -> v1: udelay will waste CPU cycles in UP. So call napi_schedule from > delayed work queue context. > > --- > --- > .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- > .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- > .../marvell/octeontx2/nic/otx2_common.c | 43 +++---------------- > .../marvell/octeontx2/nic/otx2_common.h | 3 +- > .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 7 +-- > .../marvell/octeontx2/nic/otx2_txrx.c | 29 ++++++++++--- > .../marvell/octeontx2/nic/otx2_txrx.h | 4 +- > 7 files changed, 42 insertions(+), 50 deletions(-) > > diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c > index 826f691de259..211c7d8a0556 100644 > --- a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c > +++ b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c > @@ -107,12 +107,13 @@ int cn10k_sq_aq_init(void *dev, u16 qidx, u16 sqb_aura) > } > > #define NPA_MAX_BURST 16 > -void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) > +int cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) > { > struct otx2_nic *pfvf = dev; > u64 ptrs[NPA_MAX_BURST]; > int num_ptrs = 1; > dma_addr_t bufptr; > + int cnt = cq->pool_ptrs; nit: please arrange local variables in new Networking code in reverse xmas tree order - longest line to shortest. > > /* Refill pool with new buffers */ > while (cq->pool_ptrs) { ...
> From: Simon Horman <horms@kernel.org> > Subject: [EXT] Re: [PATCH net v2] octeontx2-pf: Fix page pool cache index > corruption. > -void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) > > +int cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) > > { > > struct otx2_nic *pfvf = dev; > > u64 ptrs[NPA_MAX_BURST]; > > int num_ptrs = 1; > > dma_addr_t bufptr; > > + int cnt = cq->pool_ptrs; > > nit: please arrange local variables in new Networking code in reverse xmas > tree order - longest line to shortest. ACK -Ratheesh
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c index 826f691de259..211c7d8a0556 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.c @@ -107,12 +107,13 @@ int cn10k_sq_aq_init(void *dev, u16 qidx, u16 sqb_aura) } #define NPA_MAX_BURST 16 -void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) +int cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) { struct otx2_nic *pfvf = dev; u64 ptrs[NPA_MAX_BURST]; int num_ptrs = 1; dma_addr_t bufptr; + int cnt = cq->pool_ptrs; /* Refill pool with new buffers */ while (cq->pool_ptrs) { @@ -131,6 +132,7 @@ void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) num_ptrs = 1; } } + return cnt - cq->pool_ptrs; } void cn10k_sqe_flush(void *dev, struct otx2_snd_queue *sq, int size, int qidx) diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.h b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.h index 8ae96815865e..c1861f7de254 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.h +++ b/drivers/net/ethernet/marvell/octeontx2/nic/cn10k.h @@ -24,7 +24,7 @@ static inline int mtu_to_dwrr_weight(struct otx2_nic *pfvf, int mtu) return weight; } -void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); +int cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); void cn10k_sqe_flush(void *dev, struct otx2_snd_queue *sq, int size, int qidx); int cn10k_sq_aq_init(void *dev, u16 qidx, u16 sqb_aura); int cn10k_lmtst_init(struct otx2_nic *pfvf); diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c index 8511906cb4e2..997fedac3a98 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c @@ -574,20 +574,8 @@ int otx2_alloc_rbuf(struct otx2_nic *pfvf, struct otx2_pool *pool, int otx2_alloc_buffer(struct otx2_nic *pfvf, struct otx2_cq_queue *cq, dma_addr_t *dma) { - if (unlikely(__otx2_alloc_rbuf(pfvf, cq->rbpool, dma))) { - struct refill_work *work; - struct delayed_work *dwork; - - work = &pfvf->refill_wrk[cq->cq_idx]; - dwork = &work->pool_refill_work; - /* Schedule a task if no other task is running */ - if (!cq->refill_task_sched) { - cq->refill_task_sched = true; - schedule_delayed_work(dwork, - msecs_to_jiffies(100)); - } + if (unlikely(__otx2_alloc_rbuf(pfvf, cq->rbpool, dma))) return -ENOMEM; - } return 0; } @@ -1082,39 +1070,20 @@ static int otx2_cq_init(struct otx2_nic *pfvf, u16 qidx) static void otx2_pool_refill_task(struct work_struct *work) { struct otx2_cq_queue *cq; - struct otx2_pool *rbpool; struct refill_work *wrk; - int qidx, free_ptrs = 0; struct otx2_nic *pfvf; - dma_addr_t bufptr; + int qidx; wrk = container_of(work, struct refill_work, pool_refill_work.work); pfvf = wrk->pf; qidx = wrk - pfvf->refill_wrk; cq = &pfvf->qset.cq[qidx]; - rbpool = cq->rbpool; - free_ptrs = cq->pool_ptrs; - while (cq->pool_ptrs) { - if (otx2_alloc_rbuf(pfvf, rbpool, &bufptr)) { - /* Schedule a WQ if we fails to free atleast half of the - * pointers else enable napi for this RQ. - */ - if (!((free_ptrs - cq->pool_ptrs) > free_ptrs / 2)) { - struct delayed_work *dwork; - - dwork = &wrk->pool_refill_work; - schedule_delayed_work(dwork, - msecs_to_jiffies(100)); - } else { - cq->refill_task_sched = false; - } - return; - } - pfvf->hw_ops->aura_freeptr(pfvf, qidx, bufptr + OTX2_HEAD_ROOM); - cq->pool_ptrs--; - } cq->refill_task_sched = false; + + local_bh_disable(); + napi_schedule(wrk->napi); + local_bh_enable(); } int otx2_config_nix_queues(struct otx2_nic *pfvf) diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h index 4c6032ee7800..c04a8ee53a82 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h @@ -302,6 +302,7 @@ struct flr_work { struct refill_work { struct delayed_work pool_refill_work; struct otx2_nic *pf; + struct napi_struct *napi; }; /* PTPv2 originTimestamp structure */ @@ -370,7 +371,7 @@ struct dev_hw_ops { int (*sq_aq_init)(void *dev, u16 qidx, u16 sqb_aura); void (*sqe_flush)(void *dev, struct otx2_snd_queue *sq, int size, int qidx); - void (*refill_pool_ptrs)(void *dev, struct otx2_cq_queue *cq); + int (*refill_pool_ptrs)(void *dev, struct otx2_cq_queue *cq); void (*aura_freeptr)(void *dev, int aura, u64 buf); }; diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c index 70b9065f7d10..6daf4d58c25d 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c @@ -1943,6 +1943,10 @@ int otx2_stop(struct net_device *netdev) netif_tx_disable(netdev); + for (wrk = 0; wrk < pf->qset.cq_cnt; wrk++) + cancel_delayed_work_sync(&pf->refill_wrk[wrk].pool_refill_work); + devm_kfree(pf->dev, pf->refill_wrk); + otx2_free_hw_resources(pf); otx2_free_cints(pf, pf->hw.cint_cnt); otx2_disable_napi(pf); @@ -1950,9 +1954,6 @@ int otx2_stop(struct net_device *netdev) for (qidx = 0; qidx < netdev->num_tx_queues; qidx++) netdev_tx_reset_queue(netdev_get_tx_queue(netdev, qidx)); - for (wrk = 0; wrk < pf->qset.cq_cnt; wrk++) - cancel_delayed_work_sync(&pf->refill_wrk[wrk].pool_refill_work); - devm_kfree(pf->dev, pf->refill_wrk); kfree(qset->sq); kfree(qset->cq); diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c index e369baf11530..b778ed366f81 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c @@ -424,9 +424,10 @@ static int otx2_rx_napi_handler(struct otx2_nic *pfvf, return processed_cqe; } -void otx2_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) +int otx2_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) { struct otx2_nic *pfvf = dev; + int cnt = cq->pool_ptrs; dma_addr_t bufptr; while (cq->pool_ptrs) { @@ -435,6 +436,8 @@ void otx2_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq) otx2_aura_freeptr(pfvf, cq->cq_idx, bufptr + OTX2_HEAD_ROOM); cq->pool_ptrs--; } + + return cnt - cq->pool_ptrs; } static int otx2_tx_napi_handler(struct otx2_nic *pfvf, @@ -521,6 +524,7 @@ int otx2_napi_handler(struct napi_struct *napi, int budget) struct otx2_cq_queue *cq; struct otx2_qset *qset; struct otx2_nic *pfvf; + int filled_cnt = -1; cq_poll = container_of(napi, struct otx2_cq_poll, napi); pfvf = (struct otx2_nic *)cq_poll->dev; @@ -541,7 +545,7 @@ int otx2_napi_handler(struct napi_struct *napi, int budget) } if (rx_cq && rx_cq->pool_ptrs) - pfvf->hw_ops->refill_pool_ptrs(pfvf, rx_cq); + filled_cnt = pfvf->hw_ops->refill_pool_ptrs(pfvf, rx_cq); /* Clear the IRQ */ otx2_write64(pfvf, NIX_LF_CINTX_INT(cq_poll->cint_idx), BIT_ULL(0)); @@ -561,9 +565,24 @@ int otx2_napi_handler(struct napi_struct *napi, int budget) otx2_config_irq_coalescing(pfvf, i); } - /* Re-enable interrupts */ - otx2_write64(pfvf, NIX_LF_CINTX_ENA_W1S(cq_poll->cint_idx), - BIT_ULL(0)); + if (unlikely(!filled_cnt)) { + struct refill_work *work; + struct delayed_work *dwork; + + work = &pfvf->refill_wrk[cq->cq_idx]; + dwork = &work->pool_refill_work; + /* Schedule a task if no other task is running */ + if (!cq->refill_task_sched) { + work->napi = napi; + cq->refill_task_sched = true; + schedule_delayed_work(dwork, + msecs_to_jiffies(100)); + } + } else { + /* Re-enable interrupts */ + otx2_write64(pfvf, NIX_LF_CINTX_ENA_W1S(cq_poll->cint_idx), + BIT_ULL(0)); + } } return workdone; } diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.h b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.h index 9e3bfbe5c480..a82ffca8ce1b 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.h +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.h @@ -170,6 +170,6 @@ void cn10k_sqe_flush(void *dev, struct otx2_snd_queue *sq, int size, int qidx); void otx2_sqe_flush(void *dev, struct otx2_snd_queue *sq, int size, int qidx); -void otx2_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); -void cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); +int otx2_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); +int cn10k_refill_pool_ptrs(void *dev, struct otx2_cq_queue *cq); #endif /* OTX2_TXRX_H */
The access to page pool `cache' array and the `count' variable is not locked. Page pool cache access is fine as long as there is only one consumer per pool. octeontx2 driver fills in rx buffers from page pool in NAPI context. If system is stressed and could not allocate buffers, refiiling work will be delegated to a delayed workqueue. This means that there are two cosumers to the page pool cache. Either workqueue or IRQ/NAPI can be run on other CPU. This will lead to lock less access, hence corruption of cache pool indexes. To fix this issue, NAPI is rescheduled from workqueue context to refill rx buffers. Fixes: b2e3406a38f0 ("octeontx2-pf: Add support for page pool") Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com> --- ChangeLogs v1 -> v2: Added local_bh_disable() to be precise on napi scheduling. v0 -> v1: udelay will waste CPU cycles in UP. So call napi_schedule from delayed work queue context. --- --- .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- .../marvell/octeontx2/nic/otx2_common.c | 43 +++---------------- .../marvell/octeontx2/nic/otx2_common.h | 3 +- .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 7 +-- .../marvell/octeontx2/nic/otx2_txrx.c | 29 ++++++++++--- .../marvell/octeontx2/nic/otx2_txrx.h | 4 +- 7 files changed, 42 insertions(+), 50 deletions(-)