| Message ID | 1502351175-1762-1-git-send-email-aditr@vmware.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote: > From: Bryan Tan <bryantan@vmware.com> > > There is a chance of a race between arming the CQ and receiving > completions. By reporting CQ missed events any ULPs should poll > again to get the completions. > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver") > Acked-by: Aditya Sarwade <asarwade@vmware.com> > Signed-off-by: Bryan Tan <bryantan@vmware.com> > Signed-off-by: Adit Ranadive <aditr@vmware.com> > --- > drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > index 69bda61..f4ab5fa 100644 > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq, > struct pvrdma_dev *dev = to_vdev(ibcq->device); > struct pvrdma_cq *cq = to_vcq(ibcq); > u32 val = cq->cq_handle; > + unsigned long flags; > > val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ? > PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; > > + spin_lock_irqsave(&cq->cq_lock, flags); > + > pvrdma_write_uar_cq(dev, val); > > + if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) { > + unsigned int head; > + > + if (pvrdma_idx_ring_has_data(&cq->ring_state->rx, > + cq->ibcq.cqe, &head)) { The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1). > + spin_unlock_irqrestore(&cq->cq_lock, flags); > + return 1; > + } > + } > + > + spin_unlock_irqrestore(&cq->cq_lock, flags); > + > return 0; > } > > -- > 2.7.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote: > From: Bryan Tan <bryantan@vmware.com> > > There is a chance of a race between arming the CQ and receiving > completions. By reporting CQ missed events any ULPs should poll > again to get the completions. > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver") > Acked-by: Aditya Sarwade <asarwade@vmware.com> > Signed-off-by: Bryan Tan <bryantan@vmware.com> > Signed-off-by: Adit Ranadive <aditr@vmware.com> > --- > drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > index 69bda61..f4ab5fa 100644 > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq, > struct pvrdma_dev *dev = to_vdev(ibcq->device); > struct pvrdma_cq *cq = to_vcq(ibcq); > u32 val = cq->cq_handle; > + unsigned long flags; > > val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ? > PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; > > + spin_lock_irqsave(&cq->cq_lock, flags); > + > pvrdma_write_uar_cq(dev, val); > > + if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) { > + unsigned int head; > + > + if (pvrdma_idx_ring_has_data(&cq->ring_state->rx, > + cq->ibcq.cqe, &head)) { > + spin_unlock_irqrestore(&cq->cq_lock, flags); > + return 1; > + } > + } > + > + spin_unlock_irqrestore(&cq->cq_lock, flags); > + Reviewed-by: Yuval Shaia <yuval.shaia@oracle.com> > return 0; > } > > -- > 2.7.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 10, 2017 at 02:15:57PM +0300, Leon Romanovsky wrote: > On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote: > > From: Bryan Tan <bryantan@vmware.com> > > > > There is a chance of a race between arming the CQ and receiving > > completions. By reporting CQ missed events any ULPs should poll > > again to get the completions. > > > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver") > > Acked-by: Aditya Sarwade <asarwade@vmware.com> > > Signed-off-by: Bryan Tan <bryantan@vmware.com> > > Signed-off-by: Adit Ranadive <aditr@vmware.com> > > --- > > drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++ > > 1 file changed, 15 insertions(+) > > > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > index 69bda61..f4ab5fa 100644 > > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq, > > struct pvrdma_dev *dev = to_vdev(ibcq->device); > > struct pvrdma_cq *cq = to_vcq(ibcq); > > u32 val = cq->cq_handle; > > + unsigned long flags; > > > > val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ? > > PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; > > > > + spin_lock_irqsave(&cq->cq_lock, flags); > > + > > pvrdma_write_uar_cq(dev, val); > > > > + if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) { > > + unsigned int head; > > + > > + if (pvrdma_idx_ring_has_data(&cq->ring_state->rx, > > + cq->ibcq.cqe, &head)) { > > The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1). I believe that the case of PVRDMA_INVALID_IDX is only due to some bug. I do not think it is an expected behaviour, it means that ring is corrupted, am i right Adit? > > > + spin_unlock_irqrestore(&cq->cq_lock, flags); > > + return 1; > > + } > > + } > > + > > + spin_unlock_irqrestore(&cq->cq_lock, flags); > > + > > return 0; > > } > > > > -- > > 2.7.4 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> On Thu, Aug 10, 2017 at 02:15:57PM +0300, Leon Romanovsky wrote: > > On Thu, Aug 10, 2017 at 12:46:15AM -0700, Adit Ranadive wrote: > > > From: Bryan Tan <bryantan@vmware.com> > > > > > > There is a chance of a race between arming the CQ and receiving > > > completions. By reporting CQ missed events any ULPs should poll > > > again to get the completions. > > > > > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver") > > > Acked-by: Aditya Sarwade <asarwade@vmware.com> > > > Signed-off-by: Bryan Tan <bryantan@vmware.com> > > > Signed-off-by: Adit Ranadive <aditr@vmware.com> > > > --- > > > drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 15 +++++++++++++++ > > > 1 file changed, 15 insertions(+) > > > > > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > > index 69bda61..f4ab5fa 100644 > > > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c > > > @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq, > > > struct pvrdma_dev *dev = to_vdev(ibcq->device); > > > struct pvrdma_cq *cq = to_vcq(ibcq); > > > u32 val = cq->cq_handle; > > > + unsigned long flags; > > > > > > val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ? > > > PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; > > > > > > + spin_lock_irqsave(&cq->cq_lock, flags); > > > + > > > pvrdma_write_uar_cq(dev, val); > > > > > > + if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) { > > > + unsigned int head; > > > + > > > + if (pvrdma_idx_ring_has_data(&cq->ring_state->rx, > > > + cq->ibcq.cqe, &head)) { > > > > The pvrdma_idx_ring_has_data function can return PVRDMA_INVALID_IDX (-1). > > I believe that the case of PVRDMA_INVALID_IDX is only due to some bug. I do > not think it is an expected behaviour, it means that ring is corrupted, am > i right Adit? Thanks Leon, Yuval. While hitting the INVALID_IDX is really rare (we haven't hit it ever in our internal testing), we should check this for consistency purposes. Look for a v1. - Adit-- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c index 69bda61..f4ab5fa 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c @@ -65,12 +65,27 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq, struct pvrdma_dev *dev = to_vdev(ibcq->device); struct pvrdma_cq *cq = to_vcq(ibcq); u32 val = cq->cq_handle; + unsigned long flags; val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ? PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM; + spin_lock_irqsave(&cq->cq_lock, flags); + pvrdma_write_uar_cq(dev, val); + if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) { + unsigned int head; + + if (pvrdma_idx_ring_has_data(&cq->ring_state->rx, + cq->ibcq.cqe, &head)) { + spin_unlock_irqrestore(&cq->cq_lock, flags); + return 1; + } + } + + spin_unlock_irqrestore(&cq->cq_lock, flags); + return 0; }