| Message ID | 1445938846-9240-2-git-send-email-sagig@mellanox.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
On 10/27/2015 11:40 AM, Sagi Grimberg wrote: > mlx4 devices (ConnectX-2, ConnectX-3) can not issue > max_sge in a single RDMA_READ request (resulting in > a completion error). Thus, expose lower max_sge_rd > to avoid this issue. Sagi, I can hear your pain when wearing the iser target driver maintainer hat. Still, this patch is currently pure WA b/c we didn't do RCA (Root Cause Analysis) Lets wait for RCA (which might yield the same patch, BTW) and keep suffering in LIO Or. > > Signed-off-by: Sagi Grimberg <sagig@mellanox.com> > --- > drivers/infiniband/hw/mlx4/main.c | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c > index 3889723..46305dc 100644 > --- a/drivers/infiniband/hw/mlx4/main.c > +++ b/drivers/infiniband/hw/mlx4/main.c > @@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev) > ibdev->max_qp_wr = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE; > ibdev->max_sge = min(dev->dev->caps.max_sq_sg, > dev->dev->caps.max_rq_sg); > - ibdev->max_sge_rd = ibdev->max_sge; > + /* reserve 2 sge slots for rdma reads */ > + ibdev->max_sge_rd = ibdev->max_sge - 2; > ibdev->max_cq = dev->dev->quotas.cq; > ibdev->max_cqe = dev->dev->caps.max_cqes; > ibdev->max_mr = dev->dev->quotas.mpt; -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 27/10/2015 16:39, Or Gerlitz wrote: > On 10/27/2015 11:40 AM, Sagi Grimberg wrote: >> mlx4 devices (ConnectX-2, ConnectX-3) can not issue >> max_sge in a single RDMA_READ request (resulting in >> a completion error). Thus, expose lower max_sge_rd >> to avoid this issue. > > Sagi, Hey Or, > Still, this patch is currently pure WA b/c we didn't do RCA (Root Cause > Analysis) So from my discussions with the HW folks a RDMA_READ wqe cannot exceed 512B. The wqe control segment is 16 bytes, the rdma section is 12 bytes (rkey + raddr) and each sge is 16 bytes so the computation is: (512B-16B-12B)/16B = 30. The reason is that the HW needs to fetch the rdma_read wqe on the RX path (rdma_read response) and it has a limited buffer at that point. Perhaps a dedicated #define for that is needed here. I'll add that in the change log in v1. Cheers, Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 10/27/2015 6:03 PM, Sagi Grimberg wrote: > So from my discussions with the HW folks a RDMA_READ wqe cannot exceed > 512B. The wqe control segment is 16 bytes, the rdma section is 12 bytes > (rkey + raddr) and each sge is 16 bytes so the computation is: > > (512B-16B-12B)/16B = 30. But AFAIR, the magic number was 28... how this goes hand in hand with your findings? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> But AFAIR, the magic number was 28... how this goes hand in hand with > your findings? mlx4 max_sge is 32, and isert does max_sge - 2 = 30. So it always used 30... and I run it reliably with this for a while now. This thing exists before I was involved so I might not be familiar with all the details... Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 10/27/2015 02:40 AM, Sagi Grimberg wrote: > mlx4 devices (ConnectX-2, ConnectX-3) can not issue > max_sge in a single RDMA_READ request (resulting in > a completion error). Thus, expose lower max_sge_rd > to avoid this issue. > > Signed-off-by: Sagi Grimberg <sagig@mellanox.com> > --- > drivers/infiniband/hw/mlx4/main.c | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c > index 3889723..46305dc 100644 > --- a/drivers/infiniband/hw/mlx4/main.c > +++ b/drivers/infiniband/hw/mlx4/main.c > @@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev) > ibdev->max_qp_wr = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE; > ibdev->max_sge = min(dev->dev->caps.max_sq_sg, > dev->dev->caps.max_rq_sg); > - ibdev->max_sge_rd = ibdev->max_sge; > + /* reserve 2 sge slots for rdma reads */ > + ibdev->max_sge_rd = ibdev->max_sge - 2; > ibdev->max_cq = dev->dev->quotas.cq; > ibdev->max_cqe = dev->dev->caps.max_cqes; > ibdev->max_mr = dev->dev->quotas.mpt; Hello Sagi, Is this the same issue as what has been discussed in http://www.spinics.net/lists/linux-rdma/msg21799.html ? Thanks, Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> Hello Sagi, > > Is this the same issue as what has been discussed in > http://www.spinics.net/lists/linux-rdma/msg21799.html ? Looks like it. I think this patch addresses this issue, but lets CC Eli to comment if I'm missing something. Thanks for digging this up... Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Just discussed the issue with Sagi. Sagi will follow up with a small correction. -----Original Message----- From: Sagi Grimberg [mailto:sagig@dev.mellanox.co.il] Sent: Tuesday, October 27, 2015 11:32 AM To: Bart Van Assche; linux-rdma@vger.kernel.org; target-devel@vger.kernel.org Cc: Steve Wise; Nicholas A. Bellinger; Or Gerlitz; Doug Ledford; Eli Cohen Subject: Re: [PATCH 1/2] mlx4: Expose correct max_sge_rd limit > Hello Sagi, > > Is this the same issue as what has been discussed in > http://www.spinics.net/lists/linux-rdma/msg21799.html ? Looks like it. I think this patch addresses this issue, but lets CC Eli to comment if I'm missing something. Thanks for digging this up... Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 3889723..46305dc 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -499,7 +499,8 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev) ibdev->max_qp_wr = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE; ibdev->max_sge = min(dev->dev->caps.max_sq_sg, dev->dev->caps.max_rq_sg); - ibdev->max_sge_rd = ibdev->max_sge; + /* reserve 2 sge slots for rdma reads */ + ibdev->max_sge_rd = ibdev->max_sge - 2; ibdev->max_cq = dev->dev->quotas.cq; ibdev->max_cqe = dev->dev->caps.max_cqes; ibdev->max_mr = dev->dev->quotas.mpt;
mlx4 devices (ConnectX-2, ConnectX-3) can not issue max_sge in a single RDMA_READ request (resulting in a completion error). Thus, expose lower max_sge_rd to avoid this issue. Signed-off-by: Sagi Grimberg <sagig@mellanox.com> --- drivers/infiniband/hw/mlx4/main.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)