diff mbox

[v2,1/2] mlx4: Expose correct max_sge_rd limit

Message ID 1446031696-19022-2-git-send-email-sagig@mellanox.com (mailing list archive)
State Accepted
Headers show

Commit Message

Sagi Grimberg Oct. 28, 2015, 11:28 a.m. UTC 
mlx4 devices (ConnectX-2, ConnectX-3) has a limitation
where rdma read work queue entries cannot exceed 512 bytes.
A rdma_read wqe needs to fit in 512 bytes:
- wqe control segment (16 bytes)
- rdma segment (16 bytes)
- scatter elements (16 bytes each)

So max_sge_rd should be: (512 - 16 - 16) / 16 = 30.

Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
Reviewed-by: Steve Wise <swise@opengridcomputing.com>
---
 drivers/infiniband/hw/mlx4/main.c |    2 +-
 include/linux/mlx4/device.h       |   11 +++++++++++
 2 files changed, 12 insertions(+), 1 deletions(-)

Comments

Doug Ledford Dec. 8, 2015, 5:56 p.m. UTC | #1 
On 10/28/2015 07:28 AM, Sagi Grimberg wrote:
> mlx4 devices (ConnectX-2, ConnectX-3) has a limitation
> where rdma read work queue entries cannot exceed 512 bytes.
> A rdma_read wqe needs to fit in 512 bytes:
> - wqe control segment (16 bytes)
> - rdma segment (16 bytes)
> - scatter elements (16 bytes each)
> 
> So max_sge_rd should be: (512 - 16 - 16) / 16 = 30.
> 
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> Reviewed-by: Steve Wise <swise@opengridcomputing.com>
> ---
>  drivers/infiniband/hw/mlx4/main.c |    2 +-
>  include/linux/mlx4/device.h       |   11 +++++++++++
>  2 files changed, 12 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
> index 7026580..e0232a2 100644
> --- a/drivers/infiniband/hw/mlx4/main.c
> +++ b/drivers/infiniband/hw/mlx4/main.c
> @@ -499,7 +499,7 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev)
>  	ibdev->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
>  	ibdev->max_sge		   = min(dev->dev->caps.max_sq_sg,
>  					 dev->dev->caps.max_rq_sg);
> -	ibdev->max_sge_rd	   = ibdev->max_sge;
> +	ibdev->max_sge_rd	   = MLX4_MAX_SGE_RD;
>  	ibdev->max_cq		   = dev->dev->quotas.cq;
>  	ibdev->max_cqe		   = dev->dev->caps.max_cqes;
>  	ibdev->max_mr		   = dev->dev->quotas.mpt;

What code tree did you make this patch from?  Because in an upstream 4.4
kernel, there is no function mlx4_ib_init_device_flags(), but there is
mlx4_ib_query_device() that does much the same thing.  But in there,
there is no ibdev->max_sge_rd, there is props->max_sge_rd.  It's like
this came from an entirely different source base somewhere.

> diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
> index dac6872..3dcae41 100644
> --- a/include/linux/mlx4/device.h
> +++ b/include/linux/mlx4/device.h
> @@ -427,6 +427,17 @@ enum {
>  };
>  
>  enum {
> +	/*
> +	 * Max wqe size for rdma read is 512 bytes, so this
> +	 * limits our max_sge_rd as the wqe needs to fit:
> +	 * - ctrl segment (16 bytes)
> +	 * - rdma segment (16 bytes)
> +	 * - scatter elements (16 bytes each)
> +	 */
> +	MLX4_MAX_SGE_RD	= (512 - 16 - 16) / 16
> +};
> +
> +enum {
>  	MLX4_DEV_PMC_SUBTYPE_GUID_INFO	 = 0x14,
>  	MLX4_DEV_PMC_SUBTYPE_PORT_INFO	 = 0x15,
>  	MLX4_DEV_PMC_SUBTYPE_PKEY_TABLE	 = 0x16,
>
diff mbox

Patch

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 7026580..e0232a2 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -499,7 +499,7 @@  static int mlx4_ib_init_device_flags(struct ib_device *ibdev)
 	ibdev->max_qp_wr	   = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE;
 	ibdev->max_sge		   = min(dev->dev->caps.max_sq_sg,
 					 dev->dev->caps.max_rq_sg);
-	ibdev->max_sge_rd	   = ibdev->max_sge;
+	ibdev->max_sge_rd	   = MLX4_MAX_SGE_RD;
 	ibdev->max_cq		   = dev->dev->quotas.cq;
 	ibdev->max_cqe		   = dev->dev->caps.max_cqes;
 	ibdev->max_mr		   = dev->dev->quotas.mpt;
diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index dac6872..3dcae41 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -427,6 +427,17 @@  enum {
 };
 
 enum {
+	/*
+	 * Max wqe size for rdma read is 512 bytes, so this
+	 * limits our max_sge_rd as the wqe needs to fit:
+	 * - ctrl segment (16 bytes)
+	 * - rdma segment (16 bytes)
+	 * - scatter elements (16 bytes each)
+	 */
+	MLX4_MAX_SGE_RD	= (512 - 16 - 16) / 16
+};
+
+enum {
 	MLX4_DEV_PMC_SUBTYPE_GUID_INFO	 = 0x14,
 	MLX4_DEV_PMC_SUBTYPE_PORT_INFO	 = 0x15,
 	MLX4_DEV_PMC_SUBTYPE_PKEY_TABLE	 = 0x16,