| Message ID | 03e7f98e-d810-2299-9f3e-2c697d2699fb@sandisk.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
On Tue, Jul 19, 2016 at 09:22:03AM -0700, Bart Van Assche wrote: > For IB and RoCE, the SGE limit for a queue pair is typically lower > than what is defined by the HCA limits. For iWARP, the RDMA READ SGE > limit is defined by dev->attrs.max_sge_rd. Modify rdma_rw_max_sge() > accordingly. It's not just iWarp - Melannox IB HCAs also have a lower RDMA READ than WRITE limit. > -static inline u32 rdma_rw_max_sge(struct ib_device *dev, > +static inline u32 rdma_rw_max_sge(struct ib_device *dev, struct ib_qp *qp, > enum dma_data_direction dir) > { > + return dir == DMA_TO_DEVICE ? qp->max_send_sge : > + min_t(u32, qp->max_send_sge, dev->attrs.max_sge_rd); > } I'm a bit worried about this implicit and barely documented assumption that we'll always have a lower RDMA READ than WRITE / SEND limit. Can we fine tune the patch to add max_write_sge and max_read_sge to struct ib_qp instead? That also makes clear they apply to RDMA READ and WRITE only. Please also add a comment to ib_verbs.h to document the exact semantics of this field(s) while you're at it, e.g. that they only apply to users of the RDMA R/W API. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c index 1ad2baa..6ecdad6 100644 --- a/drivers/infiniband/core/rw.c +++ b/drivers/infiniband/core/rw.c @@ -58,11 +58,11 @@ static inline bool rdma_rw_io_needs_mr(struct ib_device *dev, u8 port_num, return false; } -static inline u32 rdma_rw_max_sge(struct ib_device *dev, +static inline u32 rdma_rw_max_sge(struct ib_device *dev, struct ib_qp *qp, enum dma_data_direction dir) { - return dir == DMA_TO_DEVICE ? - dev->attrs.max_sge : dev->attrs.max_sge_rd; + return dir == DMA_TO_DEVICE ? qp->max_send_sge : + min_t(u32, qp->max_send_sge, dev->attrs.max_sge_rd); } static inline u32 rdma_rw_fr_page_list_len(struct ib_device *dev) @@ -186,7 +186,7 @@ static int rdma_rw_init_map_wrs(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u64 remote_addr, u32 rkey, enum dma_data_direction dir) { struct ib_device *dev = qp->pd->device; - u32 max_sge = rdma_rw_max_sge(dev, dir); + u32 max_sge = rdma_rw_max_sge(dev, qp, dir); struct ib_sge *sge; u32 total_len = 0, i, j; diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 6298f54..c7f840e 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -814,6 +814,8 @@ struct ib_qp *ib_create_qp(struct ib_pd *pd, } } + qp->max_send_sge = qp_init_attr->cap.max_send_sge; + return qp; } EXPORT_SYMBOL(ib_create_qp); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 7e440d4..c44dbf6 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1449,6 +1449,7 @@ struct ib_qp { void (*event_handler)(struct ib_event *, void *); void *qp_context; u32 qp_num; + u32 max_send_sge; enum ib_qp_type qp_type; };
For IB and RoCE, the SGE limit for a queue pair is typically lower than what is defined by the HCA limits. For iWARP, the RDMA READ SGE limit is defined by dev->attrs.max_sge_rd. Modify rdma_rw_max_sge() accordingly. Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com> Cc: <stable@vger.kernel.org> #v4.7+ Cc: Christoph Hellwig <hch@lst.de> Cc: Sagi Grimberg <sagi@grimberg.me> Cc: Steve Wise <swise@opengridcomputing.com> Cc: Parav Pandit <pandit.parav@gmail.com> Cc: Nicholas Bellinger <nab@linux-iscsi.org> Cc: Laurence Oberman <loberman@redhat.com> --- drivers/infiniband/core/rw.c | 8 ++++---- drivers/infiniband/core/verbs.c | 2 ++ include/rdma/ib_verbs.h | 1 + 3 files changed, 7 insertions(+), 4 deletions(-)