[1/1] IB/iser: Remove hard coded values for cqe and send_wr

Hi Sagi,

I've created a new patch over the 21 iser patches you have mentioned early in this thread.  It is pasted at the end of this email.
If I understand correctly, this patch will be applied along with Or's TODO list.

>> ISER_MAX_RX_CQ_LEN             4096               /* This number should be calculated during create_session */

>So in iSER CQs are shared a across the device - so this number should satisfy maximum number of connections per CQ (which is currently 8).

Again, there should be no rules enforcing a CQ to support 8 connections.  The underlining hw should be able to supports more or less as it is configured to do.  Specific to the ocrdma hw, it has lower number of CQE per CQ but it has 32 CQ

>> ISER_QP_MAX_RECV_DTOS    512                 /* Why can't we use ISCSI_DEF_XMIT_CMDS_MAX here ? */

>iSER creates the connection QP when before the session is created - so it doesn't know what the user will set at cmds_max (which is potentially larger than ISCSI_DEF_XMIT_CMDS_MAX). So we allow 512 at the moment and adjust the session cmds_max accordingly. I agree that this is a >work around for the moment as we don't know at QP creation time what is the user setting of cmds_max.

Like any other drivers, we should limit this number to the range hw supports.  If user setting is within hw range then number will be adjusted accordingly.  If user setting is not within hw range, set it to default values.
What about something like this:
#define ISER_GET_MAX_XMIT_CMDS(send_wr) (send_wr - ISER_MAX_TX_MISC_PDUS - \
					ISER_MAX_RX_MISC_PDUS)	/	\
					 (1 + ISER_INFLIGHT_DATAOUTS)
cmds_max_supported = ISER_GET_MAX_XMIT_CMDS(dev_attr->max_qp_wr)

>> ISER_MAX_TX_CQ_LEN             36944           /* the mlx4 hw supports up to 3 CQ, but the ocrdma hw supports up to  32CQ with lower number of cqe per CQ */

>What led you to conclude that: "the mlx4 hw supports up to 3 CQ"?
>TX CQ length should be

I was debugging some iser target problems sometimes back.  I could be wrong with 3CQ but it's not far from the hard limit of 4CQ set by current ib/iser code

>> ISER_QP_MAX_REQ_DTOS       4618
> >ISCSI_ISER_MAX_CONN            8                  /* I am not sure what this 8 connections per CQ is.  Open-iscsi will supports 1 connection per session so this can implies either one of these two things:
>>                                                                                          1- mlx4 is limited to 8 sessions per CQ
>>                                                                                          
>> 2- mlx4 is doing something proprietary on the hw to have multiple QP 
>> per session */

>As I said, CQs are per device and shared across iscsi connections. So each CQ will support up to 8 connections per CQ. I agree we should allocate more CQs in case of more connections are opened - but we never got any CQ to overrun (even in high stress) so this still on my todo list...

>> ISER_MAX_CQ                               4                  /* Should this number be much higher or bases on the number of cpu cores on the system to distribute CQ processing per core? */

>I completely agree, This is a legacy MAX ceiling. I have a patch for that pending at Or's table. I am all for getting it to 3.18/3.19

>>
>> We are open for suggestions.

>OK,

>I'll respond to Or's reply on the TODOs.

>Sagi.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[1/1] IB/iser: Remove hard coded values for cqe and send_wr

Commit Message

Comments

Patch