diff mbox series

[1/1] IB/isert: align target max I/O size to initiator size

Message ID 20210524085215.29005-1-mgurtovoy@nvidia.com (mailing list archive)
State New
Headers show
Series [1/1] IB/isert: align target max I/O size to initiator size | expand

Commit Message

Max Gurtovoy May 24, 2021, 8:52 a.m. UTC
Since the Linux iser initiator default max I/O size set to 512KB and
since there is no handshake procedure for this size in iser protocol,
set the default max IO size of the target to 512KB as well.

For changing the default values, there is a module parameter for both
drivers.

Reviewed-by: Alaa Hleihel <alaa@nvidia.com>
Reviewed-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 drivers/infiniband/ulp/isert/ib_isert.c | 4 ++--
 drivers/infiniband/ulp/isert/ib_isert.h | 3 ---
 2 files changed, 2 insertions(+), 5 deletions(-)

Comments

Sagi Grimberg May 25, 2021, 3:54 p.m. UTC | #1
> Since the Linux iser initiator default max I/O size set to 512KB and
> since there is no handshake procedure for this size in iser protocol,
> set the default max IO size of the target to 512KB as well.
> 
> For changing the default values, there is a module parameter for both
> drivers.

Is this solving a bug?
Max Gurtovoy May 25, 2021, 4:22 p.m. UTC | #2
On 5/25/2021 6:54 PM, Sagi Grimberg wrote:
>> Since the Linux iser initiator default max I/O size set to 512KB and
>> since there is no handshake procedure for this size in iser protocol,
>> set the default max IO size of the target to 512KB as well.
>>
>> For changing the default values, there is a module parameter for both
>> drivers.
>
> Is this solving a bug?

No. Only OOB for some old connect-IB devices.

I think it's reasonable to align initiator and target defaults anyway.
Kamal Heib June 8, 2021, 10:24 a.m. UTC | #3
On 5/25/21 7:22 PM, Max Gurtovoy wrote:
> 
> On 5/25/2021 6:54 PM, Sagi Grimberg wrote:
>>> Since the Linux iser initiator default max I/O size set to 512KB and
>>> since there is no handshake procedure for this size in iser protocol,
>>> set the default max IO size of the target to 512KB as well.
>>>
>>> For changing the default values, there is a module parameter for both
>>> drivers.
>>
>> Is this solving a bug?
> 
> No. Only OOB for some old connect-IB devices.
> 
> I think it's reasonable to align initiator and target defaults anyway.
> 
> 

Actually, this patch is solving a bug when trying iser over Connect-IB, We see
the following failure when trying to do discovery:

Server:
[  124.264648] infiniband mlx5_0: create_qp:2783:(pid 83): Create QP type 2 failed
[  124.298598] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  124.364768] isert: isert_cma_handler: failed handle connect request -12
[  128.271609] infiniband mlx5_0: create_qp:2783:(pid 890): Create QP type 2 failed
[  128.311450] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  128.378995] isert: isert_cma_handler: failed handle connect request -12
[  130.668362] infiniband mlx5_0: create_qp:2783:(pid 81): Create QP type 2 failed
[  130.705869] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  130.777306] isert: isert_cma_handler: failed handle connect request -12
[  132.671161] infiniband mlx5_0: create_qp:2783:(pid 86): Create QP type 2 failed
[  132.707807] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  132.778867] isert: isert_cma_handler: failed handle connect request -12
[  132.810653] infiniband mlx5_0: create_qp:2783:(pid 19): Create QP type 2 failed
[  132.845691] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  132.912706] isert: isert_cma_handler: failed handle connect request -12
[  134.681936] infiniband mlx5_0: create_qp:2783:(pid 83): Create QP type 2 failed
[  134.718932] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  134.788804] isert: isert_cma_handler: failed handle connect request -12
[  136.678428] infiniband mlx5_0: create_qp:2783:(pid 86): Create QP type 2 failed
[  136.715859] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  136.785058] isert: isert_cma_handler: failed handle connect request -12
[  136.817414] infiniband mlx5_0: create_qp:2783:(pid 727): Create QP type 2 failed
[  136.854583] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
[  136.922975] isert: isert_cma_handler: failed handle connect request -12


Client:
$ iscsiadm -m discovery -t sendtargets -p 172.31.0.6 -I iser
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
connection failure
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: Could not perform SendTargets discovery: iSCSI PDU timed out


Thanks,
Kamal
Max Gurtovoy June 8, 2021, 11 a.m. UTC | #4
On 6/8/2021 1:24 PM, Kamal Heib wrote:
>
> On 5/25/21 7:22 PM, Max Gurtovoy wrote:
>> On 5/25/2021 6:54 PM, Sagi Grimberg wrote:
>>>> Since the Linux iser initiator default max I/O size set to 512KB and
>>>> since there is no handshake procedure for this size in iser protocol,
>>>> set the default max IO size of the target to 512KB as well.
>>>>
>>>> For changing the default values, there is a module parameter for both
>>>> drivers.
>>> Is this solving a bug?
>> No. Only OOB for some old connect-IB devices.
>>
>> I think it's reasonable to align initiator and target defaults anyway.
>>
>>
> Actually, this patch is solving a bug when trying iser over Connect-IB, We see
> the following failure when trying to do discovery:

You can work around this using the ib_isert sg_tablesize module param 
and set it to 128.

So it's more OOB behavior than a bug.

Anyway, This is good practice to be able to establish connections also 
for old devices without WAs and we also aligning to the sg_table size in 
the initiator side.

Jason/Sagi,

can you comment on this patch for 5.14 ?


>
> Server:
> [  124.264648] infiniband mlx5_0: create_qp:2783:(pid 83): Create QP type 2 failed
> [  124.298598] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  124.364768] isert: isert_cma_handler: failed handle connect request -12
> [  128.271609] infiniband mlx5_0: create_qp:2783:(pid 890): Create QP type 2 failed
> [  128.311450] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  128.378995] isert: isert_cma_handler: failed handle connect request -12
> [  130.668362] infiniband mlx5_0: create_qp:2783:(pid 81): Create QP type 2 failed
> [  130.705869] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  130.777306] isert: isert_cma_handler: failed handle connect request -12
> [  132.671161] infiniband mlx5_0: create_qp:2783:(pid 86): Create QP type 2 failed
> [  132.707807] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  132.778867] isert: isert_cma_handler: failed handle connect request -12
> [  132.810653] infiniband mlx5_0: create_qp:2783:(pid 19): Create QP type 2 failed
> [  132.845691] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  132.912706] isert: isert_cma_handler: failed handle connect request -12
> [  134.681936] infiniband mlx5_0: create_qp:2783:(pid 83): Create QP type 2 failed
> [  134.718932] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  134.788804] isert: isert_cma_handler: failed handle connect request -12
> [  136.678428] infiniband mlx5_0: create_qp:2783:(pid 86): Create QP type 2 failed
> [  136.715859] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  136.785058] isert: isert_cma_handler: failed handle connect request -12
> [  136.817414] infiniband mlx5_0: create_qp:2783:(pid 727): Create QP type 2 failed
> [  136.854583] isert: isert_create_qp: rdma_create_qp failed for cma_id -12
> [  136.922975] isert: isert_cma_handler: failed handle connect request -12
>
>
> Client:
> $ iscsiadm -m discovery -t sendtargets -p 172.31.0.6 -I iser
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: Connection to discovery portal 172.31.0.6 failed: encountered
> connection failure
> iscsiadm: connection login retries (reopen_max) 5 exceeded
> iscsiadm: Could not perform SendTargets discovery: iSCSI PDU timed out
>
>
> Thanks,
> Kamal
>
Sagi Grimberg June 8, 2021, 11:04 p.m. UTC | #5
>> On 5/25/21 7:22 PM, Max Gurtovoy wrote:
>>> On 5/25/2021 6:54 PM, Sagi Grimberg wrote:
>>>>> Since the Linux iser initiator default max I/O size set to 512KB and
>>>>> since there is no handshake procedure for this size in iser protocol,
>>>>> set the default max IO size of the target to 512KB as well.
>>>>>
>>>>> For changing the default values, there is a module parameter for both
>>>>> drivers.
>>>> Is this solving a bug?
>>> No. Only OOB for some old connect-IB devices.
>>>
>>> I think it's reasonable to align initiator and target defaults anyway.
>>>
>>>
>> Actually, this patch is solving a bug when trying iser over 
>> Connect-IB, We see
>> the following failure when trying to do discovery:
> 
> You can work around this using the ib_isert sg_tablesize module param 
> and set it to 128.
> 
> So it's more OOB behavior than a bug.
> 
> Anyway, This is good practice to be able to establish connections also 
> for old devices without WAs and we also aligning to the sg_table size in 
> the initiator side.
> 
> Jason/Sagi,
> 
> can you comment on this patch for 5.14 ?

Actually, if this is the case, why not have a fallback when creating the
QP? Seems more reasonable to have the exception for the old devices
rather than having those mandate the common denominator no?
Max Gurtovoy June 9, 2021, 8:45 a.m. UTC | #6
On 6/9/2021 2:04 AM, Sagi Grimberg wrote:
>
>>> On 5/25/21 7:22 PM, Max Gurtovoy wrote:
>>>> On 5/25/2021 6:54 PM, Sagi Grimberg wrote:
>>>>>> Since the Linux iser initiator default max I/O size set to 512KB and
>>>>>> since there is no handshake procedure for this size in iser 
>>>>>> protocol,
>>>>>> set the default max IO size of the target to 512KB as well.
>>>>>>
>>>>>> For changing the default values, there is a module parameter for 
>>>>>> both
>>>>>> drivers.
>>>>> Is this solving a bug?
>>>> No. Only OOB for some old connect-IB devices.
>>>>
>>>> I think it's reasonable to align initiator and target defaults anyway.
>>>>
>>>>
>>> Actually, this patch is solving a bug when trying iser over 
>>> Connect-IB, We see
>>> the following failure when trying to do discovery:
>>
>> You can work around this using the ib_isert sg_tablesize module param 
>> and set it to 128.
>>
>> So it's more OOB behavior than a bug.
>>
>> Anyway, This is good practice to be able to establish connections 
>> also for old devices without WAs and we also aligning to the sg_table 
>> size in the initiator side.
>>
>> Jason/Sagi,
>>
>> can you comment on this patch for 5.14 ?
>
> Actually, if this is the case, why not have a fallback when creating the
> QP? Seems more reasonable to have the exception for the old devices
> rather than having those mandate the common denominator no?

We first wanted to support 16MiB for isert but then we get a report from 
Chelsio that it will dramatically reduce the total amount of connections 
the can support.

So we created a module param and reduced the default to 1MiB. Now we 
have similar issue with Connect-IB so reducing it to 512KiB (same as the 
default for Linux iser initiator) seems reasonable.

Users that would like larger sg_table will use the module param.

I would avoid doing fallbacks for that and maintain a code that might be 
dead in a year or two.
diff mbox series

Patch

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 160efef66031..97214329c571 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -35,10 +35,10 @@  static const struct kernel_param_ops sg_tablesize_ops = {
 	.get = param_get_int,
 };
 
-static int isert_sg_tablesize = ISCSI_ISER_DEF_SG_TABLESIZE;
+static int isert_sg_tablesize = ISCSI_ISER_MIN_SG_TABLESIZE;
 module_param_cb(sg_tablesize, &sg_tablesize_ops, &isert_sg_tablesize, 0644);
 MODULE_PARM_DESC(sg_tablesize,
-		 "Number of gather/scatter entries in a single scsi command, should >= 128 (default: 256, max: 4096)");
+		 "Number of gather/scatter entries in a single scsi command, should >= 128 (default: 128, max: 4096)");
 
 static DEFINE_MUTEX(device_list_mutex);
 static LIST_HEAD(device_list);
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 6c5af13db4e0..ca8cfebe26ca 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -65,9 +65,6 @@ 
  */
 #define ISER_RX_SIZE		(ISCSI_DEF_MAX_RECV_SEG_LEN + 1024)
 
-/* Default I/O size is 1MB */
-#define ISCSI_ISER_DEF_SG_TABLESIZE 256
-
 /* Minimum I/O size is 512KB */
 #define ISCSI_ISER_MIN_SG_TABLESIZE 128