From patchwork Wed Oct 22 09:02:34 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Or Gerlitz X-Patchwork-Id: 5130531 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 6954F9F30B for ; Wed, 22 Oct 2014 09:03:30 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 85014201D3 for ; Wed, 22 Oct 2014 09:03:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 874F72011D for ; Wed, 22 Oct 2014 09:03:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752134AbaJVJD0 (ORCPT ); Wed, 22 Oct 2014 05:03:26 -0400 Received: from eu1sys200aog121.obsmtp.com ([207.126.144.151]:35716 "EHLO eu1sys200aog121.obsmtp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751347AbaJVJDY (ORCPT ); Wed, 22 Oct 2014 05:03:24 -0400 Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob121.postini.com ([207.126.147.11]) with SMTP ID DSNKVEdy1hkXTZfwmOEyA7XNHcx9IH7jLrcj@postini.com; Wed, 22 Oct 2014 09:03:23 UTC Received: from [10.223.3.34] (10.223.3.34) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Wed, 22 Oct 2014 12:02:34 +0300 Message-ID: <544772AA.7060003@mellanox.com> Date: Wed, 22 Oct 2014 12:02:34 +0300 From: Or Gerlitz User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: "Nicholas A. Bellinger" , Or Gerlitz CC: Chris Moore , "linux-rdma@vger.kernel.org" , Sagi Grimberg Subject: Re: Can someone help me understand the reason for this code in ib_isert.c? References: <462EF229174FDB4D92ACE4656EA5610051E2EEF9@CMEXMB1.ad.emulex.com> <1413954397.30983.33.camel@haakon3.risingtidesystems.com> In-Reply-To: <1413954397.30983.33.camel@haakon3.risingtidesystems.com> X-Originating-IP: [10.223.3.34] Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-8.3 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 10/22/2014 8:06 AM, Nicholas A. Bellinger wrote: > On Tue, 2014-10-21 at 00:13 +0300, Or Gerlitz wrote: >> On Mon, Oct 20, 2014 at 6:29 PM, Chris Moore wrote: >>> The following code is in isert_conn_setup_qp() in ib_isert.c: >>> >>> /* >>> * FIXME: Use devattr.max_sge - 2 for max_send_sge as >>> * work-around for RDMA_READ.. >>> */ >>> attr.cap.max_send_sge = device->dev_attr.max_sge - 2; >>> >>> It's not clear from the comment what this is a work-around for, and I wasn't able >>> to figure it out from looking at logs. >> I believe this refers to some IBTA spec corner case which comes into >> play with the max_sges advertized by mlx4, Eli, can you shed some >> light (IBTA pointer) on that? is this the case (i.e dev_attr.max_sge >> isn't always achievable) with mlx5 too? >> > It's for ConnectX-2 reporting max_sge=31 during development, while the > largest max_send_sge for stable large block RDMA reads was (is..?) 29 > SGEs. > >>> In trying to get isert working with ocrdma I ran into a problem where the code >>> thought there was only 1 send SGE available when in fact the device has 3. >>> Then I found the above work-around, which explained the problem. >> Nic, any reason for attr.cap.max_send_sge = 1 not to work OK? >> > IIRC, pre fastreg code supported max_send_sge=1 by limiting the transfer > size per RDMA read, and would issue subsequent RDMA read + offset from > completion up to the total se_cmd->data_length. > > Not sure how this works with fastreg today. Sagi..? > While on fastreg mode, for RDMA reads we use only one SGE, talking to Sagi he explained me that the problem with creating the QP with max_send_sge = 1 has to do with other flows where two or even three SGEs are needed, e.g when the iscsi/iser header (doesn't exist in RDMA ops) is taken from one spot of the memory and the data (sense) taken from another one? Nic, we need to see what is the minimum needed by the code (two?) and tweak that line to make sure it gets there as long as the device supports it, SB, makes sense? attr.cap.max_recv_sge = 1; --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 0bea577..24abbb3 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -115,9 +115,10 @@ isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id, attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS; /* * FIXME: Use devattr.max_sge - 2 for max_send_sge as - * work-around for RDMA_READ.. + * work-around for RDMA_READ.. still need to make sure to + * have at least two SGEs for SCSI responses. */ - attr.cap.max_send_sge = device->dev_attr.max_sge - 2; + attr.cap.max_send_sge = max(2, device->dev_attr.max_sge - 2); isert_conn->max_sge = attr.cap.max_send_sge;