From patchwork Wed Apr 6 17:34:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 8764081 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 2DA689F39A for ; Wed, 6 Apr 2016 17:34:45 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 0432720121 for ; Wed, 6 Apr 2016 17:34:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DECF320115 for ; Wed, 6 Apr 2016 17:34:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751930AbcDFRek (ORCPT ); Wed, 6 Apr 2016 13:34:40 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:33040 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751873AbcDFRej (ORCPT ); Wed, 6 Apr 2016 13:34:39 -0400 Received: by mail-wm0-f66.google.com with SMTP id i204so15238741wmd.0 for ; Wed, 06 Apr 2016 10:34:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=2y3eA8bi1uMfh7B3bJyF/7PP/DZZ/NLeX5JmKrVciFc=; b=g2/KmIKFG2DT5kvOuejnTOF9D/jq80Ls7KJrAq3uPiqfBuXdvetOKL8RCWicifpbSC hBdRxZOKbrGZQrOHItxOMsNZYpUud/Cg6Ym7hEGT5FtmrBnXtwtTiJeIqSUJFINx+kGc wyiuVPaBMSXEjs/AjwROks+wJUGdJOstU+NiL6Xr+vTWn9byVmnQb8owIxMx/BkK9Vx3 YEmkTUmMx5iOa6K6aHuBtuFSJWEld4gMuxMRaKz9gNELK4ktBLwSWUQOR/EokPf5V48g 0YW4eHOvuAzQlZGkGyO8bosv/fzKyVsgT2es00i2nRzocP0865bMBoyf2rxjF+bNkNHF QheQ== X-Gm-Message-State: AD7BkJLTt/zNOWvXul3+jI7KJ8jLOpd3awfwcjb/8ClB9GqRbeusyZRKmjv84VN8f12RyA== X-Received: by 10.28.36.86 with SMTP id k83mr9825521wmk.95.1459964078105; Wed, 06 Apr 2016 10:34:38 -0700 (PDT) Received: from [192.168.1.248] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id xx3sm4288316wjc.32.2016.04.06.10.34.36 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 06 Apr 2016 10:34:37 -0700 (PDT) Subject: Re: iSER initiator in 4.5 is unhappy.. To: Christoph Hellwig References: <20160404134656.GA30269@infradead.org> <5702A46F.5070109@grimberg.me> <5702A6AC.9020008@grimberg.me> <20160404225831.GA6725@infradead.org> <57037D8E.4030506@grimberg.me> <20160405094339.GA1622@infradead.org> Cc: linux-rdma@vger.kernel.org From: sagig Message-ID: <570548AB.4090802@grimberg.me> Date: Wed, 6 Apr 2016 20:34:35 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <20160405094339.GA1622@infradead.org> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP >> This implies that maybe something broke in ib_sg_to_pages: >> >> Can you run iser with this patch applied (FR or FMR): > > For FMR this won't trigger, but here is the FR run: > > [ 63.016616] iser: iser_fast_reg_mr: failed to map sg (101/102) > [ 63.023183] iser: iser_data_buf_dump: sg[0] dma_addr:0x90E15D200 > page:0xffffea0024385740 off:0x200 sz:0xe00 dma_len:0xe00 > [ 63.035466] iser: iser_data_buf_dump: sg[1] dma_addr:0x90611D000 > page:0xffffea0024184740 off:0x0 sz:0x1000 dma_len:0x1000 > [ 63.047749] iser: iser_data_buf_dump: sg[2] dma_addr:0x90EDB3000 > page:0xffffea00243b6cc0 off:0x0 sz:0x1000 dma_len:0x1000 > [ 63.060050] iser: iser_data_buf_dump: sg[3] dma_addr:0x9062CB000 > page:0xffffea002418b2c0 off:0x0 sz:0x1000 dma_len:0x1000 ... > [ 64.215764] iser: iser_data_buf_dump: sg[95] dma_addr:0x90E440000 > page:0xffffea0024391000 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.228380] iser: iser_data_buf_dump: sg[96] dma_addr:0x90E43E000 > page:0xffffea0024390f80 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.240995] iser: iser_data_buf_dump: sg[97] dma_addr:0x901586000 > page:0xffffea0024056180 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.253611] iser: iser_data_buf_dump: sg[98] dma_addr:0x90F390000 > page:0xffffea00243ce400 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.266219] iser: iser_data_buf_dump: sg[99] dma_addr:0x90660E000 > page:0xffffea0024198380 off:0x0 sz:0x2000 dma_len:0x2000 > [ 64.278836] iser: iser_data_buf_dump: sg[100] dma_addr:0x9017E6000 > page:0xffffea002405f980 off:0x0 sz:0x1000 dma_len:0x1000 > [ 64.291577] iser: iser_data_buf_dump: sg[101] dma_addr:0x93FBF1000 > page:0xffffea0024fefc40 off:0x0 sz:0x200 dma_len:0x200 > So this scatterlist seems perfectly fine (no gaps). The only thing I can think of is that we don't have enough space reserved, but I counted 127 pages in the scatterlist when iser reserves 128 (by default). Just in case I mis-counted, can you (or Max) try with the below patch: --- if (sg_tablesize > sup_sg_tablesize) { sg_tablesize = sup_sg_tablesize; -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c index ed54b388e7ad..f2ac891c432a 100644 --- a/drivers/infiniband/ulp/iser/iser_initiator.c +++ b/drivers/infiniband/ulp/iser/iser_initiator.c @@ -250,7 +250,7 @@ int iser_alloc_rx_descriptors(struct iser_conn *iser_conn, iser_conn->min_posted_rx = iser_conn->qp_max_recv_dtos >> 2; if (device->reg_ops->alloc_reg_res(ib_conn, session->scsi_cmds_max, - iser_conn->scsi_sg_tablesize)) + iser_conn->scsi_sg_tablesize + 1)) goto create_rdma_reg_res_failed; if (iser_alloc_login_buf(iser_conn)) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 40c0f4978e2f..937b97f34be1 100644 --- a/drivers/infiniband/ulp/iser/iser_verbs.c +++ b/drivers/infiniband/ulp/iser/iser_verbs.c @@ -721,7 +721,7 @@ iser_calc_scsi_params(struct iser_conn *iser_conn, sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K); sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE, - device->ib_device->attrs.max_fast_reg_page_list_len); + device->ib_device->attrs.max_fast_reg_page_list_len - 1);