From patchwork Tue Jun 20 07:23:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9798687 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D2A4360329 for ; Tue, 20 Jun 2017 07:23:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C531C27031 for ; Tue, 20 Jun 2017 07:23:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B79F628456; Tue, 20 Jun 2017 07:23:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A4DA27031 for ; Tue, 20 Jun 2017 07:23:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750990AbdFTHXI (ORCPT ); Tue, 20 Jun 2017 03:23:08 -0400 Received: from mail-wm0-f49.google.com ([74.125.82.49]:35865 "EHLO mail-wm0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750863AbdFTHXH (ORCPT ); Tue, 20 Jun 2017 03:23:07 -0400 Received: by mail-wm0-f49.google.com with SMTP id m125so12624130wmm.1 for ; Tue, 20 Jun 2017 00:23:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=eBsd4f1qdN6gL3H3N8Ap80q3aGxqtVLra1LdfuZaXQo=; b=IiSJB8OQX2MWdgq6OJpc877DtZLgUmzzeQD+ez/OdKB1QsBbdmt6K2UzCG5hvNmo6T s2t19cSH8dPCKoTMerVU8STqaVTnNiJolwtDf6qSNs8OL/IllLxnxbOVii1GXHrBmVQH G5SFw69tHqR8XgE7y7rrT+aGpcelLJULd8DwLWymPT11D5BIdQE7+6YUoAPbJ4BGellz CO7gbQMfdc1T7rK9c4bUczw6r51fQZ+QKxzuV7y8ZOOnyiXBZzIKWTEzOeFrG9Sg5tmR NVEbwu/aQBk7RxC+z1xOkWVslUoDvQsxxxlchUko0aVEnrksEzTzablSSpQhaf/YMTFA WtFg== X-Gm-Message-State: AKS2vOyTMS2p4HlQ+1Zky94OngvRHz8xDBtg3RTfHNXTr8wLherfFUEw sVfBSuGl9J55SkV/Ico= X-Received: by 10.28.111.5 with SMTP id k5mr1540586wmc.19.1497943385942; Tue, 20 Jun 2017 00:23:05 -0700 (PDT) Received: from [192.168.64.116] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id i138sm2851495wmf.30.2017.06.20.00.23.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Jun 2017 00:23:05 -0700 (PDT) Subject: Re: iSER Connection via LIO not working To: "Marciniszyn, Mike" , Thomas Rosenstein , Robert LeBlanc Cc: linux-rdma References: <2E68B5D1-49CF-4F2C-B7F3-DA20E05503B8@creamfinance.com> <2E0A092F-9530-44FB-8A6F-D76D8E94ABE0@creamfinance.com> <827EC8DD-86E7-4011-8F68-B29396B2034D@creamfinance.com> <1790271E-1A6B-492D-B4B6-701873B23A01@creamfinance.com> <32E1700B9017364D9B60AED9960492BC3431F8F1@fmsmsx120.amr.corp.intel.com> From: Sagi Grimberg Message-ID: <193ee311-f8a7-cf78-9dbc-190c89769adf@grimberg.me> Date: Tue, 20 Jun 2017 10:23:03 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <32E1700B9017364D9B60AED9960492BC3431F8F1@fmsmsx120.amr.corp.intel.com> Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP > Good job tracking this down. Thanks! > > static void > iser_calc_scsi_params(struct iser_conn *iser_conn, > unsigned int max_sectors) > { > struct iser_device *device = iser_conn->ib_conn.device; > unsigned short sg_tablesize, sup_sg_tablesize; > > sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K); > sup_sg_tablesize = min_t(unsigned, ISCSI_ISER_MAX_SG_TABLESIZE, > device->ib_device->attrs.max_fast_reg_page_list_len); > > iser_conn->scsi_sg_tablesize = min(sg_tablesize, sup_sg_tablesize); > } > > The bug is that device->ib_device->attrs.max_fast_reg_page_list_len should only be considered valid IFF IB_DEVICE_MEM_MGT_EXTENSIONS is set in > device->ib_device->device_cap_flags. The assignment you suggest should be an else on an if test of the device_cap_flags bit. Hi Guys, Good job tracking it down! I think you are the first one to test iSER on top of qib. Mike, What other variable do we need to look at to determine the maximum registered size per fmr? I was under the impression that max_fast_reg_page_list_len is not directly related to IB_DEVICE_MEM_MGT_EXTENSIONS as fmrs also accepts page lists. Anyway, Can we please please please add proper support for wr based memory registrations in qib? I have a patch piped for removing FMRs from iser altogether, now that this came up I obviously can't send it... Looking at the code, looks like qib inherits rdmavt memory registration with the new api, the only missing piece afaict is the remote invalidate and qib can turn on IB_DEVICE_MEM_MGT_EXTENSIONS. Can something like the (untested) below work: --- goto nack_op_err; @@ -1871,6 +1872,13 @@ void qib_rc_rcv(struct qib_ctxtdata *rcd, struct ib_header *hdr, goto no_immediate_data; /* FALLTHROUGH for SEND_ONLY_WITH_IMMEDIATE */ case OP(SEND_LAST_WITH_IMMEDIATE): + case OP(SEND_LAST_WITH_INVALIDATE): + if (opcode == OP(SEND_LAST_WITH_INVALIDATE)) { + wc.ex.invalidate_rkey = be32_to_cpu(ohdr->u.imm_data); + rvt_invalidate_rkey(qp, wc.ex.invalidate_rkey); + wc.wc_flags = IB_WC_WITH_INVALIDATE; + goto send_last; + } send_last_imm: wc.ex.imm_data = ohdr->u.imm_data; hdrsize += 4; -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/hw/qib/qib_rc.c b/drivers/infiniband/hw/qib/qib_rc.c index 4ddbcac5eabe..2bdfad6ecc11 100644 --- a/drivers/infiniband/hw/qib/qib_rc.c +++ b/drivers/infiniband/hw/qib/qib_rc.c @@ -1861,6 +1861,7 @@ void qib_rc_rcv(struct qib_ctxtdata *rcd, struct ib_header *hdr, case OP(SEND_ONLY): case OP(SEND_ONLY_WITH_IMMEDIATE): + case OP(SEND_ONLY_WITH_INVALIDATE): ret = qib_get_rwqe(qp, 0); if (ret < 0)