From patchwork Fri Dec 14 00:58:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10730319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A67F191E for ; Fri, 14 Dec 2018 02:23:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A69C2D092 for ; Fri, 14 Dec 2018 02:23:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7A53C2D094; Fri, 14 Dec 2018 02:23:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEB2E2D092 for ; Fri, 14 Dec 2018 02:23:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729012AbeLNCX0 (ORCPT ); Thu, 13 Dec 2018 21:23:26 -0500 Received: from com-out001.mailprotect.be ([83.217.72.83]:59025 "EHLO com-out001.mailprotect.be" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728517AbeLNCXZ (ORCPT ); Thu, 13 Dec 2018 21:23:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mailprotect.be; s=mail; h=Content-Transfer-Encoding:MIME-Version:References :In-Reply-To:Message-Id:Date:Subject:Cc:To:From:reply-to:sender:bcc: content-type; bh=xLb3Llf5CMWxwWKfF0exBkgeNSIdvXpuyfNYP/aMro0=; b=nyrgAMeT+nTI Ht03iVkzEk1wHY01PqsWgUneCiFVqP+vzD1dVNanRz6Qv9WVMnsW8QItrqLMY8+H9iHozXgpjVAx6 r7Oqg7v8cUGxGZVrNuAZFrSDzid3Ai5ifEKUD09cz8Gc9iEdao0xGPvFwoWEo8ihdbnuv8RFVQ2cU CSN1YnBBTarE/h73U0dCfiVenx65Nv9T2hktjo7s5kqi1xv2ADq58YdWAMw41cZZrajZWJ/VkgJ8P e1Y23Frp+c3SHUl612Z5ZsZgXdFtxTqQt8PrvMlsFBeG0rpoiP8q/tkH92cyleY4cEJ4rNZqH3uRj kS/093JKdYArF4W+3mtnCg==; Received: from smtp-auth.mailprotect.be ([178.208.39.155]) by com-mpt-out001.mailprotect.be with esmtp (Exim 4.89) (envelope-from ) id 1gXbp6-000Dho-BA; Fri, 14 Dec 2018 01:59:25 +0100 Received: from desktop-bart.svl.corp.google.com (unknown [104.133.8.89]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-auth.mailprotect.be (Postfix) with ESMTPSA id 7E617C0DE7; Fri, 14 Dec 2018 01:59:18 +0100 (CET) From: Bart Van Assche To: Jason Gunthorpe Cc: Doug Ledford , linux-rdma@vger.kernel.org, Bart Van Assche , Sergey Gorenko , Max Gurtovoy , Laurence Oberman Subject: [PATCH 07/13] RDMA/srp: Rework handling of the maximum information unit length Date: Thu, 13 Dec 2018 16:58:45 -0800 Message-Id: <20181214005851.176412-8-bvanassche@acm.org> X-Mailer: git-send-email 2.20.0.405.gbc1bbc6f85-goog In-Reply-To: <20181214005851.176412-1-bvanassche@acm.org> References: <20181214005851.176412-1-bvanassche@acm.org> MIME-Version: 1.0 X-Originating-IP: 178.208.39.155 X-SpamExperts-Domain: mailprotect.be X-SpamExperts-Username: 178.208.39.128/27 Authentication-Results: mailprotect.be; auth=pass smtp.auth=178.208.39.128/27@mailprotect.be X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: SB/global_tokens (0.00890387057531) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5pbGXc3QcDkfZf5an6jXMgd602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO1tLifGj39bI0bcPyaJsYTbLl/LY5YF9Vo2OhJGJjKCGCCZpQ81ZPiyihk8FOHdtvo4l V1QHumGYL6MoCWQBeDaU32qUPOJX7vhbzYcc5l71m4VKGlZauPS2ZI2/MGgfC8Av92G8VFvri8dZ 651y1LnBHvoYB1tNyzlE795H++xpIGTTJ1+gdXdk1Q+yT5nnaqsSkubKoeJsHgdbUJ7agEzXnQGj WDOitbxNOIjwRljNwEVxceuEInfFOaYFnjfi7c902MBxp+G42GIBqokZAnJ5xBHkCMCjMposz65/ e+f6NiSg6RpPztwkEB6XvRg4fu70pheH5TNN6RiG9N0Jt+0ML5i3g2GFaGQnZ3KowlU8anvoc7s0 eHtsyrH7f4vWs1KcgxQTqdSE3cxlP67ssuZxn2a3k66wqHQRkdHrnUf0XuDVW1RAVP/E7/UYy9T8 rkAMvHmBTTipZKIPq3iOj9tSKKqNUqeXR0rRzsN1a4pG17niOlhFcyO9tgQb+/TkwEEqJ2YhaHYK PX4HHcW4fmLLdh/NSb/qpYVFz2DoGXALBBlIJ0WBFPu4HenOcZSI92EP+JYPRX76oPSBtBBPLYxI dE/smxj1IgWUYKeT3voJFsCJuVlVkChAavMv6VkLdtjlDHh8k6TTdHl8m1/8O/+BJ3GMOLWW0/7C m295vYiNyLS7X0NCOkhwtiYnV40XsePthB8DPVcWbpQfMz+dJcO48JhOWtEYVZv1TAY1dfM3jHqG d7BkB7ETNreKwKapqncwRpKL9rZSlplfHnvuvZVfkv4nvWz8lA+HgKkiKsBZTgka5QxI8QziNK+7 QPGUAEhNl7HGKUVz26Ejndnif9fnShMu2z+YCLdt1O36QJzTXh1YmXSTB9eWa9TZuy8mu9uVv4l1 vdw9m8mDEnaTAdVSvSmjlkX7We5Taya/6p8GnpBkO9UBwlGCoO6DYZIkF7Fm3Hpa/F8E4PqWfene xx31H/aAwarQpYDOYx/6JtUO/GqJg4E/KNK41fXYWiMK3pv7c+m0ulsMI3nr6mT+esOUkxE+SK4C kqqQuY/ADtB7MjMm/D2rEPnNAuCCnt9B91lvN+2w6IrWsFqaJSVzMeH10tfqoGlbdTkTLaWCAXLa p1cbnmh6pFmhgqKvdyLtSa8zJ25JHkf2y1fH38x/opU= X-Report-Abuse-To: spam@com-mpt-mgt001.mailprotect.be Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move the maximum initiator to target information unit length parameter from struct srp_target_port into struct srp_rdma_ch. This patch does not change any functionality but makes the next patch easier to read. Cc: Sergey Gorenko Cc: Max Gurtovoy Cc: Laurence Oberman Signed-off-by: Bart Van Assche --- drivers/infiniband/ulp/srp/ib_srp.c | 40 +++++++++++++++++++---------- drivers/infiniband/ulp/srp/ib_srp.h | 3 ++- 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 06fcbd181bbf..de53bbf91c62 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -823,7 +823,8 @@ static u8 srp_get_subnet_timeout(struct srp_host *host) return subnet_timeout; } -static int srp_send_req(struct srp_rdma_ch *ch, bool multich) +static int srp_send_req(struct srp_rdma_ch *ch, uint32_t max_iu_len, + bool multich) { struct srp_target_port *target = ch->target; struct { @@ -852,7 +853,7 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich) req->ib_req.opcode = SRP_LOGIN_REQ; req->ib_req.tag = 0; - req->ib_req.req_it_iu_len = cpu_to_be32(target->max_iu_len); + req->ib_req.req_it_iu_len = cpu_to_be32(max_iu_len); req->ib_req.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT | SRP_BUF_FORMAT_INDIRECT); req->ib_req.req_flags = (multich ? SRP_MULTICHAN_MULTI : @@ -1145,7 +1146,8 @@ static int srp_connected_ch(struct srp_target_port *target) return c; } -static int srp_connect_ch(struct srp_rdma_ch *ch, bool multich) +static int srp_connect_ch(struct srp_rdma_ch *ch, uint32_t max_iu_len, + bool multich) { struct srp_target_port *target = ch->target; int ret; @@ -1158,7 +1160,7 @@ static int srp_connect_ch(struct srp_rdma_ch *ch, bool multich) while (1) { init_completion(&ch->done); - ret = srp_send_req(ch, multich); + ret = srp_send_req(ch, max_iu_len, multich); if (ret) goto out; ret = wait_for_completion_interruptible(&ch->done); @@ -1344,6 +1346,16 @@ static void srp_terminate_io(struct srp_rport *rport) } } +/* Calculate maximum initiator to target information unit length. */ +static uint32_t srp_max_it_iu_len(int cmd_sg_cnt) +{ + uint32_t max_iu_len = sizeof(struct srp_cmd) + SRP_MAX_ADD_CDB_LEN + + sizeof(struct srp_indirect_buf) + + cmd_sg_cnt * sizeof(struct srp_direct_buf); + + return max_iu_len; +} + /* * It is up to the caller to ensure that srp_rport_reconnect() calls are * serialized and that no concurrent srp_queuecommand(), srp_abort(), @@ -1357,6 +1369,7 @@ static int srp_rport_reconnect(struct srp_rport *rport) { struct srp_target_port *target = rport->lld_data; struct srp_rdma_ch *ch; + uint32_t max_iu_len = srp_max_it_iu_len(target->cmd_sg_cnt); int i, j, ret = 0; bool multich = false; @@ -1402,7 +1415,7 @@ static int srp_rport_reconnect(struct srp_rport *rport) ch = &target->ch[i]; if (ret) break; - ret = srp_connect_ch(ch, multich); + ret = srp_connect_ch(ch, max_iu_len, multich); multich = true; } @@ -2316,7 +2329,7 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) req = &ch->req_ring[idx]; dev = target->srp_host->srp_dev->dev; - ib_dma_sync_single_for_cpu(dev, iu->dma, target->max_iu_len, + ib_dma_sync_single_for_cpu(dev, iu->dma, ch->max_it_iu_len, DMA_TO_DEVICE); scmnd->host_scribble = (void *) req; @@ -2353,7 +2366,7 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) goto err_iu; } - ib_dma_sync_single_for_device(dev, iu->dma, target->max_iu_len, + ib_dma_sync_single_for_device(dev, iu->dma, ch->max_it_iu_len, DMA_TO_DEVICE); if (srp_post_send(ch, iu, len)) { @@ -2421,7 +2434,7 @@ static int srp_alloc_iu_bufs(struct srp_rdma_ch *ch) for (i = 0; i < target->queue_size; ++i) { ch->tx_ring[i] = srp_alloc_iu(target->srp_host, - target->max_iu_len, + ch->max_it_iu_len, GFP_KERNEL, DMA_TO_DEVICE); if (!ch->tx_ring[i]) goto err; @@ -2487,6 +2500,9 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id, if (lrsp->opcode == SRP_LOGIN_RSP) { ch->max_ti_iu_len = be32_to_cpu(lrsp->max_ti_iu_len); ch->req_lim = be32_to_cpu(lrsp->req_lim_delta); + ch->max_it_iu_len = srp_max_it_iu_len(target->cmd_sg_cnt); + WARN_ON_ONCE(ch->max_it_iu_len > + be32_to_cpu(lrsp->max_it_iu_len)); /* * Reserve credits for task management so we don't @@ -3734,6 +3750,7 @@ static ssize_t srp_create_target(struct device *dev, int ret, node_idx, node, cpu, i; unsigned int max_sectors_per_mr, mr_per_cmd = 0; bool multich = false; + uint32_t max_iu_len; target_host = scsi_host_alloc(&srp_template, sizeof (struct srp_target_port)); @@ -3839,10 +3856,7 @@ static ssize_t srp_create_target(struct device *dev, target->mr_per_cmd = mr_per_cmd; target->indirect_size = target->sg_tablesize * sizeof (struct srp_direct_buf); - target->max_iu_len = sizeof (struct srp_cmd) + - SRP_MAX_ADD_CDB_LEN + - sizeof (struct srp_indirect_buf) + - target->cmd_sg_cnt * sizeof (struct srp_direct_buf); + max_iu_len = srp_max_it_iu_len(target->cmd_sg_cnt); INIT_WORK(&target->tl_err_work, srp_tl_err_work); INIT_WORK(&target->remove_work, srp_remove_work); @@ -3897,7 +3911,7 @@ static ssize_t srp_create_target(struct device *dev, if (ret) goto err_disconnect; - ret = srp_connect_ch(ch, multich); + ret = srp_connect_ch(ch, max_iu_len, multich); if (ret) { char dst[64]; diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h index cba29e378a74..9a271ae6573b 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.h +++ b/drivers/infiniband/ulp/srp/ib_srp.h @@ -132,6 +132,7 @@ struct srp_request { /** * struct srp_rdma_ch * @comp_vector: Completion vector used by this RDMA channel. + * @max_it_iu_len: Maximum initiator-to-target information unit length. * @max_ti_iu_len: Maximum target-to-initiator information unit length. */ struct srp_rdma_ch { @@ -149,6 +150,7 @@ struct srp_rdma_ch { struct ib_fmr_pool *fmr_pool; struct srp_fr_pool *fr_pool; }; + uint32_t max_it_iu_len; uint32_t max_ti_iu_len; /* Everything above this point is used in the hot path of @@ -197,7 +199,6 @@ struct srp_target_port { u32 ch_count; u32 lkey; enum srp_target_state state; - unsigned int max_iu_len; unsigned int cmd_sg_cnt; unsigned int indirect_size; bool allow_ext_sg;