From patchwork Thu Mar 16 15:52:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9628745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E46AF604AA for ; Thu, 16 Mar 2017 15:52:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4C722029B for ; Thu, 16 Mar 2017 15:52:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C7BC828654; Thu, 16 Mar 2017 15:52:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5582B2029B for ; Thu, 16 Mar 2017 15:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754721AbdCPPwi (ORCPT ); Thu, 16 Mar 2017 11:52:38 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:33039 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752646AbdCPPwg (ORCPT ); Thu, 16 Mar 2017 11:52:36 -0400 Received: by mail-it0-f65.google.com with SMTP id g138so7613153itb.0; Thu, 16 Mar 2017 08:52:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=NMKTKoP1Qz7qCovq88Qnd0AS0VQ2F5XZ07/HbJx3hxo=; b=WEYRjKPCl2bVqFtMHw6u+Ua51rjupCkkQDw1pZjzjawASzDwES/GtBMNU+qUEgMwEa ZOOl7ZCp0uquhDzyWuMwgff5nHyNECRdM/BuOIat8Ggpq8FBHZHo45qTAtcqW46Y9Ft8 QmnqwBdbguTx2UxwrBCd8H25l9jfkCbPL3XeNb1VoWth6+CuSLXMvAyL/pox6vUz+xfp CWGoUvf2MinuHwJ7bHEJTZ+Fktxv8a6p99tbp26o4O2TC5Ux7c5dj3l77F21hfYYxgKt 8xxM+zbm0Mz4u0h/iP4pwoU6DregMRFyspjeBBE2ZoqVcoF+B/1lilmmUvAx7Ogsa3kr 4ttg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=NMKTKoP1Qz7qCovq88Qnd0AS0VQ2F5XZ07/HbJx3hxo=; b=Y+0ZUcY5wLioPVENLISjaQd2RCKDrLMSjiWDhgQ/QF8Mek5C95iJj5DxJ/E/mSE05P DJtXLue+OEkNrUJJHUZLuh3tPoMYcAiat2g3FTqZTFDtmr/3nkfkGhuq4bvbv+0jZFP7 t8ukhh8Vs4OuiT1KSZJVY/M/YziK1P9nMjvFQWpK6kqP1Z9oa8vVwa3yS2b0DK0+uojj xPm/hL/sx+F6bVYvGHh9v+NELcOnBnkRj8EAlTFsJmwWtJX6pkQcq/GeUzNn4Zngg7Zk y5x84bznXP9vFySYraHMnHkPr0zMAneLqmFl8atQFuJumU8nk/eWSvoQNhmOjoC3szRJ zIFw== X-Gm-Message-State: AFeK/H0FN+Z1cWz1c0G+ZnwVODwSmVLCnFR6sHGYDQhSqobv43Jsk4hoGTHtrAIh2Ct4Nw== X-Received: by 10.36.143.10 with SMTP id k10mr11013387itd.55.1489679555156; Thu, 16 Mar 2017 08:52:35 -0700 (PDT) Received: from klimt.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:6aa0]) by smtp.gmail.com with ESMTPSA id g204sm1789299ita.31.2017.03.16.08.52.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Mar 2017 08:52:34 -0700 (PDT) Subject: [PATCH v1 01/14] svcrdma: Move send_wr to svc_rdma_op_ctxt From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 16 Mar 2017 11:52:34 -0400 Message-ID: <20170316155234.4482.94225.stgit@klimt.1015granger.net> In-Reply-To: <20170316154132.4482.56769.stgit@klimt.1015granger.net> References: <20170316154132.4482.56769.stgit@klimt.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Clean up: Move the ib_send_wr off the stack, and move common send WR setup code into a helper. This is a refactoring change only. Signed-off-by: Chuck Lever Reviewed-by: Sagi Grimberg --- include/linux/sunrpc/svc_rdma.h | 3 + net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 12 +----- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 57 ++++++++++++++++------------ 3 files changed, 37 insertions(+), 35 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index b105f73..fa3ef11 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -85,6 +85,7 @@ struct svc_rdma_op_ctxt { enum dma_data_direction direction; int count; unsigned int mapped_sges; + struct ib_send_wr send_wr; struct ib_sge sge[RPCSVC_MAXPAGES]; struct page *pages[RPCSVC_MAXPAGES]; }; @@ -227,6 +228,8 @@ extern int rdma_read_chunk_frmr(struct svcxprt_rdma *, struct svc_rqst *, /* svc_rdma_sendto.c */ extern int svc_rdma_map_xdr(struct svcxprt_rdma *, struct xdr_buf *, struct svc_rdma_req_map *, bool); +extern void svc_rdma_build_send_wr(struct svc_rdma_op_ctxt *ctxt, + int num_sge); extern int svc_rdma_sendto(struct svc_rqst *); extern void svc_rdma_send_error(struct svcxprt_rdma *, struct rpcrdma_msg *, int); diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c index ff1df40..6741ed0 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c @@ -104,7 +104,6 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, struct xdr_buf *sndbuf = &rqst->rq_snd_buf; struct svc_rdma_op_ctxt *ctxt; struct svc_rdma_req_map *vec; - struct ib_send_wr send_wr; int ret; vec = svc_rdma_get_req_map(rdma); @@ -132,15 +131,8 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, } svc_rdma_count_mappings(rdma, ctxt); - memset(&send_wr, 0, sizeof(send_wr)); - ctxt->cqe.done = svc_rdma_wc_send; - send_wr.wr_cqe = &ctxt->cqe; - send_wr.sg_list = ctxt->sge; - send_wr.num_sge = 1; - send_wr.opcode = IB_WR_SEND; - send_wr.send_flags = IB_SEND_SIGNALED; - - ret = svc_rdma_send(rdma, &send_wr); + svc_rdma_build_send_wr(ctxt, 1); + ret = svc_rdma_send(rdma, &ctxt->send_wr); if (ret) { ret = -EIO; goto out_unmap; diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index 515221b..fdf8e3d 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -435,6 +435,28 @@ static int send_reply_chunks(struct svcxprt_rdma *xprt, return -EIO; } +/** + * svc_rdma_build_send_wr - Set up a Send Work Request + * @ctxt: op_ctxt for transmitting the Send WR + * + */ +void svc_rdma_build_send_wr(struct svc_rdma_op_ctxt *ctxt, int num_sge) +{ + struct ib_send_wr *send_wr; + + send_wr = &ctxt->send_wr; + send_wr->next = NULL; + ctxt->cqe.done = svc_rdma_wc_send; + send_wr->wr_cqe = &ctxt->cqe; + send_wr->sg_list = ctxt->sge; + send_wr->num_sge = num_sge; + send_wr->opcode = IB_WR_SEND; + send_wr->send_flags = IB_SEND_SIGNALED; + + dprintk("svcrdma: posting Send WR with %u sge(s)\n", + send_wr->num_sge); +} + /* This function prepares the portion of the RPCRDMA message to be * sent in the RDMA_SEND. This function is called after data sent via * RDMA has already been transmitted. There are three cases: @@ -460,7 +482,7 @@ static int send_reply(struct svcxprt_rdma *rdma, u32 inv_rkey) { struct svc_rdma_op_ctxt *ctxt; - struct ib_send_wr send_wr; + struct ib_send_wr *send_wr; u32 xdr_off; int sge_no; int sge_bytes; @@ -524,19 +546,14 @@ static int send_reply(struct svcxprt_rdma *rdma, pr_err("svcrdma: Too many sges (%d)\n", sge_no); goto err; } - memset(&send_wr, 0, sizeof send_wr); - ctxt->cqe.done = svc_rdma_wc_send; - send_wr.wr_cqe = &ctxt->cqe; - send_wr.sg_list = ctxt->sge; - send_wr.num_sge = sge_no; - if (inv_rkey) { - send_wr.opcode = IB_WR_SEND_WITH_INV; - send_wr.ex.invalidate_rkey = inv_rkey; - } else - send_wr.opcode = IB_WR_SEND; - send_wr.send_flags = IB_SEND_SIGNALED; - ret = svc_rdma_send(rdma, &send_wr); + svc_rdma_build_send_wr(ctxt, sge_no); + send_wr = &ctxt->send_wr; + if (inv_rkey) { + send_wr->opcode = IB_WR_SEND_WITH_INV; + send_wr->ex.invalidate_rkey = inv_rkey; + } + ret = svc_rdma_send(rdma, send_wr); if (ret) goto err; @@ -652,7 +669,6 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) void svc_rdma_send_error(struct svcxprt_rdma *xprt, struct rpcrdma_msg *rmsgp, int status) { - struct ib_send_wr err_wr; struct page *p; struct svc_rdma_op_ctxt *ctxt; enum rpcrdma_errcode err; @@ -692,17 +708,8 @@ void svc_rdma_send_error(struct svcxprt_rdma *xprt, struct rpcrdma_msg *rmsgp, } svc_rdma_count_mappings(xprt, ctxt); - /* Prepare SEND WR */ - memset(&err_wr, 0, sizeof(err_wr)); - ctxt->cqe.done = svc_rdma_wc_send; - err_wr.wr_cqe = &ctxt->cqe; - err_wr.sg_list = ctxt->sge; - err_wr.num_sge = 1; - err_wr.opcode = IB_WR_SEND; - err_wr.send_flags = IB_SEND_SIGNALED; - - /* Post It */ - ret = svc_rdma_send(xprt, &err_wr); + svc_rdma_build_send_wr(ctxt, 1); + ret = svc_rdma_send(xprt, &ctxt->send_wr); if (ret) { dprintk("svcrdma: Error %d posting send for protocol error\n", ret);