From patchwork Mon Dec 7 20:42:56 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 7789721 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 5D8C49F1C2 for ; Mon, 7 Dec 2015 20:43:03 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 8DA7220490 for ; Mon, 7 Dec 2015 20:43:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 92FA4204EB for ; Mon, 7 Dec 2015 20:43:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932998AbbLGUm7 (ORCPT ); Mon, 7 Dec 2015 15:42:59 -0500 Received: from mail-ig0-f174.google.com ([209.85.213.174]:34260 "EHLO mail-ig0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932839AbbLGUm6 (ORCPT ); Mon, 7 Dec 2015 15:42:58 -0500 Received: by igvg19 with SMTP id g19so87451086igv.1; Mon, 07 Dec 2015 12:42:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-type:content-transfer-encoding; bh=rCYT+Abef5cc8bFmJ58ZTtlxyhKiHKM1fIImPL/fVv8=; b=tXWvPDyboVHC46g4gJJMvcwqBfuR7MaKlAW+edehSZ29xpXmr01rt7sd/qmaZaOwLG z5bAnKpqERSjQsGWORnugTcTCw0pQ/DP6XFz4nifGxgSwDZFL4v5G2Qrh43dvSqwPD2f Zi80XdXrGlAwiq+QerB/2kiGEIbkgiFYcizgcBvtZYbGWllyiOH8kc8ZETSs/s5JWdJO MVbjyuX3u2GI0lWjCI2B5UjKpF9/B4PUeqpIlU4CsmVOJrfiUzEU1b+usG22EmA4nyl3 k5dWttItONEtDdFzGlUjeLkrJ6L1YOM+/MKX7imItkHq0O7QmZ+sZkalfhGsDdOnoFmP WRuA== X-Received: by 10.50.128.70 with SMTP id nm6mr329934igb.31.1449520978033; Mon, 07 Dec 2015 12:42:58 -0800 (PST) Received: from klimt.1015granger.net (c-68-46-169-226.hsd1.mi.comcast.net. [68.46.169.226]) by smtp.gmail.com with ESMTPSA id 124sm157988ioz.10.2015.12.07.12.42.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Dec 2015 12:42:57 -0800 (PST) Subject: [PATCH v3 4/6] svcrdma: Add infrastructure to send backwards direction RPC/RDMA calls From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Mon, 07 Dec 2015 15:42:56 -0500 Message-ID: <20151207204256.12988.76473.stgit@klimt.1015granger.net> In-Reply-To: <20151207203851.12988.97804.stgit@klimt.1015granger.net> References: <20151207203851.12988.97804.stgit@klimt.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID,T_RP_MATCHES_RCVD,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To support the NFSv4.1 backchannel on RDMA connections, add a mechanism for sending a backwards-direction RPC/RDMA call on a connection established by a client. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc_rdma.h | 2 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index f71c625..bf9b17b 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -215,6 +215,8 @@ extern int rdma_read_chunk_frmr(struct svcxprt_rdma *, struct svc_rqst *, extern int svc_rdma_sendto(struct svc_rqst *); extern struct rpcrdma_read_chunk * svc_rdma_get_read_chunk(struct rpcrdma_msg *); +extern int svc_rdma_bc_post_send(struct svcxprt_rdma *, + struct svc_rdma_op_ctxt *, struct xdr_buf *); /* svc_rdma_transport.c */ extern int svc_rdma_send(struct svcxprt_rdma *, struct ib_send_wr *); diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index bad5eaa..846df63 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -648,3 +648,64 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) svc_rdma_put_context(ctxt, 0); return ret; } + +/* Send a backwards direction RPC call. + * + * Caller holds the connection's mutex and has already marshaled the + * RPC/RDMA request. Before sending the request, this API also posts + * an extra receive buffer to catch the bc reply for this request. + */ +int svc_rdma_bc_post_send(struct svcxprt_rdma *rdma, + struct svc_rdma_op_ctxt *ctxt, struct xdr_buf *sndbuf) +{ + struct svc_rdma_req_map *vec; + struct ib_send_wr send_wr; + int ret; + + vec = svc_rdma_get_req_map(); + ret = map_xdr(rdma, sndbuf, vec); + if (ret) + goto out; + + /* Post a recv buffer to handle reply for this request */ + ret = svc_rdma_post_recv(rdma); + if (ret) { + pr_err("svcrdma: Failed to post bc receive buffer, err=%d. " + "Closing transport %p.\n", ret, rdma); + set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags); + ret = -ENOTCONN; + goto out; + } + + ctxt->wr_op = IB_WR_SEND; + ctxt->direction = DMA_TO_DEVICE; + ctxt->sge[0].lkey = rdma->sc_dma_lkey; + ctxt->sge[0].length = sndbuf->len; + ctxt->sge[0].addr = + ib_dma_map_page(rdma->sc_cm_id->device, ctxt->pages[0], 0, + sndbuf->len, DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdma->sc_cm_id->device, ctxt->sge[0].addr)) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } + atomic_inc(&rdma->sc_dma_used); + + memset(&send_wr, 0, sizeof send_wr); + send_wr.wr_id = (unsigned long)ctxt; + send_wr.sg_list = ctxt->sge; + send_wr.num_sge = 1; + send_wr.opcode = IB_WR_SEND; + send_wr.send_flags = IB_SEND_SIGNALED; + + ret = svc_rdma_send(rdma, &send_wr); + if (ret) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } +out: + svc_rdma_put_req_map(vec); + dprintk("svcrdma: %s returns %d\n", __func__, ret); + return ret; +}