From patchwork Thu Sep 17 20:46:22 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 7210421 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 4FE0EBEEC1 for ; Thu, 17 Sep 2015 20:46:31 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 56ACF20785 for ; Thu, 17 Sep 2015 20:46:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 69F8E20783 for ; Thu, 17 Sep 2015 20:46:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752212AbbIQUq0 (ORCPT ); Thu, 17 Sep 2015 16:46:26 -0400 Received: from mail-qg0-f43.google.com ([209.85.192.43]:35296 "EHLO mail-qg0-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752205AbbIQUqY (ORCPT ); Thu, 17 Sep 2015 16:46:24 -0400 Received: by qgt47 with SMTP id 47so23447096qgt.2; Thu, 17 Sep 2015 13:46:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-type:content-transfer-encoding; bh=/E2Hps8ztQL/2oy0Eds0pgNS/rMzyiPDTH/C8Ax7g7o=; b=z9Oiol0cTJCyM0JQTKHMvqHEq2+IWNiWbZiRgArnp6+S44HhgTeeaTHcGsxJ/1U/gx 30uvd1b7XIzyXBgRiTHGmsGDGXgsUQMWE63WU6iLFY9101/vF/5hov3yOxqzRgfPwdT2 kQyMPcjlhz3BRI55sz+lHhS1IR+AgMLFRKpJJHLtxLo1XTt2HJCkM5fbYET9kpHaJVhh 8+ZjdCglgxoOjKxLQ3WQIcM/3cpLhcVd1Y6IzhQjyBD5Sw3LbpdNk+CHXjE9aVI1Jd7l /bjdBuLT3kDXdKlliYNHyAoBNfRR4LveSDDT7DwzAD0yml7wly+558uTBtiQoYS1M9TC S+/w== X-Received: by 10.140.98.238 with SMTP id o101mr2149750qge.70.1442522784000; Thu, 17 Sep 2015 13:46:24 -0700 (PDT) Received: from manet.1015granger.net ([2604:8800:100:81fc:82ee:73ff:fe43:d64f]) by smtp.gmail.com with ESMTPSA id r5sm2052237qkr.26.2015.09.17.13.46.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 Sep 2015 13:46:23 -0700 (PDT) Subject: [PATCH v1 16/18] svcrdma: Add infrastructure to send backwards direction RPC/RDMA calls From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Thu, 17 Sep 2015 16:46:22 -0400 Message-ID: <20150917204622.19671.71471.stgit@manet.1015granger.net> In-Reply-To: <20150917202829.19671.90044.stgit@manet.1015granger.net> References: <20150917202829.19671.90044.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-3-g7d0f MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID,T_RP_MATCHES_RCVD,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To support the NFSv4.1 backchannel on RDMA connections, add a mechanism for sending a backwards-direction RPC/RDMA call on a connection established by a client. Signed-off-by: Chuck Lever --- include/linux/sunrpc/svc_rdma.h | 2 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 63 +++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index 2500dd1..42262dd 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -216,6 +216,8 @@ extern int rdma_read_chunk_frmr(struct svcxprt_rdma *, struct svc_rqst *, extern int svc_rdma_sendto(struct svc_rqst *); extern struct rpcrdma_read_chunk * svc_rdma_get_read_chunk(struct rpcrdma_msg *); +extern int svc_rdma_bc_post_send(struct svcxprt_rdma *, + struct svc_rdma_op_ctxt *, struct xdr_buf *); /* svc_rdma_transport.c */ extern int svc_rdma_send(struct svcxprt_rdma *, struct ib_send_wr *); diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index 1dfae83..0bda3a5 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -641,3 +641,66 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) svc_rdma_put_context(ctxt, 0); return ret; } + +#if defined(CONFIG_SUNRPC_BACKCHANNEL) +/* Send a backwards direction RPC call. + * + * Caller holds the connection's mutex and has already marshaled the + * RPC/RDMA request. Before sending the request, this API also posts + * an extra receive buffer to catch the bc reply for this request. + */ +int svc_rdma_bc_post_send(struct svcxprt_rdma *rdma, + struct svc_rdma_op_ctxt *ctxt, struct xdr_buf *sndbuf) +{ + struct svc_rdma_req_map *vec; + struct ib_send_wr send_wr; + int ret; + + vec = svc_rdma_get_req_map(); + ret = map_xdr(rdma, sndbuf, vec); + if (ret) + goto out; + + /* Post a recv buffer to handle reply for this request */ + ret = svc_rdma_post_recv(rdma); + if (ret) { + pr_err("svcrdma: Failed to post bc receive buffer, err=%d. " + "Closing transport %p.\n", ret, rdma); + set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags); + ret = -ENOTCONN; + goto out; + } + + ctxt->wr_op = IB_WR_SEND; + ctxt->direction = DMA_TO_DEVICE; + ctxt->sge[0].lkey = rdma->sc_dma_lkey; + ctxt->sge[0].length = sndbuf->len; + ctxt->sge[0].addr = + ib_dma_map_page(rdma->sc_cm_id->device, ctxt->pages[0], 0, + sndbuf->len, DMA_TO_DEVICE); + if (ib_dma_mapping_error(rdma->sc_cm_id->device, ctxt->sge[0].addr)) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } + atomic_inc(&rdma->sc_dma_used); + + memset(&send_wr, 0, sizeof send_wr); + send_wr.wr_id = (unsigned long)ctxt; + send_wr.sg_list = ctxt->sge; + send_wr.num_sge = 1; + send_wr.opcode = IB_WR_SEND; + send_wr.send_flags = IB_SEND_SIGNALED; + + ret = svc_rdma_send(rdma, &send_wr); + if (ret) { + svc_rdma_unmap_dma(ctxt); + ret = -EIO; + goto out; + } +out: + svc_rdma_put_req_map(vec); + pr_info("svcrdma: %s returns %d\n", __func__, ret); + return ret; +} +#endif /* CONFIG_SUNRPC_BACKCHANNEL */