From patchwork Tue Aug 23 17:53:17 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 9296149 X-Patchwork-Delegate: Trond.Myklebust@netapp.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B7E9C607F0 for ; Tue, 23 Aug 2016 18:10:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC01028CD3 for ; Tue, 23 Aug 2016 18:10:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A072828CCB; Tue, 23 Aug 2016 18:10:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D687B28CCB for ; Tue, 23 Aug 2016 18:10:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754783AbcHWSKK (ORCPT ); Tue, 23 Aug 2016 14:10:10 -0400 Received: from mail-it0-f68.google.com ([209.85.214.68]:34253 "EHLO mail-it0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753617AbcHWSKI (ORCPT ); Tue, 23 Aug 2016 14:10:08 -0400 Received: by mail-it0-f68.google.com with SMTP id e63so8869453ith.1; Tue, 23 Aug 2016 11:10:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=VuiMvUsUfJWTMh/FLoLw+R0JrUvaZBvSCa9Hn8th0+E=; b=IV/fpnzuDeLNrZ7xZlmAIA61vnXdpxBmtL/MiwaRbr+q3jUaZG0b82N2lMYmyIe1E8 OnJNDnROL8BTmpGF3AESBfKbeL+nCBoRKhJP2nXxo7aHM6+n1xSlSt98hHMmbGtxYmq+ e4KLtmZSRAf416c9uIKlusGNyB+vQPYzNLg8h5PTn0i2VOkbbYyx4eO0tao7Tc7Hty7W s/ZC/Qqduq6O70fUy4YdEmm40mRcy1AjYWfMumoPHrNbc1FfKePmkhjLb1nwKLk7Qo+5 HjUbuSsg1YFWueFKSdphRrTYp6FlBxAUqPCGYNqUvs+SXN1jgCNx2OKexA8Gv9kMFbvv Qikw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=VuiMvUsUfJWTMh/FLoLw+R0JrUvaZBvSCa9Hn8th0+E=; b=bxYWBw1fT7Aum+56E7EnyL4qNYvnTANg6MwdyTiX3oTveD7OU2lLvhnk1F/KymBENk 6C9oTPEPwDMSnFEp9cqjGzhdr75f33B0/shAnHHiPE+0pqYkiMMFHdP8sof9nLt6TGRj Oh+iRiISeZK+k4iUvwYHCfKrw97M4KIo9BEUC9OFHxmsZwVfiPRzB4B9a/uzpDyZNal2 WU2i7H5NWtBYYK/foQOhorRVKSuA5Oq6tPcYpyBADeVz9hP19zNk+Ds5m5zZI2AMdEyi ZiE6Ihr1tkFVOcFGLNDSKolTUeluu4eNB5dXdKthygeGU8DojyMG23ElvFq32enJc4Ip NdVg== X-Gm-Message-State: AEkooutB+QH+6ftXKcWjdldwWX6nys+9tw/7/nz1zy7vSm3m6jg2pWdH0lDpMAsntd/Pmg== X-Received: by 10.107.131.11 with SMTP id f11mr31852035iod.136.1471974798855; Tue, 23 Aug 2016 10:53:18 -0700 (PDT) Received: from manet.1015granger.net ([2604:8800:100:81fc:ec4:7aff:fe6c:1dce]) by smtp.gmail.com with ESMTPSA id w12sm1764386iow.29.2016.08.23.10.53.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Aug 2016 10:53:18 -0700 (PDT) Subject: [PATCH v2 09/22] xprtrdma: Replace DMA_BIDIRECTIONAL From: Chuck Lever To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Date: Tue, 23 Aug 2016 13:53:17 -0400 Message-ID: <20160823175317.13038.42072.stgit@manet.1015granger.net> In-Reply-To: <20160823174402.13038.84561.stgit@manet.1015granger.net> References: <20160823174402.13038.84561.stgit@manet.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The use of DMA_BIDIRECTIONAL is discouraged by DMA-API.txt. Fortunately, xprtrdma now knows which direction I/O is going as soon as it allocates each regbuf. The RPC Call and Reply buffers are no longer the same regbuf. They can each be labeled correctly now. The RPC Reply buffer is never part of either a Send or Receive WR, but it can be part of Reply chunk, which is mapped and registered via ->ro_map . So it is not DMA mapped when it is allocated (DMA_NONE), to avoid a double- mapping. Since Receive buffers are no longer DMA_BIDIRECTIONAL and their contents are never modified by the host CPU, DMA-API-HOWTO.txt suggests that a DMA sync before posting each buffer should be unnecessary. (See my_card_interrupt_handler). Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/backchannel.c | 5 ++- net/sunrpc/xprtrdma/transport.c | 7 +++-- net/sunrpc/xprtrdma/verbs.c | 55 +++++++++++++++++-------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 4 ++- 4 files changed, 36 insertions(+), 35 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/sunrpc/xprtrdma/backchannel.c b/net/sunrpc/xprtrdma/backchannel.c index 60fc991..ceae872 100644 --- a/net/sunrpc/xprtrdma/backchannel.c +++ b/net/sunrpc/xprtrdma/backchannel.c @@ -45,13 +45,14 @@ static int rpcrdma_bc_setup_rqst(struct rpcrdma_xprt *r_xprt, return PTR_ERR(req); req->rl_backchannel = true; - rb = rpcrdma_alloc_regbuf(ia, RPCRDMA_HDRBUF_SIZE, GFP_KERNEL); + rb = rpcrdma_alloc_regbuf(ia, RPCRDMA_HDRBUF_SIZE, + DMA_TO_DEVICE, GFP_KERNEL); if (IS_ERR(rb)) goto out_fail; req->rl_rdmabuf = rb; size = r_xprt->rx_data.inline_rsize; - rb = rpcrdma_alloc_regbuf(ia, size, GFP_KERNEL); + rb = rpcrdma_alloc_regbuf(ia, size, DMA_TO_DEVICE, GFP_KERNEL); if (IS_ERR(rb)) goto out_fail; req->rl_sendbuf = rb; diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 94dbfd3..3424691 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -490,7 +490,7 @@ rpcrdma_get_rdmabuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, if (req->rl_rdmabuf) return true; - rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, size, flags); + rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, size, DMA_TO_DEVICE, flags); if (IS_ERR(rb)) return false; @@ -517,7 +517,8 @@ rpcrdma_get_sendbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, return true; min_size = max_t(size_t, size, r_xprt->rx_data.inline_wsize); - rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, min_size, flags); + rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, min_size, + DMA_TO_DEVICE, flags); if (IS_ERR(rb)) return false; @@ -547,7 +548,7 @@ rpcrdma_get_recvbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req, if (req->rl_recvbuf && rdmab_length(req->rl_recvbuf) >= size) return true; - rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, size, flags); + rb = rpcrdma_alloc_regbuf(&r_xprt->rx_ia, size, DMA_NONE, flags); if (IS_ERR(rb)) return false; diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 9c3e58e..0bd2ace 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -865,7 +865,7 @@ rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt) goto out; rep->rr_rdmabuf = rpcrdma_alloc_regbuf(ia, cdata->inline_rsize, - GFP_KERNEL); + DMA_FROM_DEVICE, GFP_KERNEL); if (IS_ERR(rep->rr_rdmabuf)) { rc = PTR_ERR(rep->rr_rdmabuf); goto out_free; @@ -1153,27 +1153,24 @@ rpcrdma_recv_buffer_put(struct rpcrdma_rep *rep) spin_unlock(&buffers->rb_lock); } -/* - * Wrappers for internal-use kmalloc memory registration, used by buffer code. - */ - /** - * rpcrdma_alloc_regbuf - kmalloc and register memory for SEND/RECV buffers + * rpcrdma_alloc_regbuf - allocate and DMA-map memory for SEND/RECV buffers * @ia: controlling rpcrdma_ia * @size: size of buffer to be allocated, in bytes + * @direction: direction of data movement * @flags: GFP flags * - * Returns pointer to private header of an area of internally - * registered memory, or an ERR_PTR. The registered buffer follows - * the end of the private header. + * Returns an ERR_PTR, or a pointer to a regbuf, which is a + * contiguous memory region that is DMA mapped persistently, and + * is registered for local I/O. * * xprtrdma uses a regbuf for posting an outgoing RDMA SEND, or for - * receiving the payload of RDMA RECV operations. regbufs are not - * used for RDMA READ/WRITE operations, thus are registered only for - * LOCAL access. + * receiving the payload of RDMA RECV operations. During Long Calls + * or Replies they may be registered externally via ro_map. */ struct rpcrdma_regbuf * -rpcrdma_alloc_regbuf(struct rpcrdma_ia *ia, size_t size, gfp_t flags) +rpcrdma_alloc_regbuf(struct rpcrdma_ia *ia, size_t size, + enum dma_data_direction direction, gfp_t flags) { struct rpcrdma_regbuf *rb; struct ib_sge *iov; @@ -1182,15 +1179,20 @@ rpcrdma_alloc_regbuf(struct rpcrdma_ia *ia, size_t size, gfp_t flags) if (rb == NULL) goto out; + rb->rg_direction = direction; iov = &rb->rg_iov; - iov->addr = ib_dma_map_single(ia->ri_device, - (void *)rb->rg_base, size, - DMA_BIDIRECTIONAL); - if (ib_dma_mapping_error(ia->ri_device, iov->addr)) - goto out_free; - iov->length = size; iov->lkey = ia->ri_pd->local_dma_lkey; + + if (direction != DMA_NONE) { + iov->addr = ib_dma_map_single(ia->ri_device, + (void *)rb->rg_base, + rdmab_length(rb), + rb->rg_direction); + if (ib_dma_mapping_error(ia->ri_device, iov->addr)) + goto out_free; + } + return rb; out_free: @@ -1207,14 +1209,14 @@ out: void rpcrdma_free_regbuf(struct rpcrdma_ia *ia, struct rpcrdma_regbuf *rb) { - struct ib_sge *iov; - if (!rb) return; - iov = &rb->rg_iov; - ib_dma_unmap_single(ia->ri_device, - iov->addr, iov->length, DMA_BIDIRECTIONAL); + if (rb->rg_direction != DMA_NONE) { + ib_dma_unmap_single(ia->ri_device, rdmab_addr(rb), + rdmab_length(rb), rb->rg_direction); + } + kfree(rb); } @@ -1286,11 +1288,6 @@ rpcrdma_ep_post_recv(struct rpcrdma_ia *ia, recv_wr.sg_list = &rep->rr_rdmabuf->rg_iov; recv_wr.num_sge = 1; - ib_dma_sync_single_for_cpu(ia->ri_device, - rdmab_addr(rep->rr_rdmabuf), - rdmab_length(rep->rr_rdmabuf), - DMA_BIDIRECTIONAL); - rc = ib_post_recv(ia->ri_id->qp, &recv_wr, &recv_wr_fail); if (rc) goto out_postrecv; diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index c5137f0..eea7d9d 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -113,6 +113,7 @@ struct rpcrdma_ep { struct rpcrdma_regbuf { struct ib_sge rg_iov; + enum dma_data_direction rg_direction; __be32 rg_base[0] __attribute__ ((aligned(256))); }; @@ -476,7 +477,8 @@ void rpcrdma_recv_buffer_put(struct rpcrdma_rep *); void rpcrdma_defer_mr_recovery(struct rpcrdma_mw *); struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(struct rpcrdma_ia *, - size_t, gfp_t); + size_t, enum dma_data_direction, + gfp_t); void rpcrdma_free_regbuf(struct rpcrdma_ia *, struct rpcrdma_regbuf *);