From patchwork Mon Aug 1 06:57:29 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haggai Eran X-Patchwork-Id: 9253805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 180756089F for ; Mon, 1 Aug 2016 06:58:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0AE302848F for ; Mon, 1 Aug 2016 06:58:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0C7C28492; Mon, 1 Aug 2016 06:58:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E96128492 for ; Mon, 1 Aug 2016 06:58:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751472AbcHAG6n (ORCPT ); Mon, 1 Aug 2016 02:58:43 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:36177 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751197AbcHAG6m (ORCPT ); Mon, 1 Aug 2016 02:58:42 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from haggaie@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 Aug 2016 09:58:06 +0300 Received: from arch003.mtl.labs.mlnx (arch003.mtl.labs.mlnx [10.137.35.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id u716w5B1000707; Mon, 1 Aug 2016 09:58:06 +0300 From: Haggai Eran To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org Cc: linux-pci@vger.kernel.org, Stephen Bates , Liran Liss , Leon Romanovsky , Artemy Kovalyov , Jerome Glisse , Yishai Hadas , Haggai Eran Subject: [RFC 3/7] IB/core: Helpers for mapping DMA-BUF in MRs Date: Mon, 1 Aug 2016 09:57:29 +0300 Message-Id: <1470034653-9097-4-git-send-email-haggaie@mellanox.com> X-Mailer: git-send-email 1.7.11.2 In-Reply-To: <1470034653-9097-1-git-send-email-haggaie@mellanox.com> References: <1470034653-9097-1-git-send-email-haggaie@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP DMA-BUF API allows devices to export buffers for other devices to access. This can provide a means to implement peer to peer communication for RDMA. A device with internal memory would expose its memory to the PCIe BAR, and export that region as a DMA-BUF object. Then, the user or ULP would register a memory region that is backed by the DMA-BUF object. Signed-off-by: Haggai Eran --- drivers/infiniband/core/verbs.c | 60 +++++++++++++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 9 +++++++ 2 files changed, 69 insertions(+) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 6298f54b4137..633831e77a62 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include @@ -1922,3 +1923,62 @@ void ib_drain_qp(struct ib_qp *qp) ib_drain_rq(qp); } EXPORT_SYMBOL(ib_drain_qp); + +int ib_mr_attach_dmabuf(struct ib_mr *mr, struct dma_buf *dmabuf, int access) +{ + struct device *dev = mr->device->dma_device; + struct dma_buf_attachment *attach; + struct sg_table *sg; + int err; + int n; + + get_dma_buf(dmabuf); + + attach = dma_buf_attach(dmabuf, dev); + if (IS_ERR(attach)) { + dev_err(dev, "unable to attach to DMA-BUF object (from %s): %ld", + dmabuf->exp_name, PTR_ERR(attach)); + err = PTR_ERR(attach); + goto put_dma_buf; + } + + sg = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL); + if (IS_ERR(sg)) { + dev_err(dev, "unable to map a DMA-BUF object (from %s): %ld", + dmabuf->exp_name, PTR_ERR(sg)); + err = PTR_ERR(sg); + goto detach; + } + + n = ib_map_mr_sg_zbva(mr, sg->sgl, sg->nents, NULL, PAGE_SIZE); + if (unlikely(n != sg->nents)) { + dev_err(dev, "failed to map sg (%d/%d) of DMA-BUF object (from %s)\n", + n, sg->nents, dmabuf->exp_name); + err = n < 0 ? n : -EINVAL; + goto unmap; + } + + mr->attach = attach; + mr->sg = sg; + return 0; + +unmap: + dma_buf_unmap_attachment(attach, sg, DMA_BIDIRECTIONAL); +detach: + dma_buf_detach(dmabuf, attach); +put_dma_buf: + dma_buf_put(dmabuf); + return err; +} +EXPORT_SYMBOL(ib_mr_attach_dmabuf); + +void ib_mr_detach_dmabuf(struct dma_buf_attachment *attach, + struct sg_table *sg) +{ + struct dma_buf *dmabuf = attach->dmabuf; + + dma_buf_unmap_attachment(attach, sg, DMA_BIDIRECTIONAL); + dma_buf_detach(dmabuf, attach); + dma_buf_put(dmabuf); +} +EXPORT_SYMBOL(ib_mr_detach_dmabuf); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 7e440d41487a..da23ec0a06d7 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1465,6 +1465,9 @@ struct ib_mr { struct ib_uobject *uobject; /* user */ struct list_head qp_entry; /* FR */ }; + /* For MRs that expose a DMA-BUF */ + struct dma_buf_attachment *attach; + struct sg_table *sg; }; struct ib_mw { @@ -3189,4 +3192,10 @@ int ib_sg_to_pages(struct ib_mr *mr, struct scatterlist *sgl, int sg_nents, void ib_drain_rq(struct ib_qp *qp); void ib_drain_sq(struct ib_qp *qp); void ib_drain_qp(struct ib_qp *qp); + +struct dma_buf; +int ib_mr_attach_dmabuf(struct ib_mr *mr, struct dma_buf *dmabuf, int access); +void ib_mr_detach_dmabuf(struct dma_buf_attachment *attach, + struct sg_table *sg); + #endif /* IB_VERBS_H */