From patchwork Thu Jun 20 16:12:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 11007247 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 25B4014DB for ; Thu, 20 Jun 2019 16:13:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A0932873F for ; Thu, 20 Jun 2019 16:13:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E80528793; Thu, 20 Jun 2019 16:13:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8105428742 for ; Thu, 20 Jun 2019 16:13:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732351AbfFTQNp (ORCPT ); Thu, 20 Jun 2019 12:13:45 -0400 Received: from ale.deltatee.com ([207.54.116.67]:59574 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732220AbfFTQNC (ORCPT ); Thu, 20 Jun 2019 12:13:02 -0400 Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hdzgC-00046M-Ja; Thu, 20 Jun 2019 10:13:01 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.89) (envelope-from ) id 1hdzg5-0005wr-OX; Thu, 20 Jun 2019 10:12:45 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Bjorn Helgaas , Dan Williams , Sagi Grimberg , Keith Busch , Jason Gunthorpe , Stephen Bates , Logan Gunthorpe Date: Thu, 20 Jun 2019 10:12:32 -0600 Message-Id: <20190620161240.22738-21-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190620161240.22738-1-logang@deltatee.com> References: <20190620161240.22738-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, axboe@kernel.dk, hch@lst.de, bhelgaas@google.com, dan.j.williams@intel.com, sagi@grimberg.me, kbusch@kernel.org, jgg@ziepe.ca, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH 20/28] IB/core: Introduce API for initializing a RW ctx from a DMA address X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce rdma_rw_ctx_dma_init() and rdma_rw_ctx_dma_destroy() which peform the same operation as rdma_rw_ctx_init() and rdma_rw_ctx_destroy() respectively except they operate on a DMA address and length instead of an SGL. This will be used for struct page-less P2PDMA, but there's also been opinions expressed to migrate away from SGLs and struct pages in the RDMA APIs and this will likely fit with that effort. Signed-off-by: Logan Gunthorpe --- drivers/infiniband/core/rw.c | 74 ++++++++++++++++++++++++++++++------ include/rdma/rw.h | 6 +++ 2 files changed, 69 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c index 32ca8429eaae..cefa6b930bc8 100644 --- a/drivers/infiniband/core/rw.c +++ b/drivers/infiniband/core/rw.c @@ -319,6 +319,39 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, } EXPORT_SYMBOL(rdma_rw_ctx_init); +/** + * rdma_rw_ctx_dma_init - initialize a RDMA READ/WRITE context from a + * DMA address instead of SGL + * @ctx: context to initialize + * @qp: queue pair to operate on + * @port_num: port num to which the connection is bound + * @addr: DMA address to READ/WRITE from/to + * @len: length of memory to operate on + * @remote_addr:remote address to read/write (relative to @rkey) + * @rkey: remote key to operate on + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ + * + * Returns the number of WQEs that will be needed on the workqueue if + * successful, or a negative error code. + */ +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr, + u32 rkey, enum dma_data_direction dir) +{ + struct scatterlist sg; + + sg_dma_address(&sg) = addr; + sg_dma_len(&sg) = len; + + if (rdma_rw_io_needs_mr(qp->device, port_num, dir, 1)) + return rdma_rw_init_mr_wrs(ctx, qp, port_num, &sg, 1, 0, + remote_addr, rkey, dir); + else + return rdma_rw_init_single_wr(ctx, qp, &sg, 0, remote_addr, + rkey, dir); +} +EXPORT_SYMBOL(rdma_rw_ctx_dma_init); + /** * rdma_rw_ctx_signature_init - initialize a RW context with signature offload * @ctx: context to initialize @@ -566,17 +599,7 @@ int rdma_rw_ctx_post(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, } EXPORT_SYMBOL(rdma_rw_ctx_post); -/** - * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init - * @ctx: context to release - * @qp: queue pair to operate on - * @port_num: port num to which the connection is bound - * @sg: scatterlist that was used for the READ/WRITE - * @sg_cnt: number of entries in @sg - * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ - */ -void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, - struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir) +static void __rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp) { int i; @@ -596,6 +619,21 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, BUG(); break; } +} + +/** + * rdma_rw_ctx_destroy - release all resources allocated by rdma_rw_ctx_init + * @ctx: context to release + * @qp: queue pair to operate on + * @port_num: port num to which the connection is bound + * @sg: scatterlist that was used for the READ/WRITE + * @sg_cnt: number of entries in @sg + * @dir: %DMA_TO_DEVICE for RDMA WRITE, %DMA_FROM_DEVICE for RDMA READ + */ +void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, + struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir) +{ + __rdma_rw_ctx_destroy(ctx, qp); /* P2PDMA contexts do not need to be unmapped */ if (!is_pci_p2pdma_page(sg_page(sg))) @@ -603,6 +641,20 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, } EXPORT_SYMBOL(rdma_rw_ctx_destroy); +/** + * rdma_rw_ctx_dma_destroy - release all resources allocated by + * rdma_rw_ctx_dma_init + * @ctx: context to release + * @qp: queue pair to operate on + * @port_num: port num to which the connection is bound + */ +void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, + u8 port_num) +{ + __rdma_rw_ctx_destroy(ctx, qp); +} +EXPORT_SYMBOL(rdma_rw_ctx_dma_destroy); + /** * rdma_rw_ctx_destroy_signature - release all resources allocated by * rdma_rw_ctx_init_signature diff --git a/include/rdma/rw.h b/include/rdma/rw.h index 494f79ca3e62..e47f8053af6e 100644 --- a/include/rdma/rw.h +++ b/include/rdma/rw.h @@ -58,6 +58,12 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, struct scatterlist *sg, u32 sg_cnt, enum dma_data_direction dir); +int rdma_rw_ctx_dma_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, + u8 port_num, dma_addr_t addr, u32 len, u64 remote_addr, + u32 rkey, enum dma_data_direction dir); +void rdma_rw_ctx_dma_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, + u8 port_num); + int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num, struct scatterlist *sg, u32 sg_cnt, struct scatterlist *prot_sg, u32 prot_sg_cnt,