From patchwork Thu Jun 29 11:47:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 9816499 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 64C226020A for ; Thu, 29 Jun 2017 11:47:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55AE627F54 for ; Thu, 29 Jun 2017 11:47:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 489F527FB3; Thu, 29 Jun 2017 11:47:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE78027D0E for ; Thu, 29 Jun 2017 11:47:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752545AbdF2Lrw (ORCPT ); Thu, 29 Jun 2017 07:47:52 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:60606 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752675AbdF2Lrt (ORCPT ); Thu, 29 Jun 2017 07:47:49 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 29 Jun 2017 14:47:19 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id v5TBlJD5017956; Thu, 29 Jun 2017 14:47:19 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id v5TBlJKS017957; Thu, 29 Jun 2017 14:47:19 +0300 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id v5TBlJAs017956; Thu, 29 Jun 2017 14:47:19 +0300 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: dledford@redhat.com, yishaih@mellanox.com, maorg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 6/7] mlx4: Add direct verbs man pages Date: Thu, 29 Jun 2017 14:47:07 +0300 Message-Id: <1498736828-17875-7-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1498736828-17875-1-git-send-email-yishaih@mellanox.com> References: <1498736828-17875-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Maor Gottlieb Document mlx4 direct verbs feature and exposed related functions. Signed-off-by: Maor Gottlieb Reviewed-by: Yishai Hadas --- CMakeLists.txt | 1 + providers/mlx4/man/CMakeLists.txt | 4 ++ providers/mlx4/man/mlx4dv.7 | 39 ++++++++++++ providers/mlx4/man/mlx4dv_init_obj.3 | 117 +++++++++++++++++++++++++++++++++++ 4 files changed, 161 insertions(+) create mode 100644 providers/mlx4/man/CMakeLists.txt create mode 100644 providers/mlx4/man/mlx4dv.7 create mode 100644 providers/mlx4/man/mlx4dv_init_obj.3 diff --git a/CMakeLists.txt b/CMakeLists.txt index 634f327..5dcdb90 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -415,6 +415,7 @@ add_subdirectory(providers/cxgb4) # NO SPARSE add_subdirectory(providers/hns) # NO SPARSE add_subdirectory(providers/i40iw) # NO SPARSE add_subdirectory(providers/mlx4) # NO SPARSE +add_subdirectory(providers/mlx4/man) # NO SPARSE add_subdirectory(providers/mlx5) # NO SPARSE add_subdirectory(providers/mlx5/man) # NO SPARSE add_subdirectory(providers/mthca) # NO SPARSE diff --git a/providers/mlx4/man/CMakeLists.txt b/providers/mlx4/man/CMakeLists.txt new file mode 100644 index 0000000..c44c7c3 --- /dev/null +++ b/providers/mlx4/man/CMakeLists.txt @@ -0,0 +1,4 @@ +rdma_man_pages( + mlx4dv_init_obj.3 + mlx4dv.7 +) diff --git a/providers/mlx4/man/mlx4dv.7 b/providers/mlx4/man/mlx4dv.7 new file mode 100644 index 0000000..92ced99 --- /dev/null +++ b/providers/mlx4/man/mlx4dv.7 @@ -0,0 +1,39 @@ +.\" -*- nroff -*- +.\" Licensed under the OpenIB.org (MIT) - See COPYING.md +.\" +.TH MLX4DV 7 2017-04-19 1.0.0 +.SH "NAME" +mlx4dv \- Direct verbs for mlx4 devices +.br +This is low level access to mlx4 devices to perform data path operations, +without general branching performed by \fBibv_post_send\fR(3). + +.SH "DESCRIPTION" +The libibverbs API is an abstract one. It is agnostic to any underlying +provider specific implementation. While this abstraction has the advantage +of user applications portability it has a performance penalty. For some +applications optimizing performance is more important than portability. + +The mlx4 direct verbs API is intended for such applications. +It exposes mlx4 specific low level data path (send/receive/completion) +operations, allowing the application to bypass the libibverbs data path API. + +This interface consists from one hardware specific header file +with relevant inline functions and conversion logic from ibverbs structures +to mlx4 specific structures. + +The direct include of mlx4dv.h together with linkage to mlx4 library will +allow usage of this new interface. + +Once an application uses the direct flow the locking scheme is fully managed +by itself. There is an expectation that no mixed flows in the data path for both +direct/non-direct access will be by same application. + +.SH "NOTES" +.SH "SEE ALSO" +.BR ibv_post_send (3), +.BR verbs (7) + +.SH "AUTHORS" +.TP +Maor Gottlieb diff --git a/providers/mlx4/man/mlx4dv_init_obj.3 b/providers/mlx4/man/mlx4dv_init_obj.3 new file mode 100644 index 0000000..0d0a4a5 --- /dev/null +++ b/providers/mlx4/man/mlx4dv_init_obj.3 @@ -0,0 +1,117 @@ +.\" -*- nroff -*- +.\" Licensed under the OpenIB.org (MIT) - See COPYING.md +.\" +.TH MLX4DV_INIT_OBJ 3 2017-02-02 1.0.0 +.SH "NAME" +mlx4dv_init_obj \- Initialize mlx4 direct verbs object from ibv_xxx structures +.SH "SYNOPSIS" +.nf +.B #include +.sp +.BI "int mlx4dv_init_obj(struct mlx4dv_obj *obj, uint64_t obj_type); +.fi +.SH "DESCRIPTION" +.B mlx4dv_init_obj() +This function will initialize mlx4dv_xxx structs based on supplied type. The information +for initialization is taken from ibv_xx structs supplied as part of input. + +Request information of CQ marks its owned by direct verbs for all consumer index +related actions. The initialization type can be combination of several types together. +.PP +.nf +struct mlx4dv_qp { +.in +8 +uint32_t *rdb; +uint32_t *sdb; +struct { +.in +8 +uint32_t wqe_cnt; +int wqe_shift; +int offset; +.in -8 +} sq; +struct { +.in +8 +uint32_t wqe_cnt; +int wqe_shift; +int offset; +.in -8 +} rq; +struct { +.in +8 +void *buf; +size_t length; +.in -8 +} buf; +uint64_t comp_mask; +.in -8 +}; + +struct mlx4dv_cq { +.in +8 +struct { +.in +8 +void *buf; +size_t length; +.in -8 +} buf; +uint32_t cqe_cnt; +uint32_t cqn; +uint32_t *set_ci_db; +uint32_t *arm_db; +int arm_sn; +int cqe_size; +uint64_t comp_mask; +.in -8 +}; + +struct mlx4dv_srq { +.in +8 +struct { +.in +8 +void *buf; +size_t length; +.in -8 +} buf; +int wqe_shift; +int head; +int tail; +uint32_t *db; +uint64_t comp_mask; +.in -8 +}; + +struct mlx4dv_obj { +.in +8 +struct { +.in +8 +struct ibv_qp *in; +struct mlx4dv_qp *out; +.in -8 +} qp; +struct { +.in +8 +struct ibv_cq *in; +struct mlx4dv_cq *out; +.in -8 +} cq; +.in -8 +}; + +enum mlx4dv_obj_type { +.in +8 +MLX4DV_OBJ_QP = 1 << 0, +MLX4DV_OBJ_CQ = 1 << 1, +MLX4DV_OBJ_SRQ = 1 << 2, +.in -8 +}; +.fi +.SH "RETURN VALUE" +0 on success or the value of errno on failure (which indicates the failure reason). +.SH "NOTES" + * Compatibility masks (comp_mask) are in/out fields. +.SH "SEE ALSO" +.BR mlx4dv (7) +.SH "AUTHORS" +.TP +Maor Gottlieb