From patchwork Mon Sep 19 15:56:29 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 9339823 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 012A56077A for ; Mon, 19 Sep 2016 15:57:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E56AF294D5 for ; Mon, 19 Sep 2016 15:57:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA46529501; Mon, 19 Sep 2016 15:57:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38698294D5 for ; Mon, 19 Sep 2016 15:57:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752263AbcISP5M (ORCPT ); Mon, 19 Sep 2016 11:57:12 -0400 Received: from bombadil.infradead.org ([198.137.202.9]:53982 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752247AbcISP5K (ORCPT ); Mon, 19 Sep 2016 11:57:10 -0400 Received: from [4.28.11.153] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.85_2 #1 (Red Hat Linux)) id 1bm0wP-0005GK-HS; Mon, 19 Sep 2016 15:57:10 +0000 From: Christoph Hellwig To: jgunthorpe@obsidianresearch.com Cc: linux-rdma@vger.kernel.org Subject: [PATCH 5/9] move libibverbs documentation to Documentation/ Date: Mon, 19 Sep 2016 08:56:29 -0700 Message-Id: <1474300593-31922-6-git-send-email-hch@lst.de> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1474300593-31922-1-git-send-email-hch@lst.de> References: <1474300593-31922-1-git-send-email-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP And drop bits that are outdated or replaced by top-level Documentation. Signed-off-by: Christoph Hellwig --- Documentation/libibverbs.md | 58 ++++++++++++++++ libibverbs/README | 164 -------------------------------------------- 2 files changed, 58 insertions(+), 164 deletions(-) create mode 100644 Documentation/libibverbs.md delete mode 100644 libibverbs/README diff --git a/Documentation/libibverbs.md b/Documentation/libibverbs.md new file mode 100644 index 0000000..cbe076e --- /dev/null +++ b/Documentation/libibverbs.md @@ -0,0 +1,58 @@ +# Introduction + +libibverbs is a library that allows programs to use RDMA "verbs" for +direct access to RDMA (currently InfiniBand and iWARP) hardware from +userspace. For more information on RDMA verbs, see the InfiniBand +Architecture Specification vol. 1, especially chapter 11, and the RDMA +Consortium's RDMA Protocol Verbs Specification. + +# Using libibverbs + +### Device nodes + +The verbs library expects special character device files named +/dev/infiniband/uverbsN to be created. When you load the kernel +modules, including both the low-level driver for your IB hardware as +well as the ib_uverbs module, you should see one or more uverbsN +entries in /sys/class/infiniband_verbs in addition to the +/dev/infiniband/uverbsN character device files. + +To create the appropriate character device files automatically with +udev, a rule like + + KERNEL="uverbs*", NAME="infiniband/%k" + +can be used. This will create device nodes named + + /dev/infiniband/uverbs0 + +and so on. Since the RDMA userspace verbs should be safe for use by +non-privileged users, you may want to add an appropriate MODE or GROUP +to your udev rule. + +### Permissions + +To use IB verbs from userspace, a process must be able to access the +appropriate /dev/infiniband/uverbsN special device file. You can +check the permissions on this file with the command + + ls -l /dev/infiniband/uverbs* + +Make sure that the permissions on these files are such that the +user/group that your verbs program runs as can access the device file. + +To use IB verbs from userspace, a process must also have permission to +tell the kernel to lock sufficient memory for all of your registered +memory regions as well as the memory used internally by IB resources +such as queue pairs (QPs) and completion queues (CQs). To check your +resource limits, use the command + + ulimit -l + +(or "limit memorylocked" for csh-like shells). + +If you see a small number such as 32 (the units are KB) then you will +need to increase this limit. This is usually done for ordinary users +via the file /etc/security/limits.conf. More configuration may be +necessary if you are logging in via OpenSSH and your sshd is +configured to use privilege separation. diff --git a/libibverbs/README b/libibverbs/README deleted file mode 100644 index 848eb05..0000000 --- a/libibverbs/README +++ /dev/null @@ -1,164 +0,0 @@ -Introduction -============ - -libibverbs is a library that allows programs to use RDMA "verbs" for -direct access to RDMA (currently InfiniBand and iWARP) hardware from -userspace. For more information on RDMA verbs, see the InfiniBand -Architecture Specification vol. 1, especially chapter 11, and the RDMA -Consortium's RDMA Protocol Verbs Specification. - -Using libibverbs -================ - -Device nodes ------------- - -The verbs library expects special character device files named -/dev/infiniband/uverbsN to be created. When you load the kernel -modules, including both the low-level driver for your IB hardware as -well as the ib_uverbs module, you should see one or more uverbsN -entries in /sys/class/infiniband_verbs in addition to the -/dev/infiniband/uverbsN character device files. - -To create the appropriate character device files automatically with -udev, a rule like - - KERNEL="uverbs*", NAME="infiniband/%k" - -can be used. This will create device nodes named - - /dev/infiniband/uverbs0 - -and so on. Since the RDMA userspace verbs should be safe for use by -non-privileged users, you may want to add an appropriate MODE or GROUP -to your udev rule. - -Permissions ------------ - -To use IB verbs from userspace, a process must be able to access the -appropriate /dev/infiniband/uverbsN special device file. You can -check the permissions on this file with the command - - ls -l /dev/infiniband/uverbs* - -Make sure that the permissions on these files are such that the -user/group that your verbs program runs as can access the device file. - -To use IB verbs from userspace, a process must also have permission to -tell the kernel to lock sufficient memory for all of your registered -memory regions as well as the memory used internally by IB resources -such as queue pairs (QPs) and completion queues (CQs). To check your -resource limits, use the command - - ulimit -l - -(or "limit memorylocked" for csh-like shells). - -If you see a small number such as 32 (the units are KB) then you will -need to increase this limit. This is usually done for ordinary users -via the file /etc/security/limits.conf. More configuration may be -necessary if you are logging in via OpenSSH and your sshd is -configured to use privilege separation. - -Valgrind support ----------------- - -When running applications that use libibverbs under the Valgrind -memory-checking debugger, Valgrind will falsely report "read from -uninitialized" for memory that was initialized by the kernel drivers. -Specifically, Valgrind cannot see when kernel drivers write to -userspace memory, so when the process reads from that memory, Valgrind -incorrectly assumes that the memory contents are uninitialized, and -therefore raises a warning. - -libibverbs can be built with specific support for the Valgrind -memory-checking debugger by specifying the --with-valgrind command -line argument to configure. This flag enables code in libibverbs to -tell Valgrind "this memory may look uninitialized, but it's really -OK," which therefore suppresses the incorrect "read from -uninitialized" warnings. This code adds trivial overhead to the -critical performance path, so it is disabled by default. The intent -is that production users can use a "normal" build of libibverbs and -developers can use the "valgrind debug" build by simply switching -their LD_LIBRARY_PATH environment variables. - -Libibverbs needs some header files from Valgrind in order to compile -this support; it is important to use the header files from the same -version of Valgrind that will be used at run time. You may need to -specify the directory where Valgrind's header files are installed as -an argument to --with-valgrind. For example - - ./configure --with-valgrind=/opt/valgrind - -will make the libibverbs build look for valgrind headers in -/opt/valgrind/include - -Reporting bugs -============== - -Bugs should be reported to the OpenFabrics mailing list -. In your bug report, please include: - - * Information about your system: - - Linux distribution and version - - Linux kernel and version - - InfiniBand/iWARP hardware and firmware version - - ... any other relevant information - - * How to reproduce the bug. Command line arguments for a libibverbs - example program or source code that other developers can - compile and run is most convenient. - - * If the bug is a crash, the exact output printed out when the crash - occurred, including any kernel messages produced. - - * If a verbs call is mysteriously returning an error or failing, the - output of "strace -ewrite -ewrite=all ". - -Submitting patches -================== - -Patches should also be submitted to the OpenFabrics mailing list -. Please use unified diff form (the -u -option to GNU diff), and include a good description of what your patch -does and why it should be applied. If your patch fixes a bug, please -make sure to describe the bug and how your fix works. - -Please include a change to the ChangeLog file (in standard GNU -changelog format) as part of your patch. - -Make sure that your contribution can be licensed under the same -license as the original code you are patching, and that you have all -necessary permissions to release your work. - -TODO -==== - -1.1 series ----------- - -The libibverbs API and ABI are frozen for all releases in the 1.1 -series. Methods were added to struct ibv_context to implement the -following features, so it should be possible to add them in a future -release in the 1.1 series: - - * Memory window (MW) support. - - * Implement the reregister memory region (MR) verb. We will add an - extension to the IB spec to allow the application to indicate that - the region is only being extended, and that operations in progress - should _not_ fail (contrary to the IB spec, which states that - reregister must be implemented so that it behaves equivalently to a - deregister followed by a register). - -Other possibilities -------------------- - -There are no plans to implement the following features, which would be -needed for completeness but don't seem particularly useful. However, -if there is demand from application developers or an implementation is -contributed, then the feature may be added. - - * Implement the query address handle (AH) verb. - * Implement the query memory region (MR) verb.