From patchwork Fri May 18 13:04:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Pen X-Patchwork-Id: 10410641 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 61CDF6037D for ; Fri, 18 May 2018 13:06:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E21D28980 for ; Fri, 18 May 2018 13:06:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 42B6728986; Fri, 18 May 2018 13:06:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14C9428980 for ; Fri, 18 May 2018 13:06:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752236AbeERNGs (ORCPT ); Fri, 18 May 2018 09:06:48 -0400 Received: from mail-wm0-f65.google.com ([74.125.82.65]:54554 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752208AbeERNF6 (ORCPT ); Fri, 18 May 2018 09:05:58 -0400 Received: by mail-wm0-f65.google.com with SMTP id f6-v6so14011631wmc.4 for ; Fri, 18 May 2018 06:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=G4eky4YKG2wr227M+ng3HDq+YgxKq3qUBOXqjrvaFQQ=; b=uAQNwlXPZAI9k9fVNNuSSDkzhkHTufxoj8CMjKJ9OakqAmMLPgPHytgOLgwWZsCuKY h24fhFe/gtMwq97zPsJtLa95baVYXL1XefVI5WYHcnjRPR4sPe32G9CB3UA00PYHcx3L LEueRLPS8tTO2IUPI+vvkVmiI3SA4hISanHNNmiQzxCrVJMe/MphAjyMchlj4ODOeTXY UWCvwNZzGAqgSu61bcMmt+XTyuNMjOsPPjSGgAY5mIfi77c9Y65xTAO+LHlMECM5oGVz mXMSGThguWLOh+TH5kwwIwt/EBIgJ+8AZmrJEHBHCNuzhT35hr9EYOwXz3DOFl8CFSqb HwbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=G4eky4YKG2wr227M+ng3HDq+YgxKq3qUBOXqjrvaFQQ=; b=qGI3C5RqGSeGZ0EKf2zdxX506nQAR8DlLiVgT9hj+nFWvAo2N5uppPE6GSEAdAgwT9 ORlmSYA/O1NA8ZX4TMQOSC5OywjjARmRXVXaSIFgCFxXEFt7ZaDhTbFFq2x+z8umjweI WastTn6wujvFZdS0RK38AOE3p27Y1k7np6gxZW/mDk5SRQI38aGGhAah35OLPT1yNJM/ x1Dy7hn2ZYglXxIEE1/RPKH3ktx+hSUMMpw3Mc6CrMCDVDPEEWv2t1RlBWU9LA/BAcPR mOg/WVgHpJl5uqyTRac7u4ZGvrtVIc6KreRIkWdDUNLmRUdyEQUraclm+kVwyvEKGij6 atCw== X-Gm-Message-State: ALKqPwc1zOIjbgU1UIt1Ey+nbMlNLApwCxUjGChOI9XysLYQwpxwMlBG Sp9njKyUzziKH07cXkrj/kxdr3oG X-Google-Smtp-Source: AB8JxZqjjB+yUgtjtdMGidUmGlmwivbpiN40CAVhxf8W57yWeSwMHFjyx+n4Y8raNzg+9qWyPiLFyQ== X-Received: by 2002:a1c:e08:: with SMTP id 8-v6mr4506561wmo.67.1526648756375; Fri, 18 May 2018 06:05:56 -0700 (PDT) Received: from pb.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id m35-v6sm7639980wrm.51.2018.05.18.06.05.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 May 2018 06:05:55 -0700 (PDT) From: Roman Pen To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Sagi Grimberg , Bart Van Assche , Or Gerlitz , Doug Ledford , Swapnil Ingle , Danil Kipnis , Jack Wang , Roman Pen Subject: [PATCH v2 25/26] ibnbd: a bit of documentation Date: Fri, 18 May 2018 15:04:12 +0200 Message-Id: <20180518130413.16997-26-roman.penyaev@profitbricks.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20180518130413.16997-1-roman.penyaev@profitbricks.com> References: <20180518130413.16997-1-roman.penyaev@profitbricks.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP README with description of major sysfs entries. Signed-off-by: Roman Pen Signed-off-by: Danil Kipnis Cc: Jack Wang --- drivers/block/ibnbd/README | 299 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 drivers/block/ibnbd/README diff --git a/drivers/block/ibnbd/README b/drivers/block/ibnbd/README new file mode 100644 index 000000000000..bbaddd02c1c5 --- /dev/null +++ b/drivers/block/ibnbd/README @@ -0,0 +1,299 @@ +*************************************** +Infiniband Network Block Device (IBNBD) +*************************************** + +Introduction +------------ + +IBNBD (InfiniBand Network Block Device) is a pair of kernel modules +(client and server) that allow for remote access of a block device on +the server over IBTRS protocol using the RDMA (InfiniBand, RoCE, iWarp) +transport. After being mapped, the remote block devices can be accessed +on the client side as local block devices. + +I/O is transfered between client and server by the IBTRS transport +modules. The administration of IBNBD and IBTRS modules is done via +sysfs entries. + +Requirements +------------ + + IBTRS kernel modules + +Quick Start +----------- + +Server side: + # modprobe ibnbd_server + +Client side: + # modprobe ibnbd_client + # echo "sessname=blya path=ip:10.50.100.66 device_path=/dev/ram0" > \ + /sys/devices/virtual/ibnbd-client/ctl/map_device + + Where "sessname=" is a session name, a string to identify the session + on client and on server sides; "path=" is a destination IP address or + a pair of a source and a destination IPs, separated by comma. Multiple + "path=" options can be specified in order to use multipath (see IBTRS + description for details); "device_path=" is the block device to be + mapped from the server side. After the session to the server machine is + established, the mapped device will appear on the client side under + /dev/ibnbd. + + +====================== +Client Sysfs Interface +====================== + +All sysfs files that are not read-only provide the usage information on read: + +Example: + # cat /sys/devices/virtual/ibnbd-client/ctl/map_device + + > Usage: echo "sessname= path=<[srcaddr,]dstaddr> + > [path=<[srcaddr,]dstaddr>] device_path= + > [access_mode=] + > [io_mode=]" > map_device + > + > addr ::= [ ip: | ip: | gid: ] + +Entries under /sys/devices/virtual/ibnbd-client/ctl/ +======================================= + +map_device (RW) +--------------- + +Expected format is the following: + + sessname= + path=<[srcaddr,]dstaddr> [path=<[srcaddr,]dstaddr> ...] + device_path= + [access_mode=] + [io_mode=] + +Where: + +sessname: accepts a string not bigger than 256 chars, which identifies + a given session on the client and on the server. + I.e. "clt_hostname-srv_hostname" could be a natural choice. + +path: describes a connection between the client and the server by + specifying destination and, when required, the source address. + The addresses are to be provided in the following format: + + ip: + ip: + gid: + + for example: + + path=ip:10.0.0.66 + The single addr is treated as the destination. + The connection will be established to this + server from any client IP address. + + path=ip:10.0.0.66,ip:10.0.1.66 + First addr is the source address and the second + is the destination. + + If multiple "path=" options are specified multiple connection + will be established and data will be sent according to + the selected multipath policy (see IBTRS mp_policy sysfs entry + description). + +device_path: Path to the block device on the server side. Path is specified + relative to the directory on server side configured in the + 'dev_search_path' module parameter of the ibnbd_server. + The ibnbd_server prepends the received from client + with and tries to open the + / block device. On success, + a /dev/ibnbd device file, a /sys/block/ibnbd_client/ibnbd/ + directory and an entry in /sys/devices/virtual/ibnbd-client/ctl/devices + will be created. + + If 'dev_search_path' contains '%SESSNAME%', then each session can + have different devices namespace, e.g. server was configured with + the following parameter "dev_search_path=/run/ibnbd-devs/%SESSNAME%", + client has this string "sessname=blya device_path=sda", then server + will try to open: /run/ibnbd-devs/blya/sda. + +access_mode: the access_mode parameter specifies if the device is to be + mapped as "ro" read-only or "rw" read-write. The server allows + a device to be exported in rw mode only once. The "migration" + access mode has to be specified if a second mapping in read-write + mode is desired. + + By default "rw" is used. + +io_mode: the io_mode parameter specifies if the device on the server + will be opened as block device "blockio" or as file "fileio". + When the device is opened as file, the VFS page cache is used + for read I/O operations, write I/O operations bypass the page + cache and go directly to disk (except meta updates, like file + access time). + + By default "blockio" mode is used. + +Exit Codes: + +If the device is already mapped it will fail with EEXIST. If the input +has an invalid format it will return EINVAL. If the device path cannot +be found on the server, it will fail with ENOENT. + +Finding device file after mapping +--------------------------------- + +After mapping, the device file can be found by: + o The symlink /sys/devices/virtual/ibnbd-client/ctl/devices/ + points to /sys/block/. The last part of the symlink destination + is the same as the device name. By extracting the last part of the + path the path to the device /dev/ can be build. + + o /dev/block/$(cat /sys/devices/virtual/ibnbd-client/ctl/devices//dev) + +How to find the of the device is described on the next +section. + +Entries under /sys/devices/virtual/ibnbd-client/ctl/devices/ +============================================================ + +For each device mapped on the client a new symbolic link is created as +/sys/devices/virtual/ibnbd-client/ctl/devices/, which points +to the block device created by ibnbd (/sys/block/ibnbd/). +The of each device is created as follows: + +- If the 'device_path' provided during mapping contains slashes ("/"), + they are replaced by exclamation mark ("!") and used as as the + . Otherwise, the will be the same as the + "device_path" provided. + +Entries under /sys/block/ibnbd/ibnbd_client/ +=============================================== + +unmap_device (RW) +----------------- + +To unmap a volume, "normal" or "force" has to be written to: + /sys/block/ibnbd/ibnbd_client/unmap_device + +When "normal" is used, the operation will fail with EBUSY if any process +is using the device. When "force" is used, the device is also unmapped +when device is in use. All I/Os that are in progress will fail. + +Example: + + # echo "normal" > /sys/block/ibnbd0/ibnbd/unmap_device + +state (RO) +---------- + +The file contains the current state of the block device. The state file +returns "open" when the device is successfully mapped from the server +and accepting I/O requests. When the connection to the server gets +disconnected in case of an error (e.g. link failure), the state file +returns "closed" and all I/O requests submitted to it will fail with -EIO. + +session (RO) +------------ + +IBNBD uses IBTRS session to transport the data between client and +server. The entry "session" contains the name of the session, that +was used to establish the IBTRS session. It's the same name that +was passed as server parameter to the map_device entry. + +mapping_path (RO) +----------------- + +Contains the path that was passed as "device_path" to the map_device +operation. + +====================== +Server Sysfs Interface +====================== + +Entries under /sys/devices/virtual/ibnbd-server/ctl/ +==================================================== + +When a client maps a device, a directory entry with the name of the +block device is created under /sys/devices/virtual/ibnbd-server/ctl/devices/. + +Entries under /sys/devices/virtual/ibnbd-server/ctl/devices// +========================================================================== + +block_dev (link) +--------------- + +Is a symlink to the sysfs entry of the exported device. + +Example: + + block_dev -> ../../../../devices/virtual/block/ram0 + +Entries under /sys/devices/virtual/ibnbd-server/ctl/devices//sessions/ +=================================================================================== + +For each client a particular device is exported to, following directory will be +created: + +/sys/devices/virtual/ibnbd-server/ctl/devices//sessions// + +When the device is unmapped by that client, the directory will be removed. + +Entries under /sys/devices/virtual/ibnbd-server/ctl/devices//sessions/ +================================================================================================= + +read_only (RO) +-------------- + +Contains '1' if device is mapped read-only, otherwise '0'. + +mapping_path (RO) +----------------- + +Contains the relative device path provided by the user during mapping. + +============================== +IBNBD-Server Module Parameters +============================== + +dev_search_path +--------------- + +When a device is mapped from the client, the server generates the path +to the block device on the server side by concatenating dev_search_path +and the "device_path" that was specified in the map_device operation. + +The default dev_search_path is: "/". + +dev_search_path option can also contain %SESSNAME% in order to provide +different deviec namespaces for different sessions. See "device_path" +option for details. + +============================== +Protocol (ibnbd/ibnbd-proto.h) +============================== + +1. Before mapping first device from a given server, client sends an +IBNBD_MSG_SESS_INFO to the server. Server responds with +IBNBD_MSG_SESS_INFO_RSP. Currently the messages only contain the protocol +version for backward compatibility. + +2. Client requests to open a device by sending IBNBD_MSG_OPEN message. This +contains the path to the device, access mode (read-only or writable), and +io_mode which specifies if the device should be opened as block device or +using file io. Server responds to the message with IBNBD_MSG_OPEN_RSP. This +contains a 32 bit device id to be used for IOs and device "geometry" related +information: side, max_hw_sectors, etc. + +3. Client attaches IBNBD_MSG_IO to each IO message send to a device. This +message contains device id, provided by server in his ibnbd_msg_open_rsp, +sector to be accessed, read-write flags and bi_size. + +4. Client closes a device by sending IBNBD_MSG_CLOSE which contains only the +device id provided by the server. + + +Contact +------- + +Mailing list: "IBNBD/IBTRS Storage Team"