From patchwork Mon Oct 23 13:17:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10022491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6A13D60381 for ; Mon, 23 Oct 2017 13:18:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 642EF204FE for ; Mon, 23 Oct 2017 13:18:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5915528898; Mon, 23 Oct 2017 13:18:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB6A8204FE for ; Mon, 23 Oct 2017 13:18:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932167AbdJWNSg (ORCPT ); Mon, 23 Oct 2017 09:18:36 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:51930 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751444AbdJWNSa (ORCPT ); Mon, 23 Oct 2017 09:18:30 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 23 Oct 2017 15:18:23 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id v9NDINqp027885; Mon, 23 Oct 2017 16:18:23 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id v9NDIN9F004646; Mon, 23 Oct 2017 16:18:23 +0300 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id v9NDINwf004645; Mon, 23 Oct 2017 16:18:23 +0300 From: Yishai Hadas To: dledford@redhat.com Cc: linux-rdma@vger.kernel.org, yishaih@mellanox.com, majd@mellanox.com, artemyko@mellanox.com Subject: [PATCH rdma-core 08/10] verbs: Tag matching send interface Date: Mon, 23 Oct 2017 16:17:59 +0300 Message-Id: <1508764681-4531-9-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1508764681-4531-1-git-send-email-yishaih@mellanox.com> References: <1508764681-4531-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Artemy Kovalyov Add tag matching header (TMH) and rendezvous header (RVH) definitions to infiniband/tm_types.h. Add TM message building explanations to Documentation/tag_matching.md. Signed-off-by: Artemy Kovalyov Reviewed-by: Yishai Hadas --- Documentation/tag_matching.md | 52 ++++++++++++++++++++++++++++++++ debian/libibverbs-dev.install | 1 + libibverbs/CMakeLists.txt | 1 + libibverbs/tm_types.h | 70 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 124 insertions(+) create mode 100644 libibverbs/tm_types.h diff --git a/Documentation/tag_matching.md b/Documentation/tag_matching.md index 1e5a929..23d6532 100644 --- a/Documentation/tag_matching.md +++ b/Documentation/tag_matching.md @@ -234,6 +234,58 @@ While **wr_id** identifies the tag manipulation operation itself, the completions. +### Sending TM messages + +TM messages are sent using standard RC Send operations. A TM message comprises +a Tag-Matching Header (TMH), an optional Rendezvous Header (RVH), and +a payload. + +TMH and RVH are defined in infiniband/tm_types.h: + +```h +struct ibv_tmh { + uint8_t opcode; + uint8_t reserved[3]; + __be32 app_ctx; + __be64 tag; +}; +``` +```h +struct ibv_rvh { + __be64 va; + __be32 rkey; + __be32 len; +}; +``` + +The following opcodes are defined: + +* **IBV_TM_NO_TAG** - Send a message without a tag. +Such a message will always be treated as unexpected by the receiver TM-SRQ. +Any data following the opcode is ignored by the tag matching logic, and the +message is delivered in its entirety (including the opcode) to the standard +SRQ buffer. + +* **IBV_TM_OP_EAGER** - Send an eager tagged message. +The message consists of a TMH followed by payload. + +* **IBV_TM_OP_RNDV** - Send a tagged rendezvous request. +The message consists of a TMH, an RVH, and optional additional data (which may +be inspected by receiver SW if the message is deemed unexpected). The RVH must +refer to a registered buffer containing the rendezvous payload. The total +rendezvous message size must not exceed the **max_rndv_hdr_size** capability. +The Sender must consider the operation outstanding until a TM message with the +**IBV_TM_OP_FIN** opcode is received, after which the buffer may be deregistered +and freed. + +* **IBV_TM_OP_FIN** - Send a rendezvous completion indication. +The message consists of a copy of the original TMH and RVH of the rendezvous +request, apart the opcode. This message is sent after the receiver has +completed the transfer of the rendezvous payload by an RDMA-read operation. It +may be sent either by HW or SW, depending on whether the rendezvous request +was handled as expected or unexpected by the TM-SRQ. + + ### TM completion processing There are 2 types of TM completions: tag-manipulation and receive completions. diff --git a/debian/libibverbs-dev.install b/debian/libibverbs-dev.install index b854a19..026327e 100644 --- a/debian/libibverbs-dev.install +++ b/debian/libibverbs-dev.install @@ -6,6 +6,7 @@ usr/include/infiniband/opcode.h usr/include/infiniband/sa-kern-abi.h usr/include/infiniband/sa.h usr/include/infiniband/verbs.h +usr/include/infiniband/tm_types.h usr/lib/*/libibverbs*.so usr/lib/*/libmlx4.so usr/lib/*/libmlx5.so diff --git a/libibverbs/CMakeLists.txt b/libibverbs/CMakeLists.txt index 863c395..af340a8 100644 --- a/libibverbs/CMakeLists.txt +++ b/libibverbs/CMakeLists.txt @@ -5,6 +5,7 @@ publish_headers(infiniband sa-kern-abi.h sa.h verbs.h + tm_types.h ) publish_internal_headers(infiniband diff --git a/libibverbs/tm_types.h b/libibverbs/tm_types.h new file mode 100644 index 0000000..9a50d45 --- /dev/null +++ b/libibverbs/tm_types.h @@ -0,0 +1,70 @@ +/* + * Copyright (c) 2017 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ +#ifndef _TM_TYPES_H +#define _TM_TYPES_H + +#include +#include + +#ifdef __cplusplus +# define BEGIN_C_DECLS extern "C" { +# define END_C_DECLS } +#else /* !__cplusplus */ +# define BEGIN_C_DECLS +# define END_C_DECLS +#endif /* __cplusplus */ + +BEGIN_C_DECLS + +enum ibv_tmh_op { + IBV_TMH_NO_TAG = 0, + IBV_TMH_RNDV = 1, + IBV_TMH_FIN = 2, + IBV_TMH_EAGER = 3, +}; + +struct ibv_tmh { + uint8_t opcode; /* from enum ibv_tmh_op */ + uint8_t reserved[3]; /* must be zero */ + __be32 app_ctx; /* opaque user data */ + __be64 tag; +}; + +struct ibv_rvh { + __be64 va; + __be32 rkey; + __be32 len; +}; + +END_C_DECLS +#endif /* _TM_TYPES_H */