From patchwork Sun Jan 27 15:28:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10782913 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DA596C2 for ; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8ACF12AA33 for ; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7F4A92AA36; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC47D2AA33 for ; Sun, 27 Jan 2019 15:29:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726403AbfA0P3h (ORCPT ); Sun, 27 Jan 2019 10:29:37 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:49265 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726357AbfA0P3h (ORCPT ); Sun, 27 Jan 2019 10:29:37 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 27 Jan 2019 17:29:32 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTWBl014746; Sun, 27 Jan 2019 17:29:32 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTWfd011521; Sun, 27 Jan 2019 17:29:32 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x0RFTWh7011520; Sun, 27 Jan 2019 17:29:32 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, danitg@mellanox.com, yosefe@mellanox.com, maxg@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 1/3] rdmacm: Expose rdma_init_qp_attr Date: Sun, 27 Jan 2019 17:28:55 +0200 Message-Id: <1548602938-11370-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> References: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Danit Goldberg Expose the internal function 'rdma_init_qp_attr' in librdmacm API. Applications can get the parameters for creating AH (needed for DC QP), or control QP attributes after its creation. Signed-off-by: Danit Goldberg Signed-off-by: Yossi Itigin Signed-off-by: Yishai Hadas --- debian/librdmacm-dev.install | 1 + debian/librdmacm1.symbols | 2 ++ librdmacm/CMakeLists.txt | 2 +- librdmacm/cma.c | 4 +-- librdmacm/librdmacm.map | 5 ++++ librdmacm/man/CMakeLists.txt | 1 + librdmacm/man/rdma_init_qp_attr.3.md | 54 ++++++++++++++++++++++++++++++++++++ librdmacm/rdma_cma.h | 10 +++++++ 8 files changed, 76 insertions(+), 3 deletions(-) create mode 100644 librdmacm/man/rdma_init_qp_attr.3.md diff --git a/debian/librdmacm-dev.install b/debian/librdmacm-dev.install index b8f643c..fe1b755 100644 --- a/debian/librdmacm-dev.install +++ b/debian/librdmacm-dev.install @@ -34,6 +34,7 @@ usr/share/man/man3/rdma_get_request.3 usr/share/man/man3/rdma_get_send_comp.3 usr/share/man/man3/rdma_get_src_port.3 usr/share/man/man3/rdma_getaddrinfo.3 +usr/share/man/man3/rdma_init_qp_attr.3 usr/share/man/man3/rdma_join_multicast.3 usr/share/man/man3/rdma_join_multicast_ex.3 usr/share/man/man3/rdma_leave_multicast.3 diff --git a/debian/librdmacm1.symbols b/debian/librdmacm1.symbols index 0fea265..0d1a54f 100644 --- a/debian/librdmacm1.symbols +++ b/debian/librdmacm1.symbols @@ -2,6 +2,7 @@ librdmacm.so.1 librdmacm1 #MINVER# * Build-Depends-Package: librdmacm-dev RDMACM_1.0@RDMACM_1.0 1.0.15 RDMACM_1.1@RDMACM_1.1 16 + RDMACM_1.2@RDMACM_1.2 23 raccept@RDMACM_1.0 1.0.16 rbind@RDMACM_1.0 1.0.16 rclose@RDMACM_1.0 1.0.16 @@ -32,6 +33,7 @@ librdmacm.so.1 librdmacm1 #MINVER# rdma_get_request@RDMACM_1.0 1.0.15 rdma_get_src_port@RDMACM_1.0 1.0.19 rdma_getaddrinfo@RDMACM_1.0 1.0.15 + rdma_init_qp_attr@RDMACM_1.2 23 rdma_join_multicast@RDMACM_1.0 1.0.15 rdma_join_multicast_ex@RDMACM_1.1 16 rdma_leave_multicast@RDMACM_1.0 1.0.15 diff --git a/librdmacm/CMakeLists.txt b/librdmacm/CMakeLists.txt index f71e407..b0329ad 100644 --- a/librdmacm/CMakeLists.txt +++ b/librdmacm/CMakeLists.txt @@ -10,7 +10,7 @@ publish_headers(infiniband rdma_library(rdmacm librdmacm.map # See Documentation/versioning.md - 1 1.1.${PACKAGE_VERSION} + 1 1.2.${PACKAGE_VERSION} acm.c addrinfo.c cma.c diff --git a/librdmacm/cma.c b/librdmacm/cma.c index 628d883..687ffd7 100644 --- a/librdmacm/cma.c +++ b/librdmacm/cma.c @@ -991,8 +991,8 @@ static int ucma_is_ud_qp(enum ibv_qp_type qp_type) return (qp_type == IBV_QPT_UD); } -static int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr *qp_attr, - int *qp_attr_mask) +int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr *qp_attr, + int *qp_attr_mask) { struct ucma_abi_init_qp_attr cmd; struct ib_uverbs_qp_attr resp; diff --git a/librdmacm/librdmacm.map b/librdmacm/librdmacm.map index 5813d74..bde1bc4 100644 --- a/librdmacm/librdmacm.map +++ b/librdmacm/librdmacm.map @@ -76,3 +76,8 @@ RDMACM_1.1 { global: rdma_join_multicast_ex; } RDMACM_1.0; + +RDMACM_1.2 { + global: + rdma_init_qp_attr; +} RDMACM_1.1; diff --git a/librdmacm/man/CMakeLists.txt b/librdmacm/man/CMakeLists.txt index d016c92..aec3b6b 100644 --- a/librdmacm/man/CMakeLists.txt +++ b/librdmacm/man/CMakeLists.txt @@ -32,6 +32,7 @@ rdma_man_pages( rdma_get_send_comp.3 rdma_get_src_port.3 rdma_getaddrinfo.3 + rdma_init_qp_attr.3.md rdma_join_multicast.3 rdma_join_multicast_ex.3 rdma_leave_multicast.3 diff --git a/librdmacm/man/rdma_init_qp_attr.3.md b/librdmacm/man/rdma_init_qp_attr.3.md new file mode 100644 index 0000000..99e812a --- /dev/null +++ b/librdmacm/man/rdma_init_qp_attr.3.md @@ -0,0 +1,54 @@ +--- +date: 2018-12-31 +footer: librdmacm +header: "Librdmacm Programmer's Manual" +layout: page +license: 'Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md' +section: 3 +title: RDMA_INIT_QP_ATTR +--- + +# NAME + +rdma_init_qp_attr - Returns qp attributes of a rdma_cm_id. + +# SYNOPSIS + +```c +#include + +int rdma_init_qp_attr(struct rdma_cm_id *id, + struct ibv_qp_attr *qp_attr, + int *qp_attr_mask); +``` +# DESCRIPTION + +**rdma_init_qp_attr()** returns qp attributes of a rdma_cm_id. + +Information about qp attributes and qp attributes mask is returned through the *qp_attr* and *qp_attr_mask* parameters. + +For details on the qp_attr structure, see ibv_modify_qp. + +# ARGUMENTS + +*id* +: RDMA identifier. + +*qp_attr* +: A reference to a qp attributes struct containing response information. + +*qp_attr_mask* +: A reference to a qp attributes mask containing response information. + +# RETURN VALUE + +**rdma_init_qp_attr()** returns 0 on success, or -1 on error. If an error occurs, errno will be set to indicate the failure reason. + +# SEE ALSO + +**rdma_cm**(7), +**ibv_modify_qp**(3) + +# AUTHOR + +Danit Goldberg diff --git a/librdmacm/rdma_cma.h b/librdmacm/rdma_cma.h index 2096a81..cf4a683 100644 --- a/librdmacm/rdma_cma.h +++ b/librdmacm/rdma_cma.h @@ -721,6 +721,16 @@ int rdma_getaddrinfo(const char *node, const char *service, void rdma_freeaddrinfo(struct rdma_addrinfo *res); +/** + * rdma_init_qp_attr - Returns QP attributes. + * @id: Communication identifier. + * @qp_attr: A reference to a QP attributes struct containing + * response information. + * @qp_attr_mask: A reference to a QP attributes mask containing + * response information. + */ +int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr *qp_attr, + int *qp_attr_mask); #ifdef __cplusplus } #endif From patchwork Sun Jan 27 15:28:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10782915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B80691E for ; Sun, 27 Jan 2019 15:29:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 192202AA33 for ; Sun, 27 Jan 2019 15:29:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D8332AA34; Sun, 27 Jan 2019 15:29:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 706252AA35 for ; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726423AbfA0P3i (ORCPT ); Sun, 27 Jan 2019 10:29:38 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:49264 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726354AbfA0P3h (ORCPT ); Sun, 27 Jan 2019 10:29:37 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 27 Jan 2019 17:29:33 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTWp3014749; Sun, 27 Jan 2019 17:29:32 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTW0k011525; Sun, 27 Jan 2019 17:29:32 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x0RFTWTf011524; Sun, 27 Jan 2019 17:29:32 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, danitg@mellanox.com, yosefe@mellanox.com, maxg@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 2/3] rdmacm: Expose CONNECT_RESPONSE event and rdma_establish() Date: Sun, 27 Jan 2019 17:28:56 +0200 Message-Id: <1548602938-11370-3-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> References: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yossi Itigin Applications which do not create a QP through rdma_create_qp() may want to postpone the ESTABLISHED event on the passive side, to let the active side complete an application-specific connection establishment phase. For example, modify a QP created by the application to RTR, or make some preparations for receiving messages from the passive side. This patch will return a new event on the active side: CONNECT_RESPONSE, instead of ESTABLISHED, if id->qp==NULL. This gives a chance to the application to perform the extra connection setup. In addition, this patch adds a new rdma_establish() API, which should be called afterwards to complete the connection and generate an ESTABLISHED event on the passive side. Signed-off-by: Yossi Itigin Signed-off-by: Danit Goldberg Signed-off-by: Yishai Hadas --- debian/librdmacm-dev.install | 1 + debian/librdmacm1.symbols | 1 + librdmacm/cma.c | 29 +++++++++++++++---- librdmacm/librdmacm.map | 1 + librdmacm/man/CMakeLists.txt | 1 + librdmacm/man/rdma_establish.3.md | 59 +++++++++++++++++++++++++++++++++++++++ librdmacm/rdma_cma.h | 18 ++++++++++++ 7 files changed, 104 insertions(+), 6 deletions(-) create mode 100644 librdmacm/man/rdma_establish.3.md diff --git a/debian/librdmacm-dev.install b/debian/librdmacm-dev.install index fe1b755..e12c300 100644 --- a/debian/librdmacm-dev.install +++ b/debian/librdmacm-dev.install @@ -22,6 +22,7 @@ usr/share/man/man3/rdma_destroy_id.3 usr/share/man/man3/rdma_destroy_qp.3 usr/share/man/man3/rdma_destroy_srq.3 usr/share/man/man3/rdma_disconnect.3 +usr/share/man/man3/rdma_establish.3 usr/share/man/man3/rdma_event_str.3 usr/share/man/man3/rdma_free_devices.3 usr/share/man/man3/rdma_get_cm_event.3 diff --git a/debian/librdmacm1.symbols b/debian/librdmacm1.symbols index 0d1a54f..996122f 100644 --- a/debian/librdmacm1.symbols +++ b/debian/librdmacm1.symbols @@ -25,6 +25,7 @@ librdmacm.so.1 librdmacm1 #MINVER# rdma_destroy_srq@RDMACM_1.0 1.0.15 rdma_disconnect@RDMACM_1.0 1.0.15 rdma_event_str@RDMACM_1.0 1.0.15 + rdma_establish@RDMACM_1.2 23 rdma_free_devices@RDMACM_1.0 1.0.15 rdma_freeaddrinfo@RDMACM_1.0 1.0.15 rdma_get_cm_event@RDMACM_1.0 1.0.15 diff --git a/librdmacm/cma.c b/librdmacm/cma.c index 687ffd7..5fbbcf5 100644 --- a/librdmacm/cma.c +++ b/librdmacm/cma.c @@ -2077,6 +2077,18 @@ static void ucma_copy_ud_event(struct cma_event *event, dst->qkey = src->qkey; } +int rdma_establish(struct rdma_cm_id *id) +{ + if (id->qp) + return ERR(EINVAL); + + /* id->qp is NULL, so ucma_process_conn_resp() will only send ACCEPT to + * the passive side, and will not attempt to modify the QP. + */ + return ucma_process_conn_resp(container_of(id, struct cma_id_private, + id)); +} + int rdma_get_cm_event(struct rdma_event_channel *channel, struct rdma_cm_event **event) { @@ -2153,12 +2165,17 @@ retry: break; case RDMA_CM_EVENT_CONNECT_RESPONSE: ucma_copy_conn_event(evt, &resp.param.conn); - evt->event.status = ucma_process_conn_resp(evt->id_priv); - if (!evt->event.status) - evt->event.event = RDMA_CM_EVENT_ESTABLISHED; - else { - evt->event.event = RDMA_CM_EVENT_CONNECT_ERROR; - evt->id_priv->connect_error = 1; + if (!evt->id_priv->id.qp) { + evt->event.event = RDMA_CM_EVENT_CONNECT_RESPONSE; + } else { + evt->event.status = + ucma_process_conn_resp(evt->id_priv); + if (!evt->event.status) + evt->event.event = RDMA_CM_EVENT_ESTABLISHED; + else { + evt->event.event = RDMA_CM_EVENT_CONNECT_ERROR; + evt->id_priv->connect_error = 1; + } } break; case RDMA_CM_EVENT_ESTABLISHED: diff --git a/librdmacm/librdmacm.map b/librdmacm/librdmacm.map index bde1bc4..7f55e84 100644 --- a/librdmacm/librdmacm.map +++ b/librdmacm/librdmacm.map @@ -79,5 +79,6 @@ RDMACM_1.1 { RDMACM_1.2 { global: + rdma_establish; rdma_init_qp_attr; } RDMACM_1.1; diff --git a/librdmacm/man/CMakeLists.txt b/librdmacm/man/CMakeLists.txt index aec3b6b..2d1efbf 100644 --- a/librdmacm/man/CMakeLists.txt +++ b/librdmacm/man/CMakeLists.txt @@ -20,6 +20,7 @@ rdma_man_pages( rdma_destroy_qp.3 rdma_destroy_srq.3 rdma_disconnect.3 + rdma_establish.3.md rdma_event_str.3 rdma_free_devices.3 rdma_get_cm_event.3 diff --git a/librdmacm/man/rdma_establish.3.md b/librdmacm/man/rdma_establish.3.md new file mode 100644 index 0000000..91f390d --- /dev/null +++ b/librdmacm/man/rdma_establish.3.md @@ -0,0 +1,59 @@ +--- +date: 2019-01-16 +footer: librdmacm +header: "Librdmacm Programmer's Manual" +layout: page +license: 'Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md' +section: 3 +title: RDMA_ESTABLISH +--- + +# NAME + +rdma_establish - Complete an active connection request. + +# SYNOPSIS + +```c +#include + +int rdma_establish(struct rdma_cm_id *id); +``` + +# DESCRIPTION + +**rdma_establish()** Acknowledge an incoming connection response event and complete the connection establishment. + +Notes: + +If a QP has not been created on the rdma_cm_id, this function should be called by the active side to complete the connection, + +after getting connect response event. + +This will trigger a connection established event on the passive side. + +This function should not be used on an rdma_cm_id on which a QP has been created. + +# ARGUMENTS + +*id* +: RDMA identifier. + +# RETURN VALUE + +**rdma_establish()** returns 0 on success, or -1 on error. If an error occurs, errno will be set to indicate the failure reason. + +# SEE ALSO + +**rdma_connect**(3), +**rdma_disconnect**(3) +**rdma_get_cm_event**(3) + +# AUTHORS + +Danit Goldberg + +Yossi Itigin + + + diff --git a/librdmacm/rdma_cma.h b/librdmacm/rdma_cma.h index cf4a683..758a3f7 100644 --- a/librdmacm/rdma_cma.h +++ b/librdmacm/rdma_cma.h @@ -442,6 +442,24 @@ void rdma_destroy_qp(struct rdma_cm_id *id); int rdma_connect(struct rdma_cm_id *id, struct rdma_conn_param *conn_param); /** + * rdma_establish - Complete an active connection request. + * @id: RDMA identifier. + * Description: + * Acknowledge an incoming connection response event and complete the + * connection establishment. + * Notes: + * If a QP has not been created on the rdma_cm_id, this function should be + * called by the active side to complete the connection, after getting connect + * response event. This will trigger a connection established event on the + * passive side. + * This function should not be used on an rdma_cm_id on which a QP has been + * created. + * See also: + * rdma_connect, rdma_disconnect, rdma_get_cm_event + */ +int rdma_establish(struct rdma_cm_id *id); + +/** * rdma_listen - Listen for incoming connection requests. * @id: RDMA identifier. * @backlog: backlog of incoming connection requests. From patchwork Sun Jan 27 15:28:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10782911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7011E159A for ; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DC742AA34 for ; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 522822AA36; Sun, 27 Jan 2019 15:29:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B24072AA34 for ; Sun, 27 Jan 2019 15:29:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726397AbfA0P3h (ORCPT ); Sun, 27 Jan 2019 10:29:37 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:49267 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726360AbfA0P3g (ORCPT ); Sun, 27 Jan 2019 10:29:36 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 27 Jan 2019 17:29:33 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTXvd014752; Sun, 27 Jan 2019 17:29:33 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x0RFTWEm011529; Sun, 27 Jan 2019 17:29:32 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x0RFTWLv011528; Sun, 27 Jan 2019 17:29:32 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, danitg@mellanox.com, yosefe@mellanox.com, maxg@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 3/3] rping: Add option to create QP independently Date: Sun, 27 Jan 2019 17:28:57 +0200 Message-Id: <1548602938-11370-4-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> References: <1548602938-11370-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yossi Itigin The option is available via "-q" flag. Signed-off-by: Yossi Itigin Signed-off-by: Yishai Hadas --- librdmacm/examples/rping.c | 112 ++++++++++++++++++++++++++++++++++++++------- librdmacm/man/rping.1 | 3 ++ 2 files changed, 99 insertions(+), 16 deletions(-) diff --git a/librdmacm/examples/rping.c b/librdmacm/examples/rping.c index 49f5dc5..993c5ad 100644 --- a/librdmacm/examples/rping.c +++ b/librdmacm/examples/rping.c @@ -145,6 +145,7 @@ struct rping_cb { struct sockaddr_storage ssource; __be16 port; /* dst port in NBO */ int verbose; /* verbose logging */ + int self_create_qp; /* Create QP not via cma */ int count; /* ping count */ int size; /* ping data size */ int validate; /* validate ping data */ @@ -190,6 +191,12 @@ static int rping_cma_event_handler(struct rdma_cm_id *cma_id, sem_post(&cb->sem); break; + case RDMA_CM_EVENT_CONNECT_RESPONSE: + DEBUG_LOG("CONNECT_RESPONSE\n"); + cb->state = CONNECTED; + sem_post(&cb->sem); + break; + case RDMA_CM_EVENT_ESTABLISHED: DEBUG_LOG("ESTABLISHED\n"); @@ -347,13 +354,67 @@ error: return ret; } +static void rping_init_conn_param(struct rping_cb *cb, + struct rdma_conn_param *conn_param) +{ + memset(conn_param, 0, sizeof(*conn_param)); + conn_param->responder_resources = 1; + conn_param->initiator_depth = 1; + conn_param->retry_count = 7; + conn_param->rnr_retry_count = 7; + if (cb->self_create_qp) + conn_param->qp_num = cb->qp->qp_num; +} + + +static int rping_self_modify_qp(struct rping_cb *cb, struct rdma_cm_id *id) +{ + struct ibv_qp_attr qp_attr; + int qp_attr_mask, ret; + + qp_attr.qp_state = IBV_QPS_INIT; + ret = rdma_init_qp_attr(id, &qp_attr, &qp_attr_mask); + if (ret) + return ret; + + ret = ibv_modify_qp(cb->qp, &qp_attr, qp_attr_mask); + if (ret) + return ret; + + qp_attr.qp_state = IBV_QPS_RTR; + ret = rdma_init_qp_attr(id, &qp_attr, &qp_attr_mask); + if (ret) + return ret; + + ret = ibv_modify_qp(cb->qp, &qp_attr, qp_attr_mask); + if (ret) + return ret; + + qp_attr.qp_state = IBV_QPS_RTS; + ret = rdma_init_qp_attr(id, &qp_attr, &qp_attr_mask); + if (ret) + return ret; + + return ibv_modify_qp(cb->qp, &qp_attr, qp_attr_mask); +} + static int rping_accept(struct rping_cb *cb) { + struct rdma_conn_param conn_param; int ret; DEBUG_LOG("accepting client connection request\n"); - ret = rdma_accept(cb->child_cm_id, NULL); + if (cb->self_create_qp) { + ret = rping_self_modify_qp(cb, cb->child_cm_id); + if (ret) + return ret; + + rping_init_conn_param(cb, &conn_param); + ret = rdma_accept(cb->child_cm_id, &conn_param); + } else { + ret = rdma_accept(cb->child_cm_id, NULL); + } if (ret) { perror("rdma_accept"); return ret; @@ -480,6 +541,7 @@ static void rping_free_buffers(struct rping_cb *cb) static int rping_create_qp(struct rping_cb *cb) { struct ibv_qp_init_attr init_attr; + struct rdma_cm_id *id; int ret; memset(&init_attr, 0, sizeof(init_attr)); @@ -491,16 +553,21 @@ static int rping_create_qp(struct rping_cb *cb) init_attr.send_cq = cb->cq; init_attr.recv_cq = cb->cq; - if (cb->server) { - ret = rdma_create_qp(cb->child_cm_id, cb->pd, &init_attr); - if (!ret) - cb->qp = cb->child_cm_id->qp; - } else { - ret = rdma_create_qp(cb->cm_id, cb->pd, &init_attr); - if (!ret) - cb->qp = cb->cm_id->qp; + if (cb->self_create_qp) { + cb->qp = ibv_create_qp(cb->pd, &init_attr); + if (!cb->qp) { + perror("ibv_create_qp"); + return -1; + } + return 0; } + id = cb->server ? cb->child_cm_id : cb->cm_id; + ret = rdma_create_qp(id, cb->pd, &init_attr); + if (!ret) + cb->qp = id->qp; + else + perror("rdma_create_qp"); return ret; } @@ -549,7 +616,6 @@ static int rping_setup_qp(struct rping_cb *cb, struct rdma_cm_id *cm_id) ret = rping_create_qp(cb); if (ret) { - perror("rdma_create_qp"); goto err3; } DEBUG_LOG("created qp %p\n", cb->qp); @@ -1011,11 +1077,7 @@ static int rping_connect_client(struct rping_cb *cb) struct rdma_conn_param conn_param; int ret; - memset(&conn_param, 0, sizeof conn_param); - conn_param.responder_resources = 1; - conn_param.initiator_depth = 1; - conn_param.retry_count = 7; - + rping_init_conn_param(cb, &conn_param); ret = rdma_connect(cb->cm_id, &conn_param); if (ret) { perror("rdma_connect"); @@ -1028,6 +1090,20 @@ static int rping_connect_client(struct rping_cb *cb) return -1; } + if (cb->self_create_qp) { + ret = rping_self_modify_qp(cb, cb->cm_id); + if (ret) { + perror("rping_modify_qp"); + return ret; + } + + ret = rdma_establish(cb->cm_id); + if (ret) { + perror("rdma_establish"); + return ret; + } + } + DEBUG_LOG("rmda_connect successful\n"); return 0; } @@ -1160,6 +1236,7 @@ static void usage(const char *name) printf("\t-a addr\t\taddress\n"); printf("\t-p port\t\tport\n"); printf("\t-P\t\tpersistent server mode allowing multiple connections\n"); + printf("\t-q\t\tuse self-created, self-modified QP\n"); } int main(int argc, char *argv[]) @@ -1182,7 +1259,7 @@ int main(int argc, char *argv[]) sem_init(&cb->sem, 0, 0); opterr = 0; - while ((op=getopt(argc, argv, "a:I:Pp:C:S:t:scvVd")) != -1) { + while ((op = getopt(argc, argv, "a:I:Pp:C:S:t:scvVdq")) != -1) { switch (op) { case 'a': ret = get_addr(optarg, (struct sockaddr *) &cb->sin); @@ -1236,6 +1313,9 @@ int main(int argc, char *argv[]) case 'd': debug++; break; + case 'q': + cb->self_create_qp = 1; + break; default: usage("rping"); ret = EINVAL; diff --git a/librdmacm/man/rping.1 b/librdmacm/man/rping.1 index 8d4f157..7ec530e 100644 --- a/librdmacm/man/rping.1 +++ b/librdmacm/man/rping.1 @@ -52,6 +52,9 @@ The size of each message transferred, in bytes. (default 100) \-P Run the server in persistent mode. This allows multiple rping clients to connect to a single server instance. The server will run until killed. +.TP +\-q +Control QP Creation/Modification directly from the application, instead of rdma_cm. .SH "NOTES" Because this test maps RDMA resources to userspace, users must ensure that they have available system resources and permissions. See the