From patchwork Mon Nov 4 10:37:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11225353 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 739FD913 for ; Mon, 4 Nov 2019 10:37:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2847821D7F for ; Mon, 4 Nov 2019 10:37:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="MZ9cf7pC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728064AbfKDKhl (ORCPT ); Mon, 4 Nov 2019 05:37:41 -0500 Received: from mail-eopbgr150084.outbound.protection.outlook.com ([40.107.15.84]:57608 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727607AbfKDKhl (ORCPT ); Mon, 4 Nov 2019 05:37:41 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=R8hzf62OHq/aFpDpyhGmcbhunahITTB41kXNtwuY1IeE4SN7W1/AES6XpF519r7PAFFC4mgpzXZZ8U2pIsLgr+bJvarFrNp5zrjzsecQ8N+UnxpT8NIrwlIU7/xVECjbB0l6grHLfC0A2paAM3g44MqF6LE+DXqR5CpXz4GQ+8Y7OMT3FpLKQynRiaQgFAhVieRm6QmezbOQIzmvwC95jZy3M3UepIolls2AuCRnaILFpk/JfIGwkjPho/kjhiQ7951gxyBPrttz66U25b1p6F8tW1bcpgYbvKkD27x3mD9rXFWjbrxF55r1OOsRz0ogNbeHA3YnzhSXwW7GZ6LddA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q7UfGriFTAUpm6pFcszGpfsvPF8JRRSFehvliyo4i78=; b=T5GdW095+yw9al8JaK3U+0WiRsgIuYm3nbEzbIeKjhSg89yfQpuvaS2rEZNZVxtwYZWQ1zJaXlZ7f+2gWN3qKCGqFKcOmDx8dRgP21tgrWvXWg6MbRXjU96WWE7h8GC9UjucjrjPYOQJe/0ELiKtHUdbL87/j6eukyLaNzDixSTuV7Iu/XqiqadfiqEC+rOIbqJngriHp15UkB/l0ERbWePE7cEXUkdiQaFuJxcsblowxfZNgFOUekXNRmZBMcIAuEmnxJyzMUk71RSfoyOB5mB072LnGAHmUdNjx/bbRy4zoDXUyyKrFJcW0slu6vpQHUz9G5bKbr1atCiv0yhF8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q7UfGriFTAUpm6pFcszGpfsvPF8JRRSFehvliyo4i78=; b=MZ9cf7pCdlKITQ3/VKWgBCbhv641QDeGqoGh7sRVNTlf8bZ4Jtc/Ku4mzKdGDPJy1W5D8QXYzQbgWftQied7Ij0xzV8ezq2B38jDzTWPrzkI+lykAk1QmEVoCP6H+FnXIrpcRy2bNEOQp3drvVwDTcADqH+fgvYqWQzOWRXdytc= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6232.eurprd05.prod.outlook.com (20.178.86.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Mon, 4 Nov 2019 10:37:25 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2408.024; Mon, 4 Nov 2019 10:37:25 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Maxim Chicherin Subject: [PATCH rdma-core 1/5] pyverbs: New CMID class Thread-Topic: [PATCH rdma-core 1/5] pyverbs: New CMID class Thread-Index: AQHVkvvZHP5KdRGueUG13fK/Rqs68Q== Date: Mon, 4 Nov 2019 10:37:25 +0000 Message-ID: <20191104103710.11196-2-noaos@mellanox.com> References: <20191104103710.11196-1-noaos@mellanox.com> In-Reply-To: <20191104103710.11196-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM4PR0701CA0039.eurprd07.prod.outlook.com (2603:10a6:200:42::49) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 0a86264b-3d2c-446d-bd43-08d76112fba2 x-ms-traffictypediagnostic: AM6PR05MB6232:|AM6PR05MB6232: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 0211965D06 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(39860400002)(376002)(136003)(366004)(346002)(396003)(199004)(189003)(25786009)(8676002)(2501003)(66066001)(386003)(6506007)(71200400001)(71190400001)(99286004)(11346002)(6436002)(8936002)(2616005)(476003)(30864003)(107886003)(1076003)(5660300002)(50226002)(6636002)(316002)(6512007)(54906003)(81166006)(446003)(81156014)(110136005)(4326008)(14454004)(6116002)(186003)(3846002)(86362001)(256004)(36756003)(6486002)(14444005)(52116002)(486006)(478600001)(66446008)(64756008)(66556008)(66476007)(66946007)(76176011)(102836004)(305945005)(26005)(2906002)(7736002)(579004)(559001);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6232;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: AXQ4ilHSR8FzBWNvYtrjLVJgoR0ZPaHmoYR7SC76uHr6MzetUgjmh9IkRKBIpNW4lqvjOnexScacEN3utQ+/oS5huModCM8asXc/x1Kv+6E5wWUGRM7SgWtFu835Vf+O4qenVloFgCLv2Oxfv9Kk9sPekHeZPbeB2FfX9pHMGrLLYLXBsCnPZn5JGizjtdko33E9OZw+F0aoiNFdmTKLtFhz3mrDED1tE2eHwWOlziOERcu/+TFNdB3eLG7lD3dhVR4UMinKlim62GvLEy7yjnk55+xVPOevoWnM6B41B0X2k97B62D8HiqPEYGiQJ0EO65d64s/Swge66YqHMPLTaFYJ4dsarc8SQD4NsXeGf3vBVF/hkb0KFrD6Voq1bPnRp7sVGeS6ilQaDxIrXKp2sI6WcoO732VLRyCB5BiVUoHlSIQHJe87q5gdYBRZTJJ MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0a86264b-3d2c-446d-bd43-08d76112fba2 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2019 10:37:25.8370 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3yIHSXSixYn9oI+ExvDTL/TpuP5VAQJYzyhyu7cuqh8ppcSlVr9Cke2P79n48TlritsvzzCCeznY1WvCKn56uQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6232 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Maxim Chicherin The CMID class represents rdma_cm_id, RDMA Connection Manager. Currently only synchronous data path is supported. Support was added to Context and PD classes to allow creation using rdmacm's API. Signed-off-by: Maxim Chicherin --- buildlib/pyverbs_functions.cmake | 2 +- pyverbs/CMakeLists.txt | 2 + pyverbs/cm_enums.pyx | 1 + pyverbs/cmid.pxd | 25 +++ pyverbs/cmid.pyx | 285 +++++++++++++++++++++++++++++++ pyverbs/device.pyx | 15 +- pyverbs/librdmacm.pxd | 106 ++++++++++++ pyverbs/librdmacm_enums.pxd | 32 ++++ pyverbs/pd.pyx | 24 ++- 9 files changed, 483 insertions(+), 9 deletions(-) create mode 120000 pyverbs/cm_enums.pyx create mode 100755 pyverbs/cmid.pxd create mode 100755 pyverbs/cmid.pyx create mode 100755 pyverbs/librdmacm.pxd create mode 100755 pyverbs/librdmacm_enums.pxd mode change 100644 => 100755 pyverbs/pd.pyx diff --git a/buildlib/pyverbs_functions.cmake b/buildlib/pyverbs_functions.cmake index 4c255054fe94..a494ec16610b 100644 --- a/buildlib/pyverbs_functions.cmake +++ b/buildlib/pyverbs_functions.cmake @@ -25,7 +25,7 @@ function(rdma_cython_module PY_MODULE LINKER_FLAGS) COMPILE_FLAGS "${CMAKE_C_FLAGS} -fPIC -fno-strict-aliasing -Wno-unused-function -Wno-redundant-decls -Wno-shadow -Wno-cast-function-type -Wno-implicit-fallthrough -Wno-unknown-warning -Wno-unknown-warning-option ${NO_VAR_TRACKING_FLAGS}" LIBRARY_OUTPUT_DIRECTORY "${BUILD_PYTHON}/${PY_MODULE}" PREFIX "") - target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs ${LINKER_FLAGS}) + target_link_libraries(${SONAME} LINK_PRIVATE ${PYTHON_LIBRARIES} ibverbs rdmacm ${LINKER_FLAGS}) install(TARGETS ${SONAME} DESTINATION ${CMAKE_INSTALL_PYTHON_ARCH_LIB}/${PY_MODULE}) endforeach() diff --git a/pyverbs/CMakeLists.txt b/pyverbs/CMakeLists.txt index 7bbb5fc841c0..de37025ce324 100755 --- a/pyverbs/CMakeLists.txt +++ b/pyverbs/CMakeLists.txt @@ -4,6 +4,8 @@ rdma_cython_module(pyverbs "" addr.pyx base.pyx + cm_enums.pyx + cmid.pyx cq.pyx device.pyx enums.pyx diff --git a/pyverbs/cm_enums.pyx b/pyverbs/cm_enums.pyx new file mode 120000 index 000000000000..bdab2b585a1d --- /dev/null +++ b/pyverbs/cm_enums.pyx @@ -0,0 +1 @@ +librdmacm_enums.pxd \ No newline at end of file diff --git a/pyverbs/cmid.pxd b/pyverbs/cmid.pxd new file mode 100755 index 000000000000..56bc755daf42 --- /dev/null +++ b/pyverbs/cmid.pxd @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) +# Copyright (c) 2018, Mellanox Technologies. All rights reserved. See COPYING file + +#cython: language_level=3 + +from pyverbs.base cimport PyverbsObject, PyverbsCM +from libc.string cimport memcpy, memset +from libc.stdlib cimport free, malloc +cimport pyverbs.librdmacm as cm + + +cdef class CMID(PyverbsCM): + cdef cm.rdma_cm_id *id + cdef object ctx + cdef object pd + cpdef close(self) + + +cdef class AddrInfo(PyverbsObject): + cdef cm.rdma_addrinfo *addr_info + cpdef close(self) + + +cdef class ConnParam(PyverbsObject): + cdef cm.rdma_conn_param conn_param \ No newline at end of file diff --git a/pyverbs/cmid.pyx b/pyverbs/cmid.pyx new file mode 100755 index 000000000000..c752feda8781 --- /dev/null +++ b/pyverbs/cmid.pyx @@ -0,0 +1,285 @@ +from pyverbs.pyverbs_error import PyverbsError +from pyverbs.device cimport PortAttr, Context +from pyverbs.qp cimport QPInitAttr, QPAttr +from pyverbs.base import PyverbsRDMAErrno +cimport pyverbs.libibverbs_enums as e +cimport pyverbs.librdmacm_enums as ce +cimport pyverbs.libibverbs as v +cimport pyverbs.librdmacm as cm +from pyverbs.pd cimport PD +from pyverbs.mr cimport MR +from pyverbs.cq cimport WC + + +cdef class ConnParam(PyverbsObject): + + def __cinit__(self, resources=1, depth=1, flow_control=0, retry=5, + rnr_retry=5, srq=0, qp_num=0): + """ + Initialize a ConnParam object over an underlying rdma_conn_param + C object which contains connection parameters. There are a few types of + port spaces in RDMACM: RDMA_PS_TCP, RDMA_PS_UDP, RDMA_PS_IB and + RDMA_PS_IPOIB. RDMA_PS_TCP resembles RC QP connection, which provides + reliable, connection-oriented QP communication. This object applies only + to RDMA_PS_TCP port space. + :param resources: Max outstanding RDMA read and atomic ops that local + side will accept from the remote side. + :param depth: Max outstanding RDMA read and atomic ops that local side + will have to the remote side. + :param flow_control: Specifies if hardware flow control is available. + :param retry: Max number of times that a send, RDMA or atomic op from + the remote peer should be retried. + :param rnr_retry: The maximum number of times that a send operation from + the remote peer should be retried on a connection + after receiving a receiver not ready (RNR) error. + :param srq: Specifies if the QP using shared receive queue, ignored if + the QP created by CMID. + :param qp_num: Specifies the QP number, ignored if the QP created by + CMID. + :return: ConnParam object + """ + memset(&self.conn_param, 0, sizeof(cm.rdma_conn_param)) + self.conn_param.responder_resources = resources + self.conn_param.initiator_depth = depth + self.conn_param.flow_control = flow_control + self.conn_param.retry_count = retry + self.conn_param.rnr_retry_count = rnr_retry + self.conn_param.srq = srq + self.conn_param.qp_num = qp_num + + def __str__(self): + print_format = '{:<4}: {:<4}\n' + return '{}: {}\n'.format('Connection parameters', "") +\ + print_format.format('responder resources', self.conn_param.responder_resources) +\ + print_format.format('initiator depth', self.conn_param.initiator_depth) +\ + print_format.format('flow control', self.conn_param.flow_control) +\ + print_format.format('retry count', self.conn_param.retry_count) +\ + print_format.format('rnr retry count', self.conn_param.rnr_retry_count) +\ + print_format.format('srq', self.conn_param.srq) +\ + print_format.format('qp number', self.conn_param.qp_num) + + +cdef class AddrInfo(PyverbsObject): + def __cinit__(self, node=None, service=None, port_space=0, flags=0): + """ + Initialize an AddrInfo object over an underlying rdma_addrinfo C object. + :param node: Name, dotted-decimal IPv4 or IPv6 hex address to resolve. + :param service: The service name or port number of the address. + :param port_space: RDMA port space used (RDMA_PS_UDP or RDMA_PS_TCP). + :param flags: Hint flags which control the operation. + :return: An AddrInfo object which contains information needed to + establish communication. + """ + cdef char* srvc = NULL + cdef char* address = NULL + cdef cm.rdma_addrinfo hints + cdef cm.rdma_addrinfo *hints_ptr = NULL + + if node is not None: + node = node.encode('utf-8') + address = node + if service is not None: + service = service.encode('utf-8') + srvc = service + if port_space != 0: + hints_ptr = &hints + memset(hints_ptr, 0, sizeof(cm.rdma_addrinfo)) + hints.ai_port_space = port_space + hints.ai_flags = flags + ret = cm.rdma_getaddrinfo(address, srvc, hints_ptr, &self.addr_info) + if ret != 0: + raise PyverbsRDMAErrno('Failed to get Address Info') + + def __dealloc__(self): + self.close() + + cpdef close(self): + self.logger.debug('Closing AddrInfo') + if self.addr_info != NULL: + cm.rdma_freeaddrinfo(self.addr_info) + self.addr_info = NULL + + +cdef class CMID(PyverbsCM): + + def __cinit__(self, object creator=None, QPInitAttr qp_init_attr=None, + PD pd=None): + """ + Initialize a CMID object over an underlying rdma_cm_id C object. + This is the main RDMA CM object which provides most of the rdmacm API. + Currently only synchronous RDMA_PS_TCP communication supported. + :param creator: For synchronous communication we need AddrInfo object in + order to establish connection. We allow creator to be + None for inner usage, see get_request method. + :param pd: Optional parameter, a PD to be associated with this CMID. + :param qp_init_attr: Optional initial QP attributes of CMID + associated QP. + :return: CMID object for synchronous communication. + """ + cdef v.ibv_qp_init_attr *init + cdef v.ibv_pd *in_pd = NULL + self.pd = None + self.ctx = None + if creator is None: + return + elif issubclass(type(creator), AddrInfo): + init = NULL if qp_init_attr is None else &qp_init_attr.attr + if pd is not None: + in_pd = pd.pd + self.pd = pd + ret = cm.rdma_create_ep(&self.id, (creator).addr_info, + in_pd, init) + if ret != 0: + raise PyverbsRDMAErrno('Failed to create CM ID') + if not (creator).addr_info.ai_flags & ce.RAI_PASSIVE: + self.ctx = Context(cmid=self) + if self.pd is None: + self.pd = PD(self) + else: + raise PyverbsRDMAErrno('Cannot create CM ID from {obj}' + .format(obj=type(creator))) + + def __dealloc__(self): + self.close() + + cpdef close(self): + self.logger.debug('Closing CMID') + if self.id != NULL: + cm.rdma_destroy_ep(self.id) + if self.ctx: + (self.ctx).context = NULL + if self.pd: + (self.pd).pd = NULL + self.id = NULL + + def get_request(self): + """ + Retrieves the next pending connection request event. The call may only + be used on listening CMIDs operating synchronously. If the call is + successful, a new CMID representing the connection request will be + returned to the user. The new CMID will reference event information + associated with the request until the user calls reject, accept, or + close on the newly created identifier. + :return: New CMID representing the connection request. + """ + to_conn = CMID() + ret = cm.rdma_get_request(self.id, &to_conn.id) + if ret != 0: + raise PyverbsRDMAErrno('Failed to get request, no connection established') + self.ctx = Context(cmid=to_conn) + self.pd = PD(to_conn) + return to_conn + + def reg_msgs(self, size): + """ + Registers a memory region for sending or receiving messages or for + RDMA operations. The registered memory may then be posted to an CMID + using post_send or post_recv methods. + :param size: The total length of the memory to register + :return: registered MR + """ + return MR(self.pd, size, e.IBV_ACCESS_LOCAL_WRITE) + + def listen(self, backlog=0): + """ + Listen for incoming connection requests or datagram service lookup. + The listen is restricted to the locally bound source address. + :param backlog: The backlog of incoming connection requests + :return: None + """ + ret = cm.rdma_listen(self.id, backlog) + if ret != 0: + raise PyverbsRDMAErrno('Listen Failed') + + def connect(self, ConnParam param=None): + """ + Initiates an active connection request to a remote destination. + :param param: Optional connection parameters + :return: None + """ + cdef cm.rdma_conn_param *conn = ¶m.conn_param if param else NULL + ret = cm.rdma_connect(self.id, conn) + if ret != 0: + raise PyverbsRDMAErrno('Failed to Connect') + + def disconnect(self): + """ + Disconnects a connection and transitions any associated QP to error + state. + :return: None + """ + ret = cm.rdma_disconnect(self.id) + if ret != 0: + raise PyverbsRDMAErrno('Failed to Disconnect') + + def accept(self, ConnParam param=None): + """ + Is called from the listening side to accept a connection or datagram + service lookup request. + :param param: Optional connection parameters + :return: None + """ + cdef cm.rdma_conn_param *conn = ¶m.conn_param if param else NULL + ret = cm.rdma_accept(self.id, conn) + if ret != 0: + raise PyverbsRDMAErrno('Failed to Accept Connection') + + def post_recv(self, MR mr not None): + """ + Posts a recv_wr via QP associated with CMID. + Context param of rdma_post_recv C function currently not supported. + :param mr: A valid MR object. + :return: None + """ + ret = cm.rdma_post_recv(self.id, NULL, mr.buf, mr.mr.length, mr.mr) + if ret != 0: + raise PyverbsRDMAErrno('Failed to Post Receive') + + def post_send(self, MR mr not None, flags=v.IBV_SEND_SIGNALED): + """ + Posts a message via QP associated with CMID. + Context param of rdma_post_send C function currently not supported. + :param mr: A valid MR object which contains message to send. + :param flags: flags for send work request. + :return: None + """ + ret = cm.rdma_post_send(self.id, NULL, mr.buf, mr.mr.length, mr.mr, + flags) + if ret != 0: + raise PyverbsRDMAErrno('Failed to Post Send') + + def get_recv_comp(self): + """ + Polls the receive CQ associated with CMID for a work completion. + :return: The retrieved WC or None if there is no completions + """ + cdef v.ibv_wc wc + ret = cm.rdma_get_recv_comp(self.id, &wc) + if ret < 0: + raise PyverbsRDMAErrno('Failed to retrieve receive completion') + elif ret == 0: + return None + return WC(wr_id=wc.wr_id, status=wc.status, opcode=wc.opcode, + vendor_err=wc.vendor_err, byte_len=wc.byte_len, + qp_num=wc.qp_num, src_qp=wc.src_qp, + imm_data=wc.imm_data, wc_flags=wc.wc_flags, + pkey_index=wc.pkey_index, slid=wc.slid, sl=wc.sl, + dlid_path_bits=wc.dlid_path_bits) + + def get_send_comp(self): + """ + Polls the send CQ associated with CMID for a work completion. + :return: The retrieved WC or None if there is no completions + """ + cdef v.ibv_wc wc + ret = cm.rdma_get_send_comp(self.id, &wc) + if ret < 0: + raise PyverbsRDMAErrno('Failed to retrieve send completion') + elif ret == 0: + return None + return WC(wr_id=wc.wr_id, status=wc.status, opcode=wc.opcode, + vendor_err=wc.vendor_err, byte_len=wc.byte_len, + qp_num=wc.qp_num, src_qp=wc.src_qp, + imm_data=wc.imm_data, wc_flags=wc.wc_flags, + pkey_index=wc.pkey_index, slid=wc.slid, sl=wc.sl, + dlid_path_bits=wc.dlid_path_bits) diff --git a/pyverbs/device.pyx b/pyverbs/device.pyx index 58a2aca27fcc..c747822b3b32 100755 --- a/pyverbs/device.pyx +++ b/pyverbs/device.pyx @@ -14,6 +14,7 @@ from .pyverbs_error import PyverbsUserError from pyverbs.base import PyverbsRDMAErrno cimport pyverbs.libibverbs_enums as e cimport pyverbs.libibverbs as v +from pyverbs.cmid cimport CMID from pyverbs.xrcd cimport XRCD from pyverbs.addr cimport GID from pyverbs.mr import DMMR @@ -79,16 +80,22 @@ cdef class Context(PyverbsCM): Initializes a Context object. The function searches the IB devices list for a device with the name provided by the user. If such a device is found, it is opened (unless provider attributes were given). + In case of cmid argument, CMID object already holds an ibv_context + initiated pointer, hence all we have to do is assign this pointer to + Context's object pointer. :param kwargs: Arguments: * *name* (str) The RDMA device's name * *attr* (object) Device-specific attributes, meaning that the device is to be opened by the provider + * *cmid* (CMID) + A CMID object (represents rdma_cm_id struct) :return: None """ cdef int count cdef v.ibv_device **dev_list + cdef CMID cmid self.pds = weakref.WeakSet() self.dms = weakref.WeakSet() @@ -99,7 +106,13 @@ cdef class Context(PyverbsCM): dev_name = kwargs.get('name') provider_attr = kwargs.get('attr') - if dev_name is not None: + cmid = kwargs.get('cmid') + + if cmid is not None: + self.context = cmid.id.verbs + cmid.ctx = self + return + elif dev_name is not None: self.name = dev_name else: raise PyverbsUserError('Device name must be provided') diff --git a/pyverbs/librdmacm.pxd b/pyverbs/librdmacm.pxd new file mode 100755 index 000000000000..935a4ae24e87 --- /dev/null +++ b/pyverbs/librdmacm.pxd @@ -0,0 +1,106 @@ +include 'libibverbs.pxd' +include 'librdmacm_enums.pxd' +from libc.stdint cimport uint8_t, uint32_t + +cdef extern from '': + + cdef struct rdma_cm_id: + ibv_context *verbs + rdma_event_channel *channel + void *context + ibv_qp *qp + rdma_port_space ps + uint8_t port_num + rdma_cm_event *event + ibv_comp_channel *send_cq_channel + ibv_cq *send_cq + ibv_comp_channel *recv_cq_channel + ibv_cq *recv_cq + ibv_srq *srq + ibv_pd *pd + ibv_qp_type qp_type + + cdef struct rdma_event_channel: + int fd + + cdef struct rdma_conn_param: + const void *private_data + uint8_t private_data_len + uint8_t responder_resources + uint8_t initiator_depth + uint8_t flow_control + uint8_t retry_count + uint8_t rnr_retry_count + uint8_t srq + uint32_t qp_num + + cdef struct rdma_ud_param: + const void *private_data + uint8_t private_data_len + ibv_ah_attr ah_attr + uint32_t qp_num + uint32_t qkey + + cdef union param: + rdma_conn_param conn + rdma_ud_param ud + + cdef struct rdma_cm_event: + rdma_cm_id *id + rdma_cm_id *listen_id + rdma_cm_event_type event + int status + param param + + cdef struct rdma_addrinfo: + int ai_flags + int ai_family + int ai_qp_type + int ai_port_space + int ai_src_len + int ai_dst_len + sockaddr *ai_src_addr + sockaddr *ai_dst_addr + char *ai_src_canonname + char *ai_dst_canonname + size_t ai_route_len + void *ai_route + size_t ai_connect_len + void *ai_connect + rdma_addrinfo *ai_next + +# These non rdmacm structs defined in one of rdma_cma.h's included header files + cdef struct sockaddr: + unsigned short sa_family + char sa_data[14] + + cdef struct in_addr: + uint32_t s_addr + + cdef struct sockaddr_in: + short sin_family + unsigned short sin_port + in_addr sin_addr + char sin_zero[8] + + int rdma_create_ep(rdma_cm_id **id, rdma_addrinfo *res, + ibv_pd *pd, ibv_qp_init_attr *qp_init_attr) + void rdma_destroy_ep(rdma_cm_id *id) + int rdma_get_request(rdma_cm_id *listen, rdma_cm_id **id) + int rdma_connect(rdma_cm_id *id, rdma_conn_param *conn_param) + int rdma_disconnect(rdma_cm_id *id) + int rdma_listen(rdma_cm_id *id, int backlog) + int rdma_accept(rdma_cm_id *id, rdma_conn_param *conn_param) + int rdma_getaddrinfo(char *node, char *service, rdma_addrinfo *hints, + rdma_addrinfo **res) + void rdma_freeaddrinfo(rdma_addrinfo *res) + +cdef extern from '': + int rdma_post_recv(rdma_cm_id *id, void *context, void *addr, + size_t length, ibv_mr *mr) + int rdma_post_send(rdma_cm_id *id, void *context, void *addr, + size_t length, ibv_mr *mr, int flags) + int rdma_get_send_comp(rdma_cm_id *id, ibv_wc *wc) + int rdma_get_recv_comp(rdma_cm_id *id, ibv_wc *wc) + ibv_mr *rdma_reg_msgs(rdma_cm_id *id, void *addr, size_t length) + int rdma_dereg_mr(ibv_mr *mr) diff --git a/pyverbs/librdmacm_enums.pxd b/pyverbs/librdmacm_enums.pxd new file mode 100755 index 000000000000..22a3648fb4a6 --- /dev/null +++ b/pyverbs/librdmacm_enums.pxd @@ -0,0 +1,32 @@ +cdef extern from '': + + cpdef enum rdma_cm_event_type: + RDMA_CM_EVENT_ADDR_RESOLVED + RDMA_CM_EVENT_ADDR_ERROR + RDMA_CM_EVENT_ROUTE_RESOLVED + RDMA_CM_EVENT_ROUTE_ERROR + RDMA_CM_EVENT_CONNECT_REQUEST + RDMA_CM_EVENT_CONNECT_RESPONSE + RDMA_CM_EVENT_CONNECT_ERROR + RDMA_CM_EVENT_UNREACHABLE + RDMA_CM_EVENT_REJECTED + RDMA_CM_EVENT_ESTABLISHED + RDMA_CM_EVENT_DISCONNECTED + RDMA_CM_EVENT_DEVICE_REMOVAL + RDMA_CM_EVENT_MULTICAST_JOIN + RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_ADDR_CHANGE + RDMA_CM_EVENT_TIMEWAIT_EXIT + + cpdef enum rdma_port_space: + RDMA_PS_IPOIB + RDMA_PS_TCP + RDMA_PS_UDP + RDMA_PS_IB + + # Hint flags which control the operation. + cpdef enum: + RAI_PASSIVE + RAI_NUMERICHOST + RAI_NOROUTE + RAI_FAMILY diff --git a/pyverbs/pd.pyx b/pyverbs/pd.pyx old mode 100644 new mode 100755 index 46cbb36009ce..d6af58f25980 --- a/pyverbs/pd.pyx +++ b/pyverbs/pd.pyx @@ -2,9 +2,10 @@ # Copyright (c) 2019, Mellanox Technologies. All rights reserved. import weakref -from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError +from pyverbs.pyverbs_error import PyverbsUserError, PyverbsError from pyverbs.base import PyverbsRDMAErrno from pyverbs.device cimport Context, DM +from pyverbs.cmid cimport CMID from .mr cimport MR, MW, DMMR from pyverbs.srq cimport SRQ from pyverbs.addr cimport AH @@ -15,18 +16,27 @@ cdef extern from 'errno.h': cdef class PD(PyverbsCM): - def __cinit__(self, Context context not None): + def __cinit__(self, object creator not None): """ Initializes a PD object. A reference for the creating Context is kept so that Python's GC will destroy the objects in the right order. :param context: The Context object creating the PD :return: The newly created PD on success """ - self.pd = v.ibv_alloc_pd(context.context) - if self.pd == NULL: - raise PyverbsRDMAErrno('Failed to allocate PD', errno) - self.ctx = context - context.add_ref(self) + if issubclass(type(creator), Context): + self.pd = v.ibv_alloc_pd((creator).context) + if self.pd == NULL: + raise PyverbsRDMAErrno('Failed to allocate PD') + self.ctx = creator + elif issubclass(type(creator), CMID): + cmid = creator + self.pd = cmid.id.pd + self.ctx = cmid.ctx + cmid.pd = self + else: + raise PyverbsUserError('Cannot create PD from {type}' + .format(type=type(creator))) + self.ctx.add_ref(self) self.logger.debug('PD: Allocated ibv_pd') self.srqs = weakref.WeakSet() self.mrs = weakref.WeakSet() From patchwork Mon Nov 4 10:37:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11225345 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2648E913 for ; Mon, 4 Nov 2019 10:37:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EA82521D7F for ; Mon, 4 Nov 2019 10:37:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="lv8NX5qp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727500AbfKDKh3 (ORCPT ); Mon, 4 Nov 2019 05:37:29 -0500 Received: from mail-eopbgr80089.outbound.protection.outlook.com ([40.107.8.89]:44166 "EHLO EUR04-VI1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727236AbfKDKh3 (ORCPT ); Mon, 4 Nov 2019 05:37:29 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NmKAHkmzMxFtIW+s7hU4T4DpjwVqarzQ8QgSPaa44uWL0HsB8QYTRZGSy1z+p2ICoyZRLzDQlxDbuDEW22Iu2OHnPoQ6sKZ6kzMy0+UPwRktsAzlXZi1SGdsXtPm9zGhU8I2kdOHqXW6pErIoX0vkV3/6q8zi9ZR0cTi0/+anFRC2dtMzcdiqZJ27bW6Rk8ORBpG6OomIIZo94XsXTA8IbH/sAdwJpx0H0zX8gjoKiMHbrFDx0VNnsrzr4ykvnVcm1Ja7PfgSwi0sbFGqyKkc+WuriduzLEfFb5jhYk6dVS/aHprhDb5bEX8SxE+pSRq4o3V4B8X+Liim8rdIw98/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a1FUTBOHaypDNLokkCZPgekc9bZo3lA15tMsPk1Kems=; b=C8VjnG8gLLFXFq+zMY3cfIEp2DxYP+xUKGpKTdtdIjXHn6L+jQ6zqv4brj50VY6Fqcsb26xu0r977ImbysR0tC9uMXdFFe67U00NIkB9+C+cE+opxP/C+iomybAbbzlYcqL63iwZU6t++AmohmdSt8IddsezGt6A8jP1pkjjtbJ8Njo3qS7aimcVVUvGJPVMCGpF/tC7w4xj5eeTJP7sUlPCmA5B6ujkY4vHLySGkZBGv48oFDzG1bF8ol3C+Qt0y5RVhFgzA63+lUoK3Plx0zN7v0p+1nU+mIslcGWoXERUiqLDoHyuz6PpAq2VaoKuuO15Fyu+HWWlyvrSJ9F27w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a1FUTBOHaypDNLokkCZPgekc9bZo3lA15tMsPk1Kems=; b=lv8NX5qp0bSAo4a+sDUVNIEl8SNfyTqA+8pkNDLyMx+Yr6885uTCe1fYdcxC38UDsPUdoikyPlOCtVElUOvsO3e5usP35Qog7GI3W5UgFzudKDwXPPYSB/WqS6ygq67ZJ+KAwALoqejBYUKvIiROaX3P/vHrSgTOmHlULE+B0Gc= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB5366.eurprd05.prod.outlook.com (20.177.191.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Mon, 4 Nov 2019 10:37:26 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2408.024; Mon, 4 Nov 2019 10:37:26 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Maxim Chicherin Subject: [PATCH rdma-core 2/5] tests: Fix PD API test Thread-Topic: [PATCH rdma-core 2/5] tests: Fix PD API test Thread-Index: AQHVkvvZTm52QF1ahkqonM2eoGSorg== Date: Mon, 4 Nov 2019 10:37:26 +0000 Message-ID: <20191104103710.11196-3-noaos@mellanox.com> References: <20191104103710.11196-1-noaos@mellanox.com> In-Reply-To: <20191104103710.11196-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM4PR0701CA0039.eurprd07.prod.outlook.com (2603:10a6:200:42::49) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 7c8de083-6224-46b5-ca08-08d76112fc35 x-ms-traffictypediagnostic: AM6PR05MB5366:|AM6PR05MB5366: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-forefront-prvs: 0211965D06 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(39860400002)(136003)(346002)(366004)(189003)(199004)(14454004)(76176011)(66066001)(6636002)(305945005)(7736002)(186003)(86362001)(6486002)(446003)(11346002)(6436002)(102836004)(6512007)(2616005)(476003)(486006)(316002)(8676002)(478600001)(4326008)(8936002)(4744005)(6116002)(81156014)(81166006)(110136005)(99286004)(54906003)(25786009)(6506007)(52116002)(26005)(50226002)(107886003)(14444005)(386003)(36756003)(256004)(66446008)(64756008)(66556008)(66946007)(66476007)(5660300002)(2501003)(2906002)(71200400001)(71190400001)(1076003)(3846002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB5366;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5mImW9dZz6WyFKTH2xRNjJrFben6c1rjyp/2CNUSo7Y7QYq9SO9ASyumiTjYqfqgRcW2PYr3r/vvo7ITPmjs934eVetYjBn1gmPl0jXr99p2YgqUUvZU7w13nOXcqZYwCZardnpYO+Kf+84/y9nNWEPKLNgjZ4zXjkfQ/imAOcsKNCd56ByenouxOmtGOC+3e20pevcUQAnIyjzw8e7tpYogZt0b+HYN+Ih1CsQPFEqVKpyEO0UHgK+5gr+cEBpMzPFHTDWPF+vRMeKpL6cHoIa+tyETb7Ph82i+KfKXWRl/NywfPGrCv8xOZPZMk9YVDSfuvTmTTy8ZtC4P9p+o6p+gNqndJpPhlSCX78VbE7lcbeCJy+2VpPN/8pWhqXCutdRCfMwuLmR4kx5xqS1MnNTvF7mhjdmrS0ZYOFN5sbzAFmWPyxlfRYwGWWVIORFd MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7c8de083-6224-46b5-ca08-08d76112fc35 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2019 10:37:26.7189 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ydAmfeHJhl2/ILhNX+7+VwNxxNLGJAgFJG9o3jpMTxd/3x1hJcaiYCnYiV8y+phYCCoPJ+5nHKzVZekS4SzTRA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5366 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Maxim Chicherin In order to support PD class in CMID, modifications was made in PD constructor, as a result one of test_pd's cases broke. Fix it to create PD properly. Signed-off-by: Maxim Chicherin --- tests/test_pd.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) mode change 100644 => 100755 tests/test_pd.py diff --git a/tests/test_pd.py b/tests/test_pd.py old mode 100644 new mode 100755 index 978cf4900146..a8d6eb2fb69f --- a/tests/test_pd.py +++ b/tests/test_pd.py @@ -48,8 +48,7 @@ class PDTest(PyverbsAPITestCase): try: PD(None) except TypeError as te: - assert 'expected pyverbs.device.Context' in te.args[0] - assert 'got NoneType' in te.args[0] + assert 'must not be None' in te.args[0] else: raise PyverbsRDMAErrno('Created a PD with None context') From patchwork Mon Nov 4 10:37:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11225349 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F72C1747 for ; Mon, 4 Nov 2019 10:37:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1529E21D7F for ; Mon, 4 Nov 2019 10:37:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="e7WNegzN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727891AbfKDKhb (ORCPT ); Mon, 4 Nov 2019 05:37:31 -0500 Received: from mail-eopbgr80089.outbound.protection.outlook.com ([40.107.8.89]:44166 "EHLO EUR04-VI1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727553AbfKDKhb (ORCPT ); Mon, 4 Nov 2019 05:37:31 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KzCu3acXkfZYmPzuceNmIz5PxM/bbapAcQTBbWNeEk1WhG3Jkmh2EIs2OpgddbGBPH+n0Cb8OsmtqY4S9CwLqphbW9urbrSYP1IPT8dgOINI6lfnpUZ4Ug71q8qjV8ZlzmhQuYehy9dxyLnJf2Z5G49RitEdCqU3XbWm1z3PCZcU7MXQ8h0zcjTPd6cfiQNkEwxjTWf0S98fCaQgVzzvSKkRmOkOB2yhQ9A9b10GKIdeasNorlGwl32oFEwDMdrQDmyUMCoK2rcMheFh3f8tDPaD7smcM5Ag/Ho22CtYf/gFbZFhB7b1iqV2WuBUqJ7Z2NDsukmPJiQRpA1/FkLW5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2I2WMeDh2vremi3XcAX2JJFKXQ0RiAVCNu4rzGF6aho=; b=IQzUdmMQ4YJcEnu7Ou1GUatEEP4/1CIv0oApQvLOkhKFlgYUYTC09YEeBcWHALOp3G9dCUe0alKh8+Nqw4TfvFP711w8Fdapn1BBlHc8SYh2p08J/p3yhE9jimG/qHjM2KkdNA2u14vnSZF24InWLwHVsUz+cFg65CgaFFyeMYtiaoJ/B+G+yGdxp5Lt3u5KqxH/lRarocFYLXQQmlZn2Seojwa4ouQOaZ4PhXTciAFuqdrHfFaB/x/3mRqAI84IgOfoo3EPkj2PUfZH8uQ6SLNuet4tMLnscyKZPKcNVCnfQPxDp+ygLhI0ZezwY6gQYYhZrlGn3F3yXElmTXZa8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2I2WMeDh2vremi3XcAX2JJFKXQ0RiAVCNu4rzGF6aho=; b=e7WNegzNqmu0eK1tJsL6Uj8tw5NQUQcNZMjqEo4AfFT/n2whDovJXKXFkZwFte3yYIcSeC2NOkOshb4f2yrKUmI2l35Twkk+Y+k0a7BjubKDkdt+Hznvk474dugZzWwMSKLsnr+vhYgcWqsYC2w3BsYHrL/qaR/WqLYRGEwqORQ= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB5366.eurprd05.prod.outlook.com (20.177.191.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Mon, 4 Nov 2019 10:37:27 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2408.024; Mon, 4 Nov 2019 10:37:27 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Maxim Chicherin Subject: [PATCH rdma-core 3/5] tests: New CMResources Class Thread-Topic: [PATCH rdma-core 3/5] tests: New CMResources Class Thread-Index: AQHVkvvaRSNnBzqSGEakdKCfJDTpmQ== Date: Mon, 4 Nov 2019 10:37:27 +0000 Message-ID: <20191104103710.11196-4-noaos@mellanox.com> References: <20191104103710.11196-1-noaos@mellanox.com> In-Reply-To: <20191104103710.11196-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM4PR0701CA0039.eurprd07.prod.outlook.com (2603:10a6:200:42::49) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: ec3bdf91-e677-4f6b-9195-08d76112fcbd x-ms-traffictypediagnostic: AM6PR05MB5366:|AM6PR05MB5366: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:826; x-forefront-prvs: 0211965D06 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(39860400002)(136003)(346002)(366004)(189003)(199004)(14454004)(76176011)(66066001)(6636002)(305945005)(7736002)(186003)(86362001)(6486002)(446003)(11346002)(6436002)(102836004)(6512007)(2616005)(476003)(486006)(316002)(8676002)(478600001)(4326008)(8936002)(6116002)(81156014)(81166006)(110136005)(99286004)(54906003)(25786009)(6506007)(52116002)(26005)(50226002)(107886003)(14444005)(386003)(36756003)(256004)(66446008)(64756008)(66556008)(66946007)(66476007)(5660300002)(2501003)(2906002)(71200400001)(71190400001)(1076003)(3846002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB5366;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: r/YPNfNE2APPNE7hXeBtQlp/DPRj3m1jYB3IojsqFdjx0MLHsaxptHIxcIgQHTq8CDaRw5iUD/eVS71/rOjURxE3KxXIYfXMqNgFgEISZaPHkZ1gDoyDltFZI0KcHfG5KKIrM/Z4gz5Q0QrEeFe+wTifs5UjhB05IPgiq0rwaMq0Ii75IKmQxzY6LHJpNzAxZ8BhvoMhECS18dNnDX3m27T6//8IqhQvA865paSgL/BvjEqbNWuHglROer36fuHEu6j5eK6x4nlfHtIOaV8n/SJwQkgMSmOKEvt9oyg0FJf3zea3YKnzruslPcTzp5iiTZu+yE0w05DNZINJ7pDHAqSyijN2969VQ9cepH6sXFv0WxoHFhGyGsE8ages2dBBWmluy4IKGaTdGtjpauB6ODqhP4MHe3uCA0d0stNBiD83r2lWHfdNm3YOMueBIt6b MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: ec3bdf91-e677-4f6b-9195-08d76112fcbd X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2019 10:37:27.5998 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: XCtudqhBwiV3eoN1h6Vh1/6I5ibKDkyzCztax2GMkmmVfYF0qvOXtlJGuJZVO7dureDBn5fDiEnJiHbpEJLPNg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5366 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Maxim Chicherin A base aggregation object for RDMACM. Currently only synchronous data path flow is supported. Signed-off-by: Maxim Chicherin --- tests/base.py | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/tests/base.py b/tests/base.py index e4e92fa2a35f..54bfc42ffe4c 100755 --- a/tests/base.py +++ b/tests/base.py @@ -11,13 +11,16 @@ from pyverbs.qp import QPCap, QPInitAttrEx, QPInitAttr, QPAttr, QP from pyverbs.addr import AHAttr, GlobalRoute from pyverbs.xrcd import XRCD, XRCDInitAttr from pyverbs.srq import SRQ, SrqInitAttrEx +from pyverbs.cmid import CMID, AddrInfo from pyverbs.device import Context +import pyverbs.cm_enums as ce import pyverbs.device as d import pyverbs.enums as e from pyverbs.pd import PD from pyverbs.cq import CQ from pyverbs.mr import MR + PATH_MTU = e.IBV_MTU_1024 MAX_DEST_RD_ATOMIC = 1 MAX_RD_ATOMIC = 1 @@ -26,6 +29,10 @@ RETRY_CNT = 7 RNR_RETRY = 7 TIMEOUT = 14 +# for rdmacm +PORT = '7471' +ZERO_ADDR = '0.0.0.0' + class PyverbsAPITestCase(unittest.TestCase): def setUp(self): @@ -144,6 +151,51 @@ class RDMATestCase(unittest.TestCase): self._add_gids_per_port(ctx, dev, port+1) +class CMResources: + """ + CMResources class is a base aggregator object which contains basic + resources for RDMA CM communication. + """ + def __init__(self, src=ZERO_ADDR, dst=ZERO_ADDR, port=PORT): + """ + :param src: Local address to bind to (for passive side) + :param dst: Destination address to connect (for active side) + :param port: Port number of the address + """ + self.is_server = True if dst == ZERO_ADDR else False + self.qp_init_attr = None + self.msg_size = 1024 + self.num_msgs = 100 + self.new_id = None + self.port = port + self.mr = None + if self.is_server: + self.ai = AddrInfo(src, self.port, ce.RDMA_PS_TCP, + ce.RAI_PASSIVE) + else: + self.ai = AddrInfo(dst, self.port, ce.RDMA_PS_TCP) + self.create_qp_init_attr() + self.cmid = CMID(creator=self.ai, qp_init_attr=self.qp_init_attr) + + def create_mr(self, cmid): + self.mr = cmid.reg_msgs(self.msg_size) + + def create_qp_init_attr(self): + self.qp_init_attr = QPInitAttr(cap=QPCap(max_recv_wr=1)) + + def listen(self): + self.cmid.listen() + self.new_id = self.cmid.get_request() + self.new_id.accept() + + def pre_run(self): + if self.is_server: + self.listen() + else: + self.cmid.connect() + self.create_mr(self.new_id if self.is_server else self.cmid) + + class BaseResources(object): """ BaseResources class is a base aggregator object which contains basic From patchwork Mon Nov 4 10:37:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11225355 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AC67913 for ; Mon, 4 Nov 2019 10:37:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D6A4E21D7D for ; Mon, 4 Nov 2019 10:37:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="lNbHpDTW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726441AbfKDKhn (ORCPT ); Mon, 4 Nov 2019 05:37:43 -0500 Received: from mail-eopbgr150084.outbound.protection.outlook.com ([40.107.15.84]:57608 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728449AbfKDKhn (ORCPT ); Mon, 4 Nov 2019 05:37:43 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=keTZ40KkcfKNMj5xgvuj6Q9h6Gx58aA8DqpsMnWyFP62gFLbyNt8dZmcG8/nGux5sqYphe/BIxy5eg8NKtUDWCn9Wd+gszX18sQozXuIxc6mXA5hP32puvxw/l810WF3gk9l9fo/TjOJVJSznzv/sWB6DGVk3hMlPupQUVwsLYg3g8Eb3sl0jftNlZNC5gL/V/NtXNB/l9yiVBNNUWMfB2kyI0aeCRyk3vER4nth4387+VKe7dmfRuZQyWWKChYFCoopEZCHI6IwyyDTe0h5jkCrm5gqboUPpCnpsvZNYPaEs77jfu/wVTnXxwCLa4lttW3XAeNPcKdBeEq/LPTCrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f0QUycYtYdILemwHJFGJJevFMM9g6sLgXKXO9Vhd6BI=; b=iuhgZeKSiwN9+1etskTYnowICCUubOnFoV8xXEAqYuR/Bn4PTvIWrJUQdrUZ1hCToIdM6GFV/YuFETCxA57f/3WWYMFW1nDukpTV9dgElPz3m15yPJDYZhQLN4qmACbpAXLg7CVxc5WwlRqS4D8emPtu6TsEXfR5e3C68lvZbemJhTNXqpPOiZEXina6NoY5/qkZIGQq3bU958QVjrbJEGOPgl1A3L4q2zc/qvAfYkuvEJhCYvwBIkB9qRhpmaa3OMAorxjSdAz7kRa4/ItsdLjLdzNhBWbFnUm/GhvLXNJwRUHHwfDSvovcVUGqbVvYi2FrM2RJgbWcDtLXtcmI8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f0QUycYtYdILemwHJFGJJevFMM9g6sLgXKXO9Vhd6BI=; b=lNbHpDTWPI6AztHBMqBIuzpE4KDoA0e+W7gkQAwNB9G9DCXAre25XuyM9T/UOtgPVPv6ByyICV/6pu9vybbNGvIdri7PWIfAO+foRl216vmyJbR28WdhS0V8zmB5Ex0xPRjkTEeayZ4YAxkSQjmO2CzkNRdBdBKyxyirDlHtSkE= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6232.eurprd05.prod.outlook.com (20.178.86.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Mon, 4 Nov 2019 10:37:28 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2408.024; Mon, 4 Nov 2019 10:37:28 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Maxim Chicherin Subject: [PATCH rdma-core 4/5] tests: Add RDMACM synchronous traffic test Thread-Topic: [PATCH rdma-core 4/5] tests: Add RDMACM synchronous traffic test Thread-Index: AQHVkvvaYLePQ614dE6cNrL/UsO/Kw== Date: Mon, 4 Nov 2019 10:37:28 +0000 Message-ID: <20191104103710.11196-5-noaos@mellanox.com> References: <20191104103710.11196-1-noaos@mellanox.com> In-Reply-To: <20191104103710.11196-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM4PR0701CA0039.eurprd07.prod.outlook.com (2603:10a6:200:42::49) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 61b28178-f672-4297-3223-08d76112fd45 x-ms-traffictypediagnostic: AM6PR05MB6232:|AM6PR05MB6232: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4714; x-forefront-prvs: 0211965D06 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(39860400002)(376002)(136003)(366004)(346002)(396003)(199004)(189003)(25786009)(8676002)(2501003)(66066001)(386003)(6506007)(71200400001)(71190400001)(99286004)(11346002)(6436002)(8936002)(2616005)(476003)(107886003)(1076003)(5660300002)(50226002)(6636002)(316002)(6512007)(54906003)(81166006)(446003)(81156014)(110136005)(4326008)(14454004)(6116002)(186003)(3846002)(86362001)(256004)(36756003)(6486002)(14444005)(52116002)(486006)(478600001)(66446008)(64756008)(66556008)(66476007)(66946007)(76176011)(102836004)(305945005)(26005)(2906002)(7736002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6232;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ghhJjxm9UNIIHWrns6mSVHGN+O6G+Y1m51X8AFFj4Zzd1s3DfC3KtQbr9SWZpEPKSeIcKndNBQcBIxz/4q5/5cBGnTGDf1ltduoKgtlNGJvYoJ/sr6pZh/mUQZfaEiFQ1v4C1XkJRlVLuo5tGw/SilsGMmLLW0QmsD1uinwcXjP+iUCyBo1MTJ29Q7Aap6VHrTmS53H+4HxdCRlBHypkba/R+tTcmxJT3n+UjTDuIApgdnbCVF+vnn1maHaaiVJpgQ8GOaBosnH0TVhlDitiA1lM4wFfJlIYgDnT8g3cn9WXzsLns+WzFbrwha/BuB8q2T4+6J8a3pfyR1iA6Vs0JFciaCbiUQPMks5k1mZyDUS4FUIp5rbqj+6dXX9o9zTFKjDTLrhpFEskvzdkCAvVIuSpxEN9doncgNMvyfeATQO2BplGgVcTzsHneBDyK/mY MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 61b28178-f672-4297-3223-08d76112fd45 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2019 10:37:28.5126 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ovGNc0U4tf3HpAQZUVASbCwzGOjvpNWuF35qbx5YIUYqzbAo2cajK4R4ayVkDwS7BXEjlOpc61t0HjyIxlphAw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6232 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Maxim Chicherin This test creates active and passive sides for RDMACM communication and validates received messages. Signed-off-by: Maxim Chicherin --- tests/CMakeLists.txt | 2 ++ tests/rdmacm_utils.py | 43 ++++++++++++++++++++++++++++++++ tests/test_rdmacm.py | 57 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+) mode change 100644 => 100755 tests/CMakeLists.txt create mode 100755 tests/rdmacm_utils.py create mode 100755 tests/test_rdmacm.py diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt old mode 100644 new mode 100755 index 960276230860..0d81d1a98fb7 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -3,6 +3,7 @@ rdma_python_test(tests __init__.py + rdmacm_utils.py test_addr.py base.py test_cq.py @@ -11,6 +12,7 @@ rdma_python_test(tests test_pd.py test_qp.py test_odp.py + test_rdmacm.py utils.py ) diff --git a/tests/rdmacm_utils.py b/tests/rdmacm_utils.py new file mode 100755 index 000000000000..59b627393e4f --- /dev/null +++ b/tests/rdmacm_utils.py @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) +# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file +""" +Provide some useful helper function for pyverbs rdmacm' tests. +""" +from tests.base import CMResources +from tests.utils import validate + + +def active_side(node): + client = CMResources(dst=node) + client.pre_run() + connected_id = client.cmid + send_msg = 'c' * client.msg_size + for _ in range(client.num_msgs): + client.mr.write(send_msg, client.msg_size) + connected_id.post_send(client.mr) + while connected_id.get_send_comp() is None: + pass + connected_id.post_recv(client.mr) + while connected_id.get_recv_comp() is None: + pass + msg_received = client.mr.read(client.msg_size, 0) + validate(msg_received, False, client.msg_size) + connected_id.disconnect() + + +def passive_side(node): + server = CMResources(src=node) + server.pre_run() + connected_id = server.new_id + send_msg = 's' * server.msg_size + for _ in range(server.num_msgs): + connected_id.post_recv(server.mr) + while connected_id.get_recv_comp() is None: + pass + msg_received = server.mr.read(server.msg_size, 0) + validate(msg_received, True, server.msg_size) + server.mr.write(send_msg, server.msg_size) + connected_id.post_send(server.mr) + while connected_id.get_send_comp() is None: + pass + connected_id.disconnect() diff --git a/tests/test_rdmacm.py b/tests/test_rdmacm.py new file mode 100755 index 000000000000..e435c635d7b2 --- /dev/null +++ b/tests/test_rdmacm.py @@ -0,0 +1,57 @@ +from tests.rdmacm_utils import active_side, passive_side +from tests.base import RDMATestCase +from multiprocessing import Process +import pyverbs.device as d +import subprocess +import unittest +import json + + +class CMTestCase(RDMATestCase): + def setUp(self): + if self.dev_name is not None: + net_name = self.get_net_name(self.dev_name) + try: + self.ip_addr = self.get_ip_address(net_name) + except KeyError: + raise unittest.SkipTest('Device {} has no net interface' + .format(self.dev_name)) + else: + dev_list = d.get_device_list() + for dev in dev_list: + net_name = self.get_net_name(dev.name.decode()) + try: + self.ip_addr = self.get_ip_address(net_name) + except IndexError: + continue + else: + self.dev_name = dev.name.decode() + break + if self.dev_name is None: + raise unittest.SkipTest('No devices with net interface') + super().setUp() + + @staticmethod + def get_net_name(dev): + process = subprocess.Popen(['ls', '/sys/class/infiniband/{}/device/net/' + .format(dev)], stdout=subprocess.PIPE) + out, err = process.communicate() + return out.decode().split('\n')[0] + + @staticmethod + def get_ip_address(ifname): + process = subprocess.Popen(['ip', '-j', 'addr', 'show', ifname], + stdout=subprocess.PIPE) + out, err = process.communicate() + loaded_json = json.loads(out.decode()) + interface = loaded_json[0]['addr_info'][0]['local'] + if 'fe80::' in interface: + interface = interface + '%' + ifname + return interface + + def test_rdmacm_sync_traffic(self): + ps = Process(target=passive_side, args=[self.ip_addr]) + ps.start() + ps.join(1) + active_side(self.ip_addr) + ps.join() From patchwork Mon Nov 4 10:37:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11225351 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DC571709 for ; Mon, 4 Nov 2019 10:37:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E6A2A21D7D for ; Mon, 4 Nov 2019 10:37:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="UFNdTzq3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727553AbfKDKhe (ORCPT ); Mon, 4 Nov 2019 05:37:34 -0500 Received: from mail-eopbgr80089.outbound.protection.outlook.com ([40.107.8.89]:44166 "EHLO EUR04-VI1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726441AbfKDKhe (ORCPT ); Mon, 4 Nov 2019 05:37:34 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P2P4tJt2mlQbwi2h1M4tOK7ZWYZe7qjF07QE2UXTVv3HFGsj/mehPfc9KHmEDNI1Vu6dkBTY3GFIFD2keZnH5EARqydHNy+BFe7CGoYCksmLwstcVYeGdT6RSt0YfMkD/2wrb2U0fIUNAtwm+5KmzozgokWSA5VQDvaklJ2rPwAvn34cPF3385JP1IxUtKGR8LA8632eMh6HLTD33MqMDfnzII+QVyyt906tnlTfOnp7bRlgRdPCngxtRTcKY7f18ZDAH4xSa0hN4XnHaRoQb19u43TBT9xSAP2GHoV2PWE+8qU3pqgxNOQXU1RvFfSIhDPgzQWON82XIOx874eSaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NzWPPKb/HHHdq/Gepknt/004Ag12cgLvwXJBuvgxKZc=; b=lQQ7sZRdCBtrgOhSq1/vlOmArgmrqCjmukcsOs7ztd1FrxY9frciZAiyYpw9n514RgilI1DFrQwtp4UcOv7upuzqn5Dnz6c559MLQZC0l/QvOQEr1vO0Y+31f74Rw8QXuOD48LOLoEHIYWHf7BAb1x+RfOY2cb2rkz48LtQCFXxKfOC2mj0ZHDRVHXqYSSL1DuF4gR3UcbZOxLBIlBlzNOLwsZcz4rqedDePh16F7Rt3lkS/5N7p7PVyMIYAMTFij5jBOlgg2A1X/kfUXMMree8q2FHikMGmAqy5llISDt5gx7rq2ui+lb4aprGznbIZcv1+nZSnxr9XC32jV6K4gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NzWPPKb/HHHdq/Gepknt/004Ag12cgLvwXJBuvgxKZc=; b=UFNdTzq33b21rIvdxHg1cwdSMypZJRdDqz8idorn6zUuien11XLgqWXnbIKcq/JeC8Tt6xFwXDDREBFE/a5NU2ewhvVWdxANq/Wegm1YHRHOTMrDkjzxUxf09kFSMSIuvdU7/WlB9QKho7OMPtyO5zyTm+Eq1ZU6UoHPnwGeC38= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB5366.eurprd05.prod.outlook.com (20.177.191.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.24; Mon, 4 Nov 2019 10:37:29 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2408.024; Mon, 4 Nov 2019 10:37:29 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Maxim Chicherin Subject: [PATCH rdma-core 5/5] Documentation: Document creation of CMID Thread-Topic: [PATCH rdma-core 5/5] Documentation: Document creation of CMID Thread-Index: AQHVkvvbqSTvVxkqCkiUGL+qRgohmg== Date: Mon, 4 Nov 2019 10:37:29 +0000 Message-ID: <20191104103710.11196-6-noaos@mellanox.com> References: <20191104103710.11196-1-noaos@mellanox.com> In-Reply-To: <20191104103710.11196-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM4PR0701CA0039.eurprd07.prod.outlook.com (2603:10a6:200:42::49) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 5711f897-91df-458c-51e9-08d76112fdc9 x-ms-traffictypediagnostic: AM6PR05MB5366:|AM6PR05MB5366: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5516; x-forefront-prvs: 0211965D06 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(39860400002)(136003)(346002)(366004)(189003)(199004)(14454004)(76176011)(66066001)(6636002)(305945005)(7736002)(186003)(86362001)(6486002)(446003)(11346002)(6436002)(102836004)(6512007)(2616005)(476003)(486006)(316002)(8676002)(478600001)(4326008)(8936002)(6116002)(81156014)(81166006)(110136005)(99286004)(54906003)(25786009)(6506007)(52116002)(26005)(50226002)(107886003)(14444005)(386003)(36756003)(256004)(66446008)(64756008)(66556008)(66946007)(66476007)(5660300002)(2501003)(2906002)(71200400001)(71190400001)(1076003)(3846002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB5366;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: RN5xzfwbiUkbO59U22oHc8cMnWDhIHQXtlzuhyj/rm4sFdsAcTFuvOC9L1J8LncnUKC5X405rO1ycNrPjxuBHycHUyayWyoFniFFz6zmW1BDMJ+quoFyPzkQzI8d4OKTa5oP9FCdZ275eTFfdysBkJ24JpgMRR1/rZKUjNAgWGZqeCmKbB9rcT4TvLZ/Oas1wjXRiCKNMosEu47XKtXPQzI16xWLctaIvfgjRmtE7aRTgRZYkymRtBH2gMWmLmOG7CVPrlG6SZwFLTGywVaB/5jQ5q/nmuuGwso8Sq+KqLxhfByqbojMvMd6nwPp+DKhnmjOPERwcHOf6mnYwCl2RYT+njxyIFnOF34DZr2fWvKPpFOFPiB6cVrbwTll8x3IpaaT6ooyqWjBkxG01MBqvTniwCDv2j3Zui56QXydm2mpclWUWhkQA/EXJzSyPxJ0 MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5711f897-91df-458c-51e9-08d76112fdc9 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2019 10:37:29.3475 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: B1FjcFtfG0YEgcgqd8wI8lyOCEfGKilfLAhWBmW23D/si9/Z3DE5J0daMrQLTxF9OkIPWD9ukbFQ5xIE7sHt5Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5366 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Maxim Chicherin Add code snippet to demonstrate creation of CMID and establish connection (synchronous control path). Signed-off-by: Maxim Chicherin --- Documentation/pyverbs.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md index 29ab9592c53c..6ff6fc5bbc37 100755 --- a/Documentation/pyverbs.md +++ b/Documentation/pyverbs.md @@ -390,3 +390,33 @@ srq_attr.comp_mask = e.IBV_SRQ_INIT_ATTR_TYPE | e.IBV_SRQ_INIT_ATTR_PD | \ e.IBV_SRQ_INIT_ATTR_CQ | e.IBV_SRQ_INIT_ATTR_XRCD srq = SRQ(ctx, srq_attr) ``` + +##### CMID +The following code snippet will demonstrate creation of a CMID object, which +represents rdma_cm_id C struct, and establish connection between two peers. +Currently only synchronous control path is supported (rdma_create_ep). +For more complex examples, please see tests/test_rdmacm. +```python +from pyverbs.qp import QPInitAttr, QPCap +from pyverbs.cmid import CMID, AddrInfo +import pyverbs.cm_enums as ce + +cap = QPCap(max_recv_wr=1) +qp_init_attr = QPInitAttr(cap=cap) +server = '11.137.14.124' +port = '7471' + +# Active side + +cai = AddrInfo(server, port, ce.RDMA_PS_TCP) +cid = CMID(creator=cai, qp_init_attr=qp_init_attr) +cid.connect() # send connection request to server + +# Passive side + +sai = AddrInfo(server, port, ce.RDMA_PS_TCP, ce.RAI_PASSIVE) +sid = CMID(creator=sai, qp_init_attr=qp_init_attr) +sid.listen() # listen for incoming connection requests +new_id = sid.get_request() # check if there are any connection requests +new_id.accept() # new_id is connected to remote peer and ready to communicate +```