From patchwork Sun Nov 17 13:30:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27A0A184E for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 07D752084C for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="rh8MQYTM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726171AbfKQNar (ORCPT ); Sun, 17 Nov 2019 08:30:47 -0500 Received: from mail-eopbgr70077.outbound.protection.outlook.com ([40.107.7.77]:21606 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726157AbfKQNaq (ORCPT ); Sun, 17 Nov 2019 08:30:46 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ijvgSLZcRp+e9oNtCuicwnSIR3Lm28df6XXNhsnvHx1yrHzEj/YAqOV769+m0kr6UPv8OXgzqJ0G0qFDI0a1sS8RqvK1fp3CpeeXcETBxTmcPhfuGZZCz7632+TAhWoTIBuu/K6NZ0On98anzcUDDnGggLDf3HmS8SA4kKGDRheTmerY+kTDe4xlIwM2KmziiVQXE0xudWF2AMMoOldjGGJvRx1nX+LX+pRNmlf0Pf3oXG3GA9uJHlDwdYrtwfm0mjok/Ma0hKESLlMwV4zaqlJcPyMzazotyH/4bM20dOjXvT27O5aXmwl5MKkF8Z571Xwpjw+SCI0MDC5AGa0nhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pZXzjWjneHPwZy1FROWJOFilDXmln5RAZ1QBYrizCqM=; b=WRXk/lSEDUotOg044YunxXVnX4TwKDczc9JVDR2p1tN/unya7eqZ53Vi0cb/eiuUUPldTkVNwtGoLvDsFyQG31ywSMq0T/8hyU5GXsFncbf0xmaU46ZntYWNpclIJi1dkwk5ZolNwflnuXJSXM5hKlrIMUnUadWNdXtde1OCvlZ5Ic5KUJ1O0FU7/ZsqYPu1WaoOtRTxW+sWDmDUwJ+r0WgQiZENz6pkIgwWHE4MYHRMB+zkzT5rFamPBwPgkfNexuaYS9NXuANfeE7bkPWcCZmy5WGSo1f8kIO3N34N6553qbdOtr97wi5yhj8r7T9yBPpun6I4Z6RqbfAFs5/OLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pZXzjWjneHPwZy1FROWJOFilDXmln5RAZ1QBYrizCqM=; b=rh8MQYTMgtdCCU9cUekAqnS550ouKLMDm9onMs6+iBCDeZ57sVLo0XLlz08zRTyxvRc1Yjs4p/YPsJHMp2pmPYdnh8NOluibdA9D9kojTgQPnsGOXY2/CRcqMUB6tP1XtRxvQq/LO1f7FVVOoxkYxn5Tb/6jSkOskqmH6O1c9Ww= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:42 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:42 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Thread-Topic: [PATCH rdma-core 1/7] pyverbs: Allow QP creation by provider Thread-Index: AQHVnUs1srj4BUSP/E2KE4hBogqBHg== Date: Sun, 17 Nov 2019 13:30:42 +0000 Message-ID: <20191117133030.10784-2-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 0baa527e-1c2e-41bc-555e-08d76b6257ca x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(14444005)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: u/0/eJ/9+ya+3wFoJU8CEG+VAfV79lgt0ACFwqg1wW8746eed3SGd/x9SyErdz6QT9gt4AL3QhVCxUuDa1GVtAhH3ylknuDj2kmWynePmpAQO0kGl7YTP+NisNImGQcXQcvwmeEXe1K7tihJrW2+vPnmsDTku5SSZOYdR0IpdVDPP/lzweHHHB+UHT/ZDRVkNzhHP73cwlVymsIRJFbFbz9Cufjkc1AT4XewMyUdn1Qzv+TBDyLlSjDtsUnCnDoRzxHL6XPjdcEOczHR5XKKmYN8gfjDgTTzY2ixXwNVYkbfNh5Q6rD6KOSWEYZCemcGXz+bk+oKUgnOS+Cj/t+by5TFEi+KOtedgGREg8sLtH40WFU/+gv/mtu3KskbodhvYN99UpVkmzon9ippuHO0VYxgFuEAcJbT6Ee3mFUEh9hcRK2ggU2I4Kn5KI1YCmZc MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0baa527e-1c2e-41bc-555e-08d76b6257ca X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:42.2644 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: dI9fDe8pCeUKi/cUq/OdbU88nxxivNv3sMdvSZl15hQEP9+1ek5ZBhztnih8oP4OnWznIkSw9/YDiR/1aPdLNw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org A QP can be created using either the generic interface or a provider specific interface. Provider QPs extend the legacy QP, which in Cython means that the legacy QP's __cinit__ will be called automatically. To avoid QP being initialized via legacy interface, add **kwargs to QP's __cinit__(). If kwargs is not empty, the QP's __cinit__ will return without creating the QP, leaving the initialization to the provider. Signed-off-by: Noa Osherovich --- pyverbs/qp.pyx | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/pyverbs/qp.pyx b/pyverbs/qp.pyx index c526f0ad1bb6..74e2ce76665e 100755 --- a/pyverbs/qp.pyx +++ b/pyverbs/qp.pyx @@ -783,7 +783,7 @@ cdef class QPAttr(PyverbsObject): cdef class QP(PyverbsCM): def __cinit__(self, object creator not None, object init_attr not None, - QPAttr qp_attr=None): + QPAttr qp_attr=None, **kwargs): """ Initializes a QP object and performs state transitions according to user request. @@ -803,11 +803,16 @@ cdef class QP(PyverbsCM): using Context). :param qp_attr: Optional QPAttr object. Will be used for QP state transitions after creation. + :param kwargs: Provider-specific QP creation attributes, meaning that + the QP will be created by the provider. :return: An initialized QP object """ cdef PD pd cdef Context ctx self.update_cqs(init_attr) + if len(kwargs) > 0: + # Leave QP initialization to the provider + return # In order to use cdef'd methods, a proper casting must be done, let's # infer the type. if issubclass(type(creator), Context): From patchwork Sun Nov 17 13:30:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248363 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63A7F1593 for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 305AE2084C for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="WxmoeHpC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726175AbfKQNav (ORCPT ); Sun, 17 Nov 2019 08:30:51 -0500 Received: from mail-eopbgr70077.outbound.protection.outlook.com ([40.107.7.77]:21606 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726151AbfKQNav (ORCPT ); Sun, 17 Nov 2019 08:30:51 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OrMw5kfcQ+lHJiS//jzytCY7pXBNxxuUSg7ldBKoiabMTGEV09GfIDJjU/XBf+GyEOgiKh+gBRIQlKKiVP0mu/15VclZdgk02RecXA8bH7g9Gzn2EhGNJuxAk4iFIdxodMBjRAZLpgAYAuwlrTUQ7qNjj3OzuKVkIRiZv4otKPhd/2CxgLEFR+xxKMObKWX7UoJnnKf/ZaJlI8134cUF7kjd/7QjRk/2azwyLR3HuhI+2TX4ognFICEDwryeRjyjZX5T+lD/ygS1yWx5nbCPvWNQQiHPRCBOD443bxNtkadFm+HQ9kAEAW4i6e7Vqt5yG7MVuUhDJjKR9u1YuGuw5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dv03pcCsy/fpJTL+sSs3uMILuZjEKAfKh2GXUOF56Lk=; b=QbpXiEnUDqAutAOAykQRENfHCTUYspXG630b9yi1nf+tkifMdJNrtqIobyTKgWDZH1SW/dtDINSpCn0Qz/ntW6W4VaBjfy3pM8htxex6yikm9lHykFBfxnQ7QOYZm0l5I310y/oQ+TqyhKjn+knMvRYHXVzDs2tsaDHI7lSLoBDZVakJ35M4K4ENIeZETgiNoFlePh0P4Gad8E/a8pl52+UCytb5VRCra6eg9uLm5veYQqegd+0jMYDfjxGhRb7rJYMGCzAk8YXfmOuxlSMEtRwUb/g1oUkeLBEkDVRoJNdtAz/D2xVbO3PjmP52Er3SW5/B94+cmWyvi55wk35wUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dv03pcCsy/fpJTL+sSs3uMILuZjEKAfKh2GXUOF56Lk=; b=WxmoeHpC77V430JNrCLnsmC7jOI12hObLvhA3miLB0ok4s8iiF3fh1ibsMtcFvPkqdZI4+azvzuNw52zo8L7usieT5pGGkosk4Wu2ENKi7h3gPuWPSBc4XbObRXFkpeyvjVTaCv6MDIgpEAhmf9sfsujwOF3fdBfAl7YCkjsUM0= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:43 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:43 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP Thread-Topic: [PATCH rdma-core 2/7] pyverbs/mlx5: Add support for mlx5 QP Thread-Index: AQHVnUs18TufquZgWEKZii7MQLBD6Q== Date: Sun, 17 Nov 2019 13:30:43 +0000 Message-ID: <20191117133030.10784-3-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 237b8f74-ff81-4d8e-cef9-08d76b625851 x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6108; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(14444005)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(30864003)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nYubCniFrFGsIzT/OjQYL7HlvUBoyFnVoSj7J2a5KOGoWq+cXg3+IsoNX+zKkBeY62p6nC2aKVapynbDsWvBVuzyVGH0M8dbKx+yr2pqYX3db4GS/gHW6oUuvK7SdwwjgGcL/5PfeyfSvt2LOEwqN6OKni3odb38vHWFUFNIAft6aZCRjgWJkX1nlvPyZLYDrbjQgeZPZ+FZvqYmk6FbO8fOIGsk3/3ZJJquA4rbp8vk3sBhzDyXJGI1ZVgVKE2BhR8LdLnlcr5BUdAK7aLsEHdN76lnObTpuQ030WLpHEcePzx9iGfB7CONg6y+sHHULRs8CEx9/8kexhvy+S2JKYIOxCxGgjKmlAZHUFmovJh8qYll+duJ6YFzGUhFVBEyHlD9YxmUU0XtMgqr9QZxugrh8bO4JjrIF++Iyj0pzPjyufA8+8uLXgeummcYepwc MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 237b8f74-ff81-4d8e-cef9-08d76b625851 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:43.1847 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: i2zdzB6h1nwKf2iTmiAZ7hUMZVJUdIYNkasbN/3mLToqB5CPToHh8aWYdhmIQjE6S94JLwAM+EtPM68Rsv6yug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add the needed declarations and structs to allow users to create a QP via the mlx5dv interface. Creation process is similar to that of a legacy QP but the user shall use an Mlx5Context and provide an Mlx5DVQPInitAttr object as well as a QPInitAttrEx object. Signed-off-by: Noa Osherovich --- pyverbs/providers/mlx5/libmlx5.pxd | 15 ++ pyverbs/providers/mlx5/mlx5dv.pxd | 11 ++ pyverbs/providers/mlx5/mlx5dv.pyx | 192 +++++++++++++++++++++++- pyverbs/providers/mlx5/mlx5dv_enums.pxd | 21 +++ 4 files changed, 236 insertions(+), 3 deletions(-) diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd index aaf75620a9bb..2c5be241a3ec 100644 --- a/pyverbs/providers/mlx5/libmlx5.pxd +++ b/pyverbs/providers/mlx5/libmlx5.pxd @@ -1,6 +1,8 @@ # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) # Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file +include 'mlx5dv_enums.pxd' + from libcpp cimport bool cimport pyverbs.libibverbs as v @@ -40,7 +42,20 @@ cdef extern from 'infiniband/mlx5dv.h': unsigned int flow_action_flags unsigned int dc_odp_caps + cdef struct mlx5dv_dc_init_attr: + mlx5dv_dc_type dc_type + unsigned long dct_access_key + + cdef struct mlx5dv_qp_init_attr: + unsigned long comp_mask + unsigned int create_flags + mlx5dv_dc_init_attr dc_init_attr + unsigned long send_ops_flags + bool mlx5dv_is_supported(v.ibv_device *device) v.ibv_context* mlx5dv_open_device(v.ibv_device *device, mlx5dv_context_attr *attr) int mlx5dv_query_device(v.ibv_context *ctx, mlx5dv_context *attrs_out) + v.ibv_qp *mlx5dv_create_qp(v.ibv_context *context, + v.ibv_qp_init_attr_ex *qp_attr, + mlx5dv_qp_init_attr *mlx5_qp_attr) diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd index d9fea82af894..918065012d30 100644 --- a/pyverbs/providers/mlx5/mlx5dv.pxd +++ b/pyverbs/providers/mlx5/mlx5dv.pxd @@ -6,6 +6,8 @@ cimport pyverbs.providers.mlx5.libmlx5 as dv from pyverbs.base cimport PyverbsObject from pyverbs.device cimport Context +from pyverbs.qp cimport QP + cdef class Mlx5Context(Context): pass @@ -15,3 +17,12 @@ cdef class Mlx5DVContextAttr(PyverbsObject): cdef class Mlx5DVContext(PyverbsObject): cdef dv.mlx5dv_context dv + +cdef class Mlx5DVDCInitAttr(PyverbsObject): + cdef dv.mlx5dv_dc_init_attr attr + +cdef class Mlx5DVQPInitAttr(PyverbsObject): + cdef dv.mlx5dv_qp_init_attr attr + +cdef class Mlx5QP(QP): + cdef object dc_type diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx index dadc9cdcceee..b4a971a5b935 100644 --- a/pyverbs/providers/mlx5/mlx5dv.pyx +++ b/pyverbs/providers/mlx5/mlx5dv.pyx @@ -6,7 +6,8 @@ cimport pyverbs.providers.mlx5.mlx5dv_enums as dve cimport pyverbs.providers.mlx5.libmlx5 as dv from pyverbs.base import PyverbsRDMAErrno cimport pyverbs.libibverbs_enums as e - +from pyverbs.qp cimport QPInitAttrEx +from pyverbs.pd cimport PD cdef class Mlx5DVContextAttr(PyverbsObject): """ @@ -179,8 +180,159 @@ cdef class Mlx5DVContext(PyverbsObject): print_format.format('DC ODP caps', self.dv.dc_odp_caps) +cdef class Mlx5DVDCInitAttr(PyverbsObject): + """ + Represents mlx5dv_dc_init_attr struct, which defines initial attributes + for DC QP creation. + """ + def __cinit__(self, dc_type=dve.MLX5DV_DCTYPE_DCI, dct_access_key=0): + """ + Initializes an Mlx5DVDCInitAttr object with the given DC type and DCT + access key. + :param dc_type: Which DC QP to create (DCI/DCT). + :param dct_access_key: Access key to be used by the DCT + :return: An initializes object + """ + self.attr.dc_type = dc_type + self.attr.dct_access_key = dct_access_key + + def __str__(self): + print_format = '{:20}: {:<20}\n' + return print_format.format('DC type', dc_type_to_str(self.attr.dc_type)) +\ + print_format.format('DCT access key', self.attr.dct_access_key) + + @property + def dc_type(self): + return self.attr.dc_type + @dc_type.setter + def dc_type(self, val): + self.attr.dc_type = val + + @property + def dct_access_key(self): + return self.attr.dct_access_key + @dct_access_key.setter + def dct_access_key(self, val): + self.attr.dct_access_key = val + + +cdef class Mlx5DVQPInitAttr(PyverbsObject): + """ + Represents mlx5dv_qp_init_attr struct, initial attributes used for mlx5 QP + creation. + """ + def __cinit__(self, comp_mask=0, create_flags=0, + Mlx5DVDCInitAttr dc_init_attr=None, send_ops_flags=0): + """ + Initializes an Mlx5DVQPInitAttr object with the given user data. + :param comp_mask: A bitmask specifying which fields are valid + :param create_flags: A bitwise OR of mlx5dv_qp_create_flags + :param dc_init_attr: Mlx5DVDCInitAttr object + :param send_ops_flags: A bitwise OR of mlx5dv_qp_create_send_ops_flags + :return: An initialized Mlx5DVQPInitAttr object + """ + self.attr.comp_mask = comp_mask + self.attr.create_flags = create_flags + self.attr.send_ops_flags = send_ops_flags + if dc_init_attr is not None: + self.attr.dc_init_attr.dc_type = dc_init_attr.dc_type + self.attr.dc_init_attr.dct_access_key = dc_init_attr.dct_access_key + + def __str__(self): + print_format = '{:20}: {:<20}\n' + return print_format.format('Comp mask', + qp_comp_mask_to_str(self.attr.comp_mask)) +\ + print_format.format('Create flags', + qp_create_flags_to_str(self.attr.create_flags)) +\ + 'DC init attr:\n' +\ + print_format.format(' DC type', + dc_type_to_str(self.attr.dc_init_attr.dc_type)) +\ + print_format.format(' DCT access key', + self.attr.dc_init_attr.dct_access_key) +\ + print_format.format('Send ops flags', + send_ops_flags_to_str(self.attr.send_ops_flags)) + + @property + def comp_mask(self): + return self.attr.comp_mask + @comp_mask.setter + def comp_mask(self, val): + self.attr.comp_mask = val + + @property + def create_flags(self): + return self.attr.create_flags + @create_flags.setter + def create_flags(self, val): + self.attr.create_flags = val + + @property + def send_ops_flags(self): + return self.attr.send_ops_flags + @send_ops_flags.setter + def send_ops_flags(self, val): + self.attr.send_ops_flags = val + + @property + def dc_type(self): + return self.attr.dc_init_attr.dc_type + @dc_type.setter + def dc_type(self, val): + self.attr.dc_init_attr.dc_type = val + + @property + def dct_access_key(self): + return self.attr.dc_init_attr.dct_access_key + @dct_access_key.setter + def dct_access_key(self, val): + self.attr.dc_init_attr.dct_access_key = val + + +cdef class Mlx5QP(QP): + def __cinit__(self, Mlx5Context context, QPInitAttrEx init_attr, + Mlx5DVQPInitAttr dv_init_attr): + """ + Initializes an mlx5 QP according to the user-provided data. + :param context: mlx5 Context object + :param init_attr: QPInitAttrEx object + :param dv_init_attr: Mlx5DVQPInitAttr object + :return: An initialized Mlx5QP + """ + cdef PD pd + self.dc_type = dv_init_attr.dc_type if dv_init_attr else 0 + if init_attr.pd is not None: + pd = init_attr.pd + pd.add_ref(self) + self.qp = \ + dv.mlx5dv_create_qp(context.context, + &init_attr.attr, + &dv_init_attr.attr if dv_init_attr is not None + else NULL) + if self.qp == NULL: + raise PyverbsRDMAErrno('Failed to create MLX5 QP.\nQPInitAttrEx ' + 'attributes:\n{}\nMLX5DVQPInitAttr:\n{}'. + format(init_attr, dv_init_attr)) + + def _get_comp_mask(self, dst): + masks = {dve.MLX5DV_DCTYPE_DCT: {'INIT': e.IBV_QP_PKEY_INDEX | + e.IBV_QP_PORT | e.IBV_QP_ACCESS_FLAGS, + 'RTR': e.IBV_QP_AV |\ + e.IBV_QP_PATH_MTU |\ + e.IBV_QP_MIN_RNR_TIMER}, + dve.MLX5DV_DCTYPE_DCI: {'INIT': e.IBV_QP_PKEY_INDEX |\ + e.IBV_QP_PORT, + 'RTR': e.IBV_QP_PATH_MTU, + 'RTS': e.IBV_QP_TIMEOUT |\ + e.IBV_QP_RETRY_CNT |\ + e.IBV_QP_RNR_RETRY | e.IBV_QP_SQ_PSN |\ + e.IBV_QP_MAX_QP_RD_ATOMIC}} + if self.dc_type == 0: + return super()._get_comp_mask(dst) + return masks[self.dc_type][dst] | e.IBV_QP_STATE + + def qpts_to_str(qp_types): - numberic_types = qp_types + numeric_types = qp_types qpts_str = '' qpts = {e.IBV_QPT_RC: 'RC', e.IBV_QPT_UC: 'UC', e.IBV_QPT_UD: 'UD', e.IBV_QPT_RAW_PACKET: 'Raw Packet', e.IBV_QPT_XRC_SEND: 'XRC Send', @@ -191,7 +343,7 @@ def qpts_to_str(qp_types): qp_types -= t if qp_types == 0: break - return qpts_str[:-2] + ' ({})'.format(numberic_types) + return qpts_str[:-2] + ' ({})'.format(numeric_types) def bitmask_to_str(bits, values): @@ -251,3 +403,37 @@ def tunnel_offloads_to_str(tun): dve.MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP:\ 'Ctrl word + MPLS over UDP'} return bitmask_to_str(tun, l) + + +def dc_type_to_str(dctype): + l = {dve.MLX5DV_DCTYPE_DCT: 'DCT', dve.MLX5DV_DCTYPE_DCI: 'DCI'} + try: + return l[dctype] + except KeyError: + return 'Unknown DC type ({dc})'.format(dc=dctype) + + +def qp_comp_mask_to_str(flags): + l = {dve.MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS: 'Create flags', + dve.MLX5DV_QP_INIT_ATTR_MASK_DC: 'DC', + dve.MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS: 'Send ops flags'} + return bitmask_to_str(flags, l) + + +def qp_create_flags_to_str(flags): + l = {dve.MLX5DV_QP_CREATE_TUNNEL_OFFLOADS: 'Tunnel offloads', + dve.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC: + 'Allow UC self loopback', + dve.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_MC: + 'Allow MC self loopback', + dve.MLX5DV_QP_CREATE_DISABLE_SCATTER_TO_CQE: 'Disable scatter to CQE', + dve.MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE: 'Allow scatter to CQE', + dve.MLX5DV_QP_CREATE_PACKET_BASED_CREDIT_MODE: + 'Packet based credit mode'} + return bitmask_to_str(flags, l) + + +def send_ops_flags_to_str(flags): + l = {dve.MLX5DV_QP_EX_WITH_MR_INTERLEAVED: 'With MR interleaved', + dve.MLX5DV_QP_EX_WITH_MR_LIST: 'With MR list'} + return bitmask_to_str(flags, l) diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd index 038a49111a3b..3f7f591e23a5 100644 --- a/pyverbs/providers/mlx5/mlx5dv_enums.pxd +++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd @@ -45,3 +45,24 @@ cdef extern from 'infiniband/mlx5dv.h': MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_SPI_STEERING = 1 << 2 MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_FULL_OFFLOAD = 1 << 3 MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_TX_IV_IS_ESN = 1 << 4 + + cpdef enum mlx5dv_qp_init_attr_mask: + MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS = 1 << 0 + MLX5DV_QP_INIT_ATTR_MASK_DC = 1 << 1 + MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS = 1 << 2 + + cpdef enum mlx5dv_qp_create_flags: + MLX5DV_QP_CREATE_TUNNEL_OFFLOADS = 1 << 0 + MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC = 1 << 1 + MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_MC = 1 << 2 + MLX5DV_QP_CREATE_DISABLE_SCATTER_TO_CQE = 1 << 3 + MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE = 1 << 4 + MLX5DV_QP_CREATE_PACKET_BASED_CREDIT_MODE = 1 << 5 + + cpdef enum mlx5dv_dc_type: + MLX5DV_DCTYPE_DCT = 1 + MLX5DV_DCTYPE_DCI = 2 + + cpdef enum mlx5dv_qp_create_send_ops_flags: + MLX5DV_QP_EX_WITH_MR_INTERLEAVED = 1 << 0 + MLX5DV_QP_EX_WITH_MR_LIST = 1 << 1 From patchwork Sun Nov 17 13:30:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8EFF71863 for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6D8D12084C for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="DebWZwvn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726177AbfKQNaw (ORCPT ); Sun, 17 Nov 2019 08:30:52 -0500 Received: from mail-eopbgr70084.outbound.protection.outlook.com ([40.107.7.84]:28158 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726157AbfKQNaw (ORCPT ); Sun, 17 Nov 2019 08:30:52 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mogX2760Vz6mmfIDBDIgtSkI+3UzbCBF3f0VAc1kdkJrsUu7Xs7/ZNzpzBJ+RJUKC5YQLJsdsio0VNzp03BGl5nngsdyx/+BWYLrYVt2YFU0g7sGBFT5gGihkmDUnuwug6zNtcQwGKA63uilZfyjSSD6oQUS1tvBHlloTZb6WKSyt4Q2IIAgzCBJ7LtOBIFrf9EgkQPAuBYHLw5ZA5vm+VA/Lx3kwRs+0kZ9SWquT/PsuzGWmiM8z1QgwM/aac6HrFg4Iz6o5vSfHq4x9lmF0lhcjhoy6MJvMwzG6V2uuGXqVPpoz7/+7Jk+7KLYQw6nNnq2WTgRN/iwb7vkdBkCsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q7SIayCHYhC20fF9Qzm1ju2qFQXjWxBlEGRdL01Kx4c=; b=ecz89g8bLyk4KLU0Wt+hUh2Jkm01gZEDgWICwy0sQlLlFZFpknw9eCBcxlJm6ZzZBhW7U4nOAmrJhzHxDmUr7ikrY0YhNsoWV7Pw2RyHdjT2cHHvEbKLIIH9FcWwPOpc9XhWxk0D+NyVhtljLGAMEisgYC1riP677dLl3gWwgei+/b5YSpAuXYJu4WxeIjxMB86kcTAxYvbu4R6b9yS1uB8lJvB99XcPBLBnZLTy5nCZ5lK8Shnmql5/DRo96nWItL2jr+yaHhkKI5d1wBkVS4yJL2YxVPodGVrunEGdcrDgLTrWmZyTbxyxg0i/HthTxWiwJKFwCbAi+ouzith8sQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q7SIayCHYhC20fF9Qzm1ju2qFQXjWxBlEGRdL01Kx4c=; b=DebWZwvn5+SJX/Vsa4Gbioq+C/JHx8Foa5yP+5mQdm/SUweCrOBbDCjuvcmD8r9f1x7Xb/IC+Q9wGWipcSqTnyTrbgrE3SPaY9W12p+AARtSOup1rwYO3Ny/tKbWo52wicjK9HvEmXyRPctkFng96DOTT9uGhizA7NNGWhYWup4= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:44 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:44 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation Thread-Topic: [PATCH rdma-core 3/7] pyverbs: Add default values for CQ creation Thread-Index: AQHVnUs2z/rCsrWdA02rkqgt36Zsig== Date: Sun, 17 Nov 2019 13:30:44 +0000 Message-ID: <20191117133030.10784-4-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 7191e802-4157-4e11-35a4-08d76b6258e8 x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(14444005)(54906003)(4744005)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: EuwbtuS4dmHucDuIWRu+SqSMR/tT0XxKKWBvlFsiD/tGKMf0XRxmMvgWQObY4rFi7Ffxdnqt8lGasS+NsPPkcmZ54eoDq6S1UXKuEL6L4GOjhL+XDmnH52ykpgigZ1cS3RRC5Qnd4cOaA8O0KeGi1eXAC9XtSaElqKGc/3RqoYy+/YHxj4iz1Q4Dzzk/GkgdIyd/aA2+31i2vpYCIP12+tQnRz8/zJ8zQmT8Tlw9DwNoQRBsdox0yo+TLQrTaDzAa+dnKMrPxahpT4M2kd9MNnIJgyhWpBagYRW+B6qjGe0GxgzzOeN9Xe1dxFXRlqqvizNnn09szPu56n2+zygY7+Is8YL7RjMrOlWLcXnnnfZ+XcwnV7O6EX9Zc3/zsYu3UJIUXHOhgxZBG/wjxCSZkoUjSnc1dbQNSu8r04KSMcEjbqu3VfV4JLy1CiDUs0C9 MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7191e802-4157-4e11-35a4-08d76b6258e8 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:44.1090 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EyfjmkEzzHSSJRgOI5jxWsCS9oHUhliejM4rxb9+szes1TEO6Z4wyjD8jDqI4KjoT/pcI82iFjAfQYuGIjH03g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org When creating a CQ, set CQ context and completion channel to None and completion vector to 0 by default, to make the simple CQ creation shorter. Signed-off-by: Noa Osherovich --- pyverbs/cq.pyx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pyverbs/cq.pyx b/pyverbs/cq.pyx index 6a96347ee979..ecfef6d7fdb9 100755 --- a/pyverbs/cq.pyx +++ b/pyverbs/cq.pyx @@ -67,8 +67,8 @@ cdef class CQ(PyverbsCM): A Completion Queue is the notification mechanism for work request completions. A CQ can have 0 or more associated QPs. """ - def __cinit__(self, Context context not None, cqe, cq_context, - CompChannel channel, comp_vector): + def __cinit__(self, Context context not None, cqe, cq_context=None, + CompChannel channel=None, comp_vector=0): """ Initializes a CQ object with the given parameters. :param context: The device's context on which to open the CQ From patchwork Sun Nov 17 13:30:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0286186D for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 95D5D2084C for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="a6xc4ezM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726183AbfKQNay (ORCPT ); Sun, 17 Nov 2019 08:30:54 -0500 Received: from mail-eopbgr70077.outbound.protection.outlook.com ([40.107.7.77]:21606 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726174AbfKQNay (ORCPT ); Sun, 17 Nov 2019 08:30:54 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dwlHb6kZ1PK2Xa4lGeSYCt1gjTVBF0Xf9RrFd1lhrQ8U276SXYoMRFA/02vQwNZ6xHTYXUSrfhG9/b88Os3gN0ojTDgsIEA5zj0RvaAZbbXbj4ZiAqzrn4jOp3tVffteB49Zt4uH1lgpil1ABz1LOKByMEoClqU7YBQZJs5f/CFC+F6EwgfjjEazkYpYxSfGKEUOGA96OttnIrmEoFQ0LnjecW2kdOixdCMXIwEuS0bMhDzb8t0gs5NT0nJW7RZcVeq4ClqYa4UvMQavCVhR+EKYiM5f3yOnk3WvM71WLYTSJIfWk7ykwHJ565ioT0EetmJWIg5Up5cW9yC9I1OaLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aUKXsDnEzb9OcPXJ4vaGdlfOm4th1l1XdoBIaPUPypc=; b=P5QI9+tdeVzfLmz4/g6wY2Aqygi8D3xzXV0oslnXmpND/pnB/ESGXdokRuUi7HY1JKmgLPhG3Z2RvnSTM47hCGNGMRL14rO9HrvRE6AsN4wSm0mATUrHEb9w3zZfeJVFHiHJ0CpQ7X+T9KXv8zIyMxDRPT7zF/APfyZURpWQr+0FBtndSyRAjUK5OpysUpHQXomAI2q3SptwrT2O1hAdLTqcyhT6mqY0rmvvqgRxZWO8cndvlsekb9jVJAUYbBFJaKKzvXsBcmynYuj8cGYQcspnjL1INXn4IgDfxEPN5rK1Hvy4u55X86Mk4YFa5V2mJrh/JeJXU86Ij5Idp9v7Gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aUKXsDnEzb9OcPXJ4vaGdlfOm4th1l1XdoBIaPUPypc=; b=a6xc4ezMLMmJzLk/V09m+dBkR/iFsQLkiAdyVvodePpX+hB++65tNNxQGLddQwgqX2xKow+AR+2471OADu8Lhc3fyI43/upjqSmGD7nMfLo7zwCKZctnujWX0wu2yn3B03NmRjb3geOjkHbPzURlpGRD6OPS7ZL7hkOxeKCDjew= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:45 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:45 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ Thread-Topic: [PATCH rdma-core 4/7] pyverbs: Add support for provider extended CQ Thread-Index: AQHVnUs3PS8eq+vKdkWnXmj9xjzAow== Date: Sun, 17 Nov 2019 13:30:44 +0000 Message-ID: <20191117133030.10784-5-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 1755bbe2-1a84-474d-e086-08d76b62596b x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nIibDxyixxv/H4CYowjCe6VSHVL/TfDf+kz8Kos0R8CP1tr48/UwJHNkncbbkLmUQektW0BrtcwPsdy8lAkcPXpbcYST32DgZ2h6jHo7cbmfayECs5xtxsB0dQZVVQH/oVBZdj+cX9ylao/qXX6cY90waw7JmDPoEikQxYGZAmPt9udv9Mn/LPpmeiJkckDzVLryDBhVmighkOfoIkIwT/Cxz6i4OBUTUv35TAORT269PjNOuLNBEGjXjihvjOdd5hmITs66QYcpSKKBFCcvu4bBa4UJBg9VUiNqnfwSR7KfWQcxqofSq0YvExP5Mr1bC5MafP0OgpooX1EIUFE9wN65tKX4QR36VbO3cvJvKD8JRIXyNxEuUU1gml1VmyV4y+8sDnZfjzxHJJMK9orhfN1Bprrzw4ltflrYFhODM37WT3nUCoHNcmUQmx6Iv9kS MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1755bbe2-1a84-474d-e086-08d76b62596b X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:44.9384 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: FzR6pi4a+Ly4QdoQMSghXeSepAc46CPzvhY1XcFoYMPwiTPc9jroGZatBZ3iPAU7K/Z8DAycZNHPTp3vauUUVg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org An extended CQ can be created using either the generic interface or a provider specific interface. Provider CQs extend the legacy extended CQ, which in Cython means that the legacy CQEX's __cinit__ will be called automatically. To avoid initialization via legacy interface, add **kwargs to CQEX's __cinit__(). If kwargs is not empty, the CQEX's __cinit__ will return without creating the CQ, leaving the initialization to the provider. Signed-off-by: Noa Osherovich --- pyverbs/cq.pyx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/pyverbs/cq.pyx b/pyverbs/cq.pyx index ecfef6d7fdb9..ac6baa618c3d 100755 --- a/pyverbs/cq.pyx +++ b/pyverbs/cq.pyx @@ -254,7 +254,8 @@ cdef class CqInitAttrEx(PyverbsObject): cdef class CQEX(PyverbsCM): - def __cinit__(self, Context context not None, CqInitAttrEx init_attr): + def __cinit__(self, Context context not None, CqInitAttrEx init_attr, + **kwargs): """ Initializes a CQEX object on the given device's context with the given attributes. @@ -262,6 +263,11 @@ cdef class CQEX(PyverbsCM): :param init_attr: Initial attributes that describe the CQ :return: The newly created CQEX on success """ + self.qps = weakref.WeakSet() + self.srqs = weakref.WeakSet() + if len(kwargs) > 0: + # Leave CQ initialization to the provider + return if init_attr is None: init_attr = CqInitAttrEx() self.cq = v.ibv_create_cq_ex(context.context, &init_attr.attr) @@ -272,8 +278,6 @@ cdef class CQEX(PyverbsCM): self.ibv_cq = v.ibv_cq_ex_to_cq(self.cq) self.context = context context.add_ref(self) - self.qps = weakref.WeakSet() - self.srqs = weakref.WeakSet() cdef add_ref(self, obj): if isinstance(obj, QP): From patchwork Sun Nov 17 13:30:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248369 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09A756C1 for ; Sun, 17 Nov 2019 13:30:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C862920855 for ; Sun, 17 Nov 2019 13:30:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="EQGPGcLQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726184AbfKQNay (ORCPT ); Sun, 17 Nov 2019 08:30:54 -0500 Received: from mail-eopbgr70084.outbound.protection.outlook.com ([40.107.7.84]:28158 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726151AbfKQNay (ORCPT ); Sun, 17 Nov 2019 08:30:54 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DMnADNJx8MC1B+WAvopaAEyxh2xnde/uMWW3MNrAhE2cmzMGrvvPJBpM5G8ikfC1FLIQuw/U0PcfjCZjdOCKfQBeHKOMEPdJea5epqhGycbrECFCJJxUVRSG8LLrk5OHh1a2NKEViE5XzZKBaCnpcRXyPVuJYsZnMktiwtJQCbRa9dwmhooM75gDnnL4mbgNSAgP+uf76rSSywTHnjCHiQDnZhCn5Ij7LK/PUKzG2PFxG4H2iQiRbOmTyknqS4OnMokwPqQAjSz5K3eAdQbj1beIJfknzV/7rrwP1PdMpL14I5dzyTBNLucNIbw2o5GRrBT/4pGzb3gaPfLv/yTEyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pgH329SK/w2fFHAN0xCJQZQNwlTyeuVFoXoCG1Nawsk=; b=lz4OaIZ9ctTP/koO6KPoH8hXq8jRTw5dSYh0cwNx78heAtBrqx4JF0HHuJEy0ia+CjcNkLUXxeFbkrE22iF7fO2sU5kJfPerVKYgIwgO48QgwLxrzOAGFi1qaUIIEJTARWSF5dX1l4QMANYbW4cZgVg4Rf/zkNEUHDxl904O5CrHlxvEBKxtUbijCVWaF3l52dlNdZBWvogIFD+JoKUiKdkLN5QIGYDO6rzHMqXveB14QJi9UYMkJvCUyUCEaCdJFnZTXKToe9PTH7SaF0rdgc8U+JyGqfjYC1WWvpBu1YvDEJ24bVrYZIu3xTtnINoeRKsQqJrs7NmS8FyFQgEMRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pgH329SK/w2fFHAN0xCJQZQNwlTyeuVFoXoCG1Nawsk=; b=EQGPGcLQIZ5Ii+zxAJsnByqLVpZHFKdTEBVOz538VXoSl4Tzyxunqrey1+XaY8NUXaLaNRcqIRD+yhnqovLZgdo9v//xKyM1vz359JTwi7WUGzT/Z5CzUrR21Apo2ZlajGJ7/u8r41/nPu9vaamILaon9pPojDRhUTrkYE2nTEU= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:45 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:45 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ Thread-Topic: [PATCH rdma-core 5/7] pyverbs/mlx5: Add support for mlx5 CQ Thread-Index: AQHVnUs3/8Fv5u9dJEigHceLDHNXGw== Date: Sun, 17 Nov 2019 13:30:45 +0000 Message-ID: <20191117133030.10784-6-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 1675ef01-9e9f-425a-2c7f-08d76b6259ee x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3276; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nCtKg9tS37ImdoaDmclouXxBRCUR/bDKZhwmvyfMPzHFlRGA0OsRHIWHBsFUvnPvuw06ajyZQO3w23P795fG9jJwSp8FkpMaKuegHXXi1uJ6DVZ+LaAAywCi3Kls4LIHb7yXfbjfN/PZoUc9liBTMYpsnZ+wxlngg7fpHxARQv94y4HseM7zZlPPxlstfvEmoUitywOxpnyjUn5sCNvtzKf6Jv2GpiOM7LXsuNicTXnZEu0o0oFJcJKJ/OajwuicqIsPG69+4G93XMLzTQDjJK0+zgKluC8ddIizSLpb0WXEkPsSHTjZWS79Dm0TaNc9gHZueu8iWDpOZ18SL1VxiGLakbjDZexkkxEv9CDw4yZ2vcqGFu1MOjJVd1EW8K5cRz4gR4z9WeFiPzn9hl6hCnNi3uyd6rUur4jiuGEal4Xx1hCETHdM/GhHWeenp4QP MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1675ef01-9e9f-425a-2c7f-08d76b6259ee X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:45.8213 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LzyHO2dXAFayySJRwwWZGYAFfzj7+hLE9f/5g5OFMGDBZ0vpHKCuP2TfDcMTUJpuf5gWsj2rrbiVwJ6QiB7/aA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add the needed infrastructure to allow users to create a CQ via the mlx5dv interface. Creation process requires CQInitAttrEx, similarly to the creation of an extended CQ, but requires an Mlx5DVCQInitAttr object as well (provided as kwargs, i.e. dv_init_attr=). Signed-off-by: Noa Osherovich --- pyverbs/providers/mlx5/libmlx5.pxd | 9 +++ pyverbs/providers/mlx5/mlx5dv.pxd | 7 ++ pyverbs/providers/mlx5/mlx5dv.pyx | 103 ++++++++++++++++++++++++ pyverbs/providers/mlx5/mlx5dv_enums.pxd | 9 +++ 4 files changed, 128 insertions(+) diff --git a/pyverbs/providers/mlx5/libmlx5.pxd b/pyverbs/providers/mlx5/libmlx5.pxd index 2c5be241a3ec..f471f2453e38 100644 --- a/pyverbs/providers/mlx5/libmlx5.pxd +++ b/pyverbs/providers/mlx5/libmlx5.pxd @@ -52,6 +52,12 @@ cdef extern from 'infiniband/mlx5dv.h': mlx5dv_dc_init_attr dc_init_attr unsigned long send_ops_flags + cdef struct mlx5dv_cq_init_attr: + unsigned long comp_mask + unsigned char cqe_comp_res_format + unsigned int flags + unsigned short cqe_size + bool mlx5dv_is_supported(v.ibv_device *device) v.ibv_context* mlx5dv_open_device(v.ibv_device *device, mlx5dv_context_attr *attr) @@ -59,3 +65,6 @@ cdef extern from 'infiniband/mlx5dv.h': v.ibv_qp *mlx5dv_create_qp(v.ibv_context *context, v.ibv_qp_init_attr_ex *qp_attr, mlx5dv_qp_init_attr *mlx5_qp_attr) + v.ibv_cq_ex *mlx5dv_create_cq(v.ibv_context *context, + v.ibv_cq_init_attr_ex *cq_attr, + mlx5dv_cq_init_attr *mlx5_cq_attr) diff --git a/pyverbs/providers/mlx5/mlx5dv.pxd b/pyverbs/providers/mlx5/mlx5dv.pxd index 918065012d30..cc9c89d82fa6 100644 --- a/pyverbs/providers/mlx5/mlx5dv.pxd +++ b/pyverbs/providers/mlx5/mlx5dv.pxd @@ -6,6 +6,7 @@ cimport pyverbs.providers.mlx5.libmlx5 as dv from pyverbs.base cimport PyverbsObject from pyverbs.device cimport Context +from pyverbs.cq cimport CQEX from pyverbs.qp cimport QP @@ -26,3 +27,9 @@ cdef class Mlx5DVQPInitAttr(PyverbsObject): cdef class Mlx5QP(QP): cdef object dc_type + +cdef class Mlx5DVCQInitAttr(PyverbsObject): + cdef dv.mlx5dv_cq_init_attr attr + +cdef class Mlx5CQ(CQEX): + pass diff --git a/pyverbs/providers/mlx5/mlx5dv.pyx b/pyverbs/providers/mlx5/mlx5dv.pyx index b4a971a5b935..e4500567ba23 100644 --- a/pyverbs/providers/mlx5/mlx5dv.pyx +++ b/pyverbs/providers/mlx5/mlx5dv.pyx @@ -7,6 +7,8 @@ cimport pyverbs.providers.mlx5.libmlx5 as dv from pyverbs.base import PyverbsRDMAErrno cimport pyverbs.libibverbs_enums as e from pyverbs.qp cimport QPInitAttrEx +from pyverbs.cq cimport CqInitAttrEx +cimport pyverbs.libibverbs as v from pyverbs.pd cimport PD cdef class Mlx5DVContextAttr(PyverbsObject): @@ -331,6 +333,107 @@ cdef class Mlx5QP(QP): return masks[self.dc_type][dst] | e.IBV_QP_STATE +cdef class Mlx5DVCQInitAttr(PyverbsObject): + """ + Represents mlx5dv_cq_init_attr struct, initial attributes used for mlx5 CQ + creation. + """ + def __cinit__(self, comp_mask=0, cqe_comp_res_format=0, flags=0, cqe_size=0): + """ + Initializes an Mlx5CQInitAttr object with zeroes as default values. + :param comp_mask: Marks which of the following fields should be + considered. Use mlx5dv_cq_init_attr_mask enum. + :param cqe_comp_res_format: The various CQE response formats of the + responder side. Use + mlx5dv_cqe_comp_res_format enum. + :param flags: A bitwise OR of the various values described in + mlx5dv_cq_init_attr_flags. + :param cqe_size: Configure the CQE size to be 64 or 128 bytes, other + values will cause the CQ creation process to fail. + Valid when MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE is set. + :return: None + """ + self.attr.comp_mask = comp_mask + self.attr.cqe_comp_res_format = cqe_comp_res_format + self.attr.flags = flags + self.attr.cqe_size = cqe_size + + @property + def comp_mask(self): + return self.attr.comp_mask + @comp_mask.setter + def comp_mask(self, val): + self.attr.comp_mask = val + + @property + def cqe_comp_res_format(self): + return self.attr.cqe_comp_res_format + @cqe_comp_res_format.setter + def cqe_comp_res_format(self, val): + self.attr.cqe_comp_res_format = val + + @property + def flags(self): + return self.attr.flags + @flags.setter + def flags(self, val): + self.attr.flags = val + + @property + def cqe_size(self): + return self.attr.cqe_size + @cqe_size.setter + def cqe_size(self, val): + self.attr.cqe_size = val + + def __str__(self): + print_format = '{:22}: {:<20}\n' + flags = {dve.MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD: + "MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD}"} + mask = {dve.MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE: + "MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE", + dve.MLX5DV_CQ_INIT_ATTR_MASK_FLAGS: + "MLX5DV_CQ_INIT_ATTR_MASK_FLAGS", + dve.MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE: + "MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE"} + fmt = {dve.MLX5DV_CQE_RES_FORMAT_HASH: "MLX5DV_CQE_RES_FORMAT_HASH", + dve.MLX5DV_CQE_RES_FORMAT_CSUM: "MLX5DV_CQE_RES_FORMAT_CSUM", + dve.MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX: + "MLX5DV_CQE_RES_FORMAT_CSUM_STRIDX"} + + return 'Mlx5DVCQInitAttr:\n' +\ + print_format.format('comp_mask', bitmask_to_str(self.comp_mask, + mask)) +\ + print_format.format('CQE compression format', + bitmask_to_str(self.cqe_comp_res_format, + fmt)) +\ + print_format.format('flags', bitmask_to_str(self.flags, + flags)) + \ + print_format.format('CQE size', self.cqe_size) + + +cdef class Mlx5CQ(CQEX): + def __cinit__(self, Mlx5Context context, CqInitAttrEx init_attr, + Mlx5DVCQInitAttr dv_init_attr): + self.cq = \ + dv.mlx5dv_create_cq(context.context, &init_attr.attr, + &dv_init_attr.attr if dv_init_attr is not None + else NULL) + if self.cq == NULL: + raise PyverbsRDMAErrno('Failed to create MLX5 CQ.\nCQInitAttrEx:\n' + '{}\nMLX5DVCQInitAttr:\n{}'. + format(init_attr, dv_init_attr)) + self.ibv_cq = v.ibv_cq_ex_to_cq(self.cq) + self.context = context + context.add_ref(self) + + def __str__(self): + print_format = '{:<22}: {:<20}\n' + return 'Mlx5 CQ:\n' +\ + print_format.format('Handle', self.cq.handle) +\ + print_format.format('CQEs', self.cq.cqe) + + def qpts_to_str(qp_types): numeric_types = qp_types qpts_str = '' diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd index 3f7f591e23a5..3859dff81ad1 100644 --- a/pyverbs/providers/mlx5/mlx5dv_enums.pxd +++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd @@ -66,3 +66,12 @@ cdef extern from 'infiniband/mlx5dv.h': cpdef enum mlx5dv_qp_create_send_ops_flags: MLX5DV_QP_EX_WITH_MR_INTERLEAVED = 1 << 0 MLX5DV_QP_EX_WITH_MR_LIST = 1 << 1 + + cpdef enum mlx5dv_cq_init_attr_mask: + MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE = 1 << 0 + MLX5DV_CQ_INIT_ATTR_MASK_FLAGS = 1 << 1 + MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE = 1 << 2 + + cpdef enum mlx5dv_cq_init_attr_flags: + MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD = 1 << 0 + MLX5DV_CQ_INIT_ATTR_FLAGS_RESERVED = 1 << 1 From patchwork Sun Nov 17 13:30:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248371 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8CE751593 for ; Sun, 17 Nov 2019 13:30:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5BADC20862 for ; Sun, 17 Nov 2019 13:30:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="lKROYRAG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726151AbfKQNa4 (ORCPT ); Sun, 17 Nov 2019 08:30:56 -0500 Received: from mail-eopbgr70077.outbound.protection.outlook.com ([40.107.7.77]:21606 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726157AbfKQNa4 (ORCPT ); Sun, 17 Nov 2019 08:30:56 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Oi2HPzx3mTQqm0ZaFh05ATArTz+/zycf8hl3XkUrBqAA+n1bHoI/5BzpQWMF8nSK9pHWWiblH3OfZrHzPUF4uaLplyd8V8Pt2UvEjymJUJA2dfriuyntNzOlv2bziZ3uSg5pfvL5muu19n0AQwr6W/nl2oVqoHh9o0/0hb2pAK7goW9pRe9dXsokdzB8VltRutaBX5iCRQcz25p7cgkX9wuf6lV+tJOml0kRNNrV5tQOIozPY2y6jcH+z6ieBncmJIkcz4454Zba3HD3oXKN71jisUqoMHwLQSco+jPEuxvsgNA085XhXsyxE/WzyAYi3A58wA7+EeKW8a6bPWbgZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=skj3jX3LRZaWmX5exEsEoXxMxhb+cRBU2uU6bl3S5KA=; b=Za9Ay06rUAohV63tWsrqRc8f1YpjM8bMhUCBd9lsMn9Hpet+7E0Koo0jPPY2YLOSv14K4Mi4QvDfDSk7gC3IjEi/hrXIF2DtzqyYF6gQ018xN84TI8C5/ds1lGeT2ETHHA4HnVnoPEK2FNcuAuunSM8wNBA5AnsVtFjk/gZl0r5SxdQbYSYWdBjs/Z6srofXRbRXZEkoLBjQrnEj+rNtWEhpcfmG0i0+dykslQwhTFfIFTIPaJjNn6hZezuWn2rJksBQ8iVA1CktY3thxXzsmCZKRlcl7N5TBZKvAxUtEce4W98ChZ5lkaSrzCc3jBGNp1theYUJilQO4cR486LsHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=skj3jX3LRZaWmX5exEsEoXxMxhb+cRBU2uU6bl3S5KA=; b=lKROYRAGRw6OhKlk3bu+FIfvfoe99J4aCdBoCIeSTy/yJTc+fLwkJduKh8UVtRBFmJBrPSPLh1EO5B4G8qvH//4jvNMoXZtWPy9OGA2JmA/JWfq5/TB8ifQsa5lM44PMR8PIbP6MWZrvQlVXO+KaK+n05SqjkODTOGvcAQoutOM= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:46 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:46 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation Thread-Topic: [PATCH rdma-core 6/7] Documentation: Add mlx5 provider to documentation Thread-Index: AQHVnUs4lquWkLu+ikG2y0vMD6U/AA== Date: Sun, 17 Nov 2019 13:30:46 +0000 Message-ID: <20191117133030.10784-7-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 5c283600-d8f4-4f0c-49c0-08d76b625a73 x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(14444005)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9g0E2EaNGmB5cLfMJLLyFoPCH/SrMzXuHJjRiLZ4RXu7jw5SGhVg6XCqlEvqs7sN3VOYDJGrUAE99QgK4MUE0zT6a9swMXh625M+DdkSxJyHWCXR1iCYQvCCIzEVB96rEdRatFjhpK1i8WwJIwwoC7B1rcmJbdQujpoxJWlQVCSzo0YEsH1DJv8gZaIj9/YM8SnVh/EsOzEDas4dr/RvY2gtewn1B9kjczomsJKnRESLwGu+4wGbgj1v/F8e4IIhNaBDGY+ROOmCPG+FgLbSMQBTt0V8kxcCOSQeHnpXCpKHrl4T32RrDgCiBWB6UcNN5ZqOzvVhnDy9orvatysi5zAX/KR7jF7whUa56KZi9uRC3oslyFZd1SJbUzOOpeAJcxqYv0tRCXXAk0mHeQMwuXXbKtt8xzz/W22X5aLdEqYPmqLxn20iB9GoOr3YRUse MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5c283600-d8f4-4f0c-49c0-08d76b625a73 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:46.6742 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4iWPg+XxU2+oIaNi+yjTpVpG97nmohz28wePubH9lLrfzQ0i2cw7ex9g8gqfvnRrg9XDxtYK0Wz/mEFTKVkFjw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add code snippets to show: - Opening an mlx5 device - Querying an mlx5 device - Creating an mlx5 QP (DCI and Raw Packet QP with mlx5-specific capabilties) - Creating an mlx5 CQ Signed-off-by: Noa Osherovich --- Documentation/pyverbs.md | 105 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md index 7e81690a064a..e405ab53b423 100755 --- a/Documentation/pyverbs.md +++ b/Documentation/pyverbs.md @@ -389,6 +389,111 @@ srq_attr.cq = cq srq_attr.comp_mask = e.IBV_SRQ_INIT_ATTR_TYPE | e.IBV_SRQ_INIT_ATTR_PD | \ e.IBV_SRQ_INIT_ATTR_CQ | e.IBV_SRQ_INIT_ATTR_XRCD srq = SRQ(ctx, srq_attr) + + +##### Open an mlx5 provider +A provider is essentially a Context with driver-specific extra features. As +such, it inherits from Context. In legcay flow Context iterates over the IB +devices and opens the one matches the name given by the user (name= argument). +When provider attributes are also given (attr=), the Context will assign the +relevant ib_device to its device member, so that the provider will be able to +open the device in its specific way as demonstated below: + +```python +import pyverbs.providers.mlx5.mlx5dv as m +from pyverbs.pd import PD +attr = m.Mlx5DVContextAttr() # Default values are fine +ctx = m.Mlx5Context(attr=attr, name='rocep0s8f0') +# The provider context can be used as a regular Context, e.g.: +pd = PD(ctx) # Success +``` + +##### Query an mlx5 provider +After opening an mlx5 provider, users can use the device-specific query for +non-legacy attributes. The following snippet demonstrates how to do that. +```python +import pyverbs.providers.mlx5.mlx5dv as m +ctx = m.Mlx5Context(attr=m.Mlx5DVContextAttr(), name='ibp0s8f0') +mlx5_attrs = ctx.query_mlx5_device() +print(mlx5_attrs) +Version : 0 +Flags : CQE v1, Support CQE 128B compression, Support CQE 128B padding, Support packet based credit mode (in RC QP) +comp mask : CQE compression, SW parsing, Striding RQ, Tunnel offloads, Dynamic BF regs, Clock info update, Flow action flags +CQE compression caps: + max num : 64 + supported formats : with hash, with RX checksum CSUM, with stride index +SW parsing caps: + SW parsing offloads : + supported QP types : +Striding RQ caps: + min single stride log num of bytes: 6 + max single stride log num of bytes: 13 + min single wqe log num of strides: 9 + max single wqe log num of strides: 16 + supported QP types : Raw Packet +Tunnel offloads caps: +Max dynamic BF registers: 1024 +Max clock info update [nsec]: 1099511 +Flow action flags : 0 +``` + +##### Create an mlx5 QP +Using an Mlx5Context object, one can create either a legacy QP (creation +process is the same) or an mlx5 QP. An mlx5 QP is a QP by inheritance but its +constructor receives a keyword argument named `dv_init_attr`. If the user +provides it, the QP will be created using `mlx5dv_create_qp` rather than +`ibv_create_qp_ex`. The following snippet demonstrates how to create both a DC +(dynamically connected) QP and a Raw Packet QP which uses mlx5-specific +capabilities, unavailable using the legacy interface. Currently, pyverbs +supports only creation of a DCI. DCT support will be added in one of the +following PRs. +```python +from pyverbs.providers.mlx5.mlx5dv import Mlx5Context, Mlx5DVContextAttr +from pyverbs.providers.mlx5.mlx5dv import Mlx5DVQPInitAttr, Mlx5QP +import pyverbs.providers.mlx5.mlx5_enums as me +from pyverbs.qp import QPInitAttrEx, QPCap +import pyverbs.enums as e +from pyverbs.cq import CQ +from pyverbs.pd import PD + +with Mlx5Context(name='rocep0s8f0', attr=Mlx5DVContextAttr()) as ctx: + with PD(ctx) as pd: + with CQ(ctx, 100) as cq: + cap = QPCap(100, 0, 1, 0) + # Create a DC QP of type DCI + qia = QPInitAttrEx(cap=cap, pd=pd, scq=cq, qp_type=e.IBV_QPT_DRIVER, + comp_mask=e.IBV_QP_INIT_ATTR_PD, rcq=cq) + attr = Mlx5DVQPInitAttr(comp_mask=me.MLX5DV_QP_INIT_ATTR_MASK_DC) + attr.dc_type = me.MLX5DV_DCTYPE_DCI + + dci = Mlx5QP(ctx, qia, dv_init_attr=attr) + + # Create a Raw Packet QP using mlx5-specific capabilities + qia.qp_type = e.IBV_QPT_RAW_PACKET + attr.comp_mask = me.MLX5DV_QP_INIT_ATTR_MASK_QP_CREATE_FLAGS + attr.create_flags = me.MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE |\ + me.MLX5DV_QP_CREATE_TIR_ALLOW_SELF_LOOPBACK_UC |\ + me.MLX5DV_QP_CREATE_TUNNEL_OFFLOADS + qp = Mlx5QP(ctx, qia, dv_init_attr=attr) +``` + +##### Create an mlx5 CQ +Mlx5Context also allows users to create an mlx5 specific CQ. The Mlx5CQ inherits +from CQEX, but its constructor receives 3 parameters instead of 2. The 3rd +parameter is a keyword argument named `dv_init_attr`. If provided by the user, +the CQ will be created using `mlx5dv_create_cq`. +The following snippet shows this simple creation process. +```python +from pyverbs.providers.mlx5.mlx5dv import Mlx5Context, Mlx5DVContextAttr +from pyverbs.providers.mlx5.mlx5dv import Mlx5DVCQInitAttr, Mlx5CQ +import pyverbs.providers.mlx5.mlx5_enums as me +from pyverbs.cq import CqInitAttrEx + +with Mlx5Context(name='rocep0s8f0', attr=Mlx5DVContextAttr()) as ctx: + cqia = CqInitAttrEx() + mlx5_cqia = Mlx5DVCQInitAttr(comp_mask=me.MLX5DV_CQ_INIT_ATTR_MASK_COMPRESSED_CQE, + cqe_comp_res_format=me.MLX5DV_CQE_RES_FORMAT_CSUM) + cq = Mlx5CQ(ctx, cqia, dv_init_attr=mlx5_cqia) ``` ##### CMID From patchwork Sun Nov 17 13:30:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11248373 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE4DF6C1 for ; Sun, 17 Nov 2019 13:30:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 953DD2084C for ; Sun, 17 Nov 2019 13:30:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="EuI1i11V" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726157AbfKQNa4 (ORCPT ); Sun, 17 Nov 2019 08:30:56 -0500 Received: from mail-eopbgr70084.outbound.protection.outlook.com ([40.107.7.84]:28158 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726174AbfKQNa4 (ORCPT ); Sun, 17 Nov 2019 08:30:56 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XgX59edmJaUT3ITOCGM/94JH+CBxeOraXYCdObCv/6MweB7AqfTVmbfFteZkIYDES9SMi1H2nuZgykIWrEpfVRApRvc9tWmw9RTcswLeExlGFWiqvPZt5xfaxyGzZMH5pufmAan+JpeSKmQj90cGGR9f/F/SSwMGzbvXvkNKFmT01Q/JOihhu+ghEepRBieairaahNHKsrimYBiM0hjLPumUUFliGGBtRg0Ga+uz9MoN0pHkuVBFquiUpioVTjL6r5Yszgq33faSAxIhIEGZk1gZEW+Wl1gW8vZk66enigXgvD1/yaC32TJ6Fyp8LF2oST56xbDDnUr7HA7nzgtYzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XTU2iS7FimODVNjIUMdlN9MslOwrf/ny+FLNs6Oa8no=; b=ku5mUaVc/Zig2YaqZ6LCPMdzlTeuYi1V3vhAXqhGQ8/StWSA3RzYrAHzmZcefTAJLpWGcKVJw7m6mK6FW6mmGAb1k2SmU33eInH9n2NS/AFVN1ZR2BUFMmvpVTKV61WrF7WiEYEy5t142GDH4ncrG2Y6oS5XQupLzya2Mr8tX1NWkN1/xhHND17WOQ6wJtUtN3UpamWFCDpJiaj0Go4MwQY05dJoPQeE6x9Zjgp+WDd5aUCI1FI2jeRE70n2MI432XIjIeRV2gQdWAwVhQRjA+PEAUtj6p4MQa+sipmx05ci00DQ5PZ1DVayRn+bsT0NjbiHa5BJWE3idyODQY0Jew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XTU2iS7FimODVNjIUMdlN9MslOwrf/ny+FLNs6Oa8no=; b=EuI1i11VOottmT74DukgSo9odoQcRWKKdkhx9CTSf2rvs28e18Z1YuOxKfOmtZUPX7yuglwqviAmCjsSePkX9DcThJirvc599maRHAyeWYUz9xKZj1u/pIMOjk3MOcFJo/GG0i9sutQipYixVmnXBm/aIezjREzwX7NwAV+g4U0= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6598.eurprd05.prod.outlook.com (20.179.3.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Sun, 17 Nov 2019 13:30:47 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2451.029; Sun, 17 Nov 2019 13:30:47 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Noa Osherovich Subject: [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ Thread-Topic: [PATCH rdma-core 7/7] tests: Add traffic tests using extended CQ Thread-Index: AQHVnUs42Nt8N1GNE0yWI7VU8Tbn0A== Date: Sun, 17 Nov 2019 13:30:47 +0000 Message-ID: <20191117133030.10784-8-noaos@mellanox.com> References: <20191117133030.10784-1-noaos@mellanox.com> In-Reply-To: <20191117133030.10784-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR01CA0088.eurprd01.prod.exchangelabs.com (2603:10a6:208:10e::29) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 32f9babe-1ac5-418f-400c-08d76b625af8 x-ms-traffictypediagnostic: AM6PR05MB6598:|AM6PR05MB6598: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:144; x-forefront-prvs: 02243C58C6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(189003)(199004)(71200400001)(71190400001)(102836004)(6116002)(386003)(6506007)(256004)(11346002)(3846002)(2616005)(26005)(36756003)(446003)(5660300002)(86362001)(8936002)(186003)(64756008)(66556008)(66476007)(66946007)(7736002)(50226002)(66446008)(305945005)(66066001)(2501003)(81166006)(81156014)(8676002)(99286004)(6636002)(25786009)(76176011)(478600001)(14454004)(1076003)(2906002)(6436002)(6486002)(14444005)(54906003)(6512007)(476003)(107886003)(110136005)(52116002)(486006)(316002)(4326008);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6598;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: dtMhjGTfLYewdqOYN+1qAqyRKFnsaukLpzoaX5ajnENFA/YxI9BJMU++dpDtMbvvGXJZtypnZqg7qYIxdchu+2j+o/3zvI+ftalFoDa9mz4wCyk8SiAJEj/FWqnEBVSDEv3i1B8lTTwGF1ZHZgb8ynh94gx1HUapoIvhvX+KKaQX8xSb6Kg+iX6tcyBeuI8+3KWnaR9o3cN10dwArzkKjpHxVgES2auV5HZzVMim7WPrmOGeJ6fsJya9dCyWDv2OusoclzH6EHTCskbsd0KklOuKTUHjfoXJkZnMQrArNcR29q76JYV3t7LSGJIAkwJ+ldPQa5TF8ko4vtO89bVI8T2ByXbxhHArSPyIae2dsDiCFO2ol/9W+RldJvhVzhANK2+vaSK8mHAXPpLRHi1OEDosoAyvVfm9hhb9U2w8sf/c7VngJo3BF/4HYvpl987L MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 32f9babe-1ac5-418f-400c-08d76b625af8 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2019 13:30:47.5466 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: kDs8sRYFiOEfWr53J9uEyD30Dgb0HivHOBwXMtescGnL9kYmI3XfoT3YqYgaHVek352k4VNxea2tQEDZ8gtpfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6598 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add UD/RC/XRC traffic tests which use an extended CQ instead of the legacy one. Signed-off-by: Noa Osherovich --- pyverbs/libibverbs_enums.pxd | 3 ++ tests/CMakeLists.txt | 1 + tests/test_cqex.py | 75 ++++++++++++++++++++++++++++++++++++ tests/utils.py | 56 ++++++++++++++++++++++----- 4 files changed, 125 insertions(+), 10 deletions(-) create mode 100644 tests/test_cqex.py diff --git a/pyverbs/libibverbs_enums.pxd b/pyverbs/libibverbs_enums.pxd index 114915d0a751..4bf77d3f14f7 100755 --- a/pyverbs/libibverbs_enums.pxd +++ b/pyverbs/libibverbs_enums.pxd @@ -398,6 +398,9 @@ cdef extern from '': IBV_XRCD_INIT_ATTR_OFLAGS IBV_XRCD_INIT_ATTR_RESERVED + cpdef enum: + IBV_WC_STANDARD_FLAGS + cdef unsigned long long IBV_DEVICE_RAW_SCATTER_FCS cdef unsigned long long IBV_DEVICE_PCI_WRITE_END_PADDING diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 0d81d1a98fb7..7c6be35dc58a 100755 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -7,6 +7,7 @@ rdma_python_test(tests test_addr.py base.py test_cq.py + test_cqex.py test_device.py test_mr.py test_pd.py diff --git a/tests/test_cqex.py b/tests/test_cqex.py new file mode 100644 index 000000000000..df135a991c15 --- /dev/null +++ b/tests/test_cqex.py @@ -0,0 +1,75 @@ +from pyverbs.cq import CqInitAttrEx, CQEX +import pyverbs.enums as e +from pyverbs.mr import MR + +from tests.base import RCResources, UDResources, XRCResources, RDMATestCase +import tests.utils as u + + +def create_ex_cq(res): + """ + Create an Extended CQ using res's context and assign it to res's cq member. + IBV_WC_STANDARD_FLAGS is used for WC flags to avoid support differences + between devices. + :param res: An instance of TrafficResources + """ + wc_flags = e.IBV_WC_STANDARD_FLAGS + cia = CqInitAttrEx(cqe=2000, wc_flags=wc_flags) + res.cq = CQEX(res.ctx, cia) + + +class CqExUD(UDResources): + def create_cq(self): + create_ex_cq(self) + + def create_mr(self): + self.mr = MR(self.pd, self.msg_size + self.GRH_SIZE, + e.IBV_ACCESS_LOCAL_WRITE) + + +class CqExRC(RCResources): + def create_cq(self): + create_ex_cq(self) + + +class CqExXRC(XRCResources): + def create_cq(self): + create_ex_cq(self) + + +class CqExTestCase(RDMATestCase): + """ + Run traffic over the existing UD, RC and XRC infrastructure, but use + ibv_cq_ex instead of legacy ibv_cq + """ + def setUp(self): + super().setUp() + self.iters = 100 + self.qp_dict = {'ud': CqExUD, 'rc': CqExRC, 'xrc': CqExXRC} + + def create_players(self, qp_type): + client = self.qp_dict[qp_type](self.dev_name, self.ib_port, + self.gid_index) + server = self.qp_dict[qp_type](self.dev_name, self.ib_port, + self.gid_index) + if qp_type == 'xrc': + client.pre_run(server.psns, server.qps_num) + server.pre_run(client.psns, client.qps_num) + else: + client.pre_run(server.psn, server.qpn) + server.pre_run(client.psn, client.qpn) + return client, server + + def test_ud_traffic_cq_ex(self): + client, server = self.create_players('ud') + u.traffic(client, server, self.iters, self.gid_index, self.ib_port, + is_cq_ex=True) + + def test_rc_traffic_cq_ex(self): + client, server = self.create_players('rc') + u.traffic(client, server, self.iters, self.gid_index, self.ib_port, + is_cq_ex=True) + + def test_xrc_traffic_cq_ex(self): + client, server = self.create_players('xrc') + u.xrc_traffic(client, server, is_cq_ex=True) diff --git a/tests/utils.py b/tests/utils.py index 785309552e25..04a988a531e4 100755 --- a/tests/utils.py +++ b/tests/utils.py @@ -13,6 +13,7 @@ from pyverbs.addr import AHAttr, AH, GlobalRoute from pyverbs.wr import SGE, SendWR, RecvWR from pyverbs.qp import QPCap, QPInitAttrEx from tests.base import XRCResources +from pyverbs.cq import PollCqAttr import pyverbs.device as d import pyverbs.enums as e @@ -342,6 +343,37 @@ def poll_cq(cq, count=1): return wcs +def poll_cq_ex(cqex, count=1): + """ + Poll completions from the extended CQ. + :param cq: CQEX to poll from + :param count: How many completions to poll + :return: None + """ + poll_attr = PollCqAttr() + ret = cqex.start_poll(poll_attr) + while ret == 2: # ENOENT + ret = cqex.start_poll(poll_attr) + if ret != 0: + raise PyverbsRDMAErrno('Failed to poll CQ') + count -= 1 + if cqex.status != e.IBV_WC_SUCCESS: + raise PyverbsRDMAErrno('Completion status is {s}'. + format(s=cqex.status)) + # Now poll the rest of the packets + while count > 0: + ret = cqex.poll_next() + while ret == 2: + ret = cqex.poll_next() + if ret != 0: + raise PyverbsRDMAErrno('Failed to poll CQ') + if cqex.status != e.IBV_WC_SUCCESS: + raise PyverbsRDMAErrno('Completion status is {s}'. + format(s=cqex.status)) + count -= 1 + cqex.end_poll() + + def validate(received_str, is_server, msg_size): """ Validates the received buffer against the expected result. @@ -366,7 +398,7 @@ def validate(received_str, is_server, msg_size): format(exp=expected_str, rcv=received_str)) -def traffic(client, server, iters, gid_idx, port): +def traffic(client, server, iters, gid_idx, port, is_cq_ex=False): """ Runs basic traffic between two sides :param client: client side, clients base class is BaseTraffic @@ -374,8 +406,10 @@ def traffic(client, server, iters, gid_idx, port): :param iters: number of traffic iterations :param gid_idx: local gid index :param port: IB port + :param is_cq_ex: If True, use poll_cq_ex() rather than poll_cq() :return: """ + poll = poll_cq_ex if is_cq_ex else poll_cq s_recv_wr = get_recv_wr(server) c_recv_wr = get_recv_wr(client) post_recv(client.qp, c_recv_wr, client.num_msgs) @@ -383,21 +417,21 @@ def traffic(client, server, iters, gid_idx, port): for _ in range(iters): c_send_wr = get_send_wr(client, False) post_send(client, c_send_wr, gid_idx, port) - poll_cq(client.cq) - poll_cq(server.cq) + poll(client.cq) + poll(server.cq) post_recv(client.qp, c_recv_wr) msg_received = server.mr.read(server.msg_size, 0) validate(msg_received, True, server.msg_size) s_send_wr = get_send_wr(server, True) post_send(server, s_send_wr, gid_idx, port) - poll_cq(server.cq) - poll_cq(client.cq) + poll(server.cq) + poll(client.cq) post_recv(server.qp, s_recv_wr) msg_received = client.mr.read(client.msg_size, 0) validate(msg_received, False, client.msg_size) -def xrc_traffic(client, server): +def xrc_traffic(client, server, is_cq_ex=False): """ Runs basic xrc traffic, this function assumes that number of QPs, which server and client have are equal, server.send_qp[i] is connected to @@ -408,8 +442,10 @@ def xrc_traffic(client, server): of XRCResources class :param server: Aggregation object of the passive side, should be an instance of XRCResources class + :param is_cq_ex: If True, use poll_cq_ex() rather than poll_cq() :return: None """ + poll = poll_cq_ex if is_cq_ex else poll_cq client_srqn = client.srq.get_srq_num() server_srqn = server.srq.get_srq_num() s_recv_wr = get_recv_wr(server) @@ -421,15 +457,15 @@ def xrc_traffic(client, server): c_send_wr = get_send_wr(client, False) c_send_wr.set_qp_type_xrc(server_srqn) client.sqp_lst[i].post_send(c_send_wr) - poll_cq(client.cq) - poll_cq(server.cq) + poll(client.cq) + poll(server.cq) msg_received = server.mr.read(server.msg_size, 0) validate(msg_received, True, server.msg_size) s_send_wr = get_send_wr(server, True) s_send_wr.set_qp_type_xrc(client_srqn) server.sqp_lst[i].post_send(s_send_wr) - poll_cq(server.cq) - poll_cq(client.cq) + poll(server.cq) + poll(client.cq) msg_received = client.mr.read(client.msg_size, 0) validate(msg_received, False, client.msg_size)