From patchwork Thu Nov 14 09:37:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11243265 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEBE314ED for ; Thu, 14 Nov 2019 09:37:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B6DC820718 for ; Thu, 14 Nov 2019 09:37:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="PDHmlfnx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725977AbfKNJht (ORCPT ); Thu, 14 Nov 2019 04:37:49 -0500 Received: from mail-eopbgr60075.outbound.protection.outlook.com ([40.107.6.75]:2435 "EHLO EUR04-DB3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726369AbfKNJhs (ORCPT ); Thu, 14 Nov 2019 04:37:48 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EKfyTxbjR5DUFIQx38VWhlMmaeE9GryhegVWwWya03KNwuop81k5aWTbfM71p2zSSa7S+GWF7LBk4eH9JmU67b/G8TDuAU4sgIm99D9SWfa+GKlYd3eRp3BeEZpaue9TEcqd7dvckpqV4l1ptaIjXh9ZK7VodI3S0UlRp+7I0lif4tMW0TqLYKH8lLy+s4mv5mP7N56xesJwntx3tl0gr4Xqo8tf5v2tgkB0TlK3z5+DHFUPGUKG1NepP+Dh267UQgYbKSTWURe/ULJJt6LbiU22EUzUVpI890W1RQ/7/amSN+yeB5O21o0fO/+WScB15zcLPuQt3ue624JuXiX8tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uh8f8Zcqs4J3O6aCT6xGR+/7ZkJb0QQ8r9gkn6/r8EU=; b=N7i7J5GfXor5QQ0I5yMDW//VmSiiipqivb1bfnKIC/dAPLeVbmY2wWtTb2N3wBXWO4885dbWy/HS26tKVBldIV6+Kba73sL9+mOi7zov3jvEC14YSwqsvkj1JlKt7/4w3d6z8SdXIdn//eNH5AMoWHnQOZk6ehhOD3uaPWajAncid0ozVRXcm9IGg+7LcCDLicwZXVa01oCwuKUqn1LnZqlC6+ZmSwqfvSABrBWDTsg2osyxdw/Yc3DjUmNQLm219XrJg3EpiOa9b5fntdeLWH+BxImbb//ewWnRoUF5MfTTrvlIY3rA5aeZ7cj2imD1TplzqZFvynO2yzMeUJynIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uh8f8Zcqs4J3O6aCT6xGR+/7ZkJb0QQ8r9gkn6/r8EU=; b=PDHmlfnxnwsZxkxYK7vfLn3UtC0QeFr3OlmXpqdejLHru9l6f8JIlFkYCgq0eq1b9xT8JWwEsTRGzp2bSS/7eVEMsi9rWk1K4qMRUsQpzpwU9xfGQTyKXbudwszNgrrkxAkRfKggdLXbcjRTBmh/ub7HrXlqwr3IEcyXBKWetiM= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6230.eurprd05.prod.outlook.com (20.178.95.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.24; Thu, 14 Nov 2019 09:37:45 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2430.028; Thu, 14 Nov 2019 09:37:45 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Edward Srouji , Daria Velikovsky , Noa Osherovich Subject: [PATCH rdma-core 1/4] pyverbs: Add memory allocation class Thread-Topic: [PATCH rdma-core 1/4] pyverbs: Add memory allocation class Thread-Index: AQHVms8rzs2wkDxYSUeydp5vKVoinQ== Date: Thu, 14 Nov 2019 09:37:45 +0000 Message-ID: <20191114093732.12637-2-noaos@mellanox.com> References: <20191114093732.12637-1-noaos@mellanox.com> In-Reply-To: <20191114093732.12637-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR0402CA0012.eurprd04.prod.outlook.com (2603:10a6:208:15::25) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 4cb834d4-459c-4278-e480-08d768e64d70 x-ms-traffictypediagnostic: AM6PR05MB6230:|AM6PR05MB6230: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-forefront-prvs: 02213C82F8 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(366004)(396003)(376002)(346002)(136003)(39860400002)(199004)(189003)(3846002)(256004)(14444005)(71190400001)(6116002)(71200400001)(14454004)(2906002)(478600001)(81166006)(81156014)(50226002)(8676002)(8936002)(86362001)(66946007)(66476007)(66556008)(64756008)(66446008)(6512007)(4326008)(5660300002)(1076003)(6436002)(2501003)(107886003)(186003)(66066001)(476003)(446003)(2616005)(26005)(11346002)(6636002)(486006)(6486002)(36756003)(25786009)(110136005)(54906003)(316002)(305945005)(102836004)(386003)(52116002)(6506007)(7736002)(76176011)(99286004);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6230;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: XbNHoV0mCvmkGIddQ5hl62XBRssSSvdjnNnLjCc2e5SzZqcfgVDOqiTZrLkbGndkLe6+4L9JxmaQnTi8aftJxDhbBS5yA0NKGZzP3oF5Db7SCUeGS9y3ykmUdcbID4eRAxVKb6yICJlBVPe0LII1JVHWNtxf9rMhTlXaTharWsbnh/EHBCboR7YiwaesFAT7meNMnUYfte0Bo9IjZyB/buzaJXRSj2cYcEyfkftlLTdxN9d6a/txf0jkAhLyj3TSZwt7FUFW2KmGUKHbImegi6FapoFLnoP8hWgyxMnsoyJg+k17q/lSrh2YdhpFYvB2/gFB7P8yqlA4EPaTjohLngm9C6iHYa8QPt+5abnLxwIoBi6pPR7Pptt4PcPr1XN6GR84vfovbUM8h5etx8/OpS4DNkl7mDEzoIca9iUW1xFcK1BEiOPmCo1RcDEd3EV5 MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4cb834d4-459c-4278-e480-08d768e64d70 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Nov 2019 09:37:45.3389 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ew9QLxRzdyVIdsqXM/nrsTl1f6Bt+KMVLLDTlWDDi4tDz1BAveb1Oxzf5lBbaaxBJEfXzAdKUBa8o2+UiWEGBA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6230 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Edward Srouji Add new MemAlloc class which wraps some C memory allocation functions such as aligned_alloc, malloc and free. This allows Pyverbs' users to easily allocate and free memory in C style, i.e when the memory address must be passed to a driver API. Signed-off-by: Edward Srouji Reviewd-by: Daria Velikovsky Reviewd-by: Noa Osherovich --- pyverbs/base.pxd | 3 +++ pyverbs/base.pyx | 26 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/pyverbs/base.pxd b/pyverbs/base.pxd index e85f7c020e1c..e956a79915ff 100644 --- a/pyverbs/base.pxd +++ b/pyverbs/base.pxd @@ -9,3 +9,6 @@ cdef class PyverbsObject(object): cdef class PyverbsCM(PyverbsObject): cpdef close(self) + +cdef class MemAlloc(object): + pass diff --git a/pyverbs/base.pyx b/pyverbs/base.pyx index 8b3e6741ae19..a41cfc748ad0 100644 --- a/pyverbs/base.pyx +++ b/pyverbs/base.pyx @@ -3,8 +3,14 @@ import logging from pyverbs.pyverbs_error import PyverbsRDMAError +from libc.stdlib cimport malloc, free +from libc.stdint cimport uintptr_t from libc.errno cimport errno +cdef extern from 'stdlib.h': + void *aligned_alloc(size_t alignment, size_t size) + + cpdef PyverbsRDMAErrno(str msg): return PyverbsRDMAError(msg, errno) @@ -58,3 +64,23 @@ cdef class PyverbsCM(PyverbsObject): cpdef close(self): pass + + +cdef class MemAlloc(object): + @staticmethod + def malloc(size): + ptr = malloc(size) + if not ptr: + raise MemoryError('Failed to allocate memory') + return ptr + + @staticmethod + def aligned_alloc(size, alignment=8): + ptr = aligned_alloc(alignment, size) + if not ptr: + raise MemoryError('Failed to allocate memory') + return ptr + + @staticmethod + def free(ptr): + free(ptr) From patchwork Thu Nov 14 09:37:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11243267 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADF9C14E5 for ; Thu, 14 Nov 2019 09:37:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7410120715 for ; Thu, 14 Nov 2019 09:37:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="QQyQpFGi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726369AbfKNJhu (ORCPT ); Thu, 14 Nov 2019 04:37:50 -0500 Received: from mail-eopbgr40065.outbound.protection.outlook.com ([40.107.4.65]:56641 "EHLO EUR03-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726386AbfKNJhu (ORCPT ); Thu, 14 Nov 2019 04:37:50 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nE/9ylSlFOxKwaoqd63CnvkGwa4R8FN3UEsyVY08zh8uRzOUXNMZALCYl+cJYkf+caiiV1yDHpSVBOU/nh3/pvrGqN3ixxCKTzMEtDxh3EN+u7WjN6uzGVGeQVI/WoEMjpeu6vrYtA0Lb/DoiZTnOsQdUwUxrs+tNiSz0BoOypV2lcEW34g7ZnO1iKPo0FuePxwXTN0+XeEi4uplwKrEXxNYgG16i7SQCHw9+1/y9/Uk5z9NNcSkCNNlJXJYl9X5+28WXx1VSaF9rYq7KpHwGZyIB40SmiwKxuInS1wimMA2Bc9N1vaVVIE4fWKFe6Xve555qfqPZylSfjpkAHsd3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pLM7NKUd4C5O18cSQwepSE5NVLblO9P4T2v2EAtus3w=; b=SnKteHlvIdu3ZmudhAdpBuJZ4z7VkVy0tk/nVx07BIYCkvIBgvzk6+KUV1dr5pTw++LQpmKZ7jwMhb83ATBWTUa5zpHmokLIo+avKW3qDd3YB3WdFLVp/rBrpuOiLL7Mlszjw7rfJ+J8g1wECm8OaQGzfQH7WFz83dotFp6ekB7Fm3pi4Q6ZnJS2gSuShcYqWgTtg3VNhB1aq+2JoWL2sWEr7WCAY4D4V4Qsed69Q4rMEeaelnKhJv1gwRBph47bX8PXQvWWDiWn++CfxW53+TTbv2sOFLd2XT5XG5kIWx8d0RKUeS4YgPGDBIG/PPmBh9tWQiiwJRiYIBhv5W9R/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pLM7NKUd4C5O18cSQwepSE5NVLblO9P4T2v2EAtus3w=; b=QQyQpFGiYszE8tJtJIZY81AEeUtR64gl7ziENcMQVB5F21wPTkUUBJwNnmNtrxWshjHlWGl5lJew8zRk6ykPPuIN+8uDlVAoPPK02lfcBLiTI4i686XmI2uy/xxuuVSd2mpp0RbpSIsx6g/2J8vYhcfZ04z7VaJPagz3NSuTK+Q= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB5783.eurprd05.prod.outlook.com (20.178.86.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23; Thu, 14 Nov 2019 09:37:46 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2430.028; Thu, 14 Nov 2019 09:37:46 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Edward Srouji , Daria Velikovsky , Noa Osherovich Subject: [PATCH rdma-core 2/4] pyverbs: Introduce ParentDomain class Thread-Topic: [PATCH rdma-core 2/4] pyverbs: Introduce ParentDomain class Thread-Index: AQHVms8sbhl+P+e5bkW4yXUEEbCwjg== Date: Thu, 14 Nov 2019 09:37:46 +0000 Message-ID: <20191114093732.12637-3-noaos@mellanox.com> References: <20191114093732.12637-1-noaos@mellanox.com> In-Reply-To: <20191114093732.12637-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR0402CA0012.eurprd04.prod.outlook.com (2603:10a6:208:15::25) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: c813bb2a-c96c-4715-1e97-08d768e64e46 x-ms-traffictypediagnostic: AM6PR05MB5783:|AM6PR05MB5783: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3968; x-forefront-prvs: 02213C82F8 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(396003)(376002)(136003)(366004)(346002)(39860400002)(189003)(199004)(52116002)(4326008)(8936002)(8676002)(81166006)(81156014)(50226002)(7736002)(3846002)(6116002)(6486002)(305945005)(36756003)(107886003)(2906002)(6436002)(6512007)(66946007)(478600001)(66476007)(66446008)(64756008)(66556008)(71200400001)(110136005)(99286004)(54906003)(316002)(256004)(25786009)(5660300002)(1076003)(71190400001)(6506007)(26005)(86362001)(186003)(76176011)(14444005)(486006)(2501003)(476003)(11346002)(2616005)(6636002)(66066001)(386003)(14454004)(30864003)(446003)(102836004);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB5783;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: YpaCULZjTjYV1/eckutPMQBaQFrEXLdGA2L1696E5hU4boqlBZIiIPN2Cb12oi6xl9eqNM8E6NftZgzT3VgFnXKvxvQEwMQ3InrLi+qlq8Sa2857Y5N4Q/LrjcWPsfLk716i7TS0erv68rH7vw+s4qsS+9hC/0L2Fv2Vf96GEDQk2BhMtpiMWloaBXf4g8+W178DlxE51wlOoqSzT9wtCDoy+Nylhp6QntQmbwj5qYYgPJAVTLbqGeJW0aQvAeGOPXMo+/WO73m4ywc9huTZp3oIpGvksw2GJCJNPdr2Pgmg0eBgBnbKoAXQdUXtTl0TWk0TlfjIuG2dDhpPilhjmJtHe2QYz3x/Ltxq/r7fiY90Sd+2BcvKTF5jT3BIYnxA1zZCszrxKupNE+M4cRomXIKNo2r2VOpPiG7+/ABIoiYPKVPhXekm5iAuSCpNzH+1 MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: c813bb2a-c96c-4715-1e97-08d768e64e46 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Nov 2019 09:37:46.6779 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: NW+LWoJHbH1qP3HF8KvrSThs0hccfUnzo2wJVozaL+I6y6K9EY5gbL8fEFLGNHXENkg43sber32ejCzgGwTyRw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5783 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Edward Srouji The parent domain object extends the normal PD (protection domain) and can be used interchangeably with it. This patch adds ParentDomain class, in order to allow the user to allocate parent domain in a user-friendly way. In addition, ParentDomainContext class was added which is used as the pd_context and it includes PD (protection domain) object, alloc and free Python functions. This allows the user to implement allocators in Python that are callable from the C callbacks wrappers via driver code. Signed-off-by: Edward Srouji Reviewd-by: Daria Velikovsky Reviewd-by: Noa Osherovich --- pyverbs/libibverbs.pxd | 11 ++ pyverbs/libibverbs_enums.pxd | 7 ++ pyverbs/pd.pxd | 17 +++ pyverbs/pd.pyx | 150 ++++++++++++++++++++++-- pyverbs/providers/mlx5/mlx5dv_enums.pxd | 11 ++ pyverbs/srq.pyx | 2 +- 6 files changed, 187 insertions(+), 11 deletions(-) diff --git a/pyverbs/libibverbs.pxd b/pyverbs/libibverbs.pxd index 02137d81e2d3..ad8d8bacc541 100755 --- a/pyverbs/libibverbs.pxd +++ b/pyverbs/libibverbs.pxd @@ -454,6 +454,15 @@ cdef extern from 'infiniband/verbs.h': ibv_qp_type qp_type; unsigned int events_completed; + cdef struct ibv_parent_domain_init_attr: + ibv_pd *pd; + uint32_t comp_mask; + void *(*alloc)(ibv_pd *pd, void *pd_context, size_t size, + size_t alignment, uint64_t resource_type); + void (*free)(ibv_pd *pd, void *pd_context, void *ptr, + uint64_t resource_type); + void *pd_context; + ibv_device **ibv_get_device_list(int *n) void ibv_free_device_list(ibv_device **list) ibv_context *ibv_open_device(ibv_device *device) @@ -540,3 +549,5 @@ cdef extern from 'infiniband/verbs.h': int ibv_destroy_srq(ibv_srq *srq) int ibv_post_srq_recv(ibv_srq *srq, ibv_recv_wr *recv_wr, ibv_recv_wr **bad_recv_wr) + ibv_pd *ibv_alloc_parent_domain(ibv_context *context, + ibv_parent_domain_init_attr *attr) diff --git a/pyverbs/libibverbs_enums.pxd b/pyverbs/libibverbs_enums.pxd index 114915d0a751..66066e2c37fd 100755 --- a/pyverbs/libibverbs_enums.pxd +++ b/pyverbs/libibverbs_enums.pxd @@ -401,6 +401,13 @@ cdef extern from '': cdef unsigned long long IBV_DEVICE_RAW_SCATTER_FCS cdef unsigned long long IBV_DEVICE_PCI_WRITE_END_PADDING + cpdef enum ibv_parent_domain_init_attr_mask: + IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS + IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT + + cdef void *IBV_ALLOCATOR_USE_DEFAULT + _IBV_DEVICE_RAW_SCATTER_FCS = IBV_DEVICE_RAW_SCATTER_FCS _IBV_DEVICE_PCI_WRITE_END_PADDING = IBV_DEVICE_PCI_WRITE_END_PADDING +_IBV_ALLOCATOR_USE_DEFAULT = IBV_ALLOCATOR_USE_DEFAULT diff --git a/pyverbs/pd.pxd b/pyverbs/pd.pxd index 6dd9c2959ed3..cb0b4715ba0d 100644 --- a/pyverbs/pd.pxd +++ b/pyverbs/pd.pxd @@ -3,6 +3,7 @@ #cython: language_level=3 +from pyverbs.base cimport PyverbsObject from pyverbs.device cimport Context cimport pyverbs.libibverbs as v from .base cimport PyverbsCM @@ -17,3 +18,19 @@ cdef class PD(PyverbsCM): cdef object mws cdef object ahs cdef object qps + cdef object parent_domains + +cdef class ParentDomainInitAttr(PyverbsObject): + cdef v.ibv_parent_domain_init_attr init_attr + cdef object pd + cdef object alloc + cdef object dealloc + +cdef class ParentDomain(PD): + cdef object protection_domain + pass + +cdef class ParentDomainContext(PyverbsObject): + cdef object p_alloc + cdef object p_free + cdef object pd diff --git a/pyverbs/pd.pyx b/pyverbs/pd.pyx index a4fc76f0fe5b..b639c3dcc74d 100644 --- a/pyverbs/pd.pyx +++ b/pyverbs/pd.pyx @@ -2,35 +2,43 @@ # Copyright (c) 2019, Mellanox Technologies. All rights reserved. import weakref -from pyverbs.pyverbs_error import PyverbsRDMAError, PyverbsError +from pyverbs.pyverbs_error import PyverbsError, PyverbsUserError from pyverbs.base import PyverbsRDMAErrno -from pyverbs.device cimport Context, DM +from pyverbs.device cimport Context +from libc.stdint cimport uintptr_t +from libc.errno cimport errno from .mr cimport MR, MW, DMMR from pyverbs.srq cimport SRQ from pyverbs.addr cimport AH from pyverbs.qp cimport QP -from libc.errno cimport errno cdef class PD(PyverbsCM): - def __cinit__(self, Context context not None): + def __cinit__(self, Context context not None, **kwargs): """ Initializes a PD object. A reference for the creating Context is kept so that Python's GC will destroy the objects in the right order. :param context: The Context object creating the PD - :return: The newly created PD on success + :param kwargs: Arguments: + * *attr* (object) + If provided PD will not be allocated, leaving the allocation to + be made by an inheriting class """ - self.pd = v.ibv_alloc_pd(context.context) - if self.pd == NULL: - raise PyverbsRDMAErrno('Failed to allocate PD', errno) + # If there's a Parent Domain attribute skip PD allocation + # since this is done by the Parent Domain class + if not kwargs.get('attr'): + self.pd = v.ibv_alloc_pd(context.context) + if self.pd == NULL: + raise PyverbsRDMAErrno('Failed to allocate PD', errno) + self.logger.debug('PD: Allocated ibv_pd') self.ctx = context context.add_ref(self) - self.logger.debug('PD: Allocated ibv_pd') self.srqs = weakref.WeakSet() self.mrs = weakref.WeakSet() self.mws = weakref.WeakSet() self.ahs = weakref.WeakSet() self.qps = weakref.WeakSet() + self.parent_domains = weakref.WeakSet() def __dealloc__(self): """ @@ -48,7 +56,8 @@ cdef class PD(PyverbsCM): :return: None """ self.logger.debug('Closing PD') - self.close_weakrefs([self.qps, self.ahs, self.mws, self.mrs, self.srqs]) + self.close_weakrefs([self.parent_domains, self.qps, self.ahs, self.mws, + self.mrs, self.srqs]) if self.pd != NULL: rc = v.ibv_dealloc_pd(self.pd) if rc != 0: @@ -67,5 +76,126 @@ cdef class PD(PyverbsCM): self.qps.add(obj) elif isinstance(obj, SRQ): self.srqs.add(obj) + elif isinstance(obj, ParentDomain): + self.parent_domains.add(obj) else: raise PyverbsError('Unrecognized object type') + + +cdef void *pd_alloc(v.ibv_pd *pd, void *pd_context, size_t size, + size_t alignment, v.uint64_t resource_type): + """ + Parent Domain allocator wrapper. This function is used to wrap a + user-defined Python alloc function which should be a part of pd_context. + :param pd: Parent domain + :param pd_context: User-specific context of type ParentDomainContext + :param size: Size of the requested buffer + :param alignment: Alignment of the requested buffer + :param resource_type: Vendor-specific resource type + :return: Pointer to the allocated buffer, or NULL to designate an error. + It may also return IBV_ALLOCATOR_USE_DEFAULT asking the callee to + allocate the buffer using the default allocator. + + """ + cdef ParentDomainContext pd_ctx + pd_ctx = pd_context + ptr = pd_ctx.p_alloc(pd_ctx.pd, pd_ctx, size, alignment, + resource_type) + return ptr + + +cdef void pd_free(v.ibv_pd *pd, void *pd_context, void *ptr, + v.uint64_t resource_type): + """ + Parent Domain deallocator wrapper. This function is used to wrap a + user-defined Python free function which should be part of pd_context. + :param pd: Parent domain + :param pd_context: User-specific context of type ParentDomainContext + :param ptr: Pointer to the buffer to be freed + :param resource_type: Vendor-specific resource type + """ + cdef ParentDomainContext pd_ctx + pd_ctx = pd_context + pd_ctx.p_free(pd_ctx.pd, pd_ctx, ptr, resource_type) + + +cdef class ParentDomainContext(PyverbsObject): + def __cinit__(self, PD pd, alloc_func, free_func): + """ + Initializes ParentDomainContext object which is used as a pd_context. + It contains the relevant fields in order to allow the user to write + alloc and free functions in Python + :param pd: PD object that represents the ibv_pd which is passed to the + creation of the Parent Domain + :param alloc_func: Python alloc function + :param free_func: Python free function + """ + self.pd = pd + self.p_alloc = alloc_func + self.p_free = free_func + + +cdef class ParentDomainInitAttr(PyverbsObject): + def __cinit__(self, PD pd not None, ParentDomainContext pd_context=None): + """ + Represents ibv_parent_domain_init_attr C struct + :param pd: PD to initialize the ParentDomain with + :param pd_context: ParentDomainContext object including the alloc and + free Python callbacks + """ + self.pd = pd + self.init_attr.pd = pd.pd + if pd_context: + self.init_attr.alloc = pd_alloc + self.init_attr.free = pd_free + self.init_attr.pd_context = pd_context + # The only way to use Python callbacks is to pass the (Python) + # functions through pd_context. Hence, we must set PD_CONTEXT + # in the comp mask. + self.init_attr.comp_mask = v.IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT | \ + v.IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS + + @property + def comp_mask(self): + return self.init_attr.comp_mask + + +cdef class ParentDomain(PD): + def __cinit__(self, Context context not None, **kwargs): + """ + Initializes ParentDomain object which represents a parent domain of + ibv_pd C struct type + :param context: Device context + :param kwargs: Arguments: + * *attr* (object) + Attribute of type ParentDomainInitAttr to initialize the + ParentDomain with + """ + cdef ParentDomainInitAttr attr + attr = kwargs.get('attr') + if attr is None: + raise PyverbsUserError('ParentDomain must take attr') + (attr.pd).add_ref(self) + self.protection_domain = attr.pd + self.pd = v.ibv_alloc_parent_domain(context.context, &attr.init_attr) + if self.pd == NULL: + raise PyverbsRDMAErrno('Failed to allocate Parent Domain') + self.logger.debug('Allocated ParentDomain') + + def __dealloc__(self): + self.__close(True) + + cpdef close(self): + self.__close() + + def __close(self, from_dealloc=False): + """ + The close function can be called either explicitly by the user, or + implicitly (from __dealloc__). In the case it was called by dealloc, + the close function of the PD would have been already called, thus + freeing the PD of this parent domain and no need to dealloc it again + :param from_dealloc: Indicates whether the close was called via dealloc + """ + self.logger.debug('Closing ParentDomain') + if not from_dealloc: + super(ParentDomain, self).close() diff --git a/pyverbs/providers/mlx5/mlx5dv_enums.pxd b/pyverbs/providers/mlx5/mlx5dv_enums.pxd index 038a49111a3b..b02da9bf5001 100644 --- a/pyverbs/providers/mlx5/mlx5dv_enums.pxd +++ b/pyverbs/providers/mlx5/mlx5dv_enums.pxd @@ -45,3 +45,14 @@ cdef extern from 'infiniband/mlx5dv.h': MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_SPI_STEERING = 1 << 2 MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_FULL_OFFLOAD = 1 << 3 MLX5DV_FLOW_ACTION_FLAGS_ESP_AES_GCM_TX_IV_IS_ESN = 1 << 4 + + cdef unsigned long long MLX5DV_RES_TYPE_QP + cdef unsigned long long MLX5DV_RES_TYPE_RWQ + cdef unsigned long long MLX5DV_RES_TYPE_DBR + cdef unsigned long long MLX5DV_RES_TYPE_SRQ + + +_MLX5DV_RES_TYPE_QP = MLX5DV_RES_TYPE_QP +_MLX5DV_RES_TYPE_RWQ = MLX5DV_RES_TYPE_RWQ +_MLX5DV_RES_TYPE_DBR = MLX5DV_RES_TYPE_DBR +_MLX5DV_RES_TYPE_SRQ = MLX5DV_RES_TYPE_SRQ diff --git a/pyverbs/srq.pyx b/pyverbs/srq.pyx index 9cad4cafdd83..a60aa5dcb0e5 100755 --- a/pyverbs/srq.pyx +++ b/pyverbs/srq.pyx @@ -125,7 +125,7 @@ cdef class SRQ(PyverbsCM): def __cinit__(self, object creator not None, object attr not None): self.srq = NULL self.cq = None - if type(creator) == PD: + if isinstance(creator, PD): self._create_srq(creator, attr) elif type(creator) == Context: self._create_srq_ex(creator, attr) From patchwork Thu Nov 14 09:37:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11243269 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F26F514ED for ; Thu, 14 Nov 2019 09:37:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C98A320715 for ; Thu, 14 Nov 2019 09:37:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="VAYhmn3I" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726386AbfKNJhv (ORCPT ); Thu, 14 Nov 2019 04:37:51 -0500 Received: from mail-eopbgr60075.outbound.protection.outlook.com ([40.107.6.75]:2435 "EHLO EUR04-DB3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726410AbfKNJhv (ORCPT ); Thu, 14 Nov 2019 04:37:51 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kFoI9HyTr2eUbwhZ7hPhsR6nDMQWGEWQ+Cx+ASt8Ow5w/mvdhKS+huoI/MN6A0ZMDRE5I9Ewq55fcidkSx6IoDqabqTUHhD0nTVNFsSND5owjlWKZOmDS2SckRAaYSGikowhUKikYd5TAJyCC6BfFXySlpcKhh4ezwUC2JFYWoanumuk5xsCkeDFjD1zhuhR38CKvkCRp3ckAIutVtLPLzQ20YWNrhUma11KnmRWNHdHsOL1NEj/MaidQCpQGdcUH2soFWI4c4lRgaUsjYP1dxFDjhp+nXZJt6p3GM9G+Ke/3Ktkodbegr4rQaVReTIjmvRuNsvJWrQsoXjWrGejHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aZcKBdP2dsVtvU086ZRfEL6XQZXH+w42iFgS6d+BBRo=; b=AwmZTkz72Z3umFDRLcCPTqYWDR+jsmv7sQ1NJsD7/JB0XCJBde+V9rJCb7/AMoH9xNrW1gWdjOOaxttyEW8KZSW//u11HNQyyFpCd4WQy0YcAVav06OWkbxPluiCrTSHoWpwaUPWh6xlcJZKTHcjRMLGc9tdIY/BGPhsx8EeL8P+o3p/y9PRBm7ZSdiXKB58HxPxrlo4YcXwinPwB3xYscCRe752KVKBcEbhQSXS7jaozz0mCVzgwJPdMGOLivU4yyeDRHZ+gq7bprHK+X79AxZS/wMPfhOf6NB49SJQ0n7kOVNTjf5VnwqnICTY04guOw0KJdKmQUCOxG04SQJZYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aZcKBdP2dsVtvU086ZRfEL6XQZXH+w42iFgS6d+BBRo=; b=VAYhmn3I3JC/n2aAgrWpg6m3Is3rk+TXPvDgpaBH2ZBFCxQToG5xqP9OBsmQ5byFPsx/ZJ8YqJkJ5wPtnohBpeIA0bLxxwV7RW+RWa8QvxGbUk78QradVHpRyDGhDJDUz/lKBVhDoeUVaHsQjq3dLVNwP0ZmCx/cVLxN2OKL9XQ= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6230.eurprd05.prod.outlook.com (20.178.95.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.24; Thu, 14 Nov 2019 09:37:47 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2430.028; Thu, 14 Nov 2019 09:37:47 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Edward Srouji , Noa Osherovich Subject: [PATCH rdma-core 3/4] pyverbs: Document ParentDomain class and add a simple example Thread-Topic: [PATCH rdma-core 3/4] pyverbs: Document ParentDomain class and add a simple example Thread-Index: AQHVms8sn8Z/uBrjf0egaxUqQvxt5g== Date: Thu, 14 Nov 2019 09:37:47 +0000 Message-ID: <20191114093732.12637-4-noaos@mellanox.com> References: <20191114093732.12637-1-noaos@mellanox.com> In-Reply-To: <20191114093732.12637-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR0402CA0012.eurprd04.prod.outlook.com (2603:10a6:208:15::25) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 8d903c2e-5789-43f0-0cc2-08d768e64f0b x-ms-traffictypediagnostic: AM6PR05MB6230:|AM6PR05MB6230: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6108; x-forefront-prvs: 02213C82F8 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(366004)(396003)(376002)(346002)(136003)(39860400002)(199004)(189003)(3846002)(256004)(71190400001)(6116002)(71200400001)(14454004)(2906002)(478600001)(81166006)(81156014)(50226002)(8676002)(8936002)(86362001)(66946007)(66476007)(66556008)(64756008)(66446008)(6512007)(4326008)(5660300002)(1076003)(6436002)(2501003)(107886003)(186003)(66066001)(476003)(446003)(2616005)(26005)(11346002)(6636002)(486006)(6486002)(36756003)(25786009)(110136005)(54906003)(316002)(305945005)(102836004)(386003)(52116002)(6506007)(7736002)(76176011)(99286004);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6230;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: urWd002J+VM5GuDvROX4kvTCQ/ijDYvNzbbnMz55S8AAiy5EbKChIc1DKCvHBeFyrSxcwh9hHk/PUCAfEl9djq4x44e0HmTqC1SGdEYrenFgfRgdWShqOvQN94hqOdLW5j7MvQH4r4m6b/UEDCHVeSECTA2bU7vqQXCr2c7yZfZMt1ZTuR+8OiCjXJT1BSWhsRJRmEgZMqHu9tiMPCpDTpwORX9vjLxfffCT6EH/qHp0G8X5OPXt76geJ39j/FAitWC9u1wgVOkU1xq+NgvqRj33jCSmzIoXv+YC6MZYBARcTAiUMG44F+/gVydkNnV3TeejVw82CF44VRfaY/FLlxiNuz7qT52kyZghIqgXwsc6nXrZl0CUlC2Wn9YpKfvQjgdwY79EFy2148GRvH+ft0nPZMSiuysbA/WL4mBtxKcBMYOyY8KeaUK7Q59sNdUD MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d903c2e-5789-43f0-0cc2-08d768e64f0b X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Nov 2019 09:37:47.6271 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zLIK4aQE2nFhuEI+pk5Dn/ju7xvetbbe3qamzNbDwn1Z46FYum3Z3JiUyQzmoZoZBsO3XcFSHc0FF9Mcpr4b2A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6230 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Edward Srouji Add a documentation for ParentDomain with a simple code snippet demonstrating the creation of the object with Python defined allocators. Signed-off-by: Edward Srouji Reviewd-by: Noa Osherovich --- Documentation/pyverbs.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md index 29ab9592c53c..db6c6b99905e 100755 --- a/Documentation/pyverbs.md +++ b/Documentation/pyverbs.md @@ -390,3 +390,30 @@ srq_attr.comp_mask = e.IBV_SRQ_INIT_ATTR_TYPE | e.IBV_SRQ_INIT_ATTR_PD | \ e.IBV_SRQ_INIT_ATTR_CQ | e.IBV_SRQ_INIT_ATTR_XRCD srq = SRQ(ctx, srq_attr) ``` + +##### ParentDomain +The following code demonstrates the creation of Parent Domain object. +In this example, a simple Python allocator is defined. It uses MemAlloc class to +allocate aligned memory using a C style aligned_alloc. +```python +from pyverbs.pd import PD, ParentDomainInitAttr, ParentDomain, \ + ParentDomainContext +from pyverbs.device import Context +from pyverbs.base import MemAlloc + + +def alloc_p_func(pd, context, size, alignment, resource_type): + p = MemAlloc.aligned_alloc(size, alignment) + return p + + +def free_p_func(pd, context, ptr, resource_type): + MemAlloc.free(ptr) + + +ctx = Context(name='rocep0s8f0') +pd = PD(ctx) +pd_ctx = ParentDomainContext(pd, alloc_p_func, free_p_func) +pd_attr = ParentDomainInitAttr(pd=pd, pd_context=pd_ctx) +parent_domain = ParentDomain(ctx, attr=pd_attr) +``` From patchwork Thu Nov 14 09:37:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noa Osherovich X-Patchwork-Id: 11243271 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74045930 for ; Thu, 14 Nov 2019 09:37:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 49E8E20709 for ; Thu, 14 Nov 2019 09:37:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="KznzUaKJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726115AbfKNJhz (ORCPT ); Thu, 14 Nov 2019 04:37:55 -0500 Received: from mail-eopbgr60063.outbound.protection.outlook.com ([40.107.6.63]:60673 "EHLO EUR04-DB3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726410AbfKNJhz (ORCPT ); Thu, 14 Nov 2019 04:37:55 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f9qpi3Av9H+8R57LZxdt1OgYcs2vaq4bBWEJ4YSlCH/ZsaUwpTYdvqQL6aJpD3grAd289Fb9HZruUrqi5z7u6aEH12Uch6ezi2RnvCT/7HpbjoH/NaERfpqhpWPxNSAGVCObbaTwXrzBOp79xEVB++ti2zqOlIG/s+RzrdPrk2Nxue1WozL5yZlzzTihz8kkOUAGCONJHyir8m/WzULpMj+HlxdsVgOwhJUggyJs9Q2r62+5j3MLIVallAhG2Jfr8rVZdC/MQlQ9H+iERuZiBaZdEUC7Afv+4owqsOzmUoUFGxyBY7YMUeFFVtsJ896IXBRxDjjgAKecHt14TsHWaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FynntZjT6aPM/WYQ1esZjLei2Xa+aKaFu6Qth1Ho5tk=; b=Jy2N3/QYwBdVCqsUIPAI4UEjx173CwclF19DvXv3jbajwCq3pcSTzxrLsODUrx8gn6TqREsm6Ya3c4XcnfDU80kkyzi8aWsVOLpIx10OH9Gocz/8wz1qqqTOsFA4t3AARV4EClo6xA0L44drXsQhiHUBihYiHNBtHCnCYEV2pQ9FsPlW+Tz4pLKqgqmqun3JxBFqihwcvE1unP8pJkWjEbv5hhJu7rwSTCdKSAnUnq2XDWSpzlAcso6pNRtoK7lzMpxdRIOx5YiXYxHyZdrcNcZUFjwANzKVeXCRibDaLhcwVqIMAcKHe4ulKS/26OncegVT61yO+8OERTjQmYO/bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FynntZjT6aPM/WYQ1esZjLei2Xa+aKaFu6Qth1Ho5tk=; b=KznzUaKJPITdB8kVa9PMAhk6QwESTWde/C7ZuyhFuCqVCRJgFeLrfLDitIGOxXzMP2Gsh0Eszk5aoP8wHl5TF4gc8UZGLSnxfjeywLYqwkkRYD3fzVxvlzACG8h1EteoXRXNPYny2BqCSak2g+xlaVreHK5jx9I7N0yIhJ++szk= Received: from AM6PR05MB4968.eurprd05.prod.outlook.com (20.177.33.17) by AM6PR05MB6230.eurprd05.prod.outlook.com (20.178.95.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2430.24; Thu, 14 Nov 2019 09:37:48 +0000 Received: from AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff]) by AM6PR05MB4968.eurprd05.prod.outlook.com ([fe80::505f:e3f8:3f87:e3ff%7]) with mapi id 15.20.2430.028; Thu, 14 Nov 2019 09:37:48 +0000 From: Noa Osherovich To: "dledford@redhat.com" , Jason Gunthorpe , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , Edward Srouji , Noa Osherovich Subject: [PATCH rdma-core 4/4] tests: Add a test for parent domain Thread-Topic: [PATCH rdma-core 4/4] tests: Add a test for parent domain Thread-Index: AQHVms8tNrNtgWDAi0O3CkLKEjwuIQ== Date: Thu, 14 Nov 2019 09:37:48 +0000 Message-ID: <20191114093732.12637-5-noaos@mellanox.com> References: <20191114093732.12637-1-noaos@mellanox.com> In-Reply-To: <20191114093732.12637-1-noaos@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-clientproxiedby: AM0PR0402CA0012.eurprd04.prod.outlook.com (2603:10a6:208:15::25) To AM6PR05MB4968.eurprd05.prod.outlook.com (2603:10a6:20b:4::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=noaos@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [94.188.199.18] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: da29954c-56e3-4ac9-ee92-08d768e64f97 x-ms-traffictypediagnostic: AM6PR05MB6230:|AM6PR05MB6230: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 02213C82F8 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(366004)(396003)(376002)(346002)(136003)(39860400002)(199004)(189003)(3846002)(256004)(14444005)(71190400001)(6116002)(71200400001)(14454004)(2906002)(478600001)(81166006)(81156014)(50226002)(8676002)(8936002)(86362001)(66946007)(66476007)(66556008)(64756008)(66446008)(6512007)(4326008)(5660300002)(1076003)(6436002)(2501003)(107886003)(186003)(66066001)(476003)(446003)(2616005)(26005)(11346002)(6636002)(486006)(6486002)(36756003)(25786009)(110136005)(54906003)(316002)(305945005)(102836004)(386003)(52116002)(6506007)(7736002)(76176011)(99286004);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB6230;H:AM6PR05MB4968.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E+3GMSRvW053rcWS7UFJubVbBOQj3ZFgv4TeS/sGvoB6s/ccc3ZmYNe+/3aYgeTcofqQkrhScUGkuzdm5EHWLcewQ4s2HDf2Q8ivDKp+EAvRap9GgtGQ1OiWiOOQ06pB/V/9LsqTBfdMkem/wR4fmXailTJZrxES0FtIlv/udPFzF64IEpASj+gU/A2hnPoZiqvmoAGVVACEteod1WXbV4xIipEE9z2iIctuVSwhCz5qNt1wZ0m3HoAO0bkXc9z/R0DlRFrWLiFoW8zhfcsW7VoUXJvErSC8FO990tqwpZU3ZVoDhmdMgk38Ke/QifuwX8kCXf+XQvJK7kfISvyyEHtQIoWrncxoLirq7RydbAbBb7In6OUfNB4ZDwHDQc43lm5JAuJF/a+fHyWW67WQ9Wimmii2v/fn8zQapzH1u7+hHKRTpBHZfG+Y1EcJhugo MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: da29954c-56e3-4ac9-ee92-08d768e64f97 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Nov 2019 09:37:48.5459 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: l4UF8o7sbP9KVPGBjviBhAEH1xzn0MMgAV3vRQ5cHTLhzR3ati+swT99CnB19v6nOido5yYL93f1E/QOn/zTvw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB6230 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Edward Srouji The test checks parent domain creation without custom allocators, with custom allocators that return USE_DEFAULT (indicates the driver to use the default allocator) and custom allocators with local memory allocation. In addition, in each case a QP is created in order to verify the usage of the allocators. Signed-off-by: Edward Srouji Reviewd-by: Noa Osherovich --- tests/CMakeLists.txt | 1 + tests/test_parent_domain.py | 86 +++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 tests/test_parent_domain.py diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 960276230860..4a8e4f6607c8 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -8,6 +8,7 @@ rdma_python_test(tests test_cq.py test_device.py test_mr.py + test_parent_domain.py test_pd.py test_qp.py test_odp.py diff --git a/tests/test_parent_domain.py b/tests/test_parent_domain.py new file mode 100644 index 000000000000..5731f6256d04 --- /dev/null +++ b/tests/test_parent_domain.py @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) +# Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. See COPYING file +""" +Test module for Pyverbs' ParentDomain. +""" +from pyverbs.pd import ParentDomainInitAttr, ParentDomain, ParentDomainContext +from pyverbs.pyverbs_error import PyverbsRDMAError +from pyverbs.srq import SrqAttr, SrqInitAttr, SRQ +from pyverbs.qp import QPInitAttr, QP +from tests.base import BaseResources +from tests.base import RDMATestCase +from pyverbs.base import MemAlloc +import pyverbs.device as d +import pyverbs.enums as e +from pyverbs.cq import CQ +import tests.utils as u +import unittest + + +class ParentDomainRes(BaseResources): + def __init__(self, dev_name, ib_port=None, gid_index=None): + super().__init__(dev_name=dev_name, ib_port=ib_port, + gid_index=gid_index) + # Parent Domain will be created according to the test + self.pd_ctx = None + self.parent_domain = None + + +class ParentDomainTestCase(RDMATestCase): + def setUp(self): + if self.dev_name is None: + dev = d.get_device_list()[-1] + self.dev_name = dev.name.decode() + self.pd_res = ParentDomainRes(self.dev_name) + + def _create_parent_domain_with_allocators(self, alloc_func, free_func): + if alloc_func and free_func: + self.pd_res.pd_ctx = ParentDomainContext(self.pd_res.pd, alloc_func, + free_func) + pd_attr = ParentDomainInitAttr(pd=self.pd_res.pd, + pd_context=self.pd_res.pd_ctx) + try: + self.pd_res.parent_domain = ParentDomain(self.pd_res.ctx, + attr=pd_attr) + except PyverbsRDMAError as ex: + if 'not supported' in str(ex) or 'not implemented' in str(ex): + raise unittest.SkipTest('Parent Domain is not supported on this device') + raise ex + + def _create_rdma_objects(self): + cq = CQ(self.pd_res.ctx, 100, None, None, 0) + dev_attr = self.pd_res.ctx.query_device() + qp_cap = u.random_qp_cap(dev_attr) + qia = QPInitAttr(scq=cq, rcq=cq, cap=qp_cap) + qia.qp_type = e.IBV_QPT_RC + QP(self.pd_res.parent_domain, qia) + srq_init_attr = SrqInitAttr(SrqAttr()) + SRQ(self.pd_res.parent_domain, srq_init_attr) + + def test_without_allocators(self): + self._create_parent_domain_with_allocators(None, None) + self._create_rdma_objects() + self.pd_res.parent_domain.close() + + def test_default_allocators(self): + def alloc_p_func(pd, context, size, alignment, resource_type): + return e._IBV_ALLOCATOR_USE_DEFAULT + + def free_p_func(pd, context, ptr, resource_type): + return e._IBV_ALLOCATOR_USE_DEFAULT + + self._create_parent_domain_with_allocators(alloc_p_func, free_p_func) + self._create_rdma_objects() + self.pd_res.parent_domain.close() + + def test_mem_align_allocators(self): + def alloc_p_func(pd, context, size, alignment, resource_type): + p = MemAlloc.aligned_alloc(size, alignment) + return p + + def free_p_func(pd, context, ptr, resource_type): + MemAlloc.free(ptr) + + self._create_parent_domain_with_allocators(alloc_p_func, free_p_func) + self._create_rdma_objects() + self.pd_res.parent_domain.close()