From patchwork Mon Sep 23 23:49:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 11157737 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F19C31668 for ; Mon, 23 Sep 2019 23:49:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C5D9F21D6C for ; Mon, 23 Sep 2019 23:49:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=vmware.com header.i=@vmware.com header.b="FyeuMbmQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391461AbfIWXti (ORCPT ); Mon, 23 Sep 2019 19:49:38 -0400 Received: from mail-eopbgr690069.outbound.protection.outlook.com ([40.107.69.69]:35713 "EHLO NAM04-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729276AbfIWXti (ORCPT ); Mon, 23 Sep 2019 19:49:38 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dwHy3szSf2ccITNzuiHlHt1d3ORKyzdZVWWhCNhrNy7oLLa2j1sClWK+hZhPpMXV7io/n0yjvAQoa2yqlukCDK3yX57aKgcbibCCXx6I85NjLEpVtTg+aBw6tODCCs+6WcM8RGwskf+N29tsh1wAJCHPaj6H1uWMqmYOTl0q3cWjlhiqS9YyIEafLDkjH3pJLgiJFLsUae4kliXnWHnqBCkyJAjLlLFC6PBXj3Kav6R6ZNVft4+IvPIlTw155KyywM9y7I2rYtYUbs7cxDF0hQTKQdVfRkTAKQ/oWsixGVPzCVsIA4WDUOvhZqNJrjvVho3HuDlR7Y+zkrmasgJqbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kS8y2tp3e/ixPXzH44FkJjTVoSIm0u836gvhFsxJVHA=; b=UKoxERjBMX9tOhjXr6wVSbw/hXb2FhJHM+wz2epMOOAeBey0uFynUOYGFDxSxtmCGlsiCDdYAC8FGDi5ZDPbpHPis2VpUn+tCxWPzWShTENkOZGnZvg7D+ss4YMDbnCpBOk/yE8VjL2r9Yx1m1S6O9PpididAv7C3GYnNjKH+EE8su0he1Df2LY0qXUd/m6UUE/G3PLcnTcPKGmqON9KBV1xedN6EjiedW4okxnx143RfQcPV4j0hQwg9PFCKBFCNPSV6eVEECHdtOD4zYh39woKP1Yk2kryw8Wqa8kT4z7QBP66NRq0n1maFnYleP/JxE7kfYG7kAHjCqGtonuRxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kS8y2tp3e/ixPXzH44FkJjTVoSIm0u836gvhFsxJVHA=; b=FyeuMbmQICDHWsl5kTL42pKz4gK59DDJF6jUZyShl5+tmvRaHIsTpvpfLOPmsw4m7O9jHJOoaJanKU8TTmMsGJpPsoHAFnAr6SnYgJWCfBdDsxmwvpVwc77loIsP/1Qhxvci090LYBSE8w8VoCXn2JqbrmITK5zPla+L7xFilow= Received: from BYAPR05MB5511.namprd05.prod.outlook.com (20.177.186.28) by BYAPR05MB4455.namprd05.prod.outlook.com (52.135.205.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2305.15; Mon, 23 Sep 2019 23:49:35 +0000 Received: from BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::81ed:73c3:bc95:7d03]) by BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::81ed:73c3:bc95:7d03%5]) with mapi id 15.20.2284.023; Mon, 23 Sep 2019 23:49:35 +0000 From: Adit Ranadive To: "jgg@mellanox.com" , "leon@kernel.org" , "linux-rdma@vger.kernel.org" CC: Adit Ranadive Subject: [PATCH rdma-core v2 1/2] Update kernel headers Thread-Topic: [PATCH rdma-core v2 1/2] Update kernel headers Thread-Index: AQHVcmmNIkofFtfnkEivO4vWC26rdg== Date: Mon, 23 Sep 2019 23:49:34 +0000 Message-ID: <333074d88847396252a597993d9a51645acfffd6.1569282124.git.aditr@vmware.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR06CA0069.namprd06.prod.outlook.com (2603:10b6:a03:14b::46) To BYAPR05MB5511.namprd05.prod.outlook.com (2603:10b6:a03:1a::28) x-mailer: git-send-email 2.18.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.170.99.1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 763020b9-2a8c-4169-2a03-08d74080afcf x-ms-traffictypediagnostic: BYAPR05MB4455:|BYAPR05MB4455: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4941; x-forefront-prvs: 0169092318 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(346002)(396003)(136003)(39850400004)(376002)(366004)(199004)(189003)(8936002)(110136005)(14454004)(316002)(7736002)(25786009)(305945005)(4326008)(6486002)(256004)(2201001)(6436002)(6512007)(14444005)(107886003)(36756003)(478600001)(66066001)(3846002)(6116002)(81166006)(8676002)(81156014)(71190400001)(186003)(66946007)(76176011)(52116002)(86362001)(26005)(99286004)(6506007)(102836004)(386003)(486006)(11346002)(71200400001)(476003)(5660300002)(2906002)(446003)(66446008)(66556008)(64756008)(50226002)(2616005)(2501003)(118296001)(66476007);DIR:OUT;SFP:1101;SCL:1;SRVR:BYAPR05MB4455;H:BYAPR05MB5511.namprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: gDyWi352Kbwn684au2rL822Px2HvmMJss2nmEDYVdOeMNu+A2lcYFY6YCE1C32vIi/txWL0F3ukiyZnlSte6fds4+3ozz/6AuFQ0TjmT27kT9XeCyvVDXbqJNx0g9W44/gMI/26+elsajE5TZ8ZbNW5+DZlfQds0mSCMJ0DqstHDT6zF4XjmRUgsV8YqjTIRi+DEBhAt/RpcKZl1yUI854mrZXWhBFML1g3+BM25n6puodpRGdh5NJxPZBIKO3vs/IYVQwRE29CcLJchTshosd97VAYyB8PgZnzbo70zJKv5vJjkwqtsH5nCXL7oOnAazI8AuFW2w2NR1En7KnNZJlyw3iBSG3c6sL7YIVY2L11udusrGSih/iJYqflvjmnXwBUfHK2jXurMUu8EMkLM5bPlTpOBaAAzSpvC6peYDUc= MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-Network-Message-Id: 763020b9-2a8c-4169-2a03-08d74080afcf X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2019 23:49:34.8843 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 0AvA6Iav5ZaeEfbxV47CuXdJsHZfa8m8KXB5y8D8Zshs1woBqP5V6Jm1OItKEoonFCZN3JTJt8Qcpjwih3db0w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR05MB4455 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org To commit ?? ("RDMA/vmw_pvrdma: Use resource ids from physical device if available") Signed-off-by: Adit Ranadive --- kernel-headers/CMakeLists.txt | 1 + kernel-headers/rdma/rvt-abi.h | 66 ++++++++++++++++++++++++++++ kernel-headers/rdma/vmw_pvrdma-abi.h | 13 ++++++ 3 files changed, 80 insertions(+) create mode 100644 kernel-headers/rdma/rvt-abi.h diff --git a/kernel-headers/CMakeLists.txt b/kernel-headers/CMakeLists.txt index 50bc77e6ab6e..13624b22d81b 100644 --- a/kernel-headers/CMakeLists.txt +++ b/kernel-headers/CMakeLists.txt @@ -23,6 +23,7 @@ publish_internal_headers(rdma rdma/rdma_user_ioctl.h rdma/rdma_user_ioctl_cmds.h rdma/rdma_user_rxe.h + rdma/rvt-abi.h rdma/siw-abi.h rdma/vmw_pvrdma-abi.h ) diff --git a/kernel-headers/rdma/rvt-abi.h b/kernel-headers/rdma/rvt-abi.h new file mode 100644 index 000000000000..7c05a02d2be5 --- /dev/null +++ b/kernel-headers/rdma/rvt-abi.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ + +/* + * This file contains defines, structures, etc. that are used + * to communicate between kernel and user code. + */ + +#ifndef RVT_ABI_USER_H +#define RVT_ABI_USER_H + +#include +#include +#ifndef RDMA_ATOMIC_UAPI +#define RDMA_ATOMIC_UAPI(_type, _name) struct{ _type val; } _name +#endif + +struct rvt_wqe_sge { + __aligned_u64 addr; + __u32 length; + __u32 lkey; +}; + +/* + * This structure is used to contain the head pointer, tail pointer, + * and completion queue entries as a single memory allocation so + * it can be mmap'ed into user space. + */ +struct rvt_cq_wc { + /* index of next entry to fill */ + RDMA_ATOMIC_UAPI(__u32, head); + /* index of next ib_poll_cq() entry */ + RDMA_ATOMIC_UAPI(__u32, tail); + + /* these are actually size ibcq.cqe + 1 */ + struct ib_uverbs_wc uqueue[]; +}; + +/* + * Receive work request queue entry. + * The size of the sg_list is determined when the QP (or SRQ) is created + * and stored in qp->r_rq.max_sge (or srq->rq.max_sge). + */ +struct rvt_rwqe { + __u64 wr_id; + __u8 num_sge; + __u8 padding[7]; + struct rvt_wqe_sge sg_list[]; +}; + +/* + * This structure is used to contain the head pointer, tail pointer, + * and receive work queue entries as a single memory allocation so + * it can be mmap'ed into user space. + * Note that the wq array elements are variable size so you can't + * just index into the array to get the N'th element; + * use get_rwqe_ptr() for user space and rvt_get_rwqe_ptr() + * for kernel space. + */ +struct rvt_rwq { + /* new work requests posted to the head */ + RDMA_ATOMIC_UAPI(__u32, head); + /* receives pull requests from here. */ + RDMA_ATOMIC_UAPI(__u32, tail); + struct rvt_rwqe wq[]; +}; +#endif /* RVT_ABI_USER_H */ diff --git a/kernel-headers/rdma/vmw_pvrdma-abi.h b/kernel-headers/rdma/vmw_pvrdma-abi.h index 6e73f0274e41..1d339285550e 100644 --- a/kernel-headers/rdma/vmw_pvrdma-abi.h +++ b/kernel-headers/rdma/vmw_pvrdma-abi.h @@ -133,6 +133,10 @@ enum pvrdma_wc_flags { PVRDMA_WC_FLAGS_MAX = PVRDMA_WC_WITH_NETWORK_HDR_TYPE, }; +enum pvrdma_user_qp_create_flags { + PVRDMA_USER_QP_CREATE_USE_RESP = 1 << 0, +}; + struct pvrdma_alloc_ucontext_resp { __u32 qp_tab_size; __u32 reserved; @@ -177,6 +181,15 @@ struct pvrdma_create_qp { __u32 rbuf_size; __u32 sbuf_size; __aligned_u64 qp_addr; + __u32 flags; + __u32 reserved; +}; + +struct pvrdma_create_qp_resp { + __u32 qpn; + __u32 qp_handle; + __u32 qpn_valid; + __u32 reserved; }; /* PVRDMA masked atomic compare and swap */ From patchwork Mon Sep 23 23:49:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 11157739 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D5211668 for ; Mon, 23 Sep 2019 23:49:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D390121D56 for ; Mon, 23 Sep 2019 23:49:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=vmware.com header.i=@vmware.com header.b="HEsUVZBg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2408394AbfIWXtm (ORCPT ); Mon, 23 Sep 2019 19:49:42 -0400 Received: from mail-eopbgr690063.outbound.protection.outlook.com ([40.107.69.63]:12923 "EHLO NAM04-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729276AbfIWXtm (ORCPT ); Mon, 23 Sep 2019 19:49:42 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nLzGpkW9zobVF0UC99zu80SVSm0dOO7Euq2Yu5LxumSz4gENgG+xssDHChdIsrCeDGzID7EVPHj3V2bIoIqiMhp3C2HCG6RXzxSn8pAwGNneH8xXCf/1iTrM/EuRBy1npHS9kCWsS+8VhEkuUuNzWt6+Gk1RqFdfOG3A5tniTRMKZHjF9jBiSXoZ26KmJNCv4/uJAqAml8MWVfPUF4jBHQyNVsYKeYVBALMWZEkgmylLjdqJ6MX6Jnz6uVVAP+SAP2aR4Kp3rtUIUrUyZc0767qy+wHbcgIRdNOwhx5vE3Ckds0yiJo0lfRi09h5uEBp/oV/1rXjE/Bh5CbaJljatw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+yrH9YsM/4CewjflRkkV+obPBR2pM2j/tjQV8uVJ570=; b=HXfwxXZeUd5Z5wi9drkV07qgjVu8mqprn3bFTarGnxr5c1wIN9M+46Uw4dWonIpBoXwQ0xS+VBPa0/Zz5EgaRdM3wwW0gOt7rDvNYzA2a17mPDb9ZMfeHrJDBOOivzRb+D2iYcU3OZFaBIly2XkZ90KLT2Zn0Kk8RWRUiySWoqSXdLyxLzWjqor/O2whTbX0Rn6zqXMOr1I0xXsltMA8wAHYjZ/hRVYQLh1Q2l88AX+PPwkMQQerhHRnHsCAfdeVhtYGMYbIt5BC7lb13xxvBDQut/hQFQMRp+vaapfORgO44ze0dZKLPLwJJbXUAaiB+coqhtnIV3qptqWIv25tBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+yrH9YsM/4CewjflRkkV+obPBR2pM2j/tjQV8uVJ570=; b=HEsUVZBgYJ7UTMuqrPPZL5ZUBOeexSoQ8FpHsOiK1pYrJ7np1yQFw6Q1AN8Y8gmenUBLBGU3at9GNlDZoL2CnPq8R8l9b4NvujpJvCP7nN3qQTbIerkNtqBwDUKSzrmZ1jLVHbBVMUn/2yJsURcH1ZvzMjmN/tqwabLRAzQUCeI= Received: from BYAPR05MB5511.namprd05.prod.outlook.com (20.177.186.28) by BYAPR05MB4455.namprd05.prod.outlook.com (52.135.205.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2305.15; Mon, 23 Sep 2019 23:49:38 +0000 Received: from BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::81ed:73c3:bc95:7d03]) by BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::81ed:73c3:bc95:7d03%5]) with mapi id 15.20.2284.023; Mon, 23 Sep 2019 23:49:38 +0000 From: Adit Ranadive To: "jgg@mellanox.com" , "leon@kernel.org" , "linux-rdma@vger.kernel.org" CC: Bryan Tan , Adit Ranadive Subject: [PATCH rdma-core v2 2/2] vmw_pvrdma: Use resource ids from physical device if available Thread-Topic: [PATCH rdma-core v2 2/2] vmw_pvrdma: Use resource ids from physical device if available Thread-Index: AQHVcmmP5qJKSwXwkUuK/2qZiyULPA== Date: Mon, 23 Sep 2019 23:49:38 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR06CA0069.namprd06.prod.outlook.com (2603:10b6:a03:14b::46) To BYAPR05MB5511.namprd05.prod.outlook.com (2603:10b6:a03:1a::28) x-mailer: git-send-email 2.18.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.170.99.1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7c44e7db-e7bb-4cf1-5658-08d74080b1fb x-ms-traffictypediagnostic: BYAPR05MB4455:|BYAPR05MB4455: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3044; x-forefront-prvs: 0169092318 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(346002)(396003)(136003)(39850400004)(376002)(366004)(199004)(189003)(54906003)(8936002)(110136005)(14454004)(316002)(7736002)(25786009)(305945005)(4326008)(6486002)(256004)(2201001)(6436002)(6512007)(14444005)(107886003)(36756003)(478600001)(66066001)(3846002)(6116002)(81166006)(8676002)(81156014)(71190400001)(186003)(66946007)(76176011)(52116002)(86362001)(26005)(99286004)(6506007)(102836004)(386003)(486006)(11346002)(71200400001)(476003)(5660300002)(2906002)(446003)(66446008)(66556008)(64756008)(50226002)(2616005)(2501003)(118296001)(66476007);DIR:OUT;SFP:1101;SCL:1;SRVR:BYAPR05MB4455;H:BYAPR05MB5511.namprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: uJ130zzCjawNWiLo+fpyqiXtasDWJwrnGExuGrsTXk2SpafS1wiWlO+mN4FzSu47o2uwyFDs+u5AwXsK7ul6USMhOVePl8JMF1CMS+l8plFqSFZ982KKMLlWQ05mEgJmTBaYkk4d8PRjDP1xFRxcDMQDVVNLcLmj28Xgy3pqbZGtb9c62SYvbCohf7nKRWjU9ffv1QjGBre4DJGHp68gem5+WIoEMeNmkTkGC+FP1NiZgSW+tV5k9tZfIFl//5xxVfmqzhiITfYdFmVMzfoil3clJ0FUJ/lrEV5V04sT4j0Vjiq2y0trbT2rxoaX7hwDtWDkCJ++X2pD8m9AXq9rqfVIiOE4WvPc50notxgamk4SENcFy5Z40sNi5GIx/mHd3dH9MICzUtl34S2IgRZszdK+NRptLCeGkwgt7YbyI2U= MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7c44e7db-e7bb-4cf1-5658-08d74080b1fb X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2019 23:49:38.3674 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: hoPYvoarY7/NlRt0XukwUNxzCxGtvRXHdeqWxwwkdw+K2rgyLtbOvhZc0F218U3eqCckIOkZUQ7JZ6DeESOCPQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR05MB4455 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Bryan Tan This is the accompanying userspace change to allow applications use physical resource ids if provided by the driver/device. The create QP command is modified to specify a flag that tells the driver to send back the physical HCA's QPN and the hypervisor- generated QP handle separately. Reviewed-by: Jorgen Hansen Signed-off-by: Adit Ranadive Signed-off-by: Bryan Tan --- providers/vmw_pvrdma/pvrdma-abi.h | 2 +- providers/vmw_pvrdma/pvrdma.h | 1 + providers/vmw_pvrdma/qp.c | 24 +++++++++++++----------- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/providers/vmw_pvrdma/pvrdma-abi.h b/providers/vmw_pvrdma/pvrdma-abi.h index 77db9ddd1bb7..1a4c3c8a98f2 100644 --- a/providers/vmw_pvrdma/pvrdma-abi.h +++ b/providers/vmw_pvrdma/pvrdma-abi.h @@ -55,7 +55,7 @@ DECLARE_DRV_CMD(user_pvrdma_alloc_pd, IB_USER_VERBS_CMD_ALLOC_PD, DECLARE_DRV_CMD(user_pvrdma_create_cq, IB_USER_VERBS_CMD_CREATE_CQ, pvrdma_create_cq, pvrdma_create_cq_resp); DECLARE_DRV_CMD(user_pvrdma_create_qp, IB_USER_VERBS_CMD_CREATE_QP, - pvrdma_create_qp, empty); + pvrdma_create_qp, pvrdma_create_qp_resp); DECLARE_DRV_CMD(user_pvrdma_create_srq, IB_USER_VERBS_CMD_CREATE_SRQ, pvrdma_create_srq, pvrdma_create_srq_resp); DECLARE_DRV_CMD(user_pvrdma_alloc_ucontext, IB_USER_VERBS_CMD_GET_CONTEXT, diff --git a/providers/vmw_pvrdma/pvrdma.h b/providers/vmw_pvrdma/pvrdma.h index d90bd8096664..bb4a2db08d14 100644 --- a/providers/vmw_pvrdma/pvrdma.h +++ b/providers/vmw_pvrdma/pvrdma.h @@ -170,6 +170,7 @@ struct pvrdma_qp { struct pvrdma_wq sq; struct pvrdma_wq rq; int is_srq; + uint32_t qp_handle; }; struct pvrdma_ah { diff --git a/providers/vmw_pvrdma/qp.c b/providers/vmw_pvrdma/qp.c index ef429db93a43..966480f5abaa 100644 --- a/providers/vmw_pvrdma/qp.c +++ b/providers/vmw_pvrdma/qp.c @@ -211,9 +211,8 @@ struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, { struct pvrdma_device *dev = to_vdev(pd->context->device); struct user_pvrdma_create_qp cmd; - struct ib_uverbs_create_qp_resp resp; + struct user_pvrdma_create_qp_resp resp = {}; struct pvrdma_qp *qp; - int ret; int is_srq = !!(attr->srq); attr->cap.max_send_sge = max_t(uint32_t, 1U, attr->cap.max_send_sge); @@ -281,15 +280,18 @@ struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, cmd.rbuf_addr = (uintptr_t)qp->rbuf.buf; cmd.rbuf_size = qp->rbuf.length; cmd.qp_addr = (uintptr_t) qp; + cmd.flags = PVRDMA_USER_QP_CREATE_USE_RESP; - ret = ibv_cmd_create_qp(pd, &qp->ibv_qp, attr, - &cmd.ibv_cmd, sizeof(cmd), - &resp, sizeof(resp)); - - if (ret) + if (ibv_cmd_create_qp(pd, &qp->ibv_qp, attr, &cmd.ibv_cmd, sizeof(cmd), + &resp.ibv_resp, sizeof(resp))) goto err_free; - to_vctx(pd->context)->qp_tbl[qp->ibv_qp.qp_num & 0xFFFF] = qp; + if (resp.drv_payload.qpn_valid == PVRDMA_USER_QP_CREATE_USE_RESP) + qp->qp_handle = resp.drv_payload.qp_handle; + else + qp->qp_handle = qp->ibv_qp.qp_num; + + to_vctx(pd->context)->qp_tbl[qp->qp_handle & 0xFFFF] = qp; /* If set, each WR submitted to the SQ generate a completion entry */ if (attr->sq_sig_all) @@ -414,7 +416,7 @@ int pvrdma_destroy_qp(struct ibv_qp *ibqp) free(qp->rq.wrid); pvrdma_free_buf(&qp->rbuf); pvrdma_free_buf(&qp->sbuf); - ctx->qp_tbl[ibqp->qp_num & 0xFFFF] = NULL; + ctx->qp_tbl[qp->qp_handle & 0xFFFF] = NULL; free(qp); return 0; @@ -547,7 +549,7 @@ out: if (nreq) { udma_to_device_barrier(); pvrdma_write_uar_qp(ctx->uar, - PVRDMA_UAR_QP_SEND | ibqp->qp_num); + PVRDMA_UAR_QP_SEND | qp->qp_handle); } pthread_spin_unlock(&qp->sq.lock); @@ -630,7 +632,7 @@ int pvrdma_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, out: if (nreq) pvrdma_write_uar_qp(ctx->uar, - PVRDMA_UAR_QP_RECV | ibqp->qp_num); + PVRDMA_UAR_QP_RECV | qp->qp_handle); pthread_spin_unlock(&qp->rq.lock); return ret;