From patchwork Sun Oct 29 22:44:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bryan Tan X-Patchwork-Id: 10031781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 74F9D600C5 for ; Sun, 29 Oct 2017 22:45:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D4C528761 for ; Sun, 29 Oct 2017 22:45:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4FDBA28672; Sun, 29 Oct 2017 22:45:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5C2928672 for ; Sun, 29 Oct 2017 22:45:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751273AbdJ2WpL (ORCPT ); Sun, 29 Oct 2017 18:45:11 -0400 Received: from mail-cys01nam02on0070.outbound.protection.outlook.com ([104.47.37.70]:54068 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751152AbdJ2WpK (ORCPT ); Sun, 29 Oct 2017 18:45:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=onevmw.onmicrosoft.com; s=selector1-vmware-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3jXOi5IsTx/bj1oR1wOJ0vWEHksIj4Vjq8viiyauNrU=; b=NqJh3dck9v5Fb93WI6uzt0eRnIslq+PHYkRzDtgEeYADHxMugxZrXO5QsbaRtg9QiCB5w5FqA/VDL27EsXhXdEBWjHG2qsMldAM881v/UezFFE9sptNzAzW/m1pVVsOm6BXDRG4SAK0OWADa4Etflc7mrbQCtIEFXpyuOhnzRos= Received: from bryantan-devbox.prom.eng.vmware.com.prom.eng.vmware.com (208.91.1.34) by CO2PR05MB2773.namprd05.prod.outlook.com (10.166.200.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.197.4; Sun, 29 Oct 2017 22:45:06 +0000 Date: Sun, 29 Oct 2017 15:44:46 -0700 From: Bryan Tan To: linux-rdma@vger.kernel.org Subject: [PATCH rdma-core] vmw_pvrdma: Add SRQ support Message-ID: <20171029224429.GA21993@bryantan-devbox.prom.eng.vmware.com.prom.eng.vmware.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [208.91.1.34] X-ClientProxiedBy: HE1PR09CA0063.eurprd09.prod.outlook.com (10.174.50.31) To CO2PR05MB2773.namprd05.prod.outlook.com (10.166.200.25) X-MS-Office365-Filtering-Correlation-Id: ba8bd6eb-78d3-4a67-35fa-08d51f1eb433 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(2017052603238); SRVR:CO2PR05MB2773; X-Microsoft-Exchange-Diagnostics: 1; CO2PR05MB2773; 3:h8jpgTA5cKHvQZ2oTffm3sC6BUbYdb1I5LydNtK/rkDOTWrWaZLHsAydDoUsGXD+6uPka1A534N9b+2t4v1etdO9i5bPOBzVQ46VKuUO3WT0snBavC7Ew2512cgnU0ThLlm4oJtTTQbtE6MDsguDcTgf83z4h0lRMtUSY/Q5NSsxI4fMI+vuReGQDr/DbW6mR3RMS8hIQodrPoZ+vYTgu1z5pnvlTskRvPAmCdu6DPepak26h3kW/Qpwa57M9qqc; 25:00U75XA2JkzEdaqWZ9C2pBxhQSxxqsanGxLKXEsvN2oIfnI2hiKz8JFmdtixem2JjDZ0tL/G6Y+rbSvwIwlKgZqQHKa20PAhxX5dZYtSBGLUM3Kf2cUczjN1GQnUGJZFlFEkKncC3q99/PigWVDXVS2zq7USW4P7zD5FaHSGeqD+eHt9NRrtWNz1DwjPYb+GcdtyUqoyhBW4UgagU2ffCWy6mEa7ge4pHQO6Nx7zhe3F5zVF1Kya2jZ7wKAC3cdedspTk0eJNJS8RJvgr3uZKyTC7cfB2zbFzn1gm9wn6oluMmEZahlcwDOgFEGeT5XDzyOZDwwExNGTlWevkVEpmBQlmPfILB1HgAG868sgibY=; 31:yqXcYXMgCppYH1GBOlB4e23x1NorLCid79f0pz8pWyclsdq0KEFE3M0zEhOwDDOYc6vdbi6YkDk+NwN2QF6OtPIkw2ZfRb97jwtPV2B6bQxM+W1Q1aoWv9lgErNLUM/BQlyesGIaXxihy3iEXyBLw1WM3gEG6ESaur1C6MpXf/h6fgYgpmAQBmxhwZFJCkTeqJ2czizyNmVHZD7CPj5nvBwCr7yFOzs2bDsOtXJ+RCM= X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO2PR05MB2773: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=bryantan@vmware.com; X-Microsoft-Exchange-Diagnostics: 1; CO2PR05MB2773; 20:vb0OqRxFCC/koLwSQUKXYnXtuzrWzz39dteW9B7f9UOv46wBvhVVIVjSl/D48eYSaV8HsbnQzKStUV59dp3DqXt3KKpxxJY3OgBNVgsWoBRY43a+VKBRmjzp1Kbxz+3Ep9NA11gvK2w1UQgh84sDorKxNetICzJvP+ipleOu+HlQhFQ/pMg5DryTNnN9sBOa+fZWum/xCKXJ7/DDBg2at+FxIZ28PdPkijU4Znii3WWvdveDknGATvfD7UZq3TdhGCpXCSpo2HM/KXfL/nJWUUigzvYaQQtIdj1bcW9H4UlOhj/zp/Iu+NzAR9QTIpoMrmCL+PfZf5SRwl5vsP6UGBnRl04DWM22CNKFIJI5JCwhlXjn+fCMGOR8D/xpOB5+kpF/Cw3mmD6mWGj2rK8qfl2A4t/nPt5g4KXjWV3N2oiMxbDXfneoKfTe5My6PDJSCICvWajARt4fE4GXl+su+Aan2te6NE3Dh4N71T8awyhjihgnQ7y8nzei4Dhxw0ZI; 4:6nqwky83t/plye5eiVsqSI3lJdTlqbfN40weX1toxYhz2zBiVU3E23DMX5GhXDmZiL3/c8FcfI/En+vHEtMaajX2OVv5bKPVbVcqLHD10MFRv6YfmpdLRX+MeSg5Xo2F4hwz4bueF+DLv9EqepZU3qxYwt9cmRE2OJ02DseXyt3BZjHjGdP/do9OzwONKArXOs/uy5FPbinyzbH4C7cBs22EHsRxm3Finn7pjvcczfD38XIlSkluIcnifC0yNdLjVpq61ohJzXwhgizucljYMisKFEWsoe/gumy/PbGvGaD4U4p05JbOnWm5k1p3lAx5uYKhUYtmwTBpN+YCCscKk3Idyem7CUTDbE5Nd3ZyyU0= X-Exchange-Antispam-Report-Test: UriScan:(61668805478150)(278428928389397); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(3231020)(100000703101)(100105400095)(10201501046)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123564025)(20161123560025)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CO2PR05MB2773; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CO2PR05MB2773; X-Forefront-PRVS: 0475418F50 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(189002)(199003)(25786009)(47776003)(83506002)(305945005)(97736004)(6666003)(7736002)(5660300001)(53936002)(16526018)(55016002)(66066001)(6916009)(189998001)(68736007)(54356999)(106356001)(3846002)(16586007)(50986999)(8676002)(23726003)(478600001)(101416001)(86362001)(1076002)(81156014)(6116002)(2351001)(2361001)(58126008)(105586002)(33656002)(8936002)(316002)(81166006)(2906002)(50466002)(18370500001)(2004002); DIR:OUT; SFP:1101; SCL:1; SRVR:CO2PR05MB2773; H:bryantan-devbox.prom.eng.vmware.com.prom.eng.vmware.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CO2PR05MB2773; 23:TY4mN2J5v7WhQBNnswxgliHeXPUXHQ/PaKl1aZMg2?= =?us-ascii?Q?p85IS6URzEIxhvsY43h1Q8CiCQX4z30J6VnRGqNS5WQXZIorIlhXIva2yDD7?= =?us-ascii?Q?xJ1bIreadSmG4dHiARVBm4GB7PwGhH9rNA/hhMlY6xH23sDnsZPyfHjJLDby?= =?us-ascii?Q?zep/sgleqYqdWu77dM057ExsD0vZXst/+sxE5COOdL+nLEhoOS7E96SiVtJR?= =?us-ascii?Q?heHpTk3B1JeCIeGyVqVOVveZ+SThYzEsSTa4WSyRfL0FCbUFeaDW8lzQzYT3?= =?us-ascii?Q?UGKdhIHbNg17pTwZUsmAtnY5UlZWxA9wP9ExHNaPCsWZeSWRfIyx4cjgQzIt?= =?us-ascii?Q?BhsqKtENWEva6D/DuA7xHJind5MuH93VZA9BYcdSH8Z0wIwic7roJj758E4R?= =?us-ascii?Q?s9DsXfwMSDDFvvgdT0Tamu8vIReiDCcyo+nNX27tiskObqKLVjSaa9746+gg?= =?us-ascii?Q?zMkNeY+ThBFe8fozPLZ2/1h4OdfkQQrxCZcbAiYRQLxAAno03IbJhCT5JG3P?= =?us-ascii?Q?3GtjiQKcVB8adAEhY2K1FO76TqNaxE3IksPLPyxlCY3bIBzHpG+Sad2X60Kb?= =?us-ascii?Q?91ai49a/Hi1bXrJ1E0vqZf40zWICXC4R8zuHuYINy74nXMD1MgsBxz0pElLd?= =?us-ascii?Q?Mk/gANWqWnORPnm2WoQqgD/kfxxjrRqkvQVk9Sk0/arJKHgxHaOfDkEM7Gb1?= =?us-ascii?Q?zuJXMcTopUBTju1D3/HZKqUE3zaylYLea/s5zWElOFpQAs/U+PhyppCDcWPY?= =?us-ascii?Q?a1r2AZUhqxEFlkpeda7KnLylZ3X0+FeS1AALezOeXjlq+erYCbjO6tmDjqcw?= =?us-ascii?Q?wIx7iNV34tVaQ8fzn7YTVUcPO5Npic+dPgpG8CUZX4MiQBO0gBTQDwqoacwf?= =?us-ascii?Q?JHTn7fGG8CDIKo1/+x0Y+nesJalDY6+MKBhoGytGR5qlzprlD9qT0cdybQ7h?= =?us-ascii?Q?klZGYLJkMYO+I+hREPg/8cb8MpfqGcHP0MbExrBaae10U7eXqzi3tS4qiJDx?= =?us-ascii?Q?foGyeGXUmzFJP6aPOJS6+46bIG0HUBujb9Ar/FIUaVFBUw9HhaMRQbS2EMZb?= =?us-ascii?Q?bEQsDE=3D?= X-Microsoft-Exchange-Diagnostics: 1; CO2PR05MB2773; 6:GKdwsPBIkX3l+36zIYmmYZfKhvviPzEDTvg4941PcHWCjP1qvE7brAkppwpsa1rtttaAkwKUqk1srT884+7m/8pI96+qV+rFEEEGaS7wCZwDAYLb0YhncTNOxk+j1zWcgOPktcDP+tL51WY44jIHmir58q52ZQWDOamDRnOj18KFrMbRnFs75mOqJBgCX2nO2TdeHYsNBtOqbPqkQrS3nhs81T2uKyLQgHqGy5fIH/EZxMcHa42p5G0hP8ORw6taSFyWsySCeMs5D6Po8k4Ez0XEgETJukezcXma0cI/PCepToMeJWOwyMB5ZApFwtghiBsYMyPjYWQylBnJQ6/B7RFqFcWffV9O4gB2/cbx+Pg=; 5:LKKeqE5WAF1rCzbMwkROOyrdpQXVSaECCZuYaxcbOE1WRP78uUYus0Q1NdoMwea7COgs1eyPfMPYnXtDjINhQSPuO+ju/gnAWBbwXbD74RVbykmNZfF9fnC6Buha4d1fJJB4qhT/epy40A8WzJnJSmTxwAd5zTH9rheNZqs1FWU=; 24:15EQ6AtzEnvm0reWyWtAVflZVnHRrmgcz1zLV968GeO4Vpuc5KQGShvrHHm0dpfXuuih5z0/RGxkd9IMI9JbD42ADKlrPL4v2M40WYsZVrY=; 7:i489KRVm0/6/IHSzZE+/gJpUYx0L+bas9dQjhOJPRDieUDIBgUKso0dvjUM7e4VNJfscT/gqYZJ2scGzLqNxdG2NQ4AhPcb0iN4qneVnfZa7HZ+lC+YIo0hUWuynUe0i7WWeXkf01A0KqNs7X3g58RflFpaI4EvaCmW3lTDeYaG8GXZngUN08/zt9BC9+wqcNksUZToACh3eqr1k4VZPtTBVqRiXyKcbdhhmqKX4g4NiyAR09HXrW0+7D8W30kDW SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CO2PR05MB2773; 20:JlFo/HgNIG0IA3VG7Is50LoOda3Mp9H5IMj3CgULfb3VpWaskgkGUeQRDijmRjUSMYxdcM3unnjgIKvhVEVKdytFZNkwFPTUaUg2LrqZl1GTJfTm6Iw37qJBll8alkt257XabIB/3cT93MOPs35c2ZgdPKFhSHD0Ickg3FoSLWk= X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2017 22:45:06.2246 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ba8bd6eb-78d3-4a67-35fa-08d51f1eb433 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO2PR05MB2773 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Enable userlevel applications to use SRQs. Reviewed-by: Adit Ranadive Reviewed-by: Aditya Sarwade Reviewed-by: Jorgen Hansen Reviewed-by: Nitish Bhat Signed-off-by: Bryan Tan --- buildlib/fixup-include/rdma-vmw_pvrdma-abi.h | 4 + providers/vmw_pvrdma/pvrdma-abi-fix.h | 10 + providers/vmw_pvrdma/pvrdma.h | 41 ++++ providers/vmw_pvrdma/pvrdma_main.c | 6 + providers/vmw_pvrdma/pvrdma_ring.h | 1 - providers/vmw_pvrdma/qp.c | 319 +++++++++++++++++++++++---- 6 files changed, 333 insertions(+), 48 deletions(-) diff --git a/buildlib/fixup-include/rdma-vmw_pvrdma-abi.h b/buildlib/fixup-include/rdma-vmw_pvrdma-abi.h index c8c1d2d..464d497 100644 --- a/buildlib/fixup-include/rdma-vmw_pvrdma-abi.h +++ b/buildlib/fixup-include/rdma-vmw_pvrdma-abi.h @@ -57,6 +57,8 @@ #define PVRDMA_UAR_CQ_ARM_SOL BIT(29) /* Arm solicited bit. */ #define PVRDMA_UAR_CQ_ARM BIT(30) /* Arm bit. */ #define PVRDMA_UAR_CQ_POLL BIT(31) /* Poll bit. */ +#define PVRDMA_UAR_SRQ_OFFSET 8 /* SRQ doorbell. */ +#define PVRDMA_UAR_SRQ_RECV BIT(30) /* Recv bit. */ enum pvrdma_wr_opcode { PVRDMA_WR_RDMA_WRITE, @@ -157,6 +159,8 @@ struct pvrdma_resize_cq { struct pvrdma_create_srq { __u64 buf_addr; + __u32 buf_size; + __u32 reserved; }; struct pvrdma_create_srq_resp { diff --git a/providers/vmw_pvrdma/pvrdma-abi-fix.h b/providers/vmw_pvrdma/pvrdma-abi-fix.h index 83916db..8ff9d7c 100644 --- a/providers/vmw_pvrdma/pvrdma-abi-fix.h +++ b/providers/vmw_pvrdma/pvrdma-abi-fix.h @@ -69,6 +69,16 @@ struct user_pvrdma_create_cq_resp { struct pvrdma_create_cq_resp udata; }; +struct user_pvrdma_create_srq { + struct ibv_create_srq ibv_cmd; + struct pvrdma_create_srq udata; +}; + +struct user_pvrdma_create_srq_resp { + struct ibv_create_srq_resp ibv_resp; + struct pvrdma_create_srq_resp udata; +}; + struct user_pvrdma_create_qp { struct ibv_create_qp ibv_cmd; struct pvrdma_create_qp udata; diff --git a/providers/vmw_pvrdma/pvrdma.h b/providers/vmw_pvrdma/pvrdma.h index 7840e11..5a7b171 100644 --- a/providers/vmw_pvrdma/pvrdma.h +++ b/providers/vmw_pvrdma/pvrdma.h @@ -135,6 +135,21 @@ struct pvrdma_cq { uint32_t cqn; }; +struct pvrdma_srq { + struct ibv_srq ibv_srq; + struct pvrdma_buf buf; + pthread_spinlock_t lock; + uint64_t *wrid; + uint32_t srqn; + int wqe_cnt; + int wqe_size; + int max_gs; + int wqe_shift; + struct pvrdma_ring_state *ring_state; + uint16_t counter; + int offset; +}; + struct pvrdma_wq { uint64_t *wrid; pthread_spinlock_t lock; @@ -156,6 +171,7 @@ struct pvrdma_qp { int sq_spare_wqes; struct pvrdma_wq sq; struct pvrdma_wq rq; + int is_srq; }; struct pvrdma_ah { @@ -198,6 +214,11 @@ static inline struct pvrdma_cq *to_vcq(struct ibv_cq *ibcq) return container_of(ibcq, struct pvrdma_cq, ibv_cq); } +static inline struct pvrdma_srq *to_vsrq(struct ibv_srq *ibsrq) +{ + return container_of(ibsrq, struct pvrdma_srq, ibv_srq); +} + static inline struct pvrdma_qp *to_vqp(struct ibv_qp *ibqp) { return container_of(ibqp, struct pvrdma_qp, ibv_qp); @@ -218,6 +239,11 @@ static inline void pvrdma_write_uar_cq(void *uar, unsigned value) *(__le32 *)(uar + PVRDMA_UAR_CQ_OFFSET) = htole32(value); } +static inline void pvrdma_write_uar_srq(void *uar, unsigned int value) +{ + *(__le32 *)(uar + PVRDMA_UAR_SRQ_OFFSET) = htole32(value); +} + static inline int ibv_send_flags_to_pvrdma(int flags) { return flags; @@ -301,6 +327,21 @@ int pvrdma_store_qp(struct pvrdma_context *ctx, uint32_t qpn, struct pvrdma_qp *qp); void pvrdma_clear_qp(struct pvrdma_context *ctx, uint32_t qpn); +struct ibv_srq *pvrdma_create_srq(struct ibv_pd *pd, + struct ibv_srq_init_attr *attr); +int pvrdma_modify_srq(struct ibv_srq *srq, struct ibv_srq_attr *attr, + int attr_mask); +int pvrdma_query_srq(struct ibv_srq *srq, + struct ibv_srq_attr *attr); +int pvrdma_destroy_srq(struct ibv_srq *srq); +int pvrdma_alloc_srq_buf(struct pvrdma_device *dev, + struct ibv_srq_attr *attr, + struct pvrdma_srq *srq); +int pvrdma_post_srq_recv(struct ibv_srq *ibsrq, + struct ibv_recv_wr *wr, + struct ibv_recv_wr **bad_wr); +void pvrdma_init_srq_queue(struct pvrdma_srq *srq); + struct ibv_ah *pvrdma_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr); int pvrdma_destroy_ah(struct ibv_ah *ah); diff --git a/providers/vmw_pvrdma/pvrdma_main.c b/providers/vmw_pvrdma/pvrdma_main.c index 0826629..17736cd 100644 --- a/providers/vmw_pvrdma/pvrdma_main.c +++ b/providers/vmw_pvrdma/pvrdma_main.c @@ -69,6 +69,12 @@ static struct ibv_context_ops pvrdma_ctx_ops = { .modify_qp = pvrdma_modify_qp, .destroy_qp = pvrdma_destroy_qp, + .create_srq = pvrdma_create_srq, + .modify_srq = pvrdma_modify_srq, + .query_srq = pvrdma_query_srq, + .destroy_srq = pvrdma_destroy_srq, + .post_srq_recv = pvrdma_post_srq_recv, + .post_send = pvrdma_post_send, .post_recv = pvrdma_post_recv, .create_ah = pvrdma_create_ah, diff --git a/providers/vmw_pvrdma/pvrdma_ring.h b/providers/vmw_pvrdma/pvrdma_ring.h index e247231..565b45c 100644 --- a/providers/vmw_pvrdma/pvrdma_ring.h +++ b/providers/vmw_pvrdma/pvrdma_ring.h @@ -47,7 +47,6 @@ #ifndef __PVRDMA_RING_H__ #define __PVRDMA_RING_H__ -#include #include #define PVRDMA_INVALID_IDX -1 /* Invalid index. */ diff --git a/providers/vmw_pvrdma/qp.c b/providers/vmw_pvrdma/qp.c index f5e2486..25b4841 100644 --- a/providers/vmw_pvrdma/qp.c +++ b/providers/vmw_pvrdma/qp.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2012-2016 VMware, Inc. All rights reserved. + * Copyright (c) 2012-2017 VMware, Inc. All rights reserved. * * This program is free software; you can redistribute it and/or * modify it under the terms of EITHER the GNU General Public License @@ -54,37 +54,145 @@ int pvrdma_alloc_qp_buf(struct pvrdma_device *dev, struct ibv_qp_cap *cap, if (!qp->sq.wrid) return -1; - qp->rq.wrid = calloc(qp->rq.wqe_cnt, sizeof(uint64_t)); - if (!qp->rq.wrid) { - free(qp->sq.wrid); - return -1; - } - - /* Align page size for [rq][sq] */ - qp->rbuf.length = align(qp->rq.offset + - qp->rq.wqe_cnt * qp->rq.wqe_size, - dev->page_size); + /* Align page size for sq */ qp->sbuf.length = align(qp->sq.offset + qp->sq.wqe_cnt * qp->sq.wqe_size, dev->page_size); - qp->buf_size = qp->rbuf.length + qp->sbuf.length; - - if (pvrdma_alloc_buf(&qp->rbuf, qp->rbuf.length, dev->page_size)) { - free(qp->sq.wrid); - free(qp->rq.wrid); - return -1; - } if (pvrdma_alloc_buf(&qp->sbuf, qp->sbuf.length, dev->page_size)) { free(qp->sq.wrid); - free(qp->rq.wrid); - pvrdma_free_buf(&qp->rbuf); return -1; } - memset(qp->rbuf.buf, 0, qp->rbuf.length); memset(qp->sbuf.buf, 0, qp->sbuf.length); + if (!qp->is_srq) { + qp->rq.wrid = calloc(qp->rq.wqe_cnt, sizeof(uint64_t)); + if (!qp->rq.wrid) { + pvrdma_free_buf(&qp->sbuf); + free(qp->sq.wrid); + return -1; + } + + /* Align page size for rq */ + qp->rbuf.length = align(qp->rq.offset + + qp->rq.wqe_cnt * qp->rq.wqe_size, + dev->page_size); + + if (pvrdma_alloc_buf(&qp->rbuf, qp->rbuf.length, + dev->page_size)) { + free(qp->sq.wrid); + free(qp->rq.wrid); + pvrdma_free_buf(&qp->sbuf); + return -1; + } + memset(qp->rbuf.buf, 0, qp->rbuf.length); + } else { + qp->rbuf.buf = NULL; + qp->rbuf.length = 0; + } + + qp->buf_size = qp->rbuf.length + qp->sbuf.length; + + return 0; +} + +void pvrdma_init_srq_queue(struct pvrdma_srq *srq) +{ + srq->ring_state->rx.cons_head = 0; + srq->ring_state->rx.prod_tail = 0; +} + +struct ibv_srq *pvrdma_create_srq(struct ibv_pd *pd, + struct ibv_srq_init_attr *attr) +{ + struct pvrdma_device *dev = to_vdev(pd->context->device); + struct user_pvrdma_create_srq cmd; + struct ibv_create_srq_resp resp; + struct pvrdma_srq *srq; + int ret; + + attr->attr.max_wr = align_next_power2(max_t(uint32_t, 1U, attr->attr.max_wr)); + attr->attr.max_sge = align_next_power2(max_t(uint32_t, 1U, attr->attr.max_sge)); + + srq = malloc(sizeof(*srq)); + if (!srq) + return NULL; + + if (pthread_spin_init(&srq->lock, PTHREAD_PROCESS_PRIVATE)) + goto err; + + srq->wqe_cnt = attr->attr.max_wr; + srq->max_gs = attr->attr.max_sge; + srq->wqe_size = align_next_power2(sizeof(struct pvrdma_rq_wqe_hdr) + + sizeof(struct ibv_sge) * + srq->max_gs); + /* Page reserved for queue metadata */ + srq->offset = dev->page_size; + + if (pvrdma_alloc_srq_buf(dev, &attr->attr, srq)) + goto err_spinlock; + + srq->ring_state = srq->buf.buf; + pvrdma_init_srq_queue(srq); + + memset(&cmd, 0, sizeof(cmd)); + cmd.udata.buf_addr = (uintptr_t) srq->buf.buf; + cmd.udata.buf_size = srq->buf.length; + + ret = ibv_cmd_create_srq(pd, &srq->ibv_srq, attr, + &cmd.ibv_cmd, sizeof(cmd), + &resp, sizeof(resp)); + + if (ret) + goto err_free; + + srq->srqn = resp.srqn; + + return &srq->ibv_srq; + +err_free: + free(srq->wrid); + pvrdma_free_buf(&srq->buf); +err_spinlock: + pthread_spin_destroy(&srq->lock); +err: + free(srq); + + return NULL; +} + +int pvrdma_modify_srq(struct ibv_srq *srq, + struct ibv_srq_attr *attr, + int attr_mask) +{ + struct ibv_modify_srq cmd; + + return ibv_cmd_modify_srq(srq, attr, attr_mask, &cmd, sizeof(cmd)); +} + +int pvrdma_query_srq(struct ibv_srq *srq, + struct ibv_srq_attr *attr) +{ + struct ibv_query_srq cmd; + + return ibv_cmd_query_srq(srq, attr, &cmd, sizeof(cmd)); +} + +int pvrdma_destroy_srq(struct ibv_srq *ibsrq) +{ + struct pvrdma_srq *srq = to_vsrq(ibsrq); + int ret; + + ret = ibv_cmd_destroy_srq(ibsrq); + if (ret) + return ret; + + pthread_spin_destroy(&srq->lock); + pvrdma_free_buf(&srq->buf); + free(srq->wrid); + free(srq); + return 0; } @@ -92,8 +200,10 @@ static void pvrdma_init_qp_queue(struct pvrdma_qp *qp) { qp->sq.ring_state->cons_head = 0; qp->sq.ring_state->prod_tail = 0; - qp->rq.ring_state->cons_head = 0; - qp->rq.ring_state->prod_tail = 0; + if (qp->rq.ring_state) { + qp->rq.ring_state->cons_head = 0; + qp->rq.ring_state->prod_tail = 0; + } } struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, @@ -104,26 +214,28 @@ struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, struct ibv_create_qp_resp resp; struct pvrdma_qp *qp; int ret; + int is_srq = !!(attr->srq); - attr->cap.max_recv_sge = - align_next_power2(max_t(uint32_t, 1U, attr->cap.max_recv_sge)); - attr->cap.max_recv_wr = - align_next_power2(max_t(uint32_t, 1U, attr->cap.max_recv_wr)); attr->cap.max_send_sge = align_next_power2(max_t(uint32_t, 1U, attr->cap.max_send_sge)); attr->cap.max_send_wr = align_next_power2(max_t(uint32_t, 1U, attr->cap.max_send_wr)); + if (!is_srq) { + attr->cap.max_recv_sge = + align_next_power2(max_t(uint32_t, 1U, attr->cap.max_recv_sge)); + attr->cap.max_recv_wr = + align_next_power2(max_t(uint32_t, 1U, attr->cap.max_recv_wr)); + } else { + attr->cap.max_recv_sge = 0; + attr->cap.max_recv_wr = 0; + } + qp = calloc(1, sizeof(*qp)); if (!qp) return NULL; - qp->rq.max_gs = attr->cap.max_recv_sge; - qp->rq.wqe_cnt = attr->cap.max_recv_wr; - qp->rq.offset = 0; - qp->rq.wqe_size = align_next_power2(sizeof(struct pvrdma_rq_wqe_hdr) + - sizeof(struct ibv_sge) * - qp->rq.max_gs); + qp->is_srq = is_srq; qp->sq.max_gs = attr->cap.max_send_sge; qp->sq.wqe_cnt = attr->cap.max_send_wr; @@ -133,10 +245,18 @@ struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, sizeof(struct ibv_sge) * qp->sq.max_gs); - /* Reset attr.cap, no srq for now */ - if (attr->srq) { - attr->cap.max_recv_wr = 0; + if (!is_srq) { + qp->rq.max_gs = attr->cap.max_recv_sge; + qp->rq.wqe_cnt = attr->cap.max_recv_wr; + qp->rq.offset = 0; + qp->rq.wqe_size = align_next_power2(sizeof(struct pvrdma_rq_wqe_hdr) + + sizeof(struct ibv_sge) * + qp->rq.max_gs); + } else { + qp->rq.max_gs = 0; qp->rq.wqe_cnt = 0; + qp->rq.offset = 0; + qp->rq.wqe_size = 0; } /* Allocate [rq][sq] memory */ @@ -144,18 +264,24 @@ struct ibv_qp *pvrdma_create_qp(struct ibv_pd *pd, goto err; qp->sq.ring_state = qp->sbuf.buf; - qp->rq.ring_state = (struct pvrdma_ring *)&qp->sq.ring_state[1]; - pvrdma_init_qp_queue(qp); - - if (pthread_spin_init(&qp->sq.lock, PTHREAD_PROCESS_PRIVATE) || - pthread_spin_init(&qp->rq.lock, PTHREAD_PROCESS_PRIVATE)) + if (pthread_spin_init(&qp->sq.lock, PTHREAD_PROCESS_PRIVATE)) goto err_free; + if (!is_srq) { + qp->rq.ring_state = (struct pvrdma_ring *)&qp->sq.ring_state[1]; + if (pthread_spin_init(&qp->rq.lock, PTHREAD_PROCESS_PRIVATE)) + goto err_free; + } else { + qp->rq.ring_state = NULL; + } + + pvrdma_init_qp_queue(qp); + memset(&cmd, 0, sizeof(cmd)); - cmd.udata.rbuf_addr = (uintptr_t)qp->rbuf.buf; - cmd.udata.rbuf_size = qp->rbuf.length; cmd.udata.sbuf_addr = (uintptr_t)qp->sbuf.buf; cmd.udata.sbuf_size = qp->sbuf.length; + cmd.udata.rbuf_addr = (uintptr_t)qp->rbuf.buf; + cmd.udata.rbuf_size = qp->rbuf.length; cmd.udata.qp_addr = (uintptr_t) qp; ret = ibv_cmd_create_qp(pd, &qp->ibv_qp, attr, @@ -240,9 +366,9 @@ static void pvrdma_lock_cqs(struct ibv_qp *qp) struct pvrdma_cq *send_cq = to_vcq(qp->send_cq); struct pvrdma_cq *recv_cq = to_vcq(qp->recv_cq); - if (send_cq == recv_cq) + if (send_cq == recv_cq) { pthread_spin_lock(&send_cq->lock); - else if (send_cq->cqn < recv_cq->cqn) { + } else if (send_cq->cqn < recv_cq->cqn) { pthread_spin_lock(&send_cq->lock); pthread_spin_lock(&recv_cq->lock); } else { @@ -256,9 +382,9 @@ static void pvrdma_unlock_cqs(struct ibv_qp *qp) struct pvrdma_cq *send_cq = to_vcq(qp->send_cq); struct pvrdma_cq *recv_cq = to_vcq(qp->recv_cq); - if (send_cq == recv_cq) + if (send_cq == recv_cq) { pthread_spin_unlock(&send_cq->lock); - else if (send_cq->cqn < recv_cq->cqn) { + } else if (send_cq->cqn < recv_cq->cqn) { pthread_spin_unlock(&recv_cq->lock); pthread_spin_unlock(&send_cq->lock); } else { @@ -296,6 +422,11 @@ int pvrdma_destroy_qp(struct ibv_qp *ibqp) return 0; } +static void *get_srq_wqe(struct pvrdma_srq *srq, int n) +{ + return srq->buf.buf + srq->offset + (n * srq->wqe_size); +} + static void *get_rq_wqe(struct pvrdma_qp *qp, int n) { return qp->rbuf.buf + qp->rq.offset + (n * qp->rq.wqe_size); @@ -438,6 +569,9 @@ int pvrdma_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, int i; int ret = 0; + if (qp->is_srq) + return EINVAL; + if (!wr || !bad_wr) return EINVAL; @@ -503,3 +637,94 @@ out: pthread_spin_unlock(&qp->rq.lock); return ret; } + +int pvrdma_post_srq_recv(struct ibv_srq *ibsrq, + struct ibv_recv_wr *wr, + struct ibv_recv_wr **bad_wr) +{ + struct pvrdma_context *ctx = to_vctx(ibsrq->context); + struct pvrdma_srq *srq = to_vsrq(ibsrq); + struct pvrdma_rq_wqe_hdr *wqe_hdr; + struct ibv_sge *sge; + int nreq; + int ind; + int i; + int ret = 0; + + if (!wr || !bad_wr) + return EINVAL; + + pthread_spin_lock(&srq->lock); + + ind = pvrdma_idx(&(srq->ring_state->rx.prod_tail), srq->wqe_cnt); + if (ind < 0) { + pthread_spin_unlock(&srq->lock); + *bad_wr = wr; + return EINVAL; + } + + for (nreq = 0; wr; ++nreq, wr = wr->next) { + unsigned int tail; + + if (pvrdma_idx_ring_has_space(&srq->ring_state->rx, + srq->wqe_cnt, &tail) <= 0) { + ret = ENOMEM; + *bad_wr = wr; + break; + } + + if (wr->num_sge > srq->max_gs) { + ret = EINVAL; + *bad_wr = wr; + break; + } + + /* Fetch wqe */ + wqe_hdr = (struct pvrdma_rq_wqe_hdr *)get_srq_wqe(srq, ind); + wqe_hdr->wr_id = wr->wr_id; + wqe_hdr->num_sge = wr->num_sge; + + sge = (struct ibv_sge *)(wqe_hdr + 1); + for (i = 0; i < wr->num_sge; ++i) { + sge->addr = (uint64_t)wr->sg_list[i].addr; + sge->length = wr->sg_list[i].length; + sge->lkey = wr->sg_list[i].lkey; + sge++; + } + + pvrdma_idx_ring_inc(&srq->ring_state->rx.prod_tail, + srq->wqe_cnt); + + srq->wrid[ind] = wr->wr_id; + ind = (ind + 1) & (srq->wqe_cnt - 1); + } + + if (nreq) + pvrdma_write_uar_srq(ctx->uar, + PVRDMA_UAR_SRQ_RECV | srq->srqn); + + pthread_spin_unlock(&srq->lock); + + return ret; +} + +int pvrdma_alloc_srq_buf(struct pvrdma_device *dev, + struct ibv_srq_attr *attr, + struct pvrdma_srq *srq) +{ + srq->wrid = calloc(srq->wqe_cnt, sizeof(uint64_t)); + if (!srq->wrid) + return -1; + + srq->buf.length = align(srq->offset, dev->page_size); + srq->buf.length += 2 * align(srq->wqe_cnt * srq->wqe_size, dev->page_size); + + if (pvrdma_alloc_buf(&srq->buf, srq->buf.length, dev->page_size)) { + free(srq->wrid); + return -1; + } + + memset(srq->buf.buf, 0, srq->buf.length); + + return 0; +}