From patchwork Sun Sep 17 11:48:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 9954767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8CAF760352 for ; Sun, 17 Sep 2017 11:49:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71525287E7 for ; Sun, 17 Sep 2017 11:49:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65D312880C; Sun, 17 Sep 2017 11:49:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAAFE287E7 for ; Sun, 17 Sep 2017 11:49:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751370AbdIQLt0 (ORCPT ); Sun, 17 Sep 2017 07:49:26 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:60478 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751361AbdIQLtZ (ORCPT ); Sun, 17 Sep 2017 07:49:25 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Sep 2017 14:49:21 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id v8HBnLLx004656; Sun, 17 Sep 2017 14:49:21 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id v8HBnLNI021476; Sun, 17 Sep 2017 14:49:21 +0300 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id v8HBnIHN021465; Sun, 17 Sep 2017 14:49:18 +0300 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, majd@mellanox.com, Alexr@mellanox.com Subject: [PATCH RFC rdma-core] Verbs: Introduce resource domain Date: Sun, 17 Sep 2017 14:48:42 +0300 Message-Id: <1505648922-21346-1-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Resource domain represents a collection of IB resources (as of QP, CQ, etc.) which are accessed by an application from the same context (e.g. same thread) and as such can share hardware and software resources to allow smarter management of resources by the provider library and better performance. Today, without the resource domain, provider libraries ensure thread safety for all resources and implement some arbitrary logic regarding hardware resource management (e.g. pre-allocated limit pools, dynamic allocation, round-robin vs lru, etc.). We would like to allow the application to explicitly specify which resources are going to be used by which resource domains, and by that allow the libraries to improve the HW and SW resource utilization. When application creates several resources on the same resource domain index, the provider library can share the related hardware resources between them and as such better utilize its available resources. On the other hand if each resource is created by the application on a dedicated resource domain index it will use a dedicated hardware resource. For example if few QPs are used by the application from the same thread the hardware doorbell can be shared between them. This drops the need to allocate a dedicate doorbell per QP or share doorbells which are handled by different threads. On the other hand if few QPs are not logically related an application can enjoy the option to get a dedicated resource per QP and prevent the need to take a lock in its data path flow to protect that doorbell update operations. This RFC patch introduces the following verbs extensions to support that: 1) Extend ibv_query_device_ex to return max_resource_domains. This attribute represents range of valid [0..max_resource_domains] device domains that resources can be part of. Each driver which supports this feature will return a positive value, declaring its max unique resource domains that can be used. 2) Extend ibv_create_qp_ex to get resource_domain to enable associating a QP into a specific resource domain. Each library will use the input value to manage the related hardware resources, and reduce the amount of locks required. Application is required to use a single context when accessing any of the IB objects per each resource domain. Specifically, we plan to use this API in mlx5 driver to better manage the doorbell (UAR) hardware resource which is a limited hardware resource by exposing an option from kernel to allocate it on demand and share among QPs or attach it to a specific a QP based on application needs which are represented by the resource domain property. Signed-off-by: Yishai Hadas --- libibverbs/verbs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 8cdf8ab..b2b51c1 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -272,6 +272,7 @@ struct ibv_device_attr_ex { uint32_t max_wq_type_rq; struct ibv_packet_pacing_caps packet_pacing_caps; uint32_t raw_packet_caps; /* Use ibv_raw_packet_caps */ + uint32_t max_resource_domains; }; enum ibv_mtu { @@ -785,7 +786,8 @@ enum ibv_qp_init_attr_mask { IBV_QP_INIT_ATTR_MAX_TSO_HEADER = 1 << 3, IBV_QP_INIT_ATTR_IND_TABLE = 1 << 4, IBV_QP_INIT_ATTR_RX_HASH = 1 << 5, - IBV_QP_INIT_ATTR_RESERVED = 1 << 6 + IBV_QP_INIT_ATTR_RES_DOMAIN = 1 << 6, + IBV_QP_INIT_ATTR_RESERVED = 1 << 7 }; enum ibv_qp_create_flags { @@ -821,6 +823,7 @@ struct ibv_qp_init_attr_ex { struct ibv_rwq_ind_table *rwq_ind_tbl; struct ibv_rx_hash_conf rx_hash_conf; uint32_t source_qpn; + uint32_t resource_domain; }; enum ibv_qp_open_attr_mask {