From patchwork Mon Oct 28 09:14:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215015 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B294C913 for ; Mon, 28 Oct 2019 09:15:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 90845214E0 for ; Mon, 28 Oct 2019 09:15:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387878AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37886 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387876AbfJ1JP6 (ORCPT ); Mon, 28 Oct 2019 05:15:58 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fq4N032487; Mon, 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9FqL3031551; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FqXZ031550; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 1/6] Update kernel headers Date: Mon, 28 Oct 2019 11:14:54 +0200 Message-Id: <1572254099-30864-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org To commit 45b268543a8d ("RDMA/uapi: Fix and re-organize the usage of rdma_driver_id") Signed-off-by: Yishai Hadas --- kernel-headers/rdma/ib_user_ioctl_verbs.h | 22 ++++++++++++++++++++++ kernel-headers/rdma/rdma_user_ioctl_cmds.h | 22 ---------------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/kernel-headers/rdma/ib_user_ioctl_verbs.h b/kernel-headers/rdma/ib_user_ioctl_verbs.h index 72c7fc7..9019b2d 100644 --- a/kernel-headers/rdma/ib_user_ioctl_verbs.h +++ b/kernel-headers/rdma/ib_user_ioctl_verbs.h @@ -173,4 +173,26 @@ struct ib_uverbs_query_port_resp_ex { __u8 reserved[6]; }; +enum rdma_driver_id { + RDMA_DRIVER_UNKNOWN, + RDMA_DRIVER_MLX5, + RDMA_DRIVER_MLX4, + RDMA_DRIVER_CXGB3, + RDMA_DRIVER_CXGB4, + RDMA_DRIVER_MTHCA, + RDMA_DRIVER_BNXT_RE, + RDMA_DRIVER_OCRDMA, + RDMA_DRIVER_NES, + RDMA_DRIVER_I40IW, + RDMA_DRIVER_VMW_PVRDMA, + RDMA_DRIVER_QEDR, + RDMA_DRIVER_HNS, + RDMA_DRIVER_USNIC, + RDMA_DRIVER_RXE, + RDMA_DRIVER_HFI1, + RDMA_DRIVER_QIB, + RDMA_DRIVER_EFA, + RDMA_DRIVER_SIW, +}; + #endif diff --git a/kernel-headers/rdma/rdma_user_ioctl_cmds.h b/kernel-headers/rdma/rdma_user_ioctl_cmds.h index b8bb285..7b1ec80 100644 --- a/kernel-headers/rdma/rdma_user_ioctl_cmds.h +++ b/kernel-headers/rdma/rdma_user_ioctl_cmds.h @@ -84,26 +84,4 @@ struct ib_uverbs_ioctl_hdr { struct ib_uverbs_attr attrs[0]; }; -enum rdma_driver_id { - RDMA_DRIVER_UNKNOWN, - RDMA_DRIVER_MLX5, - RDMA_DRIVER_MLX4, - RDMA_DRIVER_CXGB3, - RDMA_DRIVER_CXGB4, - RDMA_DRIVER_MTHCA, - RDMA_DRIVER_BNXT_RE, - RDMA_DRIVER_OCRDMA, - RDMA_DRIVER_NES, - RDMA_DRIVER_I40IW, - RDMA_DRIVER_VMW_PVRDMA, - RDMA_DRIVER_QEDR, - RDMA_DRIVER_HNS, - RDMA_DRIVER_USNIC, - RDMA_DRIVER_RXE, - RDMA_DRIVER_HFI1, - RDMA_DRIVER_QIB, - RDMA_DRIVER_EFA, - RDMA_DRIVER_SIW, -}; - #endif From patchwork Mon Oct 28 09:14:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215019 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 420EC13B1 for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1F4D221D7F for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387872AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37885 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387874AbfJ1JP6 (ORCPT ); Mon, 28 Oct 2019 05:15:58 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fqi0032512; Mon, 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9FqcQ031555; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FqKf031554; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 2/6] verbs: custom parent-domain allocators Date: Mon, 28 Oct 2019 11:14:55 +0200 Message-Id: <1572254099-30864-3-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Haggai Eran Extend the parent domain object with custom allocation callbacks that can be used by user-applications to override the provider allocation. This can be used for example to add NUMA aware allocation. The new allocator receives context information about the parent domain, as well as the requested size and alignment of the buffer. It also receives a vendor-specific resource type code to allow customizing it for specific resources. The allocator then allocates the memory or returns an IBV_ALLOCATOR_USE_DEFAULT value to request that the provider driver use its own allocation method. Signed-off-by: Haggai Eran Signed-off-by: Yishai Hadas --- libibverbs/man/ibv_alloc_parent_domain.3 | 54 ++++++++++++++++++++++++++++++++ libibverbs/verbs.h | 12 +++++++ 2 files changed, 66 insertions(+) diff --git a/libibverbs/man/ibv_alloc_parent_domain.3 b/libibverbs/man/ibv_alloc_parent_domain.3 index 92b6058..6e2f356 100644 --- a/libibverbs/man/ibv_alloc_parent_domain.3 +++ b/libibverbs/man/ibv_alloc_parent_domain.3 @@ -41,11 +41,23 @@ The argument specifies the following: .PP .nf +enum ibv_parent_domain_init_attr_mask { +.in +8 +IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS = 1 << 0, +IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT = 1 << 1, +.in -8 +}; + struct ibv_parent_domain_init_attr { .in +8 struct ibv_pd *pd; /* referance to a protection domain, can't be NULL */ struct ibv_td *td; /* referance to a thread domain, or NULL */ uint32_t comp_mask; +void *(*alloc)(struct ibv_pd *pd, void *pd_context, size_t size, + size_t alignment, uint64_t resource_type); +void (*free)(struct ibv_pd *pd, void *pd_context, void *ptr, + uint64_t resource_type); +void *pd_context; .in -8 }; .fi @@ -56,6 +68,48 @@ will deallocate the parent domain as its exposed as an ibv_pd .I pd\fR. All resources created with the parent domain should be destroyed prior to deallocating the parent domain\fR. +.SH "ARGUMENTS" +.B pd +Reference to the protection domain that this parent domain uses. +.PP +.B td +An optional thread domain that the parent domain uses. +.PP +.B comp_mask +Bit-mask of optional fields in the ibv_parent_domain_init_attr struct. +.PP +.B alloc +Custom memory allocation function for this parent domain. Provider +memory allocations will use this function to allocate the needed memory. +The allocation function is passed the parent domain +.B pd +and the user-specified context +.B pd_context. +In addition, the callback receives the +.B size +and the +.B alignment +of the requested buffer, as well a vendor-specific +.B resource_type +, which is derived from the rdma_driver_id enum (upper 32 bits) and a vendor +specific resource code. +The function returns the pointer to the allocated buffer, or NULL to +designate an error. It may also return +.B IBV_ALLOCATOR_USE_DEFAULT +asking the callee to allocate the buffer using the default allocator. + +The callback makes sure the allocated buffer is initialized with zeros. It is +also the responsibility of the callback to make sure the memory cannot be +COWed, e.g. by using madvise(MADV_DONTFORK) or by allocating anonymous shared +memory. +.PP +.B free +Callback to free memory buffers that were allocated using a successful +alloc(). +.PP +.B pd_context +A pointer for additional user-specific data to be associated with this +parent domain. The pointer is passed back to the custom allocator functions. .SH "RETURN VALUE" .B ibv_alloc_parent_domain() returns a pointer to the allocated struct diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 12a33a9..d873f6d 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -1976,10 +1976,22 @@ struct ibv_cq_init_attr_ex { uint32_t flags; }; +enum ibv_parent_domain_init_attr_mask { + IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS = 1 << 0, + IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT = 1 << 1, +}; + +#define IBV_ALLOCATOR_USE_DEFAULT ((void *)-1) + struct ibv_parent_domain_init_attr { struct ibv_pd *pd; /* referance to a protection domain object, can't be NULL */ struct ibv_td *td; /* referance to a thread domain object, or NULL */ uint32_t comp_mask; + void *(*alloc)(struct ibv_pd *pd, void *pd_context, size_t size, + size_t alignment, uint64_t resource_type); + void (*free)(struct ibv_pd *pd, void *pd_context, void *ptr, + uint64_t resource_type); + void *pd_context; }; struct ibv_counters_init_attr { From patchwork Mon Oct 28 09:14:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215021 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A150B197C for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7ED5D21850 for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387879AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37883 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387873AbfJ1JP6 (ORCPT ); Mon, 28 Oct 2019 05:15:58 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fq6C032532; Mon, 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fql1031559; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FqPF031558; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 3/6] mlx5: Extend mlx5_alloc_parent_domain() to support custom allocator Date: Mon, 28 Oct 2019 11:14:56 +0200 Message-Id: <1572254099-30864-4-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Extend mlx5_alloc_parent_domain() to support custom allocator, downstream patches from this series will use this functionality. Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5.h | 5 +++++ providers/mlx5/verbs.c | 12 +++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index e63319e..e960e6f 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -352,6 +352,11 @@ struct mlx5_pd { struct mlx5_parent_domain { struct mlx5_pd mpd; struct mlx5_td *mtd; + void *(*alloc)(struct ibv_pd *pd, void *pd_context, size_t size, + size_t alignment, uint64_t resource_type); + void (*free)(struct ibv_pd *pd, void *pd_context, void *ptr, + uint64_t resource_type); + void *pd_context; }; enum { diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 0a66a33..166ea4f 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -326,7 +326,9 @@ mlx5_alloc_parent_domain(struct ibv_context *context, if (ibv_check_alloc_parent_domain(attr)) return NULL; - if (attr->comp_mask) { + if (!check_comp_mask(attr->comp_mask, + IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS | + IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT)) { errno = EINVAL; return NULL; } @@ -350,6 +352,14 @@ mlx5_alloc_parent_domain(struct ibv_context *context, &mparent_domain->mpd.ibv_pd, &mparent_domain->mpd.mprotection_domain->ibv_pd); + if (attr->comp_mask & IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS) { + mparent_domain->alloc = attr->alloc; + mparent_domain->free = attr->free; + } + + if (attr->comp_mask & IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT) + mparent_domain->pd_context = attr->pd_context; + return &mparent_domain->mpd.ibv_pd; } From patchwork Mon Oct 28 09:14:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215027 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 800BA197C for ; Mon, 28 Oct 2019 09:16:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5D92F214E0 for ; Mon, 28 Oct 2019 09:16:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387874AbfJ1JQA (ORCPT ); Mon, 28 Oct 2019 05:16:00 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37889 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727664AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:53 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9FqC7032542; Mon, 28 Oct 2019 11:15:52 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fqio031563; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FqJE031562; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 4/6] mlx5: Add custom allocation support for QP and RWQ buffers Date: Mon, 28 Oct 2019 11:14:57 +0200 Message-Id: <1572254099-30864-5-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add custom allocation support for QP and RWQ buffers by extending the internal allocation flows to consider this option. Signed-off-by: Yishai Hadas --- providers/mlx5/buf.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ providers/mlx5/cq.c | 2 +- providers/mlx5/mlx5.h | 6 +++++ providers/mlx5/mlx5dv.h | 4 ++++ providers/mlx5/verbs.c | 34 +++++++++++++++++++++++----- 5 files changed, 98 insertions(+), 7 deletions(-) diff --git a/providers/mlx5/buf.c b/providers/mlx5/buf.c index aa2fe02..b5cf391 100644 --- a/providers/mlx5/buf.c +++ b/providers/mlx5/buf.c @@ -356,6 +356,40 @@ int mlx5_alloc_buf_extern(struct mlx5_context *ctx, struct mlx5_buf *buf, return -1; } +static void mlx5_free_buf_custom(struct mlx5_context *ctx, + struct mlx5_buf *buf) +{ + struct mlx5_parent_domain *mparent_domain = buf->mparent_domain; + + mparent_domain->free(&mparent_domain->mpd.ibv_pd, + mparent_domain->pd_context, + buf->buf, + buf->resource_type); +} + +static int mlx5_alloc_buf_custom(struct mlx5_context *ctx, + struct mlx5_buf *buf, size_t size) +{ + struct mlx5_parent_domain *mparent_domain = buf->mparent_domain; + void *addr; + + addr = mparent_domain->alloc(&mparent_domain->mpd.ibv_pd, + mparent_domain->pd_context, size, + buf->req_alignment, + buf->resource_type); + if (addr == IBV_ALLOCATOR_USE_DEFAULT) + return 1; + + if (addr || size == 0) { + buf->buf = addr; + buf->length = size; + buf->type = MLX5_ALLOC_TYPE_CUSTOM; + return 0; + } + + return -1; +} + int mlx5_alloc_prefered_buf(struct mlx5_context *mctx, struct mlx5_buf *buf, size_t size, int page_size, @@ -364,6 +398,14 @@ int mlx5_alloc_prefered_buf(struct mlx5_context *mctx, { int ret; + if (type == MLX5_ALLOC_TYPE_CUSTOM) { + ret = mlx5_alloc_buf_custom(mctx, buf, size); + if (ret <= 0) + return ret; + + /* Fallback - default allocation is required */ + } + /* * Fallback mechanism priority: * huge pages @@ -426,6 +468,10 @@ int mlx5_free_actual_buf(struct mlx5_context *ctx, struct mlx5_buf *buf) mlx5_free_buf_extern(ctx, buf); break; + case MLX5_ALLOC_TYPE_CUSTOM: + mlx5_free_buf_custom(ctx, buf); + break; + default: fprintf(stderr, "Bad allocation type\n"); } @@ -458,12 +504,20 @@ static uint32_t mlx5_get_block_order(uint32_t v) return r; } +bool mlx5_is_custom_alloc(struct ibv_pd *pd) +{ + struct mlx5_parent_domain *mparent_domain = to_mparent_domain(pd); + + return (mparent_domain && mparent_domain->alloc && mparent_domain->free); +} + bool mlx5_is_extern_alloc(struct mlx5_context *context) { return context->extern_alloc.alloc && context->extern_alloc.free; } void mlx5_get_alloc_type(struct mlx5_context *context, + struct ibv_pd *pd, const char *component, enum mlx5_alloc_type *alloc_type, enum mlx5_alloc_type default_type) @@ -472,6 +526,11 @@ void mlx5_get_alloc_type(struct mlx5_context *context, char *env_value; char name[128]; + if (mlx5_is_custom_alloc(pd)) { + *alloc_type = MLX5_ALLOC_TYPE_CUSTOM; + return; + } + if (mlx5_is_extern_alloc(context)) { *alloc_type = MLX5_ALLOC_TYPE_EXTERNAL; return; diff --git a/providers/mlx5/cq.c b/providers/mlx5/cq.c index b9b47df..26edd86 100644 --- a/providers/mlx5/cq.c +++ b/providers/mlx5/cq.c @@ -1861,7 +1861,7 @@ int mlx5_alloc_cq_buf(struct mlx5_context *mctx, struct mlx5_cq *cq, if (mlx5_use_huge("HUGE_CQ")) default_type = MLX5_ALLOC_TYPE_HUGE; - mlx5_get_alloc_type(mctx, MLX5_CQ_PREFIX, &type, default_type); + mlx5_get_alloc_type(mctx, NULL, MLX5_CQ_PREFIX, &type, default_type); ret = mlx5_alloc_prefered_buf(mctx, buf, align(nent * cqe_sz, dev->page_size), diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index e960e6f..1c13390 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -173,6 +173,7 @@ enum mlx5_alloc_type { MLX5_ALLOC_TYPE_PREFER_HUGE, MLX5_ALLOC_TYPE_PREFER_CONTIG, MLX5_ALLOC_TYPE_EXTERNAL, + MLX5_ALLOC_TYPE_CUSTOM, MLX5_ALLOC_TYPE_ALL }; @@ -334,6 +335,9 @@ struct mlx5_buf { int base; struct mlx5_hugetlb_mem *hmem; enum mlx5_alloc_type type; + uint64_t resource_type; + size_t req_alignment; + struct mlx5_parent_domain *mparent_domain; }; struct mlx5_td { @@ -780,10 +784,12 @@ int mlx5_alloc_prefered_buf(struct mlx5_context *mctx, const char *component); int mlx5_free_actual_buf(struct mlx5_context *ctx, struct mlx5_buf *buf); void mlx5_get_alloc_type(struct mlx5_context *context, + struct ibv_pd *pd, const char *component, enum mlx5_alloc_type *alloc_type, enum mlx5_alloc_type default_alloc_type); int mlx5_use_huge(const char *key); +bool mlx5_is_custom_alloc(struct ibv_pd *pd); bool mlx5_is_extern_alloc(struct mlx5_context *context); int mlx5_alloc_buf_extern(struct mlx5_context *ctx, struct mlx5_buf *buf, size_t size); diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h index d5e8e0c..0ad9768 100644 --- a/providers/mlx5/mlx5dv.h +++ b/providers/mlx5/mlx5dv.h @@ -59,6 +59,10 @@ extern "C" { #define MLX5DV_ALWAYS_INLINE inline #endif + +#define MLX5DV_RES_TYPE_QP ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 1) +#define MLX5DV_RES_TYPE_RWQ ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 2) + enum { MLX5_RCV_DBR = 0, MLX5_SND_DBR = 1, diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 166ea4f..513af5e 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -1554,8 +1554,14 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context, if (mlx5_use_huge(qp_huge_key)) default_alloc_type = MLX5_ALLOC_TYPE_HUGE; - mlx5_get_alloc_type(to_mctx(context), MLX5_QP_PREFIX, &alloc_type, - default_alloc_type); + mlx5_get_alloc_type(to_mctx(context), attr->pd, MLX5_QP_PREFIX, + &alloc_type, default_alloc_type); + + if (alloc_type == MLX5_ALLOC_TYPE_CUSTOM) { + qp->buf.mparent_domain = to_mparent_domain(attr->pd); + qp->buf.req_alignment = to_mdev(context->device)->page_size; + qp->buf.resource_type = MLX5DV_RES_TYPE_QP; + } err = mlx5_alloc_prefered_buf(to_mctx(context), &qp->buf, align(qp->buf_size, to_mdev @@ -1569,12 +1575,20 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context, goto ex_wrid; } - memset(qp->buf.buf, 0, qp->buf_size); + if (qp->buf.type != MLX5_ALLOC_TYPE_CUSTOM) + memset(qp->buf.buf, 0, qp->buf_size); if (attr->qp_type == IBV_QPT_RAW_PACKET || qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) { size_t aligned_sq_buf_size = align(qp->sq_buf_size, to_mdev(context->device)->page_size); + + if (alloc_type == MLX5_ALLOC_TYPE_CUSTOM) { + qp->sq_buf.mparent_domain = to_mparent_domain(attr->pd); + qp->sq_buf.req_alignment = to_mdev(context->device)->page_size; + qp->sq_buf.resource_type = MLX5DV_RES_TYPE_QP; + } + /* For Raw Packet QP, allocate a separate buffer for the SQ */ err = mlx5_alloc_prefered_buf(to_mctx(context), &qp->sq_buf, aligned_sq_buf_size, @@ -1586,7 +1600,8 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context, goto rq_buf; } - memset(qp->sq_buf.buf, 0, aligned_sq_buf_size); + if (qp->sq_buf.type != MLX5_ALLOC_TYPE_CUSTOM) + memset(qp->sq_buf.buf, 0, aligned_sq_buf_size); } return 0; @@ -3121,13 +3136,14 @@ static void mlx5_free_rwq_buf(struct mlx5_rwq *rwq, struct ibv_context *context) } static int mlx5_alloc_rwq_buf(struct ibv_context *context, + struct ibv_pd *pd, struct mlx5_rwq *rwq, int size) { int err; enum mlx5_alloc_type alloc_type; - mlx5_get_alloc_type(to_mctx(context), MLX5_RWQ_PREFIX, + mlx5_get_alloc_type(to_mctx(context), pd, MLX5_RWQ_PREFIX, &alloc_type, MLX5_ALLOC_TYPE_ANON); rwq->rq.wrid = malloc(rwq->rq.wqe_cnt * sizeof(uint64_t)); @@ -3136,6 +3152,12 @@ static int mlx5_alloc_rwq_buf(struct ibv_context *context, return -1; } + if (alloc_type == MLX5_ALLOC_TYPE_CUSTOM) { + rwq->buf.mparent_domain = to_mparent_domain(pd); + rwq->buf.req_alignment = to_mdev(context->device)->page_size; + rwq->buf.resource_type = MLX5DV_RES_TYPE_RWQ; + } + err = mlx5_alloc_prefered_buf(to_mctx(context), &rwq->buf, align(rwq->buf_size, to_mdev (context->device)->page_size), @@ -3186,7 +3208,7 @@ static struct ibv_wq *create_wq(struct ibv_context *context, } rwq->buf_size = ret; - if (mlx5_alloc_rwq_buf(context, rwq, ret)) + if (mlx5_alloc_rwq_buf(context, attr->pd, rwq, ret)) goto err; mlx5_init_rwq_indices(rwq); From patchwork Mon Oct 28 09:14:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215025 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B068913 for ; Mon, 28 Oct 2019 09:16:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EC6E021850 for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387875AbfJ1JQA (ORCPT ); Mon, 28 Oct 2019 05:16:00 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37887 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387877AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:53 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9FqCZ032570; Mon, 28 Oct 2019 11:15:53 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fqd5031567; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FqXU031566; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 5/6] mlx5: Add custom allocation support for DBR Date: Mon, 28 Oct 2019 11:14:58 +0200 Message-Id: <1572254099-30864-6-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add custom allocation support for DBR by extending the internal allocation/free flows to consider this option. Signed-off-by: Yishai Hadas --- providers/mlx5/dbrec.c | 34 ++++++++++++++++++++++++++++++++-- providers/mlx5/mlx5.h | 9 +++++++-- providers/mlx5/mlx5dv.h | 1 + providers/mlx5/verbs.c | 47 +++++++++++++++++++++++++++-------------------- 4 files changed, 67 insertions(+), 24 deletions(-) diff --git a/providers/mlx5/dbrec.c b/providers/mlx5/dbrec.c index d6a69a5..5ef3d16 100644 --- a/providers/mlx5/dbrec.c +++ b/providers/mlx5/dbrec.c @@ -85,12 +85,31 @@ static struct mlx5_db_page *__add_page(struct mlx5_context *context) return page; } -__be32 *mlx5_alloc_dbrec(struct mlx5_context *context) +__be32 *mlx5_alloc_dbrec(struct mlx5_context *context, struct ibv_pd *pd, + bool *custom_alloc) { struct mlx5_db_page *page; __be32 *db = NULL; int i, j; + if (mlx5_is_custom_alloc(pd)) { + struct mlx5_parent_domain *mparent_domain = to_mparent_domain(pd); + + db = mparent_domain->alloc(&mparent_domain->mpd.ibv_pd, + mparent_domain->pd_context, 8, 8, + MLX5DV_RES_TYPE_DBR); + + if (db == IBV_ALLOCATOR_USE_DEFAULT) + goto default_alloc; + + if (!db) + return NULL; + + *custom_alloc = true; + return db; + } + +default_alloc: pthread_mutex_lock(&context->db_list_mutex); for (page = context->db_list; page; page = page->next) @@ -118,12 +137,23 @@ out: return db; } -void mlx5_free_db(struct mlx5_context *context, __be32 *db) +void mlx5_free_db(struct mlx5_context *context, __be32 *db, struct ibv_pd *pd, + bool custom_alloc) { struct mlx5_db_page *page; uintptr_t ps = to_mdev(context->ibv_ctx.context.device)->page_size; int i; + if (custom_alloc) { + struct mlx5_parent_domain *mparent_domain = to_mparent_domain(pd); + + mparent_domain->free(&mparent_domain->mpd.ibv_pd, + mparent_domain->pd_context, + db, + MLX5DV_RES_TYPE_DBR); + return; + } + pthread_mutex_lock(&context->db_list_mutex); for (page = context->db_list; page; page = page->next) diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 1c13390..9a5cd6b 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -438,6 +438,7 @@ struct mlx5_srq { int waitq_head; int waitq_tail; __be32 *db; + bool custom_db; uint16_t counter; int wq_sig; struct ibv_qp *cmd_qp; @@ -553,6 +554,7 @@ struct mlx5_qp { struct mlx5_wq sq; __be32 *db; + bool custom_db; struct mlx5_wq rq; int wq_sig; uint32_t qp_cap_cache; @@ -582,6 +584,7 @@ struct mlx5_rwq { int buf_size; struct mlx5_wq rq; __be32 *db; + bool custom_db; void *pbuff; __be32 *recv_db; int wq_sig; @@ -795,8 +798,10 @@ int mlx5_alloc_buf_extern(struct mlx5_context *ctx, struct mlx5_buf *buf, size_t size); void mlx5_free_buf_extern(struct mlx5_context *ctx, struct mlx5_buf *buf); -__be32 *mlx5_alloc_dbrec(struct mlx5_context *context); -void mlx5_free_db(struct mlx5_context *context, __be32 *db); +__be32 *mlx5_alloc_dbrec(struct mlx5_context *context, struct ibv_pd *pd, + bool *custom_alloc); +void mlx5_free_db(struct mlx5_context *context, __be32 *db, struct ibv_pd *pd, + bool custom_alloc); int mlx5_query_device(struct ibv_context *context, struct ibv_device_attr *attr); diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h index 0ad9768..ac291eb 100644 --- a/providers/mlx5/mlx5dv.h +++ b/providers/mlx5/mlx5dv.h @@ -62,6 +62,7 @@ extern "C" { #define MLX5DV_RES_TYPE_QP ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 1) #define MLX5DV_RES_TYPE_RWQ ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 2) +#define MLX5DV_RES_TYPE_DBR ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 3) enum { MLX5_RCV_DBR = 0, diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 513af5e..1e026af 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -749,7 +749,7 @@ static struct ibv_cq_ex *create_cq(struct ibv_context *context, goto err_spl; } - cq->dbrec = mlx5_alloc_dbrec(to_mctx(context)); + cq->dbrec = mlx5_alloc_dbrec(to_mctx(context), NULL, NULL); if (!cq->dbrec) { mlx5_dbg(fp, MLX5_DBG_CQ, "\n"); goto err_buf; @@ -834,7 +834,7 @@ static struct ibv_cq_ex *create_cq(struct ibv_context *context, return &cq->ibv_cq; err_db: - mlx5_free_db(to_mctx(context), cq->dbrec); + mlx5_free_db(to_mctx(context), cq->dbrec, NULL, 0); err_buf: mlx5_free_cq_buf(to_mctx(context), &cq->buf_a); @@ -968,7 +968,7 @@ int mlx5_destroy_cq(struct ibv_cq *cq) if (ret) return ret; - mlx5_free_db(to_mctx(cq->context), to_mcq(cq)->dbrec); + mlx5_free_db(to_mctx(cq->context), to_mcq(cq)->dbrec, NULL, 0); mlx5_free_cq_buf(to_mctx(cq->context), to_mcq(cq)->active_buf); free(to_mcq(cq)); @@ -1028,13 +1028,14 @@ struct ibv_srq *mlx5_create_srq(struct ibv_pd *pd, goto err; } - srq->db = mlx5_alloc_dbrec(to_mctx(pd->context)); + srq->db = mlx5_alloc_dbrec(to_mctx(pd->context), pd, &srq->custom_db); if (!srq->db) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err_free; } - *srq->db = 0; + if (!srq->custom_db) + *srq->db = 0; cmd.buf_addr = (uintptr_t) srq->buf.buf; cmd.db_addr = (uintptr_t) srq->db; @@ -1077,7 +1078,7 @@ err_destroy: err_db: pthread_mutex_unlock(&ctx->srq_table_mutex); - mlx5_free_db(to_mctx(pd->context), srq->db); + mlx5_free_db(to_mctx(pd->context), srq->db, pd, srq->custom_db); err_free: free(srq->wrid); @@ -1128,7 +1129,7 @@ int mlx5_destroy_srq(struct ibv_srq *srq) else mlx5_clear_srq(ctx, msrq->srqn); - mlx5_free_db(ctx, msrq->db); + mlx5_free_db(ctx, msrq->db, srq->pd, msrq->custom_db); mlx5_free_buf(&msrq->buf); free(msrq->tm_list); free(msrq->wrid); @@ -2033,14 +2034,16 @@ static struct ibv_qp *create_qp(struct ibv_context *context, mlx5_spinlock_init_pd(&qp->rq.lock, attr->pd)) goto err_free_qp_buf; - qp->db = mlx5_alloc_dbrec(ctx); + qp->db = mlx5_alloc_dbrec(ctx, attr->pd, &qp->custom_db); if (!qp->db) { mlx5_dbg(fp, MLX5_DBG_QP, "\n"); goto err_free_qp_buf; } - qp->db[MLX5_RCV_DBR] = 0; - qp->db[MLX5_SND_DBR] = 0; + if (!qp->custom_db) { + qp->db[MLX5_RCV_DBR] = 0; + qp->db[MLX5_SND_DBR] = 0; + } cmd.buf_addr = (uintptr_t) qp->buf.buf; cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET || @@ -2147,7 +2150,7 @@ err_free_uidx: mlx5_clear_uidx(ctx, usr_idx); err_rq_db: - mlx5_free_db(to_mctx(context), qp->db); + mlx5_free_db(to_mctx(context), qp->db, attr->pd, qp->custom_db); err_free_qp_buf: mlx5_free_qp_buf(ctx, qp); @@ -2270,7 +2273,7 @@ int mlx5_destroy_qp(struct ibv_qp *ibqp) mlx5_clear_uidx(ctx, qp->rsc.rsn); if (qp->dc_type != MLX5DV_DCTYPE_DCT) { - mlx5_free_db(ctx, qp->db); + mlx5_free_db(ctx, qp->db, ibqp->pd, qp->custom_db); mlx5_free_qp_buf(ctx, qp); } free: @@ -2892,13 +2895,14 @@ struct ibv_srq *mlx5_create_srq_ex(struct ibv_context *context, goto err; } - msrq->db = mlx5_alloc_dbrec(ctx); + msrq->db = mlx5_alloc_dbrec(ctx, attr->pd, &msrq->custom_db); if (!msrq->db) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err_free; } - *msrq->db = 0; + if (!msrq->custom_db) + *msrq->db = 0; cmd.buf_addr = (uintptr_t)msrq->buf.buf; cmd.db_addr = (uintptr_t)msrq->db; @@ -2990,7 +2994,7 @@ err_free_uidx: pthread_mutex_unlock(&ctx->srq_table_mutex); err_free_db: - mlx5_free_db(ctx, msrq->db); + mlx5_free_db(ctx, msrq->db, attr->pd, msrq->custom_db); err_free: free(msrq->wrid); @@ -3216,12 +3220,15 @@ static struct ibv_wq *create_wq(struct ibv_context *context, if (mlx5_spinlock_init_pd(&rwq->rq.lock, attr->pd)) goto err_free_rwq_buf; - rwq->db = mlx5_alloc_dbrec(ctx); + rwq->db = mlx5_alloc_dbrec(ctx, attr->pd, &rwq->custom_db); if (!rwq->db) goto err_free_rwq_buf; - rwq->db[MLX5_RCV_DBR] = 0; - rwq->db[MLX5_SND_DBR] = 0; + if (!rwq->custom_db) { + rwq->db[MLX5_RCV_DBR] = 0; + rwq->db[MLX5_SND_DBR] = 0; + } + rwq->pbuff = rwq->buf.buf + rwq->rq.offset; rwq->recv_db = &rwq->db[MLX5_RCV_DBR]; cmd.buf_addr = (uintptr_t)rwq->buf.buf; @@ -3278,7 +3285,7 @@ static struct ibv_wq *create_wq(struct ibv_context *context, err_create: mlx5_clear_uidx(ctx, cmd.user_index); err_free_db_rec: - mlx5_free_db(to_mctx(context), rwq->db); + mlx5_free_db(to_mctx(context), rwq->db, attr->pd, rwq->custom_db); err_free_rwq_buf: mlx5_free_rwq_buf(rwq, context); err: @@ -3342,7 +3349,7 @@ int mlx5_destroy_wq(struct ibv_wq *wq) __mlx5_cq_clean(to_mcq(wq->cq), rwq->rsc.rsn, NULL); mlx5_spin_unlock(&to_mcq(wq->cq)->lock); mlx5_clear_uidx(to_mctx(wq->context), rwq->rsc.rsn); - mlx5_free_db(to_mctx(wq->context), rwq->db); + mlx5_free_db(to_mctx(wq->context), rwq->db, wq->pd, rwq->custom_db); mlx5_free_rwq_buf(rwq, wq->context); free(rwq); From patchwork Mon Oct 28 09:14:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11215017 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15BA71515 for ; Mon, 28 Oct 2019 09:16:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F118E21850 for ; Mon, 28 Oct 2019 09:15:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387876AbfJ1JP7 (ORCPT ); Mon, 28 Oct 2019 05:15:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37884 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387872AbfJ1JP6 (ORCPT ); Mon, 28 Oct 2019 05:15:58 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 28 Oct 2019 11:15:53 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Frvv032588; Mon, 28 Oct 2019 11:15:53 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x9S9Fqr1031571; Mon, 28 Oct 2019 11:15:52 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x9S9FquJ031570; Mon, 28 Oct 2019 11:15:52 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, haggaie@mellanox.com, jgg@mellanox.com, maorg@mellanox.com Subject: [PATCH rdma-core 6/6] mlx5: Add custom allocation support for SRQ buffer Date: Mon, 28 Oct 2019 11:14:59 +0200 Message-Id: <1572254099-30864-7-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> References: <1572254099-30864-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add custom allocation support for SRQ buffer by extending the internal allocation flow to consider this option. As part of this change other options that were missed as of "extern allocator" became applicable as well. Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5.h | 3 ++- providers/mlx5/mlx5dv.h | 1 + providers/mlx5/srq.c | 25 ++++++++++++++++++++----- providers/mlx5/verbs.c | 10 +++++----- 4 files changed, 28 insertions(+), 11 deletions(-) diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 9a5cd6b..953abe2 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -71,6 +71,7 @@ enum { #define MLX5_QP_PREFIX "MLX_QP" #define MLX5_MR_PREFIX "MLX_MR" #define MLX5_RWQ_PREFIX "MLX_RWQ" +#define MLX5_SRQ_PREFIX "MLX_SRQ" #define MLX5_MAX_LOG2_CONTIG_BLOCK_SIZE 23 #define MLX5_MIN_LOG2_CONTIG_BLOCK_SIZE 12 @@ -863,7 +864,7 @@ int mlx5_query_srq(struct ibv_srq *srq, struct ibv_srq_attr *attr); int mlx5_destroy_srq(struct ibv_srq *srq); int mlx5_alloc_srq_buf(struct ibv_context *context, struct mlx5_srq *srq, - uint32_t nwr); + uint32_t nwr, struct ibv_pd *pd); void mlx5_complete_odp_fault(struct mlx5_srq *srq, int ind); void mlx5_free_srq_wqe(struct mlx5_srq *srq, int ind); int mlx5_post_srq_recv(struct ibv_srq *ibsrq, diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h index ac291eb..1f49352 100644 --- a/providers/mlx5/mlx5dv.h +++ b/providers/mlx5/mlx5dv.h @@ -63,6 +63,7 @@ extern "C" { #define MLX5DV_RES_TYPE_QP ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 1) #define MLX5DV_RES_TYPE_RWQ ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 2) #define MLX5DV_RES_TYPE_DBR ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 3) +#define MLX5DV_RES_TYPE_SRQ ((uint64_t)RDMA_DRIVER_MLX5 << 32 | 4) enum { MLX5_RCV_DBR = 0, diff --git a/providers/mlx5/srq.c b/providers/mlx5/srq.c index 1c15656..e9568c6 100644 --- a/providers/mlx5/srq.c +++ b/providers/mlx5/srq.c @@ -250,13 +250,14 @@ static void set_srq_buf_ll(struct mlx5_srq *srq, int start, int end) } int mlx5_alloc_srq_buf(struct ibv_context *context, struct mlx5_srq *srq, - uint32_t max_wr) + uint32_t max_wr, struct ibv_pd *pd) { int size; int buf_size; struct mlx5_context *ctx; uint32_t orig_max_wr = max_wr; bool have_wq = true; + enum mlx5_alloc_type alloc_type; ctx = to_mctx(context); @@ -296,11 +297,25 @@ int mlx5_alloc_srq_buf(struct ibv_context *context, struct mlx5_srq *srq, srq->max = align_queue_size(max_wr); buf_size = srq->max * size; - if (mlx5_alloc_buf(&srq->buf, buf_size, - to_mdev(context->device)->page_size)) + mlx5_get_alloc_type(ctx, pd, MLX5_SRQ_PREFIX, &alloc_type, + MLX5_ALLOC_TYPE_ANON); + + if (alloc_type == MLX5_ALLOC_TYPE_CUSTOM) { + srq->buf.mparent_domain = to_mparent_domain(pd); + srq->buf.req_alignment = to_mdev(context->device)->page_size; + srq->buf.resource_type = MLX5DV_RES_TYPE_SRQ; + } + + if (mlx5_alloc_prefered_buf(ctx, + &srq->buf, buf_size, + to_mdev(context->device)->page_size, + alloc_type, + MLX5_SRQ_PREFIX)) return -1; - memset(srq->buf.buf, 0, buf_size); + if (srq->buf.type != MLX5_ALLOC_TYPE_CUSTOM) + memset(srq->buf.buf, 0, buf_size); + srq->head = 0; srq->tail = align_queue_size(orig_max_wr + 1) - 1; if (have_wq) { @@ -313,7 +328,7 @@ int mlx5_alloc_srq_buf(struct ibv_context *context, struct mlx5_srq *srq, srq->wrid = malloc(srq->max * sizeof(*srq->wrid)); if (!srq->wrid) { - mlx5_free_buf(&srq->buf); + mlx5_free_actual_buf(ctx, &srq->buf); return -1; } diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 1e026af..786a75e 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -1023,7 +1023,7 @@ struct ibv_srq *mlx5_create_srq(struct ibv_pd *pd, srq->max_gs = attr->attr.max_sge; srq->counter = 0; - if (mlx5_alloc_srq_buf(pd->context, srq, attr->attr.max_wr)) { + if (mlx5_alloc_srq_buf(pd->context, srq, attr->attr.max_wr, pd)) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err; } @@ -1082,7 +1082,7 @@ err_db: err_free: free(srq->wrid); - mlx5_free_buf(&srq->buf); + mlx5_free_actual_buf(ctx, &srq->buf); err: free(srq); @@ -1130,7 +1130,7 @@ int mlx5_destroy_srq(struct ibv_srq *srq) mlx5_clear_srq(ctx, msrq->srqn); mlx5_free_db(ctx, msrq->db, srq->pd, msrq->custom_db); - mlx5_free_buf(&msrq->buf); + mlx5_free_actual_buf(ctx, &msrq->buf); free(msrq->tm_list); free(msrq->wrid); free(msrq->op); @@ -2890,7 +2890,7 @@ struct ibv_srq *mlx5_create_srq_ex(struct ibv_context *context, msrq->max_gs = attr->attr.max_sge; msrq->counter = 0; - if (mlx5_alloc_srq_buf(context, msrq, attr->attr.max_wr)) { + if (mlx5_alloc_srq_buf(context, msrq, attr->attr.max_wr, attr->pd)) { fprintf(stderr, "%s-%d:\n", __func__, __LINE__); goto err; } @@ -2998,7 +2998,7 @@ err_free_db: err_free: free(msrq->wrid); - mlx5_free_buf(&msrq->buf); + mlx5_free_actual_buf(ctx, &msrq->buf); err: free(msrq);