diff mbox series

[rdma-core,2/6] verbs: custom parent-domain allocators

Message ID 1572254099-30864-3-git-send-email-yishaih@mellanox.com (mailing list archive)
State Not Applicable
Headers show
Series verbs: Custom parent-domain allocators | expand

Commit Message

Yishai Hadas Oct. 28, 2019, 9:14 a.m. UTC
From: Haggai Eran <haggaie@mellanox.com>

Extend the parent domain object with custom allocation callbacks that
can be used by user-applications to override the provider allocation.

This can be used for example to add NUMA aware allocation.

The new allocator receives context information about the parent domain,
as well as the requested size and alignment of the buffer. It also
receives a vendor-specific resource type code to allow customizing it
for specific resources.

The allocator then allocates the memory or returns an
IBV_ALLOCATOR_USE_DEFAULT value to request that the provider driver use
its own allocation method.

Signed-off-by: Haggai Eran <haggaie@mellanox.com>
Signed-off-by: Yishai Hadas <yishaih@mellanox.com>
---
 libibverbs/man/ibv_alloc_parent_domain.3 | 54 ++++++++++++++++++++++++++++++++
 libibverbs/verbs.h                       | 12 +++++++
 2 files changed, 66 insertions(+)

Comments

Leon Romanovsky Oct. 28, 2019, 9:48 a.m. UTC | #1
On Mon, Oct 28, 2019 at 11:14:55AM +0200, Yishai Hadas wrote:
> From: Haggai Eran <haggaie@mellanox.com>
>
> Extend the parent domain object with custom allocation callbacks that
> can be used by user-applications to override the provider allocation.
>
> This can be used for example to add NUMA aware allocation.
>
> The new allocator receives context information about the parent domain,
> as well as the requested size and alignment of the buffer. It also
> receives a vendor-specific resource type code to allow customizing it
> for specific resources.
>
> The allocator then allocates the memory or returns an
> IBV_ALLOCATOR_USE_DEFAULT value to request that the provider driver use
> its own allocation method.
>
> Signed-off-by: Haggai Eran <haggaie@mellanox.com>
> Signed-off-by: Yishai Hadas <yishaih@mellanox.com>
> ---
>  libibverbs/man/ibv_alloc_parent_domain.3 | 54 ++++++++++++++++++++++++++++++++
>  libibverbs/verbs.h                       | 12 +++++++
>  2 files changed, 66 insertions(+)
>

It is unclear to me how and maybe it is not possible for this API. but I
would expect any changes in public API be accompanied by relevant tests.

Thanks
Yishai Hadas Oct. 29, 2019, 10:16 a.m. UTC | #2
On 10/28/2019 11:48 AM, Leon Romanovsky wrote:
> On Mon, Oct 28, 2019 at 11:14:55AM +0200, Yishai Hadas wrote:
>> From: Haggai Eran <haggaie@mellanox.com>
>>
>> Extend the parent domain object with custom allocation callbacks that
>> can be used by user-applications to override the provider allocation.
>>
>> This can be used for example to add NUMA aware allocation.
>>
>> The new allocator receives context information about the parent domain,
>> as well as the requested size and alignment of the buffer. It also
>> receives a vendor-specific resource type code to allow customizing it
>> for specific resources.
>>
>> The allocator then allocates the memory or returns an
>> IBV_ALLOCATOR_USE_DEFAULT value to request that the provider driver use
>> its own allocation method.
>>
>> Signed-off-by: Haggai Eran <haggaie@mellanox.com>
>> Signed-off-by: Yishai Hadas <yishaih@mellanox.com>
>> ---
>>   libibverbs/man/ibv_alloc_parent_domain.3 | 54 ++++++++++++++++++++++++++++++++
>>   libibverbs/verbs.h                       | 12 +++++++
>>   2 files changed, 66 insertions(+)
>>
> 
> It is unclear to me how and maybe it is not possible for this API. but I
> would expect any changes in public API be accompanied by relevant tests.
> 

The current pyverbs infrastructure has some lack in this area to enable 
adding a simple test over.
The infrastructure team is working to find a solution and may push it 
once be ready later on.

Yishai
diff mbox series

Patch

diff --git a/libibverbs/man/ibv_alloc_parent_domain.3 b/libibverbs/man/ibv_alloc_parent_domain.3
index 92b6058..6e2f356 100644
--- a/libibverbs/man/ibv_alloc_parent_domain.3
+++ b/libibverbs/man/ibv_alloc_parent_domain.3
@@ -41,11 +41,23 @@  The
 argument specifies the following:
 .PP
 .nf
+enum ibv_parent_domain_init_attr_mask {
+.in +8
+IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS = 1 << 0,
+IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT = 1 << 1,
+.in -8
+};
+
 struct ibv_parent_domain_init_attr {
 .in +8
 struct ibv_pd *pd; /* referance to a protection domain, can't be NULL */
 struct ibv_td *td; /* referance to a thread domain, or NULL */
 uint32_t comp_mask;
+void *(*alloc)(struct ibv_pd *pd, void *pd_context, size_t size,
+               size_t alignment, uint64_t resource_type);
+void (*free)(struct ibv_pd *pd, void *pd_context, void *ptr,
+             uint64_t resource_type);
+void *pd_context;
 .in -8
 };
 .fi
@@ -56,6 +68,48 @@  will deallocate the parent domain as its exposed as an ibv_pd
 .I pd\fR.
 All resources created with the parent domain
 should be destroyed prior to deallocating the parent domain\fR.
+.SH "ARGUMENTS"
+.B pd
+Reference to the protection domain that this parent domain uses.
+.PP
+.B td
+An optional thread domain that the parent domain uses.
+.PP
+.B comp_mask
+Bit-mask of optional fields in the ibv_parent_domain_init_attr struct.
+.PP
+.B alloc
+Custom memory allocation function for this parent domain. Provider
+memory allocations will use this function to allocate the needed memory.
+The allocation function is passed the parent domain
+.B pd
+and the user-specified context
+.B pd_context.
+In addition, the callback receives the
+.B size
+and the
+.B alignment
+of the requested buffer, as well a vendor-specific
+.B resource_type
+, which is derived from the rdma_driver_id enum (upper 32 bits) and a vendor
+specific resource code.
+The function returns the pointer to the allocated buffer, or NULL to
+designate an error.  It may also return
+.B IBV_ALLOCATOR_USE_DEFAULT
+asking the callee to allocate the buffer using the default allocator.
+
+The callback makes sure the allocated buffer is initialized with zeros. It is
+also the responsibility of the callback to make sure the memory cannot be
+COWed, e.g. by using madvise(MADV_DONTFORK) or by allocating anonymous shared
+memory.
+.PP
+.B free
+Callback to free memory buffers that were allocated using a successful
+alloc().
+.PP
+.B pd_context
+A pointer for additional user-specific data to be associated with this
+parent domain. The pointer is passed back to the custom allocator functions.
 .SH "RETURN VALUE"
 .B ibv_alloc_parent_domain()
 returns a pointer to the allocated struct
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 12a33a9..d873f6d 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -1976,10 +1976,22 @@  struct ibv_cq_init_attr_ex {
 	uint32_t		flags;
 };
 
+enum ibv_parent_domain_init_attr_mask {
+	IBV_PARENT_DOMAIN_INIT_ATTR_ALLOCATORS = 1 << 0,
+	IBV_PARENT_DOMAIN_INIT_ATTR_PD_CONTEXT = 1 << 1,
+};
+
+#define IBV_ALLOCATOR_USE_DEFAULT ((void *)-1)
+
 struct ibv_parent_domain_init_attr {
 	struct ibv_pd *pd; /* referance to a protection domain object, can't be NULL */
 	struct ibv_td *td; /* referance to a thread domain object, or NULL */
 	uint32_t comp_mask;
+	void *(*alloc)(struct ibv_pd *pd, void *pd_context, size_t size,
+		       size_t alignment, uint64_t resource_type);
+	void (*free)(struct ibv_pd *pd, void *pd_context, void *ptr,
+		     uint64_t resource_type);
+	void *pd_context;
 };
 
 struct ibv_counters_init_attr {