diff mbox

[v3,rdma-next,2/5] RDMA/nldev: add provider-specific resource tracking

Message ID 39b6b4ad37dc30697ce4eacace136c26abc54dd6.1522433107.git.swise@opengridcomputing.com (mailing list archive)
State Changes Requested
Delegated to: Jason Gunthorpe
Headers show

Commit Message

Steve Wise March 30, 2018, 6:03 p.m. UTC 
Each provider can register a "fill entry" function with the restrack core.
This function will be called when filling out a resource, allowing the
provider to add provider-specific details.  The details consist of a
nltable of nested attributes, that are in the form of <key, [print-type],
value> tuples.  Both key and value attributes are mandatory.  The key
nlattr must be a string, and the value nlattr can be one of the provider
attributes that are generic, but typed, allowing the attributes to be
validated.  Currently the provider nlattr types include string, s32,
u32, s64, and u64.  The print-type nlattr allows a provider to specify
an alternative display format for user tools displaying the attribute.
For example, a u32 attribute will default to "%u", but a print-type
attribute can be included for it to be displayed in hex.  This allows
the user tool to print the number in the format desired by the provider
driver.

More attrs can be defined as they become needed by providers.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/nldev.c    | 37 +++++++++++++++++++++++++++++++++++++
 drivers/infiniband/core/restrack.c |  1 +
 include/rdma/restrack.h            |  9 +++++++++
 include/uapi/rdma/rdma_netlink.h   | 20 ++++++++++++++++++++
 4 files changed, 67 insertions(+)

Comments

Jason Gunthorpe April 20, 2018, 6:09 p.m. UTC | #1 
On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
> index 884843e..1a680a3 100644
> +++ b/drivers/infiniband/core/nldev.c
> @@ -95,8 +95,25 @@
>  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type = NLA_NESTED },
>  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
>  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type = NLA_U32 },
> +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type = NLA_NESTED },
> +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type = NLA_NESTED },
> +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type = NLA_NUL_STRING,
> +				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
> +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
> +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
> +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
> +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
> +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },

Why do we need 64 and 32 bit version of this? Pass everything as s64
or u64?

> +static int provider_fill_res_entry(struct rdma_restrack_root *resroot,
> +				   struct sk_buff *msg,
> +				   struct rdma_restrack_entry *res)
> +{
> +	return resroot->fill_res_entry ?
> +		resroot->fill_res_entry(msg, res) : 0;
> +}

Just init resroot->fill_res_entry to a function that returns 0 and
drop provider_fill_res_entry..

> +	/*
> +	 * provider-specific attributes.
> +	 */
> +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested table */
> +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */

Document exactly what values PRINT_TYPE can take on, say, in an enum.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe April 20, 2018, 6:11 p.m. UTC | #2 
On Fri, Apr 20, 2018 at 12:09:59PM -0600, Jason Gunthorpe wrote:

> > +	/*
> > +	 * provider-specific attributes.
> > +	 */
> > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> 
> Document exactly what values PRINT_TYPE can take on, say, in an enum.

oops, never mind.  I see it now.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Wise April 20, 2018, 6:20 p.m. UTC | #3 
> 
> On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> > diff --git a/drivers/infiniband/core/nldev.c
> b/drivers/infiniband/core/nldev.c
> > index 884843e..1a680a3 100644
> > +++ b/drivers/infiniband/core/nldev.c
> > @@ -95,8 +95,25 @@
> >  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type =
> NLA_NESTED },
> >  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
> >  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type =
> NLA_U32 },
> > +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type =
> NLA_NESTED },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type =
> NLA_NESTED },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type =
> NLA_NUL_STRING,
> > +				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN
> },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
> > +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },
> 
> Why do we need 64 and 32 bit version of this? Pass everything as s64
> or u64?
>

To save skb space.  You and Leon both seem to like only 64b.  Guess 64b is
it then...

 
> > +static int provider_fill_res_entry(struct rdma_restrack_root *resroot,
> > +				   struct sk_buff *msg,
> > +				   struct rdma_restrack_entry *res)
> > +{
> > +	return resroot->fill_res_entry ?
> > +		resroot->fill_res_entry(msg, res) : 0;
> > +}
> 
> Just init resroot->fill_res_entry to a function that returns 0 and
> drop provider_fill_res_entry..
> 

Ok.

> > +	/*
> > +	 * provider-specific attributes.
> > +	 */
> > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> 
> Document exactly what values PRINT_TYPE can take on, say, in an enum.

It is in an enum.  See rdma_nldev_print_type.   I'll add a comment here.

Thanks!

Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe April 20, 2018, 6:26 p.m. UTC | #4 
On Fri, Apr 20, 2018 at 01:20:44PM -0500, Steve Wise wrote:
> > 
> > On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> > > diff --git a/drivers/infiniband/core/nldev.c
> > b/drivers/infiniband/core/nldev.c
> > > index 884843e..1a680a3 100644
> > > +++ b/drivers/infiniband/core/nldev.c
> > > @@ -95,8 +95,25 @@
> > >  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type =
> > NLA_NESTED },
> > >  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
> > >  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type =
> > NLA_U32 },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type =
> > NLA_NESTED },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type =
> > NLA_NESTED },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type =
> > NLA_NUL_STRING,
> > > +				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN
> > },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
> > > +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },
> > 
> > Why do we need 64 and 32 bit version of this? Pass everything as s64
> > or u64?
> >
> 
> To save skb space.  You and Leon both seem to like only 64b.  Guess 64b is
> it then...

Is space important? The u64 does get a bit expensive.

I guess it doesn't matter either way.

> > > +	/*
> > > +	 * provider-specific attributes.
> > > +	 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> > table */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> > 
> > Document exactly what values PRINT_TYPE can take on, say, in an enum.
> 
> It is in an enum.  See rdma_nldev_print_type.   I'll add a comment here.

ah yes, saying 'u8' in the comment when there is an enum confuses :)
But still need to specify the type, right

 /* u8 values from enum rdma_nldev_print_type */

?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Wise April 20, 2018, 6:27 p.m. UTC | #5 
> 
> On Fri, Apr 20, 2018 at 01:20:44PM -0500, Steve Wise wrote:
> > >
> > > On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> > > > diff --git a/drivers/infiniband/core/nldev.c
> > > b/drivers/infiniband/core/nldev.c
> > > > index 884843e..1a680a3 100644
> > > > +++ b/drivers/infiniband/core/nldev.c
> > > > @@ -95,8 +95,25 @@
> > > >  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type =
> > > NLA_NESTED },
> > > >  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32
},
> > > >  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type =
> > > NLA_U32 },
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type =
> > > NLA_NESTED },
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type =
> > > NLA_NESTED },
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type =
> > > NLA_NUL_STRING,
> > > > +				    .len =
RDMA_NLDEV_ATTR_ENTRY_STRLEN
> > > },
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8
},
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32
},
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32
},
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64
},
> > > > +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64
},
> > >
> > > Why do we need 64 and 32 bit version of this? Pass everything as s64
> > > or u64?
> > >
> >
> > To save skb space.  You and Leon both seem to like only 64b.  Guess 64b
is
> > it then...
> 
> Is space important? The u64 does get a bit expensive.
> 
> I guess it doesn't matter either way.
> 
> > > > +	/*
> > > > +	 * provider-specific attributes.
> > > > +	 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> > > table */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> > >
> > > Document exactly what values PRINT_TYPE can take on, say, in an enum.
> >
> > It is in an enum.  See rdma_nldev_print_type.   I'll add a comment here.
> 
> ah yes, saying 'u8' in the comment when there is an enum confuses :)
> But still need to specify the type, right
> 
>  /* u8 values from enum rdma_nldev_print_type */
> 
> ?

That's good.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe April 20, 2018, 6:45 p.m. UTC | #6 
On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
>  enum rdma_nldev_attr {
>  	/* don't change the order or add anything between, this is ABI! */
>  	RDMA_NLDEV_ATTR_UNSPEC,
> @@ -390,6 +399,17 @@ enum rdma_nldev_attr {
>  	RDMA_NLDEV_ATTR_RES_PD_ENTRY,		/* nested table */
>  	RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY,	/* u32 */
>  	RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY,	/* u32 */
> +	/*
> +	 * provider-specific attributes.
> +	 */
> +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested table */
> +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> +	RDMA_NLDEV_ATTR_PROVIDER_S32,		/* s32 */
> +	RDMA_NLDEV_ATTR_PROVIDER_U32,		/* u32 */
> +	RDMA_NLDEV_ATTR_PROVIDER_S64,		/* s64 */
> +	RDMA_NLDEV_ATTR_PROVIDER_U64,		/* u64 */

Also this is a good place to use our new DRIVER_ID thing.

If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
then the userspace at least knows what driver sent the data..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Wise April 20, 2018, 7:01 p.m. UTC | #7 
> 
> On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> >  enum rdma_nldev_attr {
> >  	/* don't change the order or add anything between, this is ABI! */
> >  	RDMA_NLDEV_ATTR_UNSPEC,
> > @@ -390,6 +399,17 @@ enum rdma_nldev_attr {
> >  	RDMA_NLDEV_ATTR_RES_PD_ENTRY,		/* nested table */
> >  	RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY,	/* u32 */
> >  	RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY,	/* u32 */
> > +	/*
> > +	 * provider-specific attributes.
> > +	 */
> > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> table */
> > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> > +	RDMA_NLDEV_ATTR_PROVIDER_S32,		/* s32 */
> > +	RDMA_NLDEV_ATTR_PROVIDER_U32,		/* u32 */
> > +	RDMA_NLDEV_ATTR_PROVIDER_S64,		/* s64 */
> > +	RDMA_NLDEV_ATTR_PROVIDER_U64,		/* u64 */
> 
> Also this is a good place to use our new DRIVER_ID thing.
> 

DRIVER_ID?

>
> If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> then the userspace at least knows what driver sent the data..
> 

The provider-specific data comes along with the core resource attributes for
a given resource query.  It isn't like an application can query just
provider attributes.  It queries a resource or set of them or all of them,
and it gets back core+provide attrs for each resource.

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe April 20, 2018, 7:16 p.m. UTC | #8 
On Fri, Apr 20, 2018 at 02:01:14PM -0500, Steve Wise wrote:
> 
> > 
> > On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> > >  enum rdma_nldev_attr {
> > >  	/* don't change the order or add anything between, this is ABI! */
> > >  	RDMA_NLDEV_ATTR_UNSPEC,
> > > @@ -390,6 +399,17 @@ enum rdma_nldev_attr {
> > >  	RDMA_NLDEV_ATTR_RES_PD_ENTRY,		/* nested table */
> > >  	RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY,	/* u32 */
> > >  	RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY,	/* u32 */
> > > +	/*
> > > +	 * provider-specific attributes.
> > > +	 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> > table */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_S32,		/* s32 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_U32,		/* u32 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_S64,		/* s64 */
> > > +	RDMA_NLDEV_ATTR_PROVIDER_U64,		/* u64 */
> > 
> > Also this is a good place to use our new DRIVER_ID thing.
> > 
> 
> DRIVER_ID?

enum rdma_driver_id {
        RDMA_DRIVER_UNKNOWN,
        RDMA_DRIVER_MLX5,
        RDMA_DRIVER_MLX4,
        RDMA_DRIVER_CXGB3,
        RDMA_DRIVER_CXGB4,
        RDMA_DRIVER_MTHCA,
        RDMA_DRIVER_BNXT_RE,
        RDMA_DRIVER_OCRDMA,
        RDMA_DRIVER_NES,
        RDMA_DRIVER_I40IW,
        RDMA_DRIVER_VMW_PVRDMA,
        RDMA_DRIVER_QEDR,
        RDMA_DRIVER_HNS,
        RDMA_DRIVER_USNIC,
        RDMA_DRIVER_RXE,
        RDMA_DRIVER_HFI1,
        RDMA_DRIVER_QIB,
};

> >
> > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > then the userspace at least knows what driver sent the data..
> > 
> 
> The provider-specific data comes along with the core resource attributes for
> a given resource query.  It isn't like an application can query just
> provider attributes.  It queries a resource or set of them or all of them,
> and it gets back core+provide attrs for each resource.

It is not about the query, it is about having the data be
self-describing.. There is no easy way to guess what the underlying
device is to interpret the strings.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Leon Romanovsky April 23, 2018, 1:19 p.m. UTC | #9 
On Fri, Apr 20, 2018 at 01:16:39PM -0600, Jason Gunthorpe wrote:
> On Fri, Apr 20, 2018 at 02:01:14PM -0500, Steve Wise wrote:
> >
> > >
> > > On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> > > >  enum rdma_nldev_attr {
> > > >  	/* don't change the order or add anything between, this is ABI! */
> > > >  	RDMA_NLDEV_ATTR_UNSPEC,
> > > > @@ -390,6 +399,17 @@ enum rdma_nldev_attr {
> > > >  	RDMA_NLDEV_ATTR_RES_PD_ENTRY,		/* nested table */
> > > >  	RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY,	/* u32 */
> > > >  	RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY,	/* u32 */
> > > > +	/*
> > > > +	 * provider-specific attributes.
> > > > +	 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested
> > > table */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_S32,		/* s32 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_U32,		/* u32 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_S64,		/* s64 */
> > > > +	RDMA_NLDEV_ATTR_PROVIDER_U64,		/* u64 */
> > >
> > > Also this is a good place to use our new DRIVER_ID thing.
> > >
> >
> > DRIVER_ID?
>
> enum rdma_driver_id {
>         RDMA_DRIVER_UNKNOWN,
>         RDMA_DRIVER_MLX5,
>         RDMA_DRIVER_MLX4,
>         RDMA_DRIVER_CXGB3,
>         RDMA_DRIVER_CXGB4,
>         RDMA_DRIVER_MTHCA,
>         RDMA_DRIVER_BNXT_RE,
>         RDMA_DRIVER_OCRDMA,
>         RDMA_DRIVER_NES,
>         RDMA_DRIVER_I40IW,
>         RDMA_DRIVER_VMW_PVRDMA,
>         RDMA_DRIVER_QEDR,
>         RDMA_DRIVER_HNS,
>         RDMA_DRIVER_USNIC,
>         RDMA_DRIVER_RXE,
>         RDMA_DRIVER_HFI1,
>         RDMA_DRIVER_QIB,
> };
>
> > >
> > > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > > then the userspace at least knows what driver sent the data..
> > >
> >
> > The provider-specific data comes along with the core resource attributes for
> > a given resource query.  It isn't like an application can query just
> > provider attributes.  It queries a resource or set of them or all of them,
> > and it gets back core+provide attrs for each resource.
>
> It is not about the query, it is about having the data be
> self-describing.. There is no easy way to guess what the underlying
> device is to interpret the strings.

Not really, in order to get data from the kernel, user will send query
with specific ib_device index and this index will be returned back.

There is no need in this enum for anything except future strace support.

Thanks

>
> Jason
Steve Wise April 23, 2018, 3:44 p.m. UTC | #10 
On 4/20/2018 1:09 PM, Jason Gunthorpe wrote:
> On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
>> diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
>> index 884843e..1a680a3 100644
>> +++ b/drivers/infiniband/core/nldev.c
>> @@ -95,8 +95,25 @@
>>  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type = NLA_NESTED },
>>  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
>>  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type = NLA_U32 },
>> +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type = NLA_NESTED },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type = NLA_NESTED },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type = NLA_NUL_STRING,
>> +				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
>> +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },
> Why do we need 64 and 32 bit version of this? Pass everything as s64
> or u64?
>
>> +static int provider_fill_res_entry(struct rdma_restrack_root *resroot,
>> +				   struct sk_buff *msg,
>> +				   struct rdma_restrack_entry *res)
>> +{
>> +	return resroot->fill_res_entry ?
>> +		resroot->fill_res_entry(msg, res) : 0;
>> +}
> Just init resroot->fill_res_entry to a function that returns 0 and
> drop provider_fill_res_entry..

Why is calling a function that returns 0 better than testing for the
null function pointer?

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Leon Romanovsky April 24, 2018, 7:09 a.m. UTC | #11 
On Mon, Apr 23, 2018 at 10:44:10AM -0500, Steve Wise wrote:
>
> On 4/20/2018 1:09 PM, Jason Gunthorpe wrote:
> > On Fri, Mar 30, 2018 at 11:03:36AM -0700, Steve Wise wrote:
> >> diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
> >> index 884843e..1a680a3 100644
> >> +++ b/drivers/infiniband/core/nldev.c
> >> @@ -95,8 +95,25 @@
> >>  	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type = NLA_NESTED },
> >>  	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
> >>  	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type = NLA_U32 },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type = NLA_NESTED },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type = NLA_NESTED },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type = NLA_NUL_STRING,
> >> +				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
> >> +	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },
> > Why do we need 64 and 32 bit version of this? Pass everything as s64
> > or u64?
> >
> >> +static int provider_fill_res_entry(struct rdma_restrack_root *resroot,
> >> +				   struct sk_buff *msg,
> >> +				   struct rdma_restrack_entry *res)
> >> +{
> >> +	return resroot->fill_res_entry ?
> >> +		resroot->fill_res_entry(msg, res) : 0;
> >> +}
> > Just init resroot->fill_res_entry to a function that returns 0 and
> > drop provider_fill_res_entry..
>
> Why is calling a function that returns 0 better than testing for the
> null function pointer?

I guess the reason to it that it will allow to us to call this
function unconditionally and get rid of provider_fill_res_entry()
entirely.

Thanks

>
> Steve.
Jason Gunthorpe April 30, 2018, 2:45 p.m. UTC | #12 
On Mon, Apr 23, 2018 at 04:19:00PM +0300, Leon Romanovsky wrote:
> > > > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > > > then the userspace at least knows what driver sent the data..
> > > >
> > >
> > > The provider-specific data comes along with the core resource attributes for
> > > a given resource query.  It isn't like an application can query just
> > > provider attributes.  It queries a resource or set of them or all of them,
> > > and it gets back core+provide attrs for each resource.
> >
> > It is not about the query, it is about having the data be
> > self-describing.. There is no easy way to guess what the underlying
> > device is to interpret the strings.
> 
> Not really, in order to get data from the kernel, user will send query
> with specific ib_device index and this index will be returned back.

The rdma device index doesn't tell you what driver is behind this
without a lot of complicated work. The driver id is intended for this
purpose.

(driver_id, driver_string) should uniquely describe the value returned
and the driver_id needs to be available through netlink someplace

> There is no need in this enum for anything except future strace support.

I want to use it for driver binding in rdma-core as well.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Leon Romanovsky April 30, 2018, 2:52 p.m. UTC | #13 
On Mon, Apr 30, 2018 at 08:45:10AM -0600, Jason Gunthorpe wrote:
> On Mon, Apr 23, 2018 at 04:19:00PM +0300, Leon Romanovsky wrote:
> > > > > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > > > > then the userspace at least knows what driver sent the data..
> > > > >
> > > >
> > > > The provider-specific data comes along with the core resource attributes for
> > > > a given resource query.  It isn't like an application can query just
> > > > provider attributes.  It queries a resource or set of them or all of them,
> > > > and it gets back core+provide attrs for each resource.
> > >
> > > It is not about the query, it is about having the data be
> > > self-describing.. There is no easy way to guess what the underlying
> > > device is to interpret the strings.
> >
> > Not really, in order to get data from the kernel, user will send query
> > with specific ib_device index and this index will be returned back.
>
> The rdma device index doesn't tell you what driver is behind this
> without a lot of complicated work. The driver id is intended for this
> purpose.

I don't see why rdmatool user should care about which driver is connected
to specific ib_device.

>
> (driver_id, driver_string) should uniquely describe the value returned
> and the driver_id needs to be available through netlink someplace
>
> > There is no need in this enum for anything except future strace support.
>
> I want to use it for driver binding in rdma-core as well.

Good, but it is orthogonal to Steve's work.

>
> Jason
Jason Gunthorpe April 30, 2018, 11:02 p.m. UTC | #14 
On Mon, Apr 30, 2018 at 05:52:49PM +0300, Leon Romanovsky wrote:
> On Mon, Apr 30, 2018 at 08:45:10AM -0600, Jason Gunthorpe wrote:
> > On Mon, Apr 23, 2018 at 04:19:00PM +0300, Leon Romanovsky wrote:
> > > > > > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > > > > > then the userspace at least knows what driver sent the data..
> > > > > >
> > > > >
> > > > > The provider-specific data comes along with the core resource attributes for
> > > > > a given resource query.  It isn't like an application can query just
> > > > > provider attributes.  It queries a resource or set of them or all of them,
> > > > > and it gets back core+provide attrs for each resource.
> > > >
> > > > It is not about the query, it is about having the data be
> > > > self-describing.. There is no easy way to guess what the underlying
> > > > device is to interpret the strings.
> > >
> > > Not really, in order to get data from the kernel, user will send query
> > > with specific ib_device index and this index will be returned back.
> >
> > The rdma device index doesn't tell you what driver is behind this
> > without a lot of complicated work. The driver id is intended for this
> > purpose.
> 
> I don't see why rdmatool user should care about which driver is connected
> to specific ib_device.

rdmatool shouldn't, but if Steve makes a custom tool only for cxgb4
then it should be able to know that it is getting counter data from
the right driver and not some other random place.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Leon Romanovsky May 1, 2018, 8:09 a.m. UTC | #15 
On Mon, Apr 30, 2018 at 05:02:09PM -0600, Jason Gunthorpe wrote:
> On Mon, Apr 30, 2018 at 05:52:49PM +0300, Leon Romanovsky wrote:
> > On Mon, Apr 30, 2018 at 08:45:10AM -0600, Jason Gunthorpe wrote:
> > > On Mon, Apr 23, 2018 at 04:19:00PM +0300, Leon Romanovsky wrote:
> > > > > > > If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
> > > > > > > then the userspace at least knows what driver sent the data..
> > > > > > >
> > > > > >
> > > > > > The provider-specific data comes along with the core resource attributes for
> > > > > > a given resource query.  It isn't like an application can query just
> > > > > > provider attributes.  It queries a resource or set of them or all of them,
> > > > > > and it gets back core+provide attrs for each resource.
> > > > >
> > > > > It is not about the query, it is about having the data be
> > > > > self-describing.. There is no easy way to guess what the underlying
> > > > > device is to interpret the strings.
> > > >
> > > > Not really, in order to get data from the kernel, user will send query
> > > > with specific ib_device index and this index will be returned back.
> > >
> > > The rdma device index doesn't tell you what driver is behind this
> > > without a lot of complicated work. The driver id is intended for this
> > > purpose.
> >
> > I don't see why rdmatool user should care about which driver is connected
> > to specific ib_device.
>
> rdmatool shouldn't, but if Steve makes a custom tool only for cxgb4
> then it should be able to know that it is getting counter data from
> the right driver and not some other random place.

I think that "-p" option of rdmatool gives to Steve needed flexibility
to provide aesthetic display, while preserving scriptability.

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Wise May 1, 2018, 2:22 p.m. UTC | #16 
On 5/1/2018 3:09 AM, Leon Romanovsky wrote:
> On Mon, Apr 30, 2018 at 05:02:09PM -0600, Jason Gunthorpe wrote:
>> On Mon, Apr 30, 2018 at 05:52:49PM +0300, Leon Romanovsky wrote:
>>> On Mon, Apr 30, 2018 at 08:45:10AM -0600, Jason Gunthorpe wrote:
>>>> On Mon, Apr 23, 2018 at 04:19:00PM +0300, Leon Romanovsky wrote:
>>>>>>>> If you send that integer when the RDMA_NLDEV_ATTR_PROVIDER is opened
>>>>>>>> then the userspace at least knows what driver sent the data..
>>>>>>>>
>>>>>>> The provider-specific data comes along with the core resource attributes for
>>>>>>> a given resource query.  It isn't like an application can query just
>>>>>>> provider attributes.  It queries a resource or set of them or all of them,
>>>>>>> and it gets back core+provide attrs for each resource.
>>>>>> It is not about the query, it is about having the data be
>>>>>> self-describing.. There is no easy way to guess what the underlying
>>>>>> device is to interpret the strings.
>>>>> Not really, in order to get data from the kernel, user will send query
>>>>> with specific ib_device index and this index will be returned back.
>>>> The rdma device index doesn't tell you what driver is behind this
>>>> without a lot of complicated work. The driver id is intended for this
>>>> purpose.
>>> I don't see why rdmatool user should care about which driver is connected
>>> to specific ib_device.
>> rdmatool shouldn't, but if Steve makes a custom tool only for cxgb4
>> then it should be able to know that it is getting counter data from
>> the right driver and not some other random place.
> I think that "-p" option of rdmatool gives to Steve needed flexibility
> to provide aesthetic display, while preserving scriptability.
>
> Thanks
>

After implementing "-p" and also considering that without "-p" all the
data is on one single row, any tool I implement, if I really need/want
one, would probably be with grep/awk/etc...

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 884843e..1a680a3 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -95,8 +95,25 @@ 
 	[RDMA_NLDEV_ATTR_RES_PD_ENTRY]		= { .type = NLA_NESTED },
 	[RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY]	= { .type = NLA_U32 },
 	[RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY] = { .type = NLA_U32 },
+	[RDMA_NLDEV_ATTR_PROVIDER]		= { .type = NLA_NESTED },
+	[RDMA_NLDEV_ATTR_PROVIDER_ENTRY]	= { .type = NLA_NESTED },
+	[RDMA_NLDEV_ATTR_PROVIDER_STRING]	= { .type = NLA_NUL_STRING,
+				    .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
+	[RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE]	= { .type = NLA_U8 },
+	[RDMA_NLDEV_ATTR_PROVIDER_S32]		= { .type = NLA_S32 },
+	[RDMA_NLDEV_ATTR_PROVIDER_U32]		= { .type = NLA_U32 },
+	[RDMA_NLDEV_ATTR_PROVIDER_S64]		= { .type = NLA_S64 },
+	[RDMA_NLDEV_ATTR_PROVIDER_U64]		= { .type = NLA_U64 },
 };
 
+static int provider_fill_res_entry(struct rdma_restrack_root *resroot,
+				   struct sk_buff *msg,
+				   struct rdma_restrack_entry *res)
+{
+	return resroot->fill_res_entry ?
+		resroot->fill_res_entry(msg, res) : 0;
+}
+
 static int fill_nldev_handle(struct sk_buff *msg, struct ib_device *device)
 {
 	if (nla_put_u32(msg, RDMA_NLDEV_ATTR_DEV_INDEX, device->index))
@@ -264,6 +281,7 @@  static int fill_res_qp_entry(struct sk_buff *msg, struct netlink_callback *cb,
 			     struct rdma_restrack_entry *res, uint32_t port)
 {
 	struct ib_qp *qp = container_of(res, struct ib_qp, res);
+	struct rdma_restrack_root *resroot = &qp->device->res;
 	struct ib_qp_init_attr qp_init_attr;
 	struct nlattr *entry_attr;
 	struct ib_qp_attr qp_attr;
@@ -313,6 +331,9 @@  static int fill_res_qp_entry(struct sk_buff *msg, struct netlink_callback *cb,
 	if (fill_res_name_pid(msg, res))
 		goto err;
 
+	if (provider_fill_res_entry(resroot, msg, res))
+		goto err;
+
 	nla_nest_end(msg, entry_attr);
 	return 0;
 
@@ -328,6 +349,7 @@  static int fill_res_cm_id_entry(struct sk_buff *msg,
 {
 	struct rdma_id_private *id_priv =
 				container_of(res, struct rdma_id_private, res);
+	struct rdma_restrack_root *resroot = &id_priv->id.device->res;
 	struct rdma_cm_id *cm_id = &id_priv->id;
 	struct nlattr *entry_attr;
 
@@ -369,6 +391,9 @@  static int fill_res_cm_id_entry(struct sk_buff *msg,
 	if (fill_res_name_pid(msg, res))
 		goto err;
 
+	if (provider_fill_res_entry(resroot, msg, res))
+		goto err;
+
 	nla_nest_end(msg, entry_attr);
 	return 0;
 
@@ -382,6 +407,7 @@  static int fill_res_cq_entry(struct sk_buff *msg, struct netlink_callback *cb,
 			     struct rdma_restrack_entry *res, uint32_t port)
 {
 	struct ib_cq *cq = container_of(res, struct ib_cq, res);
+	struct rdma_restrack_root *resroot = &cq->device->res;
 	struct nlattr *entry_attr;
 
 	entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_CQ_ENTRY);
@@ -402,6 +428,9 @@  static int fill_res_cq_entry(struct sk_buff *msg, struct netlink_callback *cb,
 	if (fill_res_name_pid(msg, res))
 		goto err;
 
+	if (provider_fill_res_entry(resroot, msg, res))
+		goto err;
+
 	nla_nest_end(msg, entry_attr);
 	return 0;
 
@@ -415,6 +444,7 @@  static int fill_res_mr_entry(struct sk_buff *msg, struct netlink_callback *cb,
 			     struct rdma_restrack_entry *res, uint32_t port)
 {
 	struct ib_mr *mr = container_of(res, struct ib_mr, res);
+	struct rdma_restrack_root *resroot = &mr->pd->device->res;
 	struct nlattr *entry_attr;
 
 	entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_MR_ENTRY);
@@ -438,6 +468,9 @@  static int fill_res_mr_entry(struct sk_buff *msg, struct netlink_callback *cb,
 	if (fill_res_name_pid(msg, res))
 		goto err;
 
+	if (provider_fill_res_entry(resroot, msg, res))
+		goto err;
+
 	nla_nest_end(msg, entry_attr);
 	return 0;
 
@@ -451,6 +484,7 @@  static int fill_res_pd_entry(struct sk_buff *msg, struct netlink_callback *cb,
 			     struct rdma_restrack_entry *res, uint32_t port)
 {
 	struct ib_pd *pd = container_of(res, struct ib_pd, res);
+	struct rdma_restrack_root *resroot = &pd->device->res;
 	struct nlattr *entry_attr;
 
 	entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_RES_PD_ENTRY);
@@ -477,6 +511,9 @@  static int fill_res_pd_entry(struct sk_buff *msg, struct netlink_callback *cb,
 	if (fill_res_name_pid(msg, res))
 		goto err;
 
+	if (provider_fill_res_entry(resroot, msg, res))
+		goto err;
+
 	nla_nest_end(msg, entry_attr);
 	return 0;
 
diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c
index efddd13..78fc6d9 100644
--- a/drivers/infiniband/core/restrack.c
+++ b/drivers/infiniband/core/restrack.c
@@ -15,6 +15,7 @@ 
 void rdma_restrack_init(struct rdma_restrack_root *res)
 {
 	init_rwsem(&res->rwsem);
+	res->fill_res_entry = NULL;
 }
 
 static const char *type2str(enum rdma_restrack_type type)
diff --git a/include/rdma/restrack.h b/include/rdma/restrack.h
index f3b3e35..51f400e 100644
--- a/include/rdma/restrack.h
+++ b/include/rdma/restrack.h
@@ -44,6 +44,8 @@  enum rdma_restrack_type {
 };
 
 #define RDMA_RESTRACK_HASH_BITS	8
+struct rdma_restrack_entry;
+
 /**
  * struct rdma_restrack_root - main resource tracking management
  * entity, per-device
@@ -57,6 +59,13 @@  struct rdma_restrack_root {
 	 * @hash: global database for all resources per-device
 	 */
 	DECLARE_HASHTABLE(hash, RDMA_RESTRACK_HASH_BITS);
+	/**
+	 * @fill_res_entry: provider-specific fill function
+	 *
+	 * Allows rdma providers to add their own restrack attributes.
+	 */
+	int (*fill_res_entry)(struct sk_buff *msg,
+			      struct rdma_restrack_entry *entry);
 };
 
 /**
diff --git a/include/uapi/rdma/rdma_netlink.h b/include/uapi/rdma/rdma_netlink.h
index 84b3f63..862c238 100644
--- a/include/uapi/rdma/rdma_netlink.h
+++ b/include/uapi/rdma/rdma_netlink.h
@@ -249,6 +249,15 @@  enum rdma_nldev_command {
 	RDMA_NLDEV_NUM_OPS
 };
 
+enum {
+	RDMA_NLDEV_ATTR_ENTRY_STRLEN = 16,
+};
+
+enum rdma_nldev_print_type {
+	RDMA_NLDEV_PRINT_TYPE_UNSPEC,
+	RDMA_NLDEV_PRINT_TYPE_HEX,
+};
+
 enum rdma_nldev_attr {
 	/* don't change the order or add anything between, this is ABI! */
 	RDMA_NLDEV_ATTR_UNSPEC,
@@ -390,6 +399,17 @@  enum rdma_nldev_attr {
 	RDMA_NLDEV_ATTR_RES_PD_ENTRY,		/* nested table */
 	RDMA_NLDEV_ATTR_RES_LOCAL_DMA_LKEY,	/* u32 */
 	RDMA_NLDEV_ATTR_RES_UNSAFE_GLOBAL_RKEY,	/* u32 */
+	/*
+	 * provider-specific attributes.
+	 */
+	RDMA_NLDEV_ATTR_PROVIDER,		/* nested table */
+	RDMA_NLDEV_ATTR_PROVIDER_ENTRY,		/* nested table */
+	RDMA_NLDEV_ATTR_PROVIDER_STRING,	/* string */
+	RDMA_NLDEV_ATTR_PROVIDER_PRINT_TYPE,	/* u8 */
+	RDMA_NLDEV_ATTR_PROVIDER_S32,		/* s32 */
+	RDMA_NLDEV_ATTR_PROVIDER_U32,		/* u32 */
+	RDMA_NLDEV_ATTR_PROVIDER_S64,		/* s64 */
+	RDMA_NLDEV_ATTR_PROVIDER_U64,		/* u64 */
 
 	RDMA_NLDEV_ATTR_MAX
 };