| Message ID | 20190530122422.32283-1-shamir.rabinovitch@oracle.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | ib_pd should not have ib_uobject | expand |
On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > This patch set complete the cleanup done in the driver/verbs/uverbs > where the dependency of the code in the ib_x uobject pointer was > removed. > > The uobject pointer is removed from the ib_pd as last step > before I can start adding the pd sharing code. > > The rdma/netlink code now don't have dependency in the ib_pd > uobject pointer and can report multiple context id that point > to same ib_pd. > > Using iproute2 I can test the modified rdma/netlink: > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > Changelog: > > v1->v2 > * 1 patch from v1 applied (Jason) > * Fix uobj_get_obj_read macro (Jason) > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > * Fix uobj_get_obj_read macro (Jason) > * rdma/netlink can now work as before (Leon) > > v2->v3: > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > Shamir Rabinovitch (4): > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > RDMA/uverbs: uobj_put_obj_read macro should be removed > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > drivers/infiniband/core/nldev.c | 129 +++++++++++- > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- > drivers/infiniband/core/verbs.c | 1 - > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - > drivers/infiniband/hw/mlx5/main.c | 1 - > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- > include/rdma/ib_verbs.h | 1 - > include/rdma/uverbs_std_types.h | 11 +- > include/uapi/rdma/rdma_netlink.h | 3 + > 9 files changed, 273 insertions(+), 95 deletions(-) > > -- > 2.20.1 > Jason, Leon, can you please review this patch set ? Anything missing from my side here? Next patch set include ib_mr cleanup (Yuval). Thanks
On Wed, Jun 05, 2019 at 10:21:26AM +0300, Shamir Rabinovitch wrote: > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > > This patch set complete the cleanup done in the driver/verbs/uverbs > > where the dependency of the code in the ib_x uobject pointer was > > removed. > > > > The uobject pointer is removed from the ib_pd as last step > > before I can start adding the pd sharing code. > > > > The rdma/netlink code now don't have dependency in the ib_pd > > uobject pointer and can report multiple context id that point > > to same ib_pd. > > > > Using iproute2 I can test the modified rdma/netlink: > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > > > Changelog: > > > > v1->v2 > > * 1 patch from v1 applied (Jason) > > * Fix uobj_get_obj_read macro (Jason) > > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > > * Fix uobj_get_obj_read macro (Jason) > > * rdma/netlink can now work as before (Leon) > > > > v2->v3: > > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > > > Shamir Rabinovitch (4): > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > > RDMA/uverbs: uobj_put_obj_read macro should be removed > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > > > drivers/infiniband/core/nldev.c | 129 +++++++++++- > > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- > > drivers/infiniband/core/verbs.c | 1 - > > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - > > drivers/infiniband/hw/mlx5/main.c | 1 - > > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- > > include/rdma/ib_verbs.h | 1 - > > include/rdma/uverbs_std_types.h | 11 +- > > include/uapi/rdma/rdma_netlink.h | 3 + > > 9 files changed, 273 insertions(+), 95 deletions(-) > > > > -- > > 2.20.1 > > > > Jason, Leon, can you please review this patch set ? I'm sorry for the delay. > > Anything missing from my side here? Can you please post rdmatool output for shared PD? In such case, all those shared PD need to have same PDN. Thanks
On Wed, Jun 05, 2019 at 11:25:49AM +0300, Leon Romanovsky wrote: > On Wed, Jun 05, 2019 at 10:21:26AM +0300, Shamir Rabinovitch wrote: > > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > > > This patch set complete the cleanup done in the driver/verbs/uverbs > > > where the dependency of the code in the ib_x uobject pointer was > > > removed. > > > > > > The uobject pointer is removed from the ib_pd as last step > > > before I can start adding the pd sharing code. > > > > > > The rdma/netlink code now don't have dependency in the ib_pd > > > uobject pointer and can report multiple context id that point > > > to same ib_pd. > > > > > > Using iproute2 I can test the modified rdma/netlink: > > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > > > > > Changelog: > > > > > > v1->v2 > > > * 1 patch from v1 applied (Jason) > > > * Fix uobj_get_obj_read macro (Jason) > > > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > > > * Fix uobj_get_obj_read macro (Jason) > > > * rdma/netlink can now work as before (Leon) > > > > > > v2->v3: > > > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > > > > > Shamir Rabinovitch (4): > > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > > > RDMA/uverbs: uobj_put_obj_read macro should be removed > > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > > > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > > > > > drivers/infiniband/core/nldev.c | 129 +++++++++++- > > > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- > > > drivers/infiniband/core/verbs.c | 1 - > > > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - > > > drivers/infiniband/hw/mlx5/main.c | 1 - > > > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- > > > include/rdma/ib_verbs.h | 1 - > > > include/rdma/uverbs_std_types.h | 11 +- > > > include/uapi/rdma/rdma_netlink.h | 3 + > > > 9 files changed, 273 insertions(+), 95 deletions(-) > > > > > > -- > > > 2.20.1 > > > > > > > Jason, Leon, can you please review this patch set ? > > I'm sorry for the delay. > > > > > Anything missing from my side here? > > Can you please post rdmatool output for shared PD? > In such case, all those shared PD need to have same PDN. Leon I do not have all the pieces in place for this yes. I only tested that current rdmatool can cope with netlink message that has nested context ids as it will happen in shared pd case. I'll do that once we have the export_to_fd & import_pd & import_mr verbs in place. > > Thanks
On Wed, Jun 05, 2019 at 04:02:45PM +0300, Shamir Rabinovitch wrote: > On Wed, Jun 05, 2019 at 11:25:49AM +0300, Leon Romanovsky wrote: > > On Wed, Jun 05, 2019 at 10:21:26AM +0300, Shamir Rabinovitch wrote: > > > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > > > > This patch set complete the cleanup done in the driver/verbs/uverbs > > > > where the dependency of the code in the ib_x uobject pointer was > > > > removed. > > > > > > > > The uobject pointer is removed from the ib_pd as last step > > > > before I can start adding the pd sharing code. > > > > > > > > The rdma/netlink code now don't have dependency in the ib_pd > > > > uobject pointer and can report multiple context id that point > > > > to same ib_pd. > > > > > > > > Using iproute2 I can test the modified rdma/netlink: > > > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > > > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > > > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > > > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > > > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > > > > > > > Changelog: > > > > > > > > v1->v2 > > > > * 1 patch from v1 applied (Jason) > > > > * Fix uobj_get_obj_read macro (Jason) > > > > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > > > > * Fix uobj_get_obj_read macro (Jason) > > > > * rdma/netlink can now work as before (Leon) > > > > > > > > v2->v3: > > > > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > > > > > > > Shamir Rabinovitch (4): > > > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > > > > RDMA/uverbs: uobj_put_obj_read macro should be removed > > > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > > > > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > > > > > > > drivers/infiniband/core/nldev.c | 129 +++++++++++- > > > > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- > > > > drivers/infiniband/core/verbs.c | 1 - > > > > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - > > > > drivers/infiniband/hw/mlx5/main.c | 1 - > > > > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- > > > > include/rdma/ib_verbs.h | 1 - > > > > include/rdma/uverbs_std_types.h | 11 +- > > > > include/uapi/rdma/rdma_netlink.h | 3 + > > > > 9 files changed, 273 insertions(+), 95 deletions(-) > > > > > > > > -- > > > > 2.20.1 > > > > > > > > > > Jason, Leon, can you please review this patch set ? > > > > I'm sorry for the delay. > > > > > > > > Anything missing from my side here? > > > > Can you please post rdmatool output for shared PD? > > In such case, all those shared PD need to have same PDN. > > Leon I do not have all the pieces in place for this yes. > > I only tested that current rdmatool can cope with netlink message that > has nested context ids as it will happen in shared pd case. Thanks, multiple PDNs worried me and I was afraid that they come from shared PDs. You will still need to mock something to see that you print them correctly. > > I'll do that once we have the export_to_fd & import_pd & import_mr verbs > in place. > > > > > Thanks
On Wed, Jun 05, 2019 at 05:46:51PM +0300, Leon Romanovsky wrote: > On Wed, Jun 05, 2019 at 04:02:45PM +0300, Shamir Rabinovitch wrote: > > On Wed, Jun 05, 2019 at 11:25:49AM +0300, Leon Romanovsky wrote: > > > On Wed, Jun 05, 2019 at 10:21:26AM +0300, Shamir Rabinovitch wrote: > > > > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > > > > > This patch set complete the cleanup done in the driver/verbs/uverbs > > > > > where the dependency of the code in the ib_x uobject pointer was > > > > > removed. > > > > > > > > > > The uobject pointer is removed from the ib_pd as last step > > > > > before I can start adding the pd sharing code. > > > > > > > > > > The rdma/netlink code now don't have dependency in the ib_pd > > > > > uobject pointer and can report multiple context id that point > > > > > to same ib_pd. > > > > > > > > > > Using iproute2 I can test the modified rdma/netlink: > > > > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > > > > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > > > > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > > > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > > > > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > > > > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > > > > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > > > > > > > > > Changelog: > > > > > > > > > > v1->v2 > > > > > * 1 patch from v1 applied (Jason) > > > > > * Fix uobj_get_obj_read macro (Jason) > > > > > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > > > > > * Fix uobj_get_obj_read macro (Jason) > > > > > * rdma/netlink can now work as before (Leon) > > > > > > > > > > v2->v3: > > > > > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > > > > > > > > > Shamir Rabinovitch (4): > > > > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > > > > > RDMA/uverbs: uobj_put_obj_read macro should be removed > > > > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > > > > > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > > > > > > > > > drivers/infiniband/core/nldev.c | 129 +++++++++++- > > > > > drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- > > > > > drivers/infiniband/core/verbs.c | 1 - > > > > > drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - > > > > > drivers/infiniband/hw/mlx5/main.c | 1 - > > > > > drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- > > > > > include/rdma/ib_verbs.h | 1 - > > > > > include/rdma/uverbs_std_types.h | 11 +- > > > > > include/uapi/rdma/rdma_netlink.h | 3 + > > > > > 9 files changed, 273 insertions(+), 95 deletions(-) > > > > > > > > > > -- > > > > > 2.20.1 > > > > > > > > > > > > > Jason, Leon, can you please review this patch set ? > > > > > > I'm sorry for the delay. > > > > > > > > > > > Anything missing from my side here? > > > > > > Can you please post rdmatool output for shared PD? > > > In such case, all those shared PD need to have same PDN. > > > > Leon I do not have all the pieces in place for this yes. > > > > I only tested that current rdmatool can cope with netlink message that > > has nested context ids as it will happen in shared pd case. > > Thanks, multiple PDNs worried me and I was afraid that they come from > shared PDs. You will still need to mock something to see that you > print them correctly. I tried to check this by iterating twice on the code in 'fill_res_pd_entry' for user pd. kernel confirm that we have duplicate lines for each pd: [ 4707.051279] fill_res_pd_entry: iter 0 [ 4707.052371] fill_res_pd_entry: iter 1 [ 4707.053426] fill_res_pd_entry: iter 0 [ 4707.054529] fill_res_pd_entry: iter 1 But the rdmatool prints remain as-is: [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] dev mlx4_0 pdn 2 local_dma_lkey 0x0 users 2 ctxn 0 pid 12918 comm ib_send_bw dev mlx4_0 pdn 3 local_dma_lkey 0x0 users 2 ctxn 1 pid 12919 comm ib_send_bw Does this make sense to you ? Since we do not have any crash in the rdmatool in this case, can we postpone this test till we have proper shared pd ? Can you please review the series ? Thanks > > > > > I'll do that once we have the export_to_fd & import_pd & import_mr verbs > > in place. > > > > > > > > Thanks
On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > This patch set complete the cleanup done in the driver/verbs/uverbs > where the dependency of the code in the ib_x uobject pointer was > removed. > > The uobject pointer is removed from the ib_pd as last step > before I can start adding the pd sharing code. > > The rdma/netlink code now don't have dependency in the ib_pd > uobject pointer and can report multiple context id that point > to same ib_pd. > > Using iproute2 I can test the modified rdma/netlink: > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > Changelog: > > v1->v2 > * 1 patch from v1 applied (Jason) > * Fix uobj_get_obj_read macro (Jason) > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > * Fix uobj_get_obj_read macro (Jason) > * rdma/netlink can now work as before (Leon) > > v2->v3: > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > Shamir Rabinovitch (4): > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > RDMA/uverbs: uobj_put_obj_read macro should be removed > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > IB/{core,hw}: ib_pd should not have ib_uobject pointer I have to think it is rather a clunky approach.. I keep moving the _cmd flow closer to the attrs flow - so to do that we should be recording the uobj 'get' in the uverbs_attr_bundle and then putting them when the handler exits automatically. Just like the attr flow does. This requires a bit of moving code around so that the write path has the bundle_priv which records all of this. What do you think? Jason
On Fri, Jun 07, 2019 at 02:51:11PM -0300, Jason Gunthorpe wrote: > On Thu, May 30, 2019 at 03:24:05PM +0300, Shamir Rabinovitch wrote: > > This patch set complete the cleanup done in the driver/verbs/uverbs > > where the dependency of the code in the ib_x uobject pointer was > > removed. > > > > The uobject pointer is removed from the ib_pd as last step > > before I can start adding the pd sharing code. > > > > The rdma/netlink code now don't have dependency in the ib_pd > > uobject pointer and can report multiple context id that point > > to same ib_pd. > > > > Using iproute2 I can test the modified rdma/netlink: > > [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 > > dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] > > dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] > > dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] > > dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] > > dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] > > dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw > > dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw > > > > Changelog: > > > > v1->v2 > > * 1 patch from v1 applied (Jason) > > * Fix uobj_get_obj_read macro (Jason) > > * Do not allocate memory when fixing uobj_get_obj_read (Jason) > > * Fix uobj_get_obj_read macro (Jason) > > * rdma/netlink can now work as before (Leon) > > > > v2->v3: > > * rdma/netlink nest multiple context ids of same ib_pd (Leon) > > > > Shamir Rabinovitch (4): > > RDMA/uverbs: uobj_get_obj_read should return the ib_uobject > > RDMA/uverbs: uobj_put_obj_read macro should be removed > > RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext > > IB/{core,hw}: ib_pd should not have ib_uobject pointer > > I have to think it is rather a clunky approach.. > > I keep moving the _cmd flow closer to the attrs flow - so to do that > we should be recording the uobj 'get' in the uverbs_attr_bundle and > then putting them when the handler exits automatically. Just like the > attr flow does. > > This requires a bit of moving code around so that the write path has > the bundle_priv which records all of this. > > What do you think? > > Jason Hi Jason, sorry for the delay, The above make perfect sense to me and can help to solve figure where uobjects are left as uobj_get_read. It do have to solve the cases where some code is doing a multiple uobj_get_read in same uberbs write call (as we had in ib_uverbs_post_send). It can be done after the shared pd patch set is adopted if it is acceptable by you. For the shared pd I'd rather to suggest the follow path to make things faster and get Yuval's help: Next patch-set: kernel: - ib_mr->uobject cleanup from (Yuval) - Add generic sharing support for ib_x object destroy in IB/core - Add export_to_fd verb - Add generic 'clone' for ib_x objects - Enable export of ib_pd - Add clone to ib_pd - Add clone of ib_pd to mlx4, mlx5, rxe - Add import_pd verb rdma-core: - Add export_to_fd verb - Add import_pd verb Follow patch-set: Kernel: - Enable export of ib_mr (Yuval) - Add clone to ib_mr (Yuval) - Add clone of ib_mr to mlx4, mlx5, rxe (Yuval) - Add import_mr verb (Yuval) rdma-core: - Add import_mr verb (Yuval) Next patch-set: rdma-core: - Add shpd pingpong sample app based on the new verbs (Yuval/me) I have some time constraint and I appreciate the time you take to review and progress this feature. Thanks
This patch set complete the cleanup done in the driver/verbs/uverbs where the dependency of the code in the ib_x uobject pointer was removed. The uobject pointer is removed from the ib_pd as last step before I can start adding the pd sharing code. The rdma/netlink code now don't have dependency in the ib_pd uobject pointer and can report multiple context id that point to same ib_pd. Using iproute2 I can test the modified rdma/netlink: [root@qemu-fc29 iproute2]# rdma/rdma res show pd dev mlx4_0 dev mlx4_0 pdn 0 local_dma_lkey 0x8000 users 4 comm [ib_core] dev mlx4_0 pdn 1 local_dma_lkey 0x8000 users 4 comm [ib_core] dev mlx4_0 pdn 2 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] dev mlx4_0 pdn 3 local_dma_lkey 0x8000 users 5 comm [ib_ipoib] dev mlx4_0 pdn 4 local_dma_lkey 0x8000 users 0 comm [ib_srp] dev mlx4_0 pdn 5 local_dma_lkey 0x8000 users 0 comm [ib_srpt] dev mlx4_0 pdn 6 local_dma_lkey 0x0 users 2 ctxn 0 pid 7693 comm ib_send_bw dev mlx4_0 pdn 7 local_dma_lkey 0x0 users 2 ctxn 1 pid 7694 comm ib_send_bw Changelog: v1->v2 * 1 patch from v1 applied (Jason) * Fix uobj_get_obj_read macro (Jason) * Do not allocate memory when fixing uobj_get_obj_read (Jason) * Fix uobj_get_obj_read macro (Jason) * rdma/netlink can now work as before (Leon) v2->v3: * rdma/netlink nest multiple context ids of same ib_pd (Leon) Shamir Rabinovitch (4): RDMA/uverbs: uobj_get_obj_read should return the ib_uobject RDMA/uverbs: uobj_put_obj_read macro should be removed RDMA/nldev: ib_pd can be pointed by multiple ib_ucontext IB/{core,hw}: ib_pd should not have ib_uobject pointer drivers/infiniband/core/nldev.c | 129 +++++++++++- drivers/infiniband/core/uverbs_cmd.c | 218 +++++++++++++-------- drivers/infiniband/core/verbs.c | 1 - drivers/infiniband/hw/hns/hns_roce_hw_v1.c | 1 - drivers/infiniband/hw/mlx5/main.c | 1 - drivers/infiniband/hw/mthca/mthca_qp.c | 3 +- include/rdma/ib_verbs.h | 1 - include/rdma/uverbs_std_types.h | 11 +- include/uapi/rdma/rdma_netlink.h | 3 + 9 files changed, 273 insertions(+), 95 deletions(-)