| Message ID | 20210628220043.9851-2-rpearsonhpe@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Replace AV by AH in UD sends | expand |
On Mon, Jun 28, 2021 at 05:00:40PM -0500, Bob Pearson wrote: > Make changes to rdma_user_rxe.h to allow indexing AH objects, passing > the index in UD send WRs to the driver and returning the index to the rxe > provider. This change will allow removing handling of the AV in the user > space provider. This change is backwards compatible with the current API > so new or old providers and drivers can work together. > > Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com> > include/uapi/rdma/rdma_user_rxe.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h > index e283c2220aba..e544832ed073 100644 > +++ b/include/uapi/rdma/rdma_user_rxe.h > @@ -98,6 +98,8 @@ struct rxe_send_wr { > __u32 remote_qpn; > __u32 remote_qkey; > __u16 pkey_index; > + __u16 reserved; > + __u32 ah_num; > } ud; > struct { > __aligned_u64 addr; > @@ -148,7 +150,12 @@ struct rxe_dma_info { > > struct rxe_send_wqe { > struct rxe_send_wr wr; > - struct rxe_av av; > + union { > + struct rxe_av av; > + struct { > + __u32 reserved[0]; > + } ex; > + }; What is this for? I didn't notice a usage? Jason
On 7/16/21 12:44 PM, Jason Gunthorpe wrote: > On Mon, Jun 28, 2021 at 05:00:40PM -0500, Bob Pearson wrote: >> Make changes to rdma_user_rxe.h to allow indexing AH objects, passing >> the index in UD send WRs to the driver and returning the index to the rxe >> provider. This change will allow removing handling of the AV in the user >> space provider. This change is backwards compatible with the current API >> so new or old providers and drivers can work together. >> >> Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com> >> include/uapi/rdma/rdma_user_rxe.h | 14 +++++++++++++- >> 1 file changed, 13 insertions(+), 1 deletion(-) >> >> diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h >> index e283c2220aba..e544832ed073 100644 >> +++ b/include/uapi/rdma/rdma_user_rxe.h >> @@ -98,6 +98,8 @@ struct rxe_send_wr { >> __u32 remote_qpn; >> __u32 remote_qkey; >> __u16 pkey_index; >> + __u16 reserved; >> + __u32 ah_num; >> } ud; >> struct { >> __aligned_u64 addr; >> @@ -148,7 +150,12 @@ struct rxe_dma_info { >> >> struct rxe_send_wqe { >> struct rxe_send_wr wr; >> - struct rxe_av av; >> + union { >> + struct rxe_av av; >> + struct { >> + __u32 reserved[0]; >> + } ex; >> + }; > > What is this for? I didn't notice a usage? > > Jason > Nothing yet. Was just pointing out that this is where we can extend the wqe without breaking ABI. I came back to this issue because I started working on implementing XRC and realized that I had to find someplace to put the xrc extended header info (the srq number) and the wqe was full up. Being dense I didn't figure out until later that the AV is only used for UD so this space is free anyway. Never the less this the patch set is still useful because IMO. Bob
On Fri, Jul 16, 2021 at 12:55:05PM -0500, Bob Pearson wrote: > On 7/16/21 12:44 PM, Jason Gunthorpe wrote: > > On Mon, Jun 28, 2021 at 05:00:40PM -0500, Bob Pearson wrote: > >> Make changes to rdma_user_rxe.h to allow indexing AH objects, passing > >> the index in UD send WRs to the driver and returning the index to the rxe > >> provider. This change will allow removing handling of the AV in the user > >> space provider. This change is backwards compatible with the current API > >> so new or old providers and drivers can work together. > >> > >> Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com> > >> include/uapi/rdma/rdma_user_rxe.h | 14 +++++++++++++- > >> 1 file changed, 13 insertions(+), 1 deletion(-) > >> > >> diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h > >> index e283c2220aba..e544832ed073 100644 > >> +++ b/include/uapi/rdma/rdma_user_rxe.h > >> @@ -98,6 +98,8 @@ struct rxe_send_wr { > >> __u32 remote_qpn; > >> __u32 remote_qkey; > >> __u16 pkey_index; > >> + __u16 reserved; > >> + __u32 ah_num; > >> } ud; > >> struct { > >> __aligned_u64 addr; > >> @@ -148,7 +150,12 @@ struct rxe_dma_info { > >> > >> struct rxe_send_wqe { > >> struct rxe_send_wr wr; > >> - struct rxe_av av; > >> + union { > >> + struct rxe_av av; > >> + struct { > >> + __u32 reserved[0]; > >> + } ex; > >> + }; > > > > What is this for? I didn't notice a usage? > > > > Jason > > > > Nothing yet. Was just pointing out that this is where we can extend the wqe without breaking ABI. > I came back to this issue because I started working on implementing XRC and realized that I had to find someplace to put the xrc extended header info (the srq number) and the wqe was full up. Being dense > I didn't figure out until later that the AV is only used for UD so this space is free anyway. > Never the less this the patch set is still useful because IMO. I think you should move the rxe_av memory from the struct rxe_send_wqe to the rxe_send_wr.ud struct by padding out the ud struct and placing the av in the proper location. Then you don't need this confusing union and the whole thing is much clearer.. Jason
diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index e283c2220aba..e544832ed073 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -98,6 +98,8 @@ struct rxe_send_wr { __u32 remote_qpn; __u32 remote_qkey; __u16 pkey_index; + __u16 reserved; + __u32 ah_num; } ud; struct { __aligned_u64 addr; @@ -148,7 +150,12 @@ struct rxe_dma_info { struct rxe_send_wqe { struct rxe_send_wr wr; - struct rxe_av av; + union { + struct rxe_av av; + struct { + __u32 reserved[0]; + } ex; + }; __u32 status; __u32 state; __aligned_u64 iova; @@ -168,6 +175,11 @@ struct rxe_recv_wqe { struct rxe_dma_info dma; }; +struct rxe_create_ah_resp { + __u32 ah_num; + __u32 reserved; +}; + struct rxe_create_cq_resp { struct mminfo mi; };
Make changes to rdma_user_rxe.h to allow indexing AH objects, passing the index in UD send WRs to the driver and returning the index to the rxe provider. This change will allow removing handling of the AV in the user space provider. This change is backwards compatible with the current API so new or old providers and drivers can work together. Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com> --- include/uapi/rdma/rdma_user_rxe.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)