| Message ID | 20220418061244.89025-1-yangx.jy@fujitsu.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | RDMA/rxe: Add RDMA Atomic Write operation | expand |
On Mon, Apr 18, 2022 at 02:12:41PM +0800, Xiao Yang wrote: > The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This > patchset makes SoftRoCE support new RDMA Atomic Write on RC service. > > I add ibv_wr_rdma_atomic_write() and a rdma_atomic_write example on my > rdma-core repository[2]. You can verify the patchset by building and > running the rdma_atomic_write example. > server: > $ ./rdma_atomic_write_server -s [server_address] -p [port_number] > client: > $ ./rdma_atomic_write_client -s [server_address] -p [port_number] We need PR to official rdma-core repo with pyverbs test to consider this code for merge. Thanks > > [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx > [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point > > v3->v4: > 1) Rebase on current wip/jgg-for-next > 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write > 3) Replace 64-bit value with 8-byte array for RDMA Atomic Write > > V2->V3: > 1) Rebase > 2) Add RDMA Atomic Write attribute for rxe device > > V1->V2: > 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D > 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value > 3) Use smp_store_release() to ensure that all prior operations have completed > > Xiao Yang (3): > RDMA/rxe: Rename send_atomic_ack() and atomic member of struct > resp_res > RDMA/rxe: Support RDMA Atomic Write operation > RDMA/rxe: Add RDMA Atomic Write attribute for rxe device > > drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ > drivers/infiniband/sw/rxe/rxe_opcode.c | 19 ++++++++ > drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++ > drivers/infiniband/sw/rxe/rxe_param.h | 5 +++ > drivers/infiniband/sw/rxe/rxe_qp.c | 4 +- > drivers/infiniband/sw/rxe/rxe_req.c | 13 +++++- > drivers/infiniband/sw/rxe/rxe_resp.c | 61 ++++++++++++++++++++------ > drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +- > include/rdma/ib_pack.h | 2 + > include/rdma/ib_verbs.h | 3 ++ > include/uapi/rdma/ib_user_verbs.h | 4 ++ > include/uapi/rdma/rdma_user_rxe.h | 1 + > 12 files changed, 103 insertions(+), 18 deletions(-) > > -- > 2.34.1 > > >
On 2022/4/20 14:54, Leon Romanovsky wrote: > We need PR to official rdma-core repo with pyverbs test to consider this > code for merge. Hi Leon, Thanks for your suggestion. I will write the corresponding pyverbs test recently. ^_^ Best Regards, Xiao Yang > > Thanks
On 2022/4/20 15:08, yangx.jy@fujitsu.com wrote: > On 2022/4/20 14:54, Leon Romanovsky wrote: >> We need PR to official rdma-core repo with pyverbs test to consider this >> code for merge. > Hi Leon, > > Thanks for your suggestion. > I will write the corresponding pyverbs test recently. ^_^ > > Best Regards, > Xiao Yang Hi Leon, Others I have introduced RDMA Atomic Write API for pyverbs and added a corresponding python test on my rdma-core repo. # bin/run_tests.py --dev rxe_enp0s3 --gid 1 -v tests.test_qpex.QpExTestCase.test_qp_ex_rc_rdma_atomic_write test_qp_ex_rc_rdma_atomic_write (tests.test_qpex.QpExTestCase) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.004s OK I hope you and others can review both kernel patch set and library patch set. Thanks a lot. Best Regards, Xiao Yang >> >> Thanks
Hi All, Ping. ^_^ Are there more comments on this patch set? Best Regards, Xiao Yang On 2022/4/18 14:12, Xiao Yang wrote: > The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This > patchset makes SoftRoCE support new RDMA Atomic Write on RC service. > > I add ibv_wr_rdma_atomic_write() and a rdma_atomic_write example on my > rdma-core repository[2]. You can verify the patchset by building and > running the rdma_atomic_write example. > server: > $ ./rdma_atomic_write_server -s [server_address] -p [port_number] > client: > $ ./rdma_atomic_write_client -s [server_address] -p [port_number] > > [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx > [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point > > v3->v4: > 1) Rebase on current wip/jgg-for-next > 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write > 3) Replace 64-bit value with 8-byte array for RDMA Atomic Write > > V2->V3: > 1) Rebase > 2) Add RDMA Atomic Write attribute for rxe device > > V1->V2: > 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D > 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value > 3) Use smp_store_release() to ensure that all prior operations have completed > > Xiao Yang (3): > RDMA/rxe: Rename send_atomic_ack() and atomic member of struct > resp_res > RDMA/rxe: Support RDMA Atomic Write operation > RDMA/rxe: Add RDMA Atomic Write attribute for rxe device > > drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ > drivers/infiniband/sw/rxe/rxe_opcode.c | 19 ++++++++ > drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++ > drivers/infiniband/sw/rxe/rxe_param.h | 5 +++ > drivers/infiniband/sw/rxe/rxe_qp.c | 4 +- > drivers/infiniband/sw/rxe/rxe_req.c | 13 +++++- > drivers/infiniband/sw/rxe/rxe_resp.c | 61 ++++++++++++++++++++------ > drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +- > include/rdma/ib_pack.h | 2 + > include/rdma/ib_verbs.h | 3 ++ > include/uapi/rdma/ib_user_verbs.h | 4 ++ > include/uapi/rdma/rdma_user_rxe.h | 1 + > 12 files changed, 103 insertions(+), 18 deletions(-) >
On Fri, May 13, 2022 at 03:46:00AM +0000, yangx.jy@fujitsu.com wrote: > Hi All, > > Ping. ^_^ > Are there more comments on this patch set? I think I said this already, but I'm waiting for all the rxe regressions to be fixed before looking any new rxe patches. Jason
On 5/13/22 06:46, Jason Gunthorpe wrote: > On Fri, May 13, 2022 at 03:46:00AM +0000, yangx.jy@fujitsu.com wrote: >> Hi All, >> >> Ping. ^_^ >> Are there more comments on this patch set? > > I think I said this already, but I'm waiting for all the rxe > regressions to be fixed before looking any new rxe patches. > > Jason I am working on them. I am trying to reproduce Bart's claim that he saw the blktests hangs go away with a merge of head of tree. Bob
On Mon, Apr 18, 2022 at 02:12:41PM +0800, Xiao Yang wrote: > The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This > patchset makes SoftRoCE support new RDMA Atomic Write on RC service. This series doesn't apply cleanly, please rebase it and re-tet it. Did you make pyverbs coverage for this? Jason
On 2022/7/4 21:45, Jason Gunthorpe wrote: > On Mon, Apr 18, 2022 at 02:12:41PM +0800, Xiao Yang wrote: >> The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This >> patchset makes SoftRoCE support new RDMA Atomic Write on RC service. > > This series doesn't apply cleanly, please rebase it and re-tet it. Hi Jason, Sure, I will rebase and retest it soon. > > Did you make pyverbs coverage for this? Yes, I have added a pyverbs API and test for RDMA Atomic Write. See the following PR: https://github.com/linux-rdma/rdma-core/pull/1179 BTW, I will check if this PR needs to be rebased as well. Best Regards, Xiao Yang > > Jason
The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This patchset makes SoftRoCE support new RDMA Atomic Write on RC service. I add ibv_wr_rdma_atomic_write() and a rdma_atomic_write example on my rdma-core repository[2]. You can verify the patchset by building and running the rdma_atomic_write example. server: $ ./rdma_atomic_write_server -s [server_address] -p [port_number] client: $ ./rdma_atomic_write_client -s [server_address] -p [port_number] [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point v3->v4: 1) Rebase on current wip/jgg-for-next 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write 3) Replace 64-bit value with 8-byte array for RDMA Atomic Write V2->V3: 1) Rebase 2) Add RDMA Atomic Write attribute for rxe device V1->V2: 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value 3) Use smp_store_release() to ensure that all prior operations have completed Xiao Yang (3): RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res RDMA/rxe: Support RDMA Atomic Write operation RDMA/rxe: Add RDMA Atomic Write attribute for rxe device drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++ drivers/infiniband/sw/rxe/rxe_opcode.c | 19 ++++++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++ drivers/infiniband/sw/rxe/rxe_param.h | 5 +++ drivers/infiniband/sw/rxe/rxe_qp.c | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 13 +++++- drivers/infiniband/sw/rxe/rxe_resp.c | 61 ++++++++++++++++++++------ drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +- include/rdma/ib_pack.h | 2 + include/rdma/ib_verbs.h | 3 ++ include/uapi/rdma/ib_user_verbs.h | 4 ++ include/uapi/rdma/rdma_user_rxe.h | 1 + 12 files changed, 103 insertions(+), 18 deletions(-)