[v4,0/6] RDMA/rxe: Add RDMA FLUSH operation

Message

Li Zhijian Aug. 5, 2022, 7:46 a.m. UTC

Hey folks,

It's been a long time since the 3rd, in the meantime, some RXE regressions have
been fixed by comminity. So It'd like to post my 4th version. feedbacks are
very welcome :).

Thanks.

These patches are going to implement a *NEW* RDMA opcode "RDMA FLUSH".
In IB SPEC 1.5[1], 2 new opcodes, ATOMIC WRITE and RDMA FLUSH were
added in the MEMORY PLACEMENT EXTENSIONS section.

This patchset makes SoftRoCE support new RDMA FLUSH on RC service.

You can verify the patchset by building and running the rdma_flush example[2].
server:
$ ./rdma_flush_server -s [server_address] -p [port_number]
client:
$ ./rdma_flush_client -s [server_address] -p [port_number]

Corresponding pyverbs and tests(tests.test_qpex.QpExTestCase.test_qp_ex_rc_rdma_flush)
are also added to rdma-core

This patches do:
- Make memory region support FLUSH access flags
- Make HCA/device support FLUSH capabilities(placement type attributes).
- Implement new packet for FLUSH request.

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/zhijianli88/rdma-core/tree/rdma-flush

CC: Xiao Yang <[e-mail]>
CC: "Gotou, Yasunori" <[e-mail]>
CC: Jason Gunthorpe <[e-mail]>
CC: Zhu Yanjun <[e-mail]
CC: Leon Romanovsky <[e-mail]>
CC: Bob Pearson <[e-mail]>
CC: Mark Bloch <[e-mail]>
CC: Wenpeng Liang <[e-mail]>
CC: Aharon Landau <[e-mail]>
CC: Tom Talpey <[e-mail]>
CC: "Gromadzki, Tomasz" <[e-mail]>
CC: Dan Williams <[e-mail]>
CC: [e-mail]
CC: [e-mail]

Can also access the kernel source in:
https://github.com/zhijianli88/linux/tree/rdma-flush
Changes log
V4:
- rework responder process
- rebase to v5.19+
- remove [7/7]: RDMA/rxe: Add RD FLUSH service support since RD is not really supported

V3:
- Just rebase and commit log and comment updates
- delete patch-1: "RDMA: mr: Introduce is_pmem", which will be combined into "Allow registering persistent flag for pmem MR only"
- delete patch-7

V2:
RDMA: mr: Introduce is_pmem
   check 1st byte to avoid crossing page boundary
   new scheme to check is_pmem # Dan

RDMA: Allow registering MR with flush access flags
   combine with [03/10] RDMA/rxe: Allow registering FLUSH flags for supported device only to this patch # Jason
   split RDMA_FLUSH to 2 capabilities

RDMA/rxe: Allow registering persistent flag for pmem MR only
   update commit message, get rid of confusing ib_check_flush_access_flags() # Tom

RDMA/rxe: Implement RC RDMA FLUSH service in requester side
   extend flush to include length field. # Tom and Tomasz

RDMA/rxe: Implement flush execution in responder side
   adjust start for WHOLE MR level # Tom
   don't support DMA mr for flush # Tom
   check flush return value

RDMA/rxe: Enable RDMA FLUSH capability for rxe device
   adjust patch's order. move it here from [04/10]
Li Zhijian (6):
  RDMA: Allow registering MR with flush access flags
  RDMA/rxe: Allow registering persistent flag for pmem MR only
  RDMA/rxe: Implement RC RDMA FLUSH service in requester side
  RDMA/rxe: Implement flush execution in responder side
  RDMA/rxe: Implement flush completion
  RDMA/rxe: Enable RDMA FLUSH capability for rxe device

 drivers/infiniband/sw/rxe/rxe_comp.c    |   4 +-
 drivers/infiniband/sw/rxe/rxe_hdr.h     |  48 ++++++++
 drivers/infiniband/sw/rxe/rxe_loc.h     |   2 +
 drivers/infiniband/sw/rxe/rxe_mr.c      |  23 +++-
 drivers/infiniband/sw/rxe/rxe_opcode.c  |  21 ++++
 drivers/infiniband/sw/rxe/rxe_opcode.h  |   4 +
 drivers/infiniband/sw/rxe/rxe_param.h   |   4 +-
 drivers/infiniband/sw/rxe/rxe_req.c     |  15 ++-
 drivers/infiniband/sw/rxe/rxe_resp.c    | 149 +++++++++++++++++++++++-
 include/rdma/ib_pack.h                  |   2 +
 include/rdma/ib_verbs.h                 |  19 ++-
 include/uapi/rdma/ib_user_ioctl_verbs.h |   2 +
 include/uapi/rdma/ib_user_verbs.h       |  14 +++
 include/uapi/rdma/rdma_user_rxe.h       |   7 ++
 14 files changed, 302 insertions(+), 12 deletions(-)

Comments

Jason Gunthorpe Aug. 26, 2022, 1:37 p.m. UTC | #1

On Fri, Aug 05, 2022 at 07:46:13AM +0000, Li Zhijian wrote:
> Hey folks,
> 
> It's been a long time since the 3rd, in the meantime, some RXE regressions have
> been fixed by comminity. So It'd like to post my 4th version. feedbacks are
> very welcome :).

I think enough work is done to start going forward with new features

Are there any comments on this series?

Jason

Xiao Yang Aug. 29, 2022, 1:36 a.m. UTC | #2

On 2022/8/26 21:37, Jason Gunthorpe wrote:
> I think enough work is done to start going forward with new features
Hi,

Good news. I hope we can start reviewing the RDMA Flush and Atomic Write 
operations. ^_^

BTW, the v5 of Atomic Write has been sent on 2022-07-08:
https://lore.kernel.org/linux-rdma/[e-mail]/
https://lore.kernel.org/linux-rdma/[e-mail]/

Best Regards,
Xiao Yang

Xiao Yang Aug. 29, 2022, 1:40 a.m. UTC | #3

On 2022/8/26 21:37, Jason Gunthorpe wrote:
> I think enough work is done to start going forward with new features
Hi,

Good news. I hope we can start reviewing the RDMA Flush and Atomic Write 
operations. ^_^

BTW, the v5 of Atomic Write has been sent on 2022-07-08:
https://lore.kernel.org/linux-rdma/[e-mail]/
https://lore.kernel.org/linux-rdma/[e-mail]/

Best Regards,
Xiao Yang