mbox series

[v4,for-next,00/12] RDMA/hns: Add support for Dynamic Context Attachment

Message ID 1627525163-1683-1-git-send-email-liangwenpeng@huawei.com (mailing list archive)
Headers show
Series RDMA/hns: Add support for Dynamic Context Attachment | expand

Message

Wenpeng Liang July 29, 2021, 2:19 a.m. UTC 
The HIP09 introduces the DCA(Dynamic Context Attachment) feature which
supports many RC QPs to share the WQE buffer in a memory pool. If a QP
enables DCA feature, the WQE's buffer will not be allocated when creating
but when the users start to post WRs. This will reduce the memory
consumption when there are too many QPs are inactive.

The related userspace series is named "libhns: Add support for Dynamic Context Attachment".

Changes since v3:
* alloc_dca_states() replaces GFP_NOWAIT with the GFP_KERNEL in #1.
* Remove the calculation of DCA percentage in #12.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1627356452-30564-1-git-send-email-liangwenpeng@huawei.com/

Changes since v2:
* Refactor flow of modify_qp and then add a shared memory mechanism to
  synchronize status of DCA, which can save more memory than previous
  scheme.
* Remove some inline functions from #1.
* Add support for dumping DCA mem status in restrack.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1620732161-27180-1-git-send-email-liweihang@huawei.com/

Changes since v1:
* Modify return type of hns_roce_enable_dca() to void.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1620650889-61650-1-git-send-email-liweihang@huawei.com/

Two RFC versions of this series has been sent before, and it's associated
with the userspace one "libhns: Add support for Dynamic Context
Attachment".

Changes since RFC v2:
* Just fix a typo in commit message of #6.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1611394994-50363-1-git-send-email-liweihang@huawei.com/

Changes since RFC v1:
* Replace all GFP_ATOMIC with GFP_NOWAIT, because the former may use
  emergency pool if no regular memory can be found.
* Change size of cap_flags of alloc_ucontext_resp from 32 to 64 to avoid
  a potential problem when pass it back to the userspace.
* Move definition of HNS_ROCE_CAP_FLAG_DCA_MODE to hns-abi.h.
* Rename free_mem_states() to free_dca_states() in #1.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1610706138-4219-1-git-send-email-liweihang@huawei.com/

Xi Wang (12):
  RDMA/hns: Introduce DCA for RC QP
  RDMA/hns: Add method for shrinking DCA memory pool
  RDMA/hns: Configure DCA mode for the userspace QP
  RDMA/hns: Refactor QP modify flow
  RDMA/hns: Add method for attaching WQE buffer
  RDMA/hns: Setup the configuration of WQE addressing to QPC
  RDMA/hns: Add method to detach WQE buffer
  RDMA/hns: Add method to query WQE buffer's address
  RDMA/hns: Add a shared memory to sync DCA status
  RDMA/hns: Sync DCA status by the shared memory
  RDMA/nldev: Add detailed CTX information support
  RDMA/hns: Dump detailed driver-specific UCTX

 drivers/infiniband/core/device.c              |    1 +
 drivers/infiniband/core/nldev.c               |    8 +-
 drivers/infiniband/hw/hns/Makefile            |    2 +-
 drivers/infiniband/hw/hns/hns_roce_dca.c      | 1438 +++++++++++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_dca.h      |   73 ++
 drivers/infiniband/hw/hns/hns_roce_device.h   |   47 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v1.c    |   12 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c    |  173 ++-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.h    |    1 +
 drivers/infiniband/hw/hns/hns_roce_main.c     |  128 ++-
 drivers/infiniband/hw/hns/hns_roce_qp.c       |  169 ++-
 drivers/infiniband/hw/hns/hns_roce_restrack.c |   50 +
 include/rdma/ib_verbs.h                       |    1 +
 include/uapi/rdma/hns-abi.h                   |   86 ++
 14 files changed, 2113 insertions(+), 76 deletions(-)
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.c
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.h

--
2.8.1

Comments

Wenpeng Liang Aug. 12, 2021, 12:42 p.m. UTC | #1 
On 2021/7/29 10:19, Wenpeng Liang wrote:
> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which
> supports many RC QPs to share the WQE buffer in a memory pool. If a QP
> enables DCA feature, the WQE's buffer will not be allocated when creating
> but when the users start to post WRs. This will reduce the memory
> consumption when there are too many QPs are inactive.
> 
> The related userspace series is named "libhns: Add support for Dynamic Context Attachment".

Hi
Does anyone have an opinion about this patchset, thanks!
Wenpeng Liang Aug. 21, 2021, 9:34 a.m. UTC | #2 
On 2021/7/29 10:19, Wenpeng Liang wrote:
> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which
> supports many RC QPs to share the WQE buffer in a memory pool. If a QP
> enables DCA feature, the WQE's buffer will not be allocated when creating
> but when the users start to post WRs. This will reduce the memory
> consumption when there are too many QPs are inactive.

Hi, Jason.
The comments of v4 have not been fixed yet, but the jgg-for-next branch has
already applied this patchset, should I send v5 or send a fix?

Thanks,
Wenpeng
Jason Gunthorpe Aug. 22, 2021, 10:25 p.m. UTC | #3 
On Sat, Aug 21, 2021 at 05:34:48PM +0800, Wenpeng Liang wrote:
> 
> 
> On 2021/7/29 10:19, Wenpeng Liang wrote:
> > The HIP09 introduces the DCA(Dynamic Context Attachment) feature which
> > supports many RC QPs to share the WQE buffer in a memory pool. If a QP
> > enables DCA feature, the WQE's buffer will not be allocated when creating
> > but when the users start to post WRs. This will reduce the memory
> > consumption when there are too many QPs are inactive.
> 
> Hi, Jason.
> The comments of v4 have not been fixed yet, but the jgg-for-next branch has
> already applied this patchset, should I send v5 or send a fix?

Oh, I made a mistake, I've fixed the branches, thanks for noticing

Jason