mbox series

[v2,RFC,0/7] RDMA/hns: Add support for Dynamic Context Attachment

Message ID 1611394994-50363-1-git-send-email-liweihang@huawei.com (mailing list archive)
Headers show
Series RDMA/hns: Add support for Dynamic Context Attachment | expand

Message

Weihang Li Jan. 23, 2021, 9:43 a.m. UTC
The hip09 introduces the DCA(Dynamic Context Attachment) feature which
supports many RC QPs to share the WQE buffer in a memory pool. If a QP
enables DCA feature, the WQE's buffer will not be allocated when creating
but when the users start to post WRs. This will reduce the memory
consumption when there are too many QPs are inactive.

Changes since v1:
* Replace all GFP_ATOMIC with GFP_NOWAIT, because the former may use
  emergency pool if no regular memory can be found.
* Change size of cap_flags of alloc_ucontext_resp from 32 to 64 to avoid
  a potential problem when pass it back to the userspace.
* Move definition of HNS_ROCE_CAP_FLAG_DCA_MODE to hns-abi.h.
* Rename free_mem_states() to free_dca_states() in #1.
* Link: https://patchwork.kernel.org/project/linux-rdma/cover/1610706138-4219-1-git-send-email-liweihang@huawei.com/

Xi Wang (7):
  RDMA/hns: Introduce DCA for RC QP
  RDMA/hns: Add method for shrinking DCA memory pool
  RDMA/hns: Configure DCA mode for the userspace QP
  RDMA/hns: Add method for attaching WQE buffer
  RDMA/hns: Setup the configuration of WQE addressing to QPC
  RDMA/hns: Add method to detach WQE buffer
  RDMA/hns: Add method to query WQE buffer's address

 drivers/infiniband/hw/hns/Makefile          |    2 +-
 drivers/infiniband/hw/hns/hns_roce_dca.c    | 1264 +++++++++++++++++++++++++++
 drivers/infiniband/hw/hns/hns_roce_dca.h    |   68 ++
 drivers/infiniband/hw/hns/hns_roce_device.h |   31 +
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c  |  223 ++++-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.h  |    3 +
 drivers/infiniband/hw/hns/hns_roce_main.c   |   27 +-
 drivers/infiniband/hw/hns/hns_roce_qp.c     |  119 ++-
 include/uapi/rdma/hns-abi.h                 |   64 ++
 9 files changed, 1754 insertions(+), 47 deletions(-)
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.c
 create mode 100644 drivers/infiniband/hw/hns/hns_roce_dca.h

Comments

Jason Gunthorpe Feb. 5, 2021, 6:03 p.m. UTC | #1
On Sat, Jan 23, 2021 at 05:43:07PM +0800, Weihang Li wrote:
> The hip09 introduces the DCA(Dynamic Context Attachment) feature which
> supports many RC QPs to share the WQE buffer in a memory pool. If a QP
> enables DCA feature, the WQE's buffer will not be allocated when creating
> but when the users start to post WRs. This will reduce the memory
> consumption when there are too many QPs are inactive.
> 
> Changes since v1:
> * Replace all GFP_ATOMIC with GFP_NOWAIT, because the former may use
>   emergency pool if no regular memory can be found.
> * Change size of cap_flags of alloc_ucontext_resp from 32 to 64 to avoid
>   a potential problem when pass it back to the userspace.
> * Move definition of HNS_ROCE_CAP_FLAG_DCA_MODE to hns-abi.h.
> * Rename free_mem_states() to free_dca_states() in #1.
> * Link: https://patchwork.kernel.org/project/linux-rdma/cover/1610706138-4219-1-git-send-email-liweihang@huawei.com/
> 
> Xi Wang (7):
>   RDMA/hns: Introduce DCA for RC QP
>   RDMA/hns: Add method for shrinking DCA memory pool
>   RDMA/hns: Configure DCA mode for the userspace QP
>   RDMA/hns: Add method for attaching WQE buffer
>   RDMA/hns: Setup the configuration of WQE addressing to QPC
>   RDMA/hns: Add method to detach WQE buffer
>   RDMA/hns: Add method to query WQE buffer's address
> 
>  drivers/infiniband/hw/hns/Makefile          |    2 +-
>  drivers/infiniband/hw/hns/hns_roce_dca.c    | 1264 +++++++++++++++++++++++++++
>  drivers/infiniband/hw/hns/hns_roce_dca.h    |   68 ++
>  drivers/infiniband/hw/hns/hns_roce_device.h |   31 +
>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c  |  223 ++++-
>  drivers/infiniband/hw/hns/hns_roce_hw_v2.h  |    3 +
>  drivers/infiniband/hw/hns/hns_roce_main.c   |   27 +-
>  drivers/infiniband/hw/hns/hns_roce_qp.c     |  119 ++-
>  include/uapi/rdma/hns-abi.h                 |   64 ++

Where are the rdma-core changes to go with this?

Jason
Weihang Li Feb. 7, 2021, 3:34 a.m. UTC | #2
On 2021/2/6 2:03, Jason Gunthorpe wrote:
> On Sat, Jan 23, 2021 at 05:43:07PM +0800, Weihang Li wrote:
>> The hip09 introduces the DCA(Dynamic Context Attachment) feature which
>> supports many RC QPs to share the WQE buffer in a memory pool. If a QP
>> enables DCA feature, the WQE's buffer will not be allocated when creating
>> but when the users start to post WRs. This will reduce the memory
>> consumption when there are too many QPs are inactive.
>>
>> Changes since v1:
>> * Replace all GFP_ATOMIC with GFP_NOWAIT, because the former may use
>>   emergency pool if no regular memory can be found.
>> * Change size of cap_flags of alloc_ucontext_resp from 32 to 64 to avoid
>>   a potential problem when pass it back to the userspace.
>> * Move definition of HNS_ROCE_CAP_FLAG_DCA_MODE to hns-abi.h.
>> * Rename free_mem_states() to free_dca_states() in #1.
>> * Link: https://patchwork.kernel.org/project/linux-rdma/cover/1610706138-4219-1-git-send-email-liweihang@huawei.com/
>>
>> Xi Wang (7):
>>   RDMA/hns: Introduce DCA for RC QP
>>   RDMA/hns: Add method for shrinking DCA memory pool
>>   RDMA/hns: Configure DCA mode for the userspace QP
>>   RDMA/hns: Add method for attaching WQE buffer
>>   RDMA/hns: Setup the configuration of WQE addressing to QPC
>>   RDMA/hns: Add method to detach WQE buffer
>>   RDMA/hns: Add method to query WQE buffer's address
>>
>>  drivers/infiniband/hw/hns/Makefile          |    2 +-
>>  drivers/infiniband/hw/hns/hns_roce_dca.c    | 1264 +++++++++++++++++++++++++++
>>  drivers/infiniband/hw/hns/hns_roce_dca.h    |   68 ++
>>  drivers/infiniband/hw/hns/hns_roce_device.h |   31 +
>>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c  |  223 ++++-
>>  drivers/infiniband/hw/hns/hns_roce_hw_v2.h  |    3 +
>>  drivers/infiniband/hw/hns/hns_roce_main.c   |   27 +-
>>  drivers/infiniband/hw/hns/hns_roce_qp.c     |  119 ++-
>>  include/uapi/rdma/hns-abi.h                 |   64 ++
> 
> Where are the rdma-core changes to go with this?
> 
> Jason
> 

I sent the userspace part just now:
https://patchwork.kernel.org/project/linux-rdma/cover/1612667574-56673-1-git-send-email-liweihang@huawei.com/

Thanks
Weihang