| Message ID | 1612667574-56673-1-git-send-email-liweihang@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | libhns: Add support for Dynamic Context Attachment | expand |
On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: > The HIP09 introduces the DCA(Dynamic Context Attachment) feature which > supports many RC QPs to share the WQE buffer in a memory pool. If a QP > enables DCA feature, the WQE's buffer will not be allocated when creating > but when the users start to post WRs. This will reduce the memory > consumption when there are too many QPs are inactive. > > Please note that we didn't find the right way to get user's configuration, > so in #4 we still use environment variable to achieve this. We will be > appreciated if anyone can provide some sugggestions. That is definately not going to work.. It should be some dv thing, dv create qp or a dv customization of the parent domain spring to mind Jason
On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: > The HIP09 introduces the DCA(Dynamic Context Attachment) feature which > supports many RC QPs to share the WQE buffer in a memory pool. If a QP > enables DCA feature, the WQE's buffer will not be allocated when creating > but when the users start to post WRs. This will reduce the memory > consumption when there are too many QPs are inactive. One a WQE buffer is allocated it still acts as a normal WQE ring buffer? So this DCA logic is to remap the send queue buffer based on demand for SQEs? How does it interact with the normal max send queue entries reported? Would like to see proper man pages explaining how this all works for rdma-core. Jason
On 2021/2/10 3:54, Jason Gunthorpe wrote: > On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: >> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which >> supports many RC QPs to share the WQE buffer in a memory pool. If a QP >> enables DCA feature, the WQE's buffer will not be allocated when creating >> but when the users start to post WRs. This will reduce the memory >> consumption when there are too many QPs are inactive. >> One a WQE buffer is allocated it still acts as a normal WQE ring > buffer? So this DCA logic is to remap the send queue buffer based on > demand for SQEs? How does it interact with the normal max send queue > entries reported? > Not exactly. If DCA is enabled, we first allocate a memory pool with a default size when opening device. Each time we trying to post WR(s) to a QP, the driver will check if current QP has WQE buffer. If not, the driver will check whether there is enough free memory in the DCA memory pool. If there is, the QP will get WQE buffer from the pool, including SQ buffer in size of max_send_wr, RQ buffer in size of max_recv_wr and extended sge buffer. If there is no enough space for the WQE buffer in the DCA pool, the driver will expand the size of pool and then assign the buffer to the QP. And if the WQE buffer of a QP is not used, this buffer would be recycled and the DCA memory pool would be shrinked. > Would like to see proper man pages explaining how this all works for > rdma-core. > > Jason > OK, will add man pages for DCA in next version. Thanks Weihang
On 2021/2/10 3:39, Jason Gunthorpe wrote: > On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: >> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which >> supports many RC QPs to share the WQE buffer in a memory pool. If a QP >> enables DCA feature, the WQE's buffer will not be allocated when creating >> but when the users start to post WRs. This will reduce the memory >> consumption when there are too many QPs are inactive. >> >> Please note that we didn't find the right way to get user's configuration, >> so in #4 we still use environment variable to achieve this. We will be >> appreciated if anyone can provide some sugggestions. > > That is definately not going to work.. It should be some dv thing, > dv create qp or a dv customization of the parent domain spring to mind > > Jason > Thank you, we will use the parameter of verbs_open_device() named private_data to allow user pass the DCA configurations into the driver according to the way of mlx5dv_open_device(). Weihang
On Sat, Feb 20, 2021 at 08:40:02AM +0000, liweihang wrote: > On 2021/2/10 3:54, Jason Gunthorpe wrote: > > On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: > >> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which > >> supports many RC QPs to share the WQE buffer in a memory pool. If a QP > >> enables DCA feature, the WQE's buffer will not be allocated when creating > >> but when the users start to post WRs. This will reduce the memory > >> consumption when there are too many QPs are inactive. > >> One a WQE buffer is allocated it still acts as a normal WQE ring > > buffer? So this DCA logic is to remap the send queue buffer based on > > demand for SQEs? How does it interact with the normal max send queue > > entries reported? > > > > Not exactly. If DCA is enabled, we first allocate a memory pool with a > default size when opening device. Each time we trying to post WR(s) to a > QP, the driver will check if current QP has WQE buffer. > > If not, the driver will check whether there is enough free memory in the > DCA memory pool. If there is, the QP will get WQE buffer from the > pool, Does that mean the QP can have a non-contiguous list of buffers? Ie it isn't just a linear ring of memory? Jason
On 2021/2/23 0:02, Jason Gunthorpe wrote: > On Sat, Feb 20, 2021 at 08:40:02AM +0000, liweihang wrote: >> On 2021/2/10 3:54, Jason Gunthorpe wrote: >>> On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: >>>> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which >>>> supports many RC QPs to share the WQE buffer in a memory pool. If a QP >>>> enables DCA feature, the WQE's buffer will not be allocated when creating >>>> but when the users start to post WRs. This will reduce the memory >>>> consumption when there are too many QPs are inactive. >>>> One a WQE buffer is allocated it still acts as a normal WQE ring >>> buffer? So this DCA logic is to remap the send queue buffer based on >>> demand for SQEs? How does it interact with the normal max send queue >>> entries reported? >>> >> >> Not exactly. If DCA is enabled, we first allocate a memory pool with a >> default size when opening device. Each time we trying to post WR(s) to a >> QP, the driver will check if current QP has WQE buffer. >> >> If not, the driver will check whether there is enough free memory in the >> DCA memory pool. If there is, the QP will get WQE buffer from the >> pool, > > Does that mean the QP can have a non-contiguous list of buffers? Ie it > isn't just a linear ring of memory? > > Jason > Yes, when using multilevel addressing, the virtual address of QP's buffers may be non-contiguous to avoid memory fragmentation. Weihang
On 2021/2/10 3:54, Jason Gunthorpe wrote: > On Sun, Feb 07, 2021 at 11:12:49AM +0800, Weihang Li wrote: >> The HIP09 introduces the DCA(Dynamic Context Attachment) feature which >> supports many RC QPs to share the WQE buffer in a memory pool. If a QP >> enables DCA feature, the WQE's buffer will not be allocated when creating >> but when the users start to post WRs. This will reduce the memory >> consumption when there are too many QPs are inactive. > > One a WQE buffer is allocated it still acts as a normal WQE ring > buffer? So this DCA logic is to remap the send queue buffer based on > demand for SQEs? How does it interact with the normal max send queue > entries reported? > > Would like to see proper man pages explaining how this all works for > rdma-core. > > Jason > Hi Jason, I'm confused about how to introduce DCA in man pages. Current man pages in rdma-core can be classifed into public and vendor-defined ones. For example, ibv_create_qp.3 in libibverbs/man and mlx5dv.7 in providers/mlx5/man, but most of them is a description for a single interface. If we want to explain how to use DCA and how does it work, should we put a hns_dca.x file in providers/hns/man? Or add a file about hns_dca_open_device() and introduce DCA in it? And another question, I know the files with a number suffix like ibv_create_qp.3 is for man pages in linux. What about the markdown files with .md suffix like ibv_fork_init.3.md? If we want to add a new one about DCA, which type should we choose? Thanks Weihang
On Wed, Feb 24, 2021 at 09:48:34AM +0000, liweihang wrote: > I'm confused about how to introduce DCA in man pages. Current man pages > in rdma-core can be classifed into public and vendor-defined ones. > For example, ibv_create_qp.3 in libibverbs/man and mlx5dv.7 in > providers/mlx5/man, but most of them is a description for a single > interface. If we want to explain how to use DCA and how does it work, > should we put a hns_dca.x file in providers/hns/man? You could do that, yes > And another question, I know the files with a number suffix like > ibv_create_qp.3 is for man pages in linux. What about the markdown files > with .md suffix like ibv_fork_init.3.md? If we want to add a new one about > DCA, which type should we choose? Always use the md files for new man pages please Jason