| Message ID | 20230629213248.3184245-1-yzhong@purestorage.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | RDMA/mlx5: align MR mem allocation size to power-of-two | expand |
On Thu, 29 Jun 2023 15:32:47 -0600, Yuanyuan Zhong wrote: > Recently I noticed the mlx5_alloc_priv_descs() do many allocations from > kmalloc-8k. It's called from following chain: > cma_cm_event_handler => nvme_rdma_cm_handler => nvme_rdma_create_queue_ib > => ib_mr_pool_init => ib_alloc_mr => __mlx5_ib_alloc_mr > => _mlx5_alloc_mkey_descs > > When the nvme-rdma asks to allocate memory regions that support maximum 257 > sg entries, the real MR memory allocation size is slightly bigger than 2048. > > [...] Applied, thanks! [1/1] RDMA/mlx5: align MR mem allocation size to power-of-two https://git.kernel.org/rdma/rdma/c/52b4bdd28c861e Best regards,
Hello, Recently I noticed the mlx5_alloc_priv_descs() do many allocations from kmalloc-8k. It's called from following chain: cma_cm_event_handler => nvme_rdma_cm_handler => nvme_rdma_create_queue_ib => ib_mr_pool_init => ib_alloc_mr => __mlx5_ib_alloc_mr => _mlx5_alloc_mkey_descs When the nvme-rdma asks to allocate memory regions that support maximum 257 sg entries, the real MR memory allocation size is slightly bigger than 2048. The MR memory allocation seems adding "add_size" to guarantee that there is enough space to find the memory aligned to 2048 (MLX5_UMR_ALIGN). The requested size is bigger than 4096 after adding the extra bytes. Then kmalloc uses 8192 as the actual allocation size. For power-of-two sizes, the alignment can be guaranteed by kmalloc() according to commit 59bb47985c1d ("mm, sl[aou]b: guarantee natural alignment for kmalloc(power-of-two)"). Here comes a draft patch that uses the lesser of existing requested size or the next power-of-two size. I'm not sure if it's better to simply allocate roundup_pow_of_two(), or if it's true to assume that slab will not have intermediate size between 2 power-of-two sizes. Also not sure if a generic memalign() will be the better approach. Can you please let me know what's the preferred fix, or fix it directly to let the allocation use the smaller slab if possible? Thanks Yuanyuan Zhong (1): RDMA/mlx5: align MR mem allocation size to power-of-two drivers/infiniband/hw/mlx5/mr.c | 5 +++++ 1 file changed, 5 insertions(+)