[v6,0/2] io_uring/rsrc: coalescing multi-hugepage registered buffers

Message

Chenliang Li July 16, 2024, 6:08 a.m. UTC

Registered buffers are stored and processed in the form of bvec array,
each bvec element typically points to a PAGE_SIZE page but can also work
with hugepages. Specifically, a buffer consisting of a hugepage is
coalesced to use only one hugepage bvec entry during registration.
This coalescing feature helps to save both the space and DMA-mapping time.

However, currently the coalescing feature doesn't work for multi-hugepage
buffers. For a buffer with several 2M hugepages, we still split it into
thousands of 4K page bvec entries while in fact, we can just use a
handful of hugepage bvecs.

This patch series enables coalescing registered buffers with more than
one hugepages. It optimizes the DMA-mapping time and saves memory for 
these kind of buffers.

Testing:

The hugepage fixed buffer I/O can be tested using fio without
modification. The fio command used in the following test is given
in [1]. There's also a liburing testcase in [2]. Also, the system
should have enough hugepages available before testing.

Perf diff of 8M(4 * 2M hugepages) fio randread test:

Before          After           Symbol
.....................................................
5.88%                           [k] __blk_rq_map_sg
3.98%           -3.95%          [k] dma_direct_map_sg
2.47%                           [k] dma_pool_alloc
1.37%           -1.36%          [k] sg_next
                +0.28%          [k] dma_map_page_attrs

Perf diff of 8M fio randwrite test:

Before          After           Symbol
......................................................
2.80%                           [k] __blk_rq_map_sg
1.74%                           [k] dma_direct_map_sg
1.61%                           [k] dma_pool_alloc
0.67%                           [k] sg_next
                +0.04%          [k] dma_map_page_attrs

The first patch prepares for adding the multi-hugepage coalescing
by storing folio_shift and folio_mask into imu, the 2nd patch
enables the feature.

---

Changes since v5:
- Reshuffle the patchset to avoid unused funtion warnings.
- Store head page of the folio and use folio offset to get rid of
  branching on bvec setups.
- Add restrictions for non border-aligned folios.
- Remove unnecessary folio_size field in io_imu_folio_data struct.

v5 : https://lore.kernel.org/io-uring/[e-mail]/T/#t

[1]
fio -iodepth=64 -rw=randread -direct=1 -ioengine=io_uring \
-bs=8M -numjobs=1 -group_reporting -mem=shmhuge -fixedbufs -hugepage-size=2M \
-filename=/dev/nvme0n1 -runtime=10s -name=test1

[2]
https://lore.kernel.org/io-uring/[e-mail]/T/#u

Chenliang Li (2):
  io_uring/rsrc: store folio shift and mask into imu
  io_uring/rsrc: enable multi-hugepage buffer coalescing

 io_uring/rsrc.c | 154 +++++++++++++++++++++++++++++++++++-------------
 io_uring/rsrc.h |  10 ++++
 2 files changed, 123 insertions(+), 41 deletions(-)


base-commit: ad00e629145b2b9f0d78aa46e204a9df7d628978