| Message ID | cover.1667384020.git.asml.silence@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | implement pcpu bio caching for IRQ I/O | expand |
On Wed, 2 Nov 2022 15:18:18 +0000, Pavel Begunkov wrote: > Add bio pcpu caching for IRQ-driven I/O. We extend the currently limited to > iopoll REQ_ALLOC_CACHE infra. Benchmarked with t/io_uring and an Optane SSD: > 2.22 -> 2.32 MIOPS for qd32 (+4.5%) and 2.60 vs 2.82 for qd128 (+8.4%). > > Works best with per-cpu queues, otherwise there might be some effects at > play, e.g. bios allocated by one cpu but freed by another, but the worst > case (always goes to mempool) doesn't show any performance degradation. > > [...] Applied, thanks! [1/6] mempool: introduce mempool_is_saturated commit: 6e4068a11413b96687a03c39814539e202de294b [2/6] bio: don't rob starving biosets of bios commit: 759aa12f19155fe4e4fb4740450b4aa4233b7d9f [3/6] bio: split pcpu cache part of bio_put into a helper commit: f25cf75a452150c243f74ab1f1836822137d5d2c [4/6] bio: add pcpu caching for non-polling bio_put commit: b99182c501c3dedeba4c9e6c92a60df1a2fee119 [5/6] bio: shrink max number of pcpu cached bios commit: 42b2b2fb6ecf1cc11eb7e75782dd7a7a17e6d958 [6/6] io_uring/rw: enable bio caches for IRQ rw commit: 12e4e8c7ab5978eb56f9d363461a8a40a8618bf4 Best regards,
Add bio pcpu caching for IRQ-driven I/O. We extend the currently limited to iopoll REQ_ALLOC_CACHE infra. Benchmarked with t/io_uring and an Optane SSD: 2.22 -> 2.32 MIOPS for qd32 (+4.5%) and 2.60 vs 2.82 for qd128 (+8.4%). Works best with per-cpu queues, otherwise there might be some effects at play, e.g. bios allocated by one cpu but freed by another, but the worst case (always goes to mempool) doesn't show any performance degradation. Currently, it's only enabled for previous REQ_ALLOC_CACHE users but will be turned on system-wide later. v2: fix botched splicing threshold checks v3: remove merged patch limit scope of flags var in bio_put_percpu_cache v4: correct outdated comment fix in-irq put -> splice modifying the non-irq safe cache list fix alloc null dereference Pavel Begunkov (6): mempool: introduce mempool_is_saturated bio: don't rob starving biosets of bios bio: split pcpu cache part of bio_put into a helper bio: add pcpu caching for non-polling bio_put bio: shrink max number of pcpu cached bios io_uring/rw: enable bio caches for IRQ rw block/bio.c | 98 +++++++++++++++++++++++++++++++---------- include/linux/mempool.h | 5 +++ io_uring/rw.c | 3 +- 3 files changed, 82 insertions(+), 24 deletions(-)