[for-next,v3,0/9] io_uring: batch multishot completions

Message

Dylan Yudaken Nov. 24, 2022, 9:35 a.m. UTC

Multishot completions currently all go through io_post_aux_cqe which will
do a lock/unlock pair of the completion spinlock, and also possibly signal
an eventfd if registered. This can slow down applications that use these
features.

This series allows the posted completions to be batched using the same
IO_URING_F_COMPLETE_DEFER as exists for non multishot completions. A
critical property of this is that all multishot completions must be
flushed to the CQ ring before the non-multishot completion (say an error)
or else ordering will break. This implies that if some completions were
deferred, then the rest must also be to keep that ordering. In order to do
this the first few patches move all the completion code into a simpler
path that defers completions when possible.

The batching is done by keeping an array of 16 CQEs, and adding to it
rather than posting immediately. If it fills up the posting happens then.

A microbenchmark was run ([1]) to test this and showed a 2.3x rps
improvment (8.3 M/s vs 19.3 M/s).

Patches 1-3 clean up the completion paths
Patch 4 introduces the cqe array
Patch 5 allows io_post_aux_cqe to use the cqe array to defer completions
Patches 6-8 are small cleanups
Patch 9 enables defered completions for multishot polled requests

v3:
 - rebase onto recent changes. A lot of duplicate changes so dropped 4 patches

v2:
 - Rebase. This includes having to deal with the allow_overflow flag
 - split io_post_aux_cqe up
 - remove msg_ring changes (as not needed)
 - add the patch 10-12 cleanups

[1]: https://github.com/DylanZA/liburing/commit/9ac66b36bcf4477bfafeff1c5f107896b7ae31cf
Run with $ make -j && ./benchmark/reg.b -s 1 -t 2000 -r 10

Dylan Yudaken (9):
  io_uring: io_req_complete_post should defer if available
  io_uring: always lock in io_apoll_task_func
  io_uring: defer all io_req_complete_failed
  io_uring: allow defer completion for aux posted cqes
  io_uring: add io_aux_cqe which allows deferred completion
  io_uring: make io_fill_cqe_aux static
  io_uring: add lockdep assertion in io_fill_cqe_aux
  io_uring: remove overflow param from io_post_aux_cqe
  io_uring: allow multishot polled reqs to defer completion

 include/linux/io_uring_types.h |   2 +
 io_uring/io_uring.c            | 101 ++++++++++++++++++++++++++-------
 io_uring/io_uring.h            |   9 ++-
 io_uring/msg_ring.c            |   4 +-
 io_uring/net.c                 |   7 ++-
 io_uring/poll.c                |   9 +--
 io_uring/rsrc.c                |   4 +-
 7 files changed, 101 insertions(+), 35 deletions(-)


base-commit: 6321e75e0bc52b0feff9643292acc92494c00e8d

Comments

Jens Axboe Nov. 24, 2022, 1:25 p.m. UTC | #1

On Thu, 24 Nov 2022 01:35:50 -0800, Dylan Yudaken wrote:
> Multishot completions currently all go through io_post_aux_cqe which will
> do a lock/unlock pair of the completion spinlock, and also possibly signal
> an eventfd if registered. This can slow down applications that use these
> features.
> 
> This series allows the posted completions to be batched using the same
> IO_URING_F_COMPLETE_DEFER as exists for non multishot completions. A
> critical property of this is that all multishot completions must be
> flushed to the CQ ring before the non-multishot completion (say an error)
> or else ordering will break. This implies that if some completions were
> deferred, then the rest must also be to keep that ordering. In order to do
> this the first few patches move all the completion code into a simpler
> path that defers completions when possible.
> 
> [...]

Applied, thanks!

[1/9] io_uring: io_req_complete_post should defer if available
      commit: 8fa737e0de7d3c4dc3d7cb9a9d9a6362d872c3f3
[2/9] io_uring: always lock in io_apoll_task_func
      commit: ca23d244ec99cc1a7a1c91f6a25bb074ea00bff1
[3/9] io_uring: defer all io_req_complete_failed
      commit: d62208d1e2d73d9949c7c58518fbd915eacad102
[4/9] io_uring: allow defer completion for aux posted cqes
      commit: 8e003ae8505a7bb728cb158198ca88912818da70
[5/9] io_uring: add io_aux_cqe which allows deferred completion
      commit: 9101a50761cf126399014cbfa518804c75c64157
[6/9] io_uring: make io_fill_cqe_aux static
      commit: 84eca39b41bc03372d647db83319ec0f6c230cda
[7/9] io_uring: add lockdep assertion in io_fill_cqe_aux
      commit: 3827133217eb3cdcb3824b22a31b76bf106a2ae3
[8/9] io_uring: remove overflow param from io_post_aux_cqe
      commit: 48ca51e666ed91f75a548a460faa60d08cfd3af6
[9/9] io_uring: allow multishot polled reqs to defer completion
      commit: 41d52216bb00504be5cf08e2d5ec8a41d16d9a67

Best regards,