[RFC,0/2] block: use eBPF to redirect IO completion

Message

Hou Tao Oct. 14, 2019, 12:28 p.m. UTC

For network stack, RPS, namely Receive Packet Steering, is used to
distribute network protocol processing from hardware-interrupted CPU
to specific CPUs and alleviating soft-irq load of the interrupted CPU.

For block layer, soft-irq (for single queue device) or hard-irq
(for multiple queue device) is used to handle IO completion, so
RPS will be useful when the soft-irq load or the hard-irq load
of a specific CPU is too high, or a specific CPU set is required
to handle IO completion.

Instead of setting the CPU set used for handling IO completion
through sysfs or procfs, we can attach an eBPF program to the
request-queue, provide some useful info (e.g., the CPU
which submits the request) to the program, and let the program
decides the proper CPU for IO completion handling.

In order to demonostrate the effect of IO completion redirection,
a test programm is built to redirect the IO completion handling
to all online CPUs or a specific CPU set:

	./test_blkdev_ccpu -d /dev/vda
or
	./test_blkdev_ccpu -d /dev/nvme0n1 -s 4,8,10-13

However I am still trying to find out a killer scenario for
the eBPF redirection, so suggestions and comments are welcome.

Regards,
Tao

Hou Tao (2):
  block: add support for redirecting IO completion through eBPF
  selftests/bpf: add test program for redirecting IO completion CPU

 block/Makefile                                |   2 +-
 block/blk-bpf.c                               | 127 +++++++++
 block/blk-mq.c                                |  22 +-
 block/blk-softirq.c                           |  27 +-
 include/linux/blkdev.h                        |   3 +
 include/linux/bpf_blkdev.h                    |   9 +
 include/linux/bpf_types.h                     |   1 +
 include/uapi/linux/bpf.h                      |   2 +
 kernel/bpf/syscall.c                          |   9 +
 tools/include/uapi/linux/bpf.h                |   2 +
 tools/lib/bpf/libbpf.c                        |   1 +
 tools/lib/bpf/libbpf_probes.c                 |   1 +
 tools/testing/selftests/bpf/Makefile          |   1 +
 .../selftests/bpf/progs/blkdev_ccpu_rr.c      |  66 +++++
 .../testing/selftests/bpf/test_blkdev_ccpu.c  | 246 ++++++++++++++++++
 15 files changed, 507 insertions(+), 12 deletions(-)
 create mode 100644 block/blk-bpf.c
 create mode 100644 include/linux/bpf_blkdev.h
 create mode 100644 tools/testing/selftests/bpf/progs/blkdev_ccpu_rr.c
 create mode 100644 tools/testing/selftests/bpf/test_blkdev_ccpu.c

Comments

Bob Liu Oct. 15, 2019, 1:20 a.m. UTC | #1

On 10/14/19 8:28 PM, Hou Tao wrote:
> For network stack, RPS, namely Receive Packet Steering, is used to
> distribute network protocol processing from hardware-interrupted CPU
> to specific CPUs and alleviating soft-irq load of the interrupted CPU.
> 
> For block layer, soft-irq (for single queue device) or hard-irq
> (for multiple queue device) is used to handle IO completion, so
> RPS will be useful when the soft-irq load or the hard-irq load
> of a specific CPU is too high, or a specific CPU set is required
> to handle IO completion.
> 
> Instead of setting the CPU set used for handling IO completion
> through sysfs or procfs, we can attach an eBPF program to the
> request-queue, provide some useful info (e.g., the CPU
> which submits the request) to the program, and let the program
> decides the proper CPU for IO completion handling.
> 

But it looks like there isn't any benefit than through sysfs/procfs?

> In order to demonostrate the effect of IO completion redirection,
> a test programm is built to redirect the IO completion handling
> to all online CPUs or a specific CPU set:
> 
> 	./test_blkdev_ccpu -d /dev/vda
> or
> 	./test_blkdev_ccpu -d /dev/nvme0n1 -s 4,8,10-13
> 
> However I am still trying to find out a killer scenario for

Speaking about scenario, perhaps attaching a filter could be useful? 
So that the data can be processed the first place.

-
Bob

> the eBPF redirection, so suggestions and comments are welcome.
> 
> Regards,
> Tao
> 
> Hou Tao (2):
>   block: add support for redirecting IO completion through eBPF
>   selftests/bpf: add test program for redirecting IO completion CPU
> 
>  block/Makefile                                |   2 +-
>  block/blk-bpf.c                               | 127 +++++++++
>  block/blk-mq.c                                |  22 +-
>  block/blk-softirq.c                           |  27 +-
>  include/linux/blkdev.h                        |   3 +
>  include/linux/bpf_blkdev.h                    |   9 +
>  include/linux/bpf_types.h                     |   1 +
>  include/uapi/linux/bpf.h                      |   2 +
>  kernel/bpf/syscall.c                          |   9 +
>  tools/include/uapi/linux/bpf.h                |   2 +
>  tools/lib/bpf/libbpf.c                        |   1 +
>  tools/lib/bpf/libbpf_probes.c                 |   1 +
>  tools/testing/selftests/bpf/Makefile          |   1 +
>  .../selftests/bpf/progs/blkdev_ccpu_rr.c      |  66 +++++
>  .../testing/selftests/bpf/test_blkdev_ccpu.c  | 246 ++++++++++++++++++
>  15 files changed, 507 insertions(+), 12 deletions(-)
>  create mode 100644 block/blk-bpf.c
>  create mode 100644 include/linux/bpf_blkdev.h
>  create mode 100644 tools/testing/selftests/bpf/progs/blkdev_ccpu_rr.c
>  create mode 100644 tools/testing/selftests/bpf/test_blkdev_ccpu.c
>