diff mbox series

[bpf-next,v2,2/2,no_merge] selftests/bpf: Benchmark runtime performance with private stack

Message ID 20240718205203.3652080-1-yonghong.song@linux.dev (mailing list archive)
State Changes Requested
Delegated to: BPF
Headers show
Series [bpf-next,v2,1/2] bpf: Support private stack for bpf progs | expand

Checks

Context Check Description
netdev/series_format success Single patches do not need cover letters
netdev/tree_selection success Clearly marked for bpf-next, async
netdev/ynl success Generated files up to date; no warnings/errors; no diff in generated;
netdev/fixes_present success Fixes tag not required for -next series
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 898 this patch: 898
netdev/build_tools success Errors and warnings before: 1 this patch: 1
netdev/cc_maintainers warning 19 maintainers not CCed: kpsingh@kernel.org dave.hansen@linux.intel.com shuah@kernel.org haoluo@google.com dsahern@kernel.org john.fastabend@gmail.com bp@alien8.de hpa@zytor.com mykolal@fb.com song@kernel.org mingo@redhat.com x86@kernel.org netdev@vger.kernel.org jolsa@kernel.org linux-kselftest@vger.kernel.org martin.lau@linux.dev eddyz87@gmail.com tglx@linutronix.de sdf@fomichev.me
netdev/build_clang success Errors and warnings before: 971 this patch: 971
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/deprecated_api success None detected
netdev/check_selftest success net selftest script(s) already in Makefile
netdev/verify_fixes success No Fixes tag
netdev/build_allmodconfig_warn success Errors and warnings before: 7633 this patch: 7633
netdev/checkpatch fail ERROR: code indent should use tabs where possible ERROR: open brace '{' following function definitions go on the next line ERROR: space required before the open parenthesis '(' WARNING: Possible repeated word: 'is' WARNING: Prefer __weak over __attribute__((weak)) WARNING: Use of volatile is usually wrong: see Documentation/process/volatile-considered-harmful.rst WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? WARNING: externs should be avoided in .c files WARNING: line length of 102 exceeds 80 columns WARNING: line length of 120 exceeds 80 columns WARNING: line length of 81 exceeds 80 columns WARNING: line length of 84 exceeds 80 columns WARNING: line length of 87 exceeds 80 columns WARNING: line length of 88 exceeds 80 columns WARNING: line length of 89 exceeds 80 columns WARNING: line length of 94 exceeds 80 columns WARNING: please, no spaces at the start of a line
netdev/build_clang_rust success No Rust files in patch. Skipping build
netdev/kdoc success Errors and warnings before: 6 this patch: 6
netdev/source_inline success Was 0 now: 0
bpf/vmtest-bpf-next-VM_Test-0 success Logs for Lint
bpf/vmtest-bpf-next-VM_Test-1 success Logs for ShellCheck
bpf/vmtest-bpf-next-VM_Test-2 success Logs for Unittests
bpf/vmtest-bpf-next-VM_Test-3 success Logs for Validate matrix.py
bpf/vmtest-bpf-next-VM_Test-5 success Logs for aarch64-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-4 success Logs for aarch64-gcc / build / build for aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-10 success Logs for aarch64-gcc / veristat
bpf/vmtest-bpf-next-VM_Test-12 success Logs for s390x-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-6 success Logs for aarch64-gcc / test (test_maps, false, 360) / test_maps on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-9 success Logs for aarch64-gcc / test (test_verifier, false, 360) / test_verifier on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-11 success Logs for s390x-gcc / build / build for s390x with gcc
bpf/vmtest-bpf-next-VM_Test-13 fail Logs for s390x-gcc / test (test_maps, false, 360) / test_maps on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-16 success Logs for s390x-gcc / test (test_verifier, false, 360) / test_verifier on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-17 success Logs for s390x-gcc / veristat
bpf/vmtest-bpf-next-VM_Test-18 success Logs for set-matrix
bpf/vmtest-bpf-next-VM_Test-19 success Logs for x86_64-gcc / build / build for x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-20 success Logs for x86_64-gcc / build-release
bpf/vmtest-bpf-next-VM_Test-28 success Logs for x86_64-llvm-17 / build / build for x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-29 success Logs for x86_64-llvm-17 / build-release / build for x86_64 with llvm-17-O2
bpf/vmtest-bpf-next-VM_Test-34 success Logs for x86_64-llvm-17 / veristat
bpf/vmtest-bpf-next-VM_Test-35 success Logs for x86_64-llvm-18 / build / build for x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-36 success Logs for x86_64-llvm-18 / build-release / build for x86_64 with llvm-18-O2
bpf/vmtest-bpf-next-VM_Test-42 success Logs for x86_64-llvm-18 / veristat
bpf/vmtest-bpf-next-VM_Test-8 success Logs for aarch64-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-7 success Logs for aarch64-gcc / test (test_progs, false, 360) / test_progs on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-14 success Logs for s390x-gcc / test (test_progs, false, 360) / test_progs on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-21 success Logs for x86_64-gcc / test (test_maps, false, 360) / test_maps on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-22 success Logs for x86_64-gcc / test (test_progs, false, 360) / test_progs on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-23 success Logs for x86_64-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-24 success Logs for x86_64-gcc / test (test_progs_no_alu32_parallel, true, 30) / test_progs_no_alu32_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-25 success Logs for x86_64-gcc / test (test_progs_parallel, true, 30) / test_progs_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-26 success Logs for x86_64-gcc / test (test_verifier, false, 360) / test_verifier on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-27 success Logs for x86_64-gcc / veristat / veristat on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-30 success Logs for x86_64-llvm-17 / test (test_maps, false, 360) / test_maps on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-33 success Logs for x86_64-llvm-17 / test (test_verifier, false, 360) / test_verifier on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-38 success Logs for x86_64-llvm-18 / test (test_progs, false, 360) / test_progs on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-37 success Logs for x86_64-llvm-18 / test (test_maps, false, 360) / test_maps on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-40 success Logs for x86_64-llvm-18 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-39 success Logs for x86_64-llvm-18 / test (test_progs_cpuv4, false, 360) / test_progs_cpuv4 on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-41 success Logs for x86_64-llvm-18 / test (test_verifier, false, 360) / test_verifier on x86_64 with llvm-18
bpf/vmtest-bpf-next-VM_Test-32 success Logs for x86_64-llvm-17 / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on x86_64 with llvm-17
bpf/vmtest-bpf-next-VM_Test-31 success Logs for x86_64-llvm-17 / test (test_progs, false, 360) / test_progs on x86_64 with llvm-17
bpf/vmtest-bpf-next-PR fail PR summary
bpf/vmtest-bpf-next-VM_Test-15 success Logs for s390x-gcc / test (test_progs_no_alu32, false, 360) / test_progs_no_alu32 on s390x with gcc

Commit Message

Yonghong Song July 18, 2024, 8:52 p.m. UTC 
This patch intends to show some benchmark results comparing a bpf
program with vs. without private stack. The patch is not intended
to land since it hacks existing kernel interface in order to
do proper comparison. The bpf program is similar to
7df4e597ea2c ("selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks")
where a raw_tp program is triggered with bpf_prog_test_run_opts() and
the raw_tp program has a loop of helper bpf_get_numa_node_id() which
will enable a fentry prog to run. The fentry prog calls three
do-nothing functions to maximumly expose the cost of private stack.

The following is the jited code for bpf prog in progs/private_stack.c
without private stack. The number of batch iterations is 4096.

subprog:
0:  f3 0f 1e fa             endbr64
4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
9:  66 90                   xchg   ax,ax
b:  55                      push   rbp
c:  48 89 e5                mov    rbp,rsp
f:  f3 0f 1e fa             endbr64
13: 31 c0                   xor    eax,eax
15: c9                      leave
16: c3                      ret

main prog:
0:  f3 0f 1e fa             endbr64
4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
9:  66 90                   xchg   ax,ax
b:  55                      push   rbp
c:  48 89 e5                mov    rbp,rsp
f:  f3 0f 1e fa             endbr64
13: 48 bf 00 e0 57 00 00    movabs rdi,0xffffc9000057e000
1a: c9 ff ff
1d: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
21: 48 83 c6 01             add    rsi,0x1
25: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
29: e8 6e 00 00 00          call   0x9c
2e: e8 69 00 00 00          call   0x9c
33: e8 64 00 00 00          call   0x9c
38: 31 c0                   xor    eax,eax
3a: c9                      leave
3b: c3                      ret

The following are the jited progs with private stack:

subprog:
0:  f3 0f 1e fa             endbr64
4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
9:  66 90                   xchg   ax,ax
b:  55                      push   rbp
c:  48 89 e5                mov    rbp,rsp
f:  f3 0f 1e fa             endbr64
13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
1a: 60 00 00
1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
24: 02 00
26: 31 c0                   xor    eax,eax
28: c9                      leave
29: c3                      ret

main prog:
0:  f3 0f 1e fa             endbr64
4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
9:  66 90                   xchg   ax,ax
b:  55                      push   rbp
c:  48 89 e5                mov    rbp,rsp
f:  f3 0f 1e fa             endbr64
13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
1a: 60 00 00
1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
24: 02 00
26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
2d: c9 ff ff
30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
34: 48 83 c6 01             add    rsi,0x1
38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
3c: 41 51                   push   r9
3e: e8 46 23 51 e1          call   0xffffffffe1512389
43: 41 59                   pop    r9
45: 41 51                   push   r9
47: e8 3d 23 51 e1          call   0xffffffffe1512389
4c: 41 59                   pop    r9
4e: 41 51                   push   r9
50: e8 34 23 51 e1          call   0xffffffffe1512389
55: 41 59                   pop    r9
57: 31 c0                   xor    eax,eax
59: c9                      leave
5a: c3                      ret

From the above, it is clear for subprog and main prog,
we have some r9 related overhead including retriving the stack
in the jit prelog code:
  movabs r9,0x607e08c1a688
  add    r9,QWORD PTR gs:0x21a00
and 'push r9' and 'pop r9' around subprog calls.

I did some benchmarking on an intel box (Intel(R) Xeon(R) D-2191A CPU @ 1.60GHz)
which has 20 cores and 80 cpus. The number of hits are in the unit
of loop iterations.

The following are two benchmark results and a few other tries show
similar results in terms of variation.
  $ ./benchs/run_bench_private_stack.sh
  no-private-stack-1:  2.152 ± 0.004M/s (drops 0.000 ± 0.000M/s)
  private-stack-1:     2.226 ± 0.003M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-8:  89.086 ± 0.674M/s (drops 0.000 ± 0.000M/s)
  private-stack-8:     90.023 ± 0.117M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-64:  1545.383 ± 3.574M/s (drops 0.000 ± 0.000M/s)
  private-stack-64:    1534.630 ± 2.063M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-512:  14591.591 ± 15.202M/s (drops 0.000 ± 0.000M/s)
  private-stack-512:   14323.796 ± 13.165M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-2048:  58680.977 ± 46.116M/s (drops 0.000 ± 0.000M/s)
  private-stack-2048:  58614.699 ± 22.031M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-4096:  119974.497 ± 90.985M/s (drops 0.000 ± 0.000M/s)
  private-stack-4096:  114841.949 ± 59.514M/s (drops 0.000 ± 0.000M/s)
  $ ./benchs/run_bench_private_stack.sh
  no-private-stack-1:  2.246 ± 0.002M/s (drops 0.000 ± 0.000M/s)
  private-stack-1:     2.232 ± 0.005M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-8:  91.446 ± 0.055M/s (drops 0.000 ± 0.000M/s)
  private-stack-8:     90.120 ± 0.069M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-64:  1578.374 ± 1.508M/s (drops 0.000 ± 0.000M/s)
  private-stack-64:    1514.909 ± 3.898M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-512:  14767.811 ± 22.399M/s (drops 0.000 ± 0.000M/s)
  private-stack-512:   14232.382 ± 227.217M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-2048:  58342.372 ± 81.519M/s (drops 0.000 ± 0.000M/s)
  private-stack-2048:  54503.335 ± 160.199M/s (drops 0.000 ± 0.000M/s)
  no-private-stack-4096:  117262.975 ± 179.802M/s (drops 0.000 ± 0.000M/s)
  private-stack-4096:  114643.523 ± 146.956M/s (drops 0.000 ± 0.000M/s)

It is is clear that private-stack is worse than non-private stack up to close 5 percents.
This can be roughly estimated based on the above jit code with no-private-stack vs. private-stack.

Although the benchmark shows up to 5% potential slowdown with private stack.
In reality, the kernel enables private stack only after stack size 64 which means
the bpf prog will do some useful things. If bpf prog uses any helper/kfunc, the
push/pop r9 overhead should be minimum compared to the overhead of helper/kfunc.
if the prog does not use a lot of helper/kfunc, there is no push/pop r9 and
the performance should be reasonable too.

With 4096 loop ierations per program run, I got
  $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack
  18.47%  bench                                              [k]
  17.29%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
  13.33%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
  11.86%  bench    [kernel.vmlinux]                          [k] migrate_enable
  11.60%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
  11.42%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
   7.87%  bench    [kernel.vmlinux]                          [k] migrate_disable
   3.71%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
   3.67%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
   0.04%  bench    bench                                     [.] btf_validate_type

  $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
    18.94%  bench                                              [k]
    16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
    15.77%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
    11.70%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
    11.48%  bench    [kernel.vmlinux]                          [k] migrate_enable
    11.30%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
     5.85%  bench    [kernel.vmlinux]                          [k] migrate_disable
     3.69%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
     3.56%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
     0.06%  bench    bench                                     [.] bpf_prog_test_run_opts

NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.

I suspect top 18.47%/18.94% perf run probably due to fentry prog bench_trigger_fentry_batch,
considering even subprog func1 takes 13.33%/16.88% time.
Overall bpf prog include trampoline takes more than 50% of the time.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 arch/x86/net/bpf_jit_comp.c                   |   5 +-
 include/linux/bpf.h                           |   3 +-
 include/uapi/linux/bpf.h                      |   3 +
 kernel/bpf/core.c                             |   3 +-
 kernel/bpf/syscall.c                          |   4 +-
 kernel/bpf/verifier.c                         |   1 +
 tools/include/uapi/linux/bpf.h                |   3 +
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   6 +
 .../bpf/benchs/bench_private_stack.c          | 149 ++++++++++++++++++
 .../bpf/benchs/run_bench_private_stack.sh     |  11 ++
 .../selftests/bpf/progs/private_stack.c       |  37 +++++
 12 files changed, 222 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_private_stack.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh
 create mode 100644 tools/testing/selftests/bpf/progs/private_stack.c

Comments

Yonghong Song July 18, 2024, 9:44 p.m. UTC | #1 
On 7/18/24 1:52 PM, Yonghong Song wrote:
> This patch intends to show some benchmark results comparing a bpf
> program with vs. without private stack. The patch is not intended
> to land since it hacks existing kernel interface in order to
> do proper comparison. The bpf program is similar to
> 7df4e597ea2c ("selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks")
> where a raw_tp program is triggered with bpf_prog_test_run_opts() and
> the raw_tp program has a loop of helper bpf_get_numa_node_id() which
> will enable a fentry prog to run. The fentry prog calls three
> do-nothing functions to maximumly expose the cost of private stack.
>
> The following is the jited code for bpf prog in progs/private_stack.c
> without private stack. The number of batch iterations is 4096.
>
> subprog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 31 c0                   xor    eax,eax
> 15: c9                      leave
> 16: c3                      ret
>
> main prog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 48 bf 00 e0 57 00 00    movabs rdi,0xffffc9000057e000
> 1a: c9 ff ff
> 1d: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
> 21: 48 83 c6 01             add    rsi,0x1
> 25: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
> 29: e8 6e 00 00 00          call   0x9c
> 2e: e8 69 00 00 00          call   0x9c
> 33: e8 64 00 00 00          call   0x9c
> 38: 31 c0                   xor    eax,eax
> 3a: c9                      leave
> 3b: c3                      ret
>
> The following are the jited progs with private stack:
>
> subprog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
> 1a: 60 00 00
> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> 24: 02 00
> 26: 31 c0                   xor    eax,eax
> 28: c9                      leave
> 29: c3                      ret
>
> main prog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
> 1a: 60 00 00
> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> 24: 02 00
> 26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
> 2d: c9 ff ff
> 30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
> 34: 48 83 c6 01             add    rsi,0x1
> 38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
> 3c: 41 51                   push   r9
> 3e: e8 46 23 51 e1          call   0xffffffffe1512389
> 43: 41 59                   pop    r9
> 45: 41 51                   push   r9
> 47: e8 3d 23 51 e1          call   0xffffffffe1512389
> 4c: 41 59                   pop    r9
> 4e: 41 51                   push   r9
> 50: e8 34 23 51 e1          call   0xffffffffe1512389
> 55: 41 59                   pop    r9
> 57: 31 c0                   xor    eax,eax
> 59: c9                      leave
> 5a: c3                      ret
>
>  From the above, it is clear for subprog and main prog,
> we have some r9 related overhead including retriving the stack
> in the jit prelog code:
>    movabs r9,0x607e08c1a688
>    add    r9,QWORD PTR gs:0x21a00
> and 'push r9' and 'pop r9' around subprog calls.
>
> I did some benchmarking on an intel box (Intel(R) Xeon(R) D-2191A CPU @ 1.60GHz)
> which has 20 cores and 80 cpus. The number of hits are in the unit
> of loop iterations.
>
> The following are two benchmark results and a few other tries show
> similar results in terms of variation.
>    $ ./benchs/run_bench_private_stack.sh
>    no-private-stack-1:  2.152 ± 0.004M/s (drops 0.000 ± 0.000M/s)
>    private-stack-1:     2.226 ± 0.003M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-8:  89.086 ± 0.674M/s (drops 0.000 ± 0.000M/s)
>    private-stack-8:     90.023 ± 0.117M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-64:  1545.383 ± 3.574M/s (drops 0.000 ± 0.000M/s)
>    private-stack-64:    1534.630 ± 2.063M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-512:  14591.591 ± 15.202M/s (drops 0.000 ± 0.000M/s)
>    private-stack-512:   14323.796 ± 13.165M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-2048:  58680.977 ± 46.116M/s (drops 0.000 ± 0.000M/s)
>    private-stack-2048:  58614.699 ± 22.031M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-4096:  119974.497 ± 90.985M/s (drops 0.000 ± 0.000M/s)
>    private-stack-4096:  114841.949 ± 59.514M/s (drops 0.000 ± 0.000M/s)
>    $ ./benchs/run_bench_private_stack.sh
>    no-private-stack-1:  2.246 ± 0.002M/s (drops 0.000 ± 0.000M/s)
>    private-stack-1:     2.232 ± 0.005M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-8:  91.446 ± 0.055M/s (drops 0.000 ± 0.000M/s)
>    private-stack-8:     90.120 ± 0.069M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-64:  1578.374 ± 1.508M/s (drops 0.000 ± 0.000M/s)
>    private-stack-64:    1514.909 ± 3.898M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-512:  14767.811 ± 22.399M/s (drops 0.000 ± 0.000M/s)
>    private-stack-512:   14232.382 ± 227.217M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-2048:  58342.372 ± 81.519M/s (drops 0.000 ± 0.000M/s)
>    private-stack-2048:  54503.335 ± 160.199M/s (drops 0.000 ± 0.000M/s)
>    no-private-stack-4096:  117262.975 ± 179.802M/s (drops 0.000 ± 0.000M/s)
>    private-stack-4096:  114643.523 ± 146.956M/s (drops 0.000 ± 0.000M/s)
>
> It is is clear that private-stack is worse than non-private stack up to close 5 percents.
> This can be roughly estimated based on the above jit code with no-private-stack vs. private-stack.
>
> Although the benchmark shows up to 5% potential slowdown with private stack.
> In reality, the kernel enables private stack only after stack size 64 which means
> the bpf prog will do some useful things. If bpf prog uses any helper/kfunc, the
> push/pop r9 overhead should be minimum compared to the overhead of helper/kfunc.
> if the prog does not use a lot of helper/kfunc, there is no push/pop r9 and
> the performance should be reasonable too.
>
> With 4096 loop ierations per program run, I got
>    $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack
>    18.47%  bench                                              [k]
>    17.29%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
>    13.33%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
>    11.86%  bench    [kernel.vmlinux]                          [k] migrate_enable
>    11.60%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
>    11.42%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
>     7.87%  bench    [kernel.vmlinux]                          [k] migrate_disable
>     3.71%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
>     3.67%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
>     0.04%  bench    bench                                     [.] btf_validate_type
>
>    $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
>      18.94%  bench                                              [k]
>      16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
>      15.77%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
>      11.70%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
>      11.48%  bench    [kernel.vmlinux]                          [k] migrate_enable
>      11.30%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
>       5.85%  bench    [kernel.vmlinux]                          [k] migrate_disable
>       3.69%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
>       3.56%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
>       0.06%  bench    bench                                     [.] bpf_prog_test_run_opts
>
> NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.

I tried with perf built with latest bpf-next and with no-private-stack, the issue still
exists. Will debug more.

>
> I suspect top 18.47%/18.94% perf run probably due to fentry prog bench_trigger_fentry_batch,
> considering even subprog func1 takes 13.33%/16.88% time.
> Overall bpf prog include trampoline takes more than 50% of the time.
>
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---
>   arch/x86/net/bpf_jit_comp.c                   |   5 +-
>   include/linux/bpf.h                           |   3 +-
>   include/uapi/linux/bpf.h                      |   3 +
>   kernel/bpf/core.c                             |   3 +-
>   kernel/bpf/syscall.c                          |   4 +-
>   kernel/bpf/verifier.c                         |   1 +
>   tools/include/uapi/linux/bpf.h                |   3 +
>   tools/testing/selftests/bpf/Makefile          |   2 +
>   tools/testing/selftests/bpf/bench.c           |   6 +
>   .../bpf/benchs/bench_private_stack.c          | 149 ++++++++++++++++++
>   .../bpf/benchs/run_bench_private_stack.sh     |  11 ++
>   .../selftests/bpf/progs/private_stack.c       |  37 +++++
>   12 files changed, 222 insertions(+), 5 deletions(-)
>   create mode 100644 tools/testing/selftests/bpf/benchs/bench_private_stack.c
>   create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh
>   create mode 100644 tools/testing/selftests/bpf/progs/private_stack.c
[...]
Kumar Kartikeya Dwivedi July 18, 2024, 9:59 p.m. UTC | #2 
On Thu, 18 Jul 2024 at 23:44, Yonghong Song <yonghong.song@linux.dev> wrote:
>
>
> On 7/18/24 1:52 PM, Yonghong Song wrote:
> > This patch intends to show some benchmark results comparing a bpf
> > program with vs. without private stack. The patch is not intended
> > to land since it hacks existing kernel interface in order to
> > do proper comparison. The bpf program is similar to
> > 7df4e597ea2c ("selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks")
> > where a raw_tp program is triggered with bpf_prog_test_run_opts() and
> > the raw_tp program has a loop of helper bpf_get_numa_node_id() which
> > will enable a fentry prog to run. The fentry prog calls three
> > do-nothing functions to maximumly expose the cost of private stack.
> >
> > The following is the jited code for bpf prog in progs/private_stack.c
> > without private stack. The number of batch iterations is 4096.
> >
> > subprog:
> > 0:  f3 0f 1e fa             endbr64
> > 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> > 9:  66 90                   xchg   ax,ax
> > b:  55                      push   rbp
> > c:  48 89 e5                mov    rbp,rsp
> > f:  f3 0f 1e fa             endbr64
> > 13: 31 c0                   xor    eax,eax
> > 15: c9                      leave
> > 16: c3                      ret
> >
> > main prog:
> > 0:  f3 0f 1e fa             endbr64
> > 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> > 9:  66 90                   xchg   ax,ax
> > b:  55                      push   rbp
> > c:  48 89 e5                mov    rbp,rsp
> > f:  f3 0f 1e fa             endbr64
> > 13: 48 bf 00 e0 57 00 00    movabs rdi,0xffffc9000057e000
> > 1a: c9 ff ff
> > 1d: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
> > 21: 48 83 c6 01             add    rsi,0x1
> > 25: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
> > 29: e8 6e 00 00 00          call   0x9c
> > 2e: e8 69 00 00 00          call   0x9c
> > 33: e8 64 00 00 00          call   0x9c
> > 38: 31 c0                   xor    eax,eax
> > 3a: c9                      leave
> > 3b: c3                      ret
> >
> > The following are the jited progs with private stack:
> >
> > subprog:
> > 0:  f3 0f 1e fa             endbr64
> > 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> > 9:  66 90                   xchg   ax,ax
> > b:  55                      push   rbp
> > c:  48 89 e5                mov    rbp,rsp
> > f:  f3 0f 1e fa             endbr64
> > 13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
> > 1a: 60 00 00
> > 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> > 24: 02 00
> > 26: 31 c0                   xor    eax,eax
> > 28: c9                      leave
> > 29: c3                      ret
> >
> > main prog:
> > 0:  f3 0f 1e fa             endbr64
> > 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> > 9:  66 90                   xchg   ax,ax
> > b:  55                      push   rbp
> > c:  48 89 e5                mov    rbp,rsp
> > f:  f3 0f 1e fa             endbr64
> > 13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
> > 1a: 60 00 00
> > 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> > 24: 02 00
> > 26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
> > 2d: c9 ff ff
> > 30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
> > 34: 48 83 c6 01             add    rsi,0x1
> > 38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
> > 3c: 41 51                   push   r9
> > 3e: e8 46 23 51 e1          call   0xffffffffe1512389
> > 43: 41 59                   pop    r9
> > 45: 41 51                   push   r9
> > 47: e8 3d 23 51 e1          call   0xffffffffe1512389
> > 4c: 41 59                   pop    r9
> > 4e: 41 51                   push   r9
> > 50: e8 34 23 51 e1          call   0xffffffffe1512389
> > 55: 41 59                   pop    r9
> > 57: 31 c0                   xor    eax,eax
> > 59: c9                      leave
> > 5a: c3                      ret
> >
> >  From the above, it is clear for subprog and main prog,
> > we have some r9 related overhead including retriving the stack
> > in the jit prelog code:
> >    movabs r9,0x607e08c1a688
> >    add    r9,QWORD PTR gs:0x21a00
> > and 'push r9' and 'pop r9' around subprog calls.
> >
> > I did some benchmarking on an intel box (Intel(R) Xeon(R) D-2191A CPU @ 1.60GHz)
> > which has 20 cores and 80 cpus. The number of hits are in the unit
> > of loop iterations.
> >
> > The following are two benchmark results and a few other tries show
> > similar results in terms of variation.
> >    $ ./benchs/run_bench_private_stack.sh
> >    no-private-stack-1:  2.152 ± 0.004M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-1:     2.226 ± 0.003M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-8:  89.086 ± 0.674M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-8:     90.023 ± 0.117M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-64:  1545.383 ± 3.574M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-64:    1534.630 ± 2.063M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-512:  14591.591 ± 15.202M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-512:   14323.796 ± 13.165M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-2048:  58680.977 ± 46.116M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-2048:  58614.699 ± 22.031M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-4096:  119974.497 ± 90.985M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-4096:  114841.949 ± 59.514M/s (drops 0.000 ± 0.000M/s)
> >    $ ./benchs/run_bench_private_stack.sh
> >    no-private-stack-1:  2.246 ± 0.002M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-1:     2.232 ± 0.005M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-8:  91.446 ± 0.055M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-8:     90.120 ± 0.069M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-64:  1578.374 ± 1.508M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-64:    1514.909 ± 3.898M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-512:  14767.811 ± 22.399M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-512:   14232.382 ± 227.217M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-2048:  58342.372 ± 81.519M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-2048:  54503.335 ± 160.199M/s (drops 0.000 ± 0.000M/s)
> >    no-private-stack-4096:  117262.975 ± 179.802M/s (drops 0.000 ± 0.000M/s)
> >    private-stack-4096:  114643.523 ± 146.956M/s (drops 0.000 ± 0.000M/s)
> >
> > It is is clear that private-stack is worse than non-private stack up to close 5 percents.
> > This can be roughly estimated based on the above jit code with no-private-stack vs. private-stack.
> >
> > Although the benchmark shows up to 5% potential slowdown with private stack.
> > In reality, the kernel enables private stack only after stack size 64 which means
> > the bpf prog will do some useful things. If bpf prog uses any helper/kfunc, the
> > push/pop r9 overhead should be minimum compared to the overhead of helper/kfunc.
> > if the prog does not use a lot of helper/kfunc, there is no push/pop r9 and
> > the performance should be reasonable too.
> >
> > With 4096 loop ierations per program run, I got
> >    $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack
> >    18.47%  bench                                              [k]
> >    17.29%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
> >    13.33%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
> >    11.86%  bench    [kernel.vmlinux]                          [k] migrate_enable
> >    11.60%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
> >    11.42%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
> >     7.87%  bench    [kernel.vmlinux]                          [k] migrate_disable
> >     3.71%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
> >     3.67%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
> >     0.04%  bench    bench                                     [.] btf_validate_type
> >
> >    $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
> >      18.94%  bench                                              [k]
> >      16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
> >      15.77%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
> >      11.70%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
> >      11.48%  bench    [kernel.vmlinux]                          [k] migrate_enable
> >      11.30%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
> >       5.85%  bench    [kernel.vmlinux]                          [k] migrate_disable
> >       3.69%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
> >       3.56%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
> >       0.06%  bench    bench                                     [.] bpf_prog_test_run_opts
> >
> > NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.
>
> I tried with perf built with latest bpf-next and with no-private-stack, the issue still
> exists. Will debug more.
>

Just as an aside, but if this doesn't work, I think you can have a
better signal-to-noise ratio if you try enabling the private stack for
XDP programs and just set up two machines, with a client sending
traffic to another and run xdp-bench [0] on the server. I think you
should observe measurable differences in throughput for
nanosecond-scale changes, especially in programs like drop which do
very little.

[0]: https://github.com/xdp-project/xdp-tools
Alexei Starovoitov July 19, 2024, 12:36 a.m. UTC | #3 
On Thu, Jul 18, 2024 at 2:44 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>
>
> >    $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
> >      18.94%  bench                                              [k]
> >      16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
> >      15.77%  bench    bpf_trampoline_6442522961                 [k]

...

> > NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.
>
> I tried with perf built with latest bpf-next and with no-private-stack, the issue still
> exists. Will debug more.

Try this fix:
https://lore.kernel.org/all/20240714065533.1112616-1-houtao@huaweicloud.com/

btw you were cc-ed on it. your @ fb goes to spam ? ;)
Yonghong Song July 19, 2024, 2:21 a.m. UTC | #4 
On 7/18/24 5:36 PM, Alexei Starovoitov wrote:
> On Thu, Jul 18, 2024 at 2:44 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>>
>>>     $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
>>>       18.94%  bench                                              [k]
>>>       16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
>>>       15.77%  bench    bpf_trampoline_6442522961                 [k]
> ...
>
>>> NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.
>> I tried with perf built with latest bpf-next and with no-private-stack, the issue still
>> exists. Will debug more.
> Try this fix:
> https://lore.kernel.org/all/20240714065533.1112616-1-houtao@huaweicloud.com/

It does fix the problem. The 'perf report' for private-stack flavor:

   18.94%  bench    bpf_prog_71f1b7d5309b304a_bench_trigger_fentry_batch  [k] bpf_prog_71f1b7d5309b304a_bench_trigger_fentry_batch
   16.50%  bench    bpf_prog_bcf7977d3b93787c_func1                       [k] bpf_prog_bcf7977d3b93787c_func1
   15.75%  bench    bpf_trampoline_6442522961                             [k] bpf_trampoline_6442522961
   11.72%  bench    [kernel.vmlinux]                                      [k] migrate_enable
   11.63%  bench    [kernel.vmlinux]                                      [k] __bpf_prog_enter_recur
   11.37%  bench    [kernel.vmlinux]                                      [k] __bpf_prog_exit_recur
    6.17%  bench    [kernel.vmlinux]                                      [k] migrate_disable
    3.59%  bench    bpf_prog_d9703036495d54b0_trigger_driver              [k] bpf_prog_d9703036495d54b0_trigger_driver
    3.51%  bench    [kernel.vmlinux]                                      [k] bpf_get_numa_node_id
    0.05%  bench    bench                                                 [.] bpf_prog_test_run_opts

>
> btw you were cc-ed on it. your @ fb goes to spam ? ;)

It is in my inbox. Sadly I skipped it since it is for perf system and I focused on
several of my patches in the last few days.
Yonghong Song July 19, 2024, 3:01 a.m. UTC | #5 
On 7/18/24 2:59 PM, Kumar Kartikeya Dwivedi wrote:
> On Thu, 18 Jul 2024 at 23:44, Yonghong Song <yonghong.song@linux.dev> wrote:
>>
>> On 7/18/24 1:52 PM, Yonghong Song wrote:
>>> This patch intends to show some benchmark results comparing a bpf
>>> program with vs. without private stack. The patch is not intended
>>> to land since it hacks existing kernel interface in order to
>>> do proper comparison. The bpf program is similar to
>>> 7df4e597ea2c ("selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks")
>>> where a raw_tp program is triggered with bpf_prog_test_run_opts() and
>>> the raw_tp program has a loop of helper bpf_get_numa_node_id() which
>>> will enable a fentry prog to run. The fentry prog calls three
>>> do-nothing functions to maximumly expose the cost of private stack.
>>>
>>> The following is the jited code for bpf prog in progs/private_stack.c
>>> without private stack. The number of batch iterations is 4096.
>>>
>>> subprog:
>>> 0:  f3 0f 1e fa             endbr64
>>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>>> 9:  66 90                   xchg   ax,ax
>>> b:  55                      push   rbp
>>> c:  48 89 e5                mov    rbp,rsp
>>> f:  f3 0f 1e fa             endbr64
>>> 13: 31 c0                   xor    eax,eax
>>> 15: c9                      leave
>>> 16: c3                      ret
>>>
>>> main prog:
>>> 0:  f3 0f 1e fa             endbr64
>>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>>> 9:  66 90                   xchg   ax,ax
>>> b:  55                      push   rbp
>>> c:  48 89 e5                mov    rbp,rsp
>>> f:  f3 0f 1e fa             endbr64
>>> 13: 48 bf 00 e0 57 00 00    movabs rdi,0xffffc9000057e000
>>> 1a: c9 ff ff
>>> 1d: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
>>> 21: 48 83 c6 01             add    rsi,0x1
>>> 25: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
>>> 29: e8 6e 00 00 00          call   0x9c
>>> 2e: e8 69 00 00 00          call   0x9c
>>> 33: e8 64 00 00 00          call   0x9c
>>> 38: 31 c0                   xor    eax,eax
>>> 3a: c9                      leave
>>> 3b: c3                      ret
>>>
>>> The following are the jited progs with private stack:
>>>
>>> subprog:
>>> 0:  f3 0f 1e fa             endbr64
>>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>>> 9:  66 90                   xchg   ax,ax
>>> b:  55                      push   rbp
>>> c:  48 89 e5                mov    rbp,rsp
>>> f:  f3 0f 1e fa             endbr64
>>> 13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
>>> 1a: 60 00 00
>>> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
>>> 24: 02 00
>>> 26: 31 c0                   xor    eax,eax
>>> 28: c9                      leave
>>> 29: c3                      ret
>>>
>>> main prog:
>>> 0:  f3 0f 1e fa             endbr64
>>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>>> 9:  66 90                   xchg   ax,ax
>>> b:  55                      push   rbp
>>> c:  48 89 e5                mov    rbp,rsp
>>> f:  f3 0f 1e fa             endbr64
>>> 13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
>>> 1a: 60 00 00
>>> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
>>> 24: 02 00
>>> 26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
>>> 2d: c9 ff ff
>>> 30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
>>> 34: 48 83 c6 01             add    rsi,0x1
>>> 38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
>>> 3c: 41 51                   push   r9
>>> 3e: e8 46 23 51 e1          call   0xffffffffe1512389
>>> 43: 41 59                   pop    r9
>>> 45: 41 51                   push   r9
>>> 47: e8 3d 23 51 e1          call   0xffffffffe1512389
>>> 4c: 41 59                   pop    r9
>>> 4e: 41 51                   push   r9
>>> 50: e8 34 23 51 e1          call   0xffffffffe1512389
>>> 55: 41 59                   pop    r9
>>> 57: 31 c0                   xor    eax,eax
>>> 59: c9                      leave
>>> 5a: c3                      ret
>>>
>>>   From the above, it is clear for subprog and main prog,
>>> we have some r9 related overhead including retriving the stack
>>> in the jit prelog code:
>>>     movabs r9,0x607e08c1a688
>>>     add    r9,QWORD PTR gs:0x21a00
>>> and 'push r9' and 'pop r9' around subprog calls.
>>>
>>> I did some benchmarking on an intel box (Intel(R) Xeon(R) D-2191A CPU @ 1.60GHz)
>>> which has 20 cores and 80 cpus. The number of hits are in the unit
>>> of loop iterations.
>>>
>>> The following are two benchmark results and a few other tries show
>>> similar results in terms of variation.
>>>     $ ./benchs/run_bench_private_stack.sh
>>>     no-private-stack-1:  2.152 ± 0.004M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-1:     2.226 ± 0.003M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-8:  89.086 ± 0.674M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-8:     90.023 ± 0.117M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-64:  1545.383 ± 3.574M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-64:    1534.630 ± 2.063M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-512:  14591.591 ± 15.202M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-512:   14323.796 ± 13.165M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-2048:  58680.977 ± 46.116M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-2048:  58614.699 ± 22.031M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-4096:  119974.497 ± 90.985M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-4096:  114841.949 ± 59.514M/s (drops 0.000 ± 0.000M/s)
>>>     $ ./benchs/run_bench_private_stack.sh
>>>     no-private-stack-1:  2.246 ± 0.002M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-1:     2.232 ± 0.005M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-8:  91.446 ± 0.055M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-8:     90.120 ± 0.069M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-64:  1578.374 ± 1.508M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-64:    1514.909 ± 3.898M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-512:  14767.811 ± 22.399M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-512:   14232.382 ± 227.217M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-2048:  58342.372 ± 81.519M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-2048:  54503.335 ± 160.199M/s (drops 0.000 ± 0.000M/s)
>>>     no-private-stack-4096:  117262.975 ± 179.802M/s (drops 0.000 ± 0.000M/s)
>>>     private-stack-4096:  114643.523 ± 146.956M/s (drops 0.000 ± 0.000M/s)
>>>
>>> It is is clear that private-stack is worse than non-private stack up to close 5 percents.
>>> This can be roughly estimated based on the above jit code with no-private-stack vs. private-stack.
>>>
>>> Although the benchmark shows up to 5% potential slowdown with private stack.
>>> In reality, the kernel enables private stack only after stack size 64 which means
>>> the bpf prog will do some useful things. If bpf prog uses any helper/kfunc, the
>>> push/pop r9 overhead should be minimum compared to the overhead of helper/kfunc.
>>> if the prog does not use a lot of helper/kfunc, there is no push/pop r9 and
>>> the performance should be reasonable too.
>>>
>>> With 4096 loop ierations per program run, I got
>>>     $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack
>>>     18.47%  bench                                              [k]
>>>     17.29%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
>>>     13.33%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
>>>     11.86%  bench    [kernel.vmlinux]                          [k] migrate_enable
>>>     11.60%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
>>>     11.42%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
>>>      7.87%  bench    [kernel.vmlinux]                          [k] migrate_disable
>>>      3.71%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
>>>      3.67%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
>>>      0.04%  bench    bench                                     [.] btf_validate_type
>>>
>>>     $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack
>>>       18.94%  bench                                              [k]
>>>       16.88%  bench    bpf_prog_bcf7977d3b93787c_func1           [k] bpf_prog_bcf7977d3b93787c_func1
>>>       15.77%  bench    bpf_trampoline_6442522961                 [k] bpf_trampoline_6442522961
>>>       11.70%  bench    [kernel.vmlinux]                          [k] __bpf_prog_enter_recur
>>>       11.48%  bench    [kernel.vmlinux]                          [k] migrate_enable
>>>       11.30%  bench    [kernel.vmlinux]                          [k] __bpf_prog_exit_recur
>>>        5.85%  bench    [kernel.vmlinux]                          [k] migrate_disable
>>>        3.69%  bench    bpf_prog_d9703036495d54b0_trigger_driver  [k] bpf_prog_d9703036495d54b0_trigger_driver
>>>        3.56%  bench    [kernel.vmlinux]                          [k] bpf_get_numa_node_id
>>>        0.06%  bench    bench                                     [.] bpf_prog_test_run_opts
>>>
>>> NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further.
>> I tried with perf built with latest bpf-next and with no-private-stack, the issue still
>> exists. Will debug more.
>>
> Just as an aside, but if this doesn't work, I think you can have a
> better signal-to-noise ratio if you try enabling the private stack for
> XDP programs and just set up two machines, with a client sending
> traffic to another and run xdp-bench [0] on the server. I think you
> should observe measurable differences in throughput for
> nanosecond-scale changes, especially in programs like drop which do
> very little.
>
> [0]: https://github.com/xdp-project/xdp-tools

Currently private stack cannot be used for xdp prog since xdp prog is 
really performance critical and we won't want even slightest performance 
loss at this point. So private stack focused on tracing related programs 
as they are easily to have nested bpf progs if there are quite some 
tracing progs at the same time. If we ineed apply private stack to xdp 
programs, I expect some performance loss and how much loss is due to bpf 
prog itself (bpf prog codes and helpers/kfuncs, see below). The example 
in this patch shows overall performance degradation around 5%. But 
considering there around 50% non-bpf programs so overall bpf prog 
degradation might be around 10% comparing to non-private-stack bpf 
programs. But this is an extreme example. In reality, the stack size 
needs to be >= 64 bytes so the bpf programs must do some meaningful work 
which means bpf prog itself will do more work and it may also call some 
helpers/kfuncs. The helpers/kfuncs will introduce push/pop operations. 
But if helper/kfunc do some meaningful work, then the relative 
performance hit with additional push/pop should be small.
bot+bpf-ci@kernel.org July 20, 2024, 12:14 a.m. UTC | #6 
Dear patch submitter,

CI has tested the following submission:
Status:     FAILURE
Name:       [bpf-next,v2,2/2,no_merge] selftests/bpf: Benchmark runtime performance with private stack
Patchwork:  https://patchwork.kernel.org/project/netdevbpf/list/?series=872349&state=*
Matrix:     https://github.com/kernel-patches/bpf/actions/runs/10015580432

Failed jobs:
test_maps-s390x-gcc: https://github.com/kernel-patches/bpf/actions/runs/10015580432/job/27687366542


Please note: this email is coming from an unmonitored mailbox. If you have
questions or feedback, please reach out to the Meta Kernel CI team at
kernel-ci@meta.com.
Alexei Starovoitov July 20, 2024, 1:08 a.m. UTC | #7 
On Thu, Jul 18, 2024 at 1:52 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>
>
> The following are the jited progs with private stack:
>
> subprog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
> 1a: 60 00 00
> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> 24: 02 00
> 26: 31 c0                   xor    eax,eax
> 28: c9                      leave
> 29: c3                      ret

Thanks for doing the benchmarking.
It's clear now that worst case overhead is ~5%.
Could you do one more benchmark such that the 'main prog'
below stays as-is with setup of r9 and push/pop r9,
but in the subprog above there is no 'movabs r9 + add r9' ?
To simulate the case when a big function with a large stack
triggers private-stack use, but it calls a subprog without
a private stack.
I think we should see a different overhead.
Obviously subprog won't have these two extra insns that setup r9
which would lead to something like ~4% slowdown vs 5%,
but I feel the overhead of pure push/pop r9 around calls
will be lower as well, because r9 is not written into inside subprog.
The CPU HW should be able to execute such push/pop faster.
I'm curious what it is.

> main prog:
> 0:  f3 0f 1e fa             endbr64
> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
> 9:  66 90                   xchg   ax,ax
> b:  55                      push   rbp
> c:  48 89 e5                mov    rbp,rsp
> f:  f3 0f 1e fa             endbr64
> 13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
> 1a: 60 00 00
> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
> 24: 02 00
> 26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
> 2d: c9 ff ff
> 30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
> 34: 48 83 c6 01             add    rsi,0x1
> 38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
> 3c: 41 51                   push   r9
> 3e: e8 46 23 51 e1          call   0xffffffffe1512389
> 43: 41 59                   pop    r9
> 45: 41 51                   push   r9
> 47: e8 3d 23 51 e1          call   0xffffffffe1512389
> 4c: 41 59                   pop    r9
> 4e: 41 51                   push   r9
> 50: e8 34 23 51 e1          call   0xffffffffe1512389
> 55: 41 59                   pop    r9
> 57: 31 c0                   xor    eax,eax
> 59: c9                      leave
> 5a: c3                      ret
>

Also pls share 'perf annotate' of JIT-ed asm.
I wonder where the hotspots are in the code.
Yonghong Song July 22, 2024, 4:33 p.m. UTC | #8 
On 7/19/24 6:08 PM, Alexei Starovoitov wrote:
> On Thu, Jul 18, 2024 at 1:52 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>>
>> The following are the jited progs with private stack:
>>
>> subprog:
>> 0:  f3 0f 1e fa             endbr64
>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>> 9:  66 90                   xchg   ax,ax
>> b:  55                      push   rbp
>> c:  48 89 e5                mov    rbp,rsp
>> f:  f3 0f 1e fa             endbr64
>> 13: 49 b9 70 a6 c1 08 7e    movabs r9,0x607e08c1a670
>> 1a: 60 00 00
>> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
>> 24: 02 00
>> 26: 31 c0                   xor    eax,eax
>> 28: c9                      leave
>> 29: c3                      ret
> Thanks for doing the benchmarking.
> It's clear now that worst case overhead is ~5%.
> Could you do one more benchmark such that the 'main prog'
> below stays as-is with setup of r9 and push/pop r9,
> but in the subprog above there is no 'movabs r9 + add r9' ?
> To simulate the case when a big function with a large stack
> triggers private-stack use, but it calls a subprog without
> a private stack.
> I think we should see a different overhead.
> Obviously subprog won't have these two extra insns that setup r9
> which would lead to something like ~4% slowdown vs 5%,
> but I feel the overhead of pure push/pop r9 around calls
> will be lower as well, because r9 is not written into inside subprog.
> The CPU HW should be able to execute such push/pop faster.
> I'm curious what it is.
Sure. Let me do an experiment with this.
>
>> main prog:
>> 0:  f3 0f 1e fa             endbr64
>> 4:  0f 1f 44 00 00          nop    DWORD PTR [rax+rax*1+0x0]
>> 9:  66 90                   xchg   ax,ax
>> b:  55                      push   rbp
>> c:  48 89 e5                mov    rbp,rsp
>> f:  f3 0f 1e fa             endbr64
>> 13: 49 b9 88 a6 c1 08 7e    movabs r9,0x607e08c1a688
>> 1a: 60 00 00
>> 1d: 65 4c 03 0c 25 00 1a    add    r9,QWORD PTR gs:0x21a00
>> 24: 02 00
>> 26: 48 bf 00 d0 5b 00 00    movabs rdi,0xffffc900005bd000
>> 2d: c9 ff ff
>> 30: 48 8b 77 00             mov    rsi,QWORD PTR [rdi+0x0]
>> 34: 48 83 c6 01             add    rsi,0x1
>> 38: 48 89 77 00             mov    QWORD PTR [rdi+0x0],rsi
>> 3c: 41 51                   push   r9
>> 3e: e8 46 23 51 e1          call   0xffffffffe1512389
>> 43: 41 59                   pop    r9
>> 45: 41 51                   push   r9
>> 47: e8 3d 23 51 e1          call   0xffffffffe1512389
>> 4c: 41 59                   pop    r9
>> 4e: 41 51                   push   r9
>> 50: e8 34 23 51 e1          call   0xffffffffe1512389
>> 55: 41 59                   pop    r9
>> 57: 31 c0                   xor    eax,eax
>> 59: c9                      leave
>> 5a: c3                      ret
>>
> Also pls share 'perf annotate' of JIT-ed asm.
> I wonder where the hotspots are in the code.
Okay, will do.
diff mbox series

Patch

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 60f5d86fb6aa..3f12f7a957ba 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3327,7 +3327,10 @@  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 skip_init_addrs:
 
 	if (bpf_enable_private_stack(prog) && !prog->private_stack_ptr) {
-		private_stack_ptr = __alloc_percpu_gfp(prog->aux->stack_depth, 8, GFP_KERNEL);
+		if (prog->aux->stack_depth == 0)
+			private_stack_ptr = __alloc_percpu_gfp(8, 8, GFP_KERNEL);
+		else
+			private_stack_ptr = __alloc_percpu_gfp(prog->aux->stack_depth, 8, GFP_KERNEL);
 		if (!private_stack_ptr) {
 			prog = orig_prog;
 			goto out_addrs;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 19a3f5355363..2f8708465c19 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1551,7 +1551,8 @@  struct bpf_prog {
 				call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
 				call_get_func_ip:1, /* Do we call get_func_ip() */
 				tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */
-				sleepable:1;	/* BPF program is sleepable */
+				sleepable:1,	/* BPF program is sleepable */
+				disable_private_stack:1; /* Disable private stack */
 	enum bpf_prog_type	type;		/* Type of BPF program */
 	enum bpf_attach_type	expected_attach_type; /* For some prog types */
 	u32			len;		/* Number of filter blocks */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 35bcf52dbc65..98af8ea8a4d6 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1409,6 +1409,9 @@  enum {
 
 /* Do not translate kernel bpf_arena pointers to user pointers */
 	BPF_F_NO_USER_CONV	= (1U << 18),
+
+/* Disable private stack */
+	BPF_F_DISABLE_PRIVATE_STACK	= (1U << 19),
 };
 
 /* Flags for BPF_PROG_QUERY. */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index f69eb0c5fe03..b5d33cf87695 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2815,14 +2815,13 @@  EXPORT_SYMBOL_GPL(bpf_prog_free);
 
 bool bpf_enable_private_stack(struct bpf_prog *prog)
 {
-	if (prog->aux->stack_depth <= 64)
+	if (prog->disable_private_stack)
 		return false;
 
 	switch (prog->aux->prog->type) {
 	case BPF_PROG_TYPE_KPROBE:
 	case BPF_PROG_TYPE_TRACEPOINT:
 	case BPF_PROG_TYPE_PERF_EVENT:
-	case BPF_PROG_TYPE_RAW_TRACEPOINT:
 		return true;
 	case BPF_PROG_TYPE_TRACING:
 		if (prog->expected_attach_type != BPF_TRACE_ITER)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 89162ddb4747..bb2b632c9c2c 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2715,7 +2715,8 @@  static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
 				 BPF_F_XDP_HAS_FRAGS |
 				 BPF_F_XDP_DEV_BOUND_ONLY |
 				 BPF_F_TEST_REG_INVARIANTS |
-				 BPF_F_TOKEN_FD))
+				 BPF_F_TOKEN_FD |
+				 BPF_F_DISABLE_PRIVATE_STACK))
 		return -EINVAL;
 
 	bpf_prog_load_fixup_attach_type(attr);
@@ -2828,6 +2829,7 @@  static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
 
 	prog->expected_attach_type = attr->expected_attach_type;
 	prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
+	prog->disable_private_stack = !!(attr->prog_flags & BPF_F_DISABLE_PRIVATE_STACK);
 	prog->aux->attach_btf = attach_btf;
 	prog->aux->attach_btf_id = attr->attach_btf_id;
 	prog->aux->dst_prog = dst_prog;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8da132a1ef28..397b2b9eed24 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19442,6 +19442,7 @@  static int jit_subprogs(struct bpf_verifier_env *env)
 			goto out_free;
 		func[i]->is_func = 1;
 		func[i]->sleepable = prog->sleepable;
+		func[i]->disable_private_stack = prog->disable_private_stack;
 		func[i]->aux->func_idx = i;
 		/* Below members will be freed only at prog->aux */
 		func[i]->aux->btf = prog->aux->btf;
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 35bcf52dbc65..98af8ea8a4d6 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1409,6 +1409,9 @@  enum {
 
 /* Do not translate kernel bpf_arena pointers to user pointers */
 	BPF_F_NO_USER_CONV	= (1U << 18),
+
+/* Disable private stack */
+	BPF_F_DISABLE_PRIVATE_STACK	= (1U << 19),
 };
 
 /* Flags for BPF_PROG_QUERY. */
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index dd49c1d23a60..44a6a43da71c 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -733,6 +733,7 @@  $(OUTPUT)/bench_local_storage_create.o: $(OUTPUT)/bench_local_storage_create.ske
 $(OUTPUT)/bench_bpf_hashmap_lookup.o: $(OUTPUT)/bpf_hashmap_lookup.skel.h
 $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h
 $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h
+$(OUTPUT)/bench_private_stack.o: $(OUTPUT)/private_stack.skel.h
 $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
 $(OUTPUT)/bench: LDLIBS += -lm
 $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -753,6 +754,7 @@  $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_local_storage_create.o \
 		 $(OUTPUT)/bench_htab_mem.o \
 		 $(OUTPUT)/bench_bpf_crypto.o \
+		 $(OUTPUT)/bench_private_stack.o \
 		 #
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c
index 627b74ae041b..4f4867cd80f9 100644
--- a/tools/testing/selftests/bpf/bench.c
+++ b/tools/testing/selftests/bpf/bench.c
@@ -282,6 +282,7 @@  extern struct argp bench_local_storage_create_argp;
 extern struct argp bench_htab_mem_argp;
 extern struct argp bench_trigger_batch_argp;
 extern struct argp bench_crypto_argp;
+extern struct argp bench_private_stack_argp;
 
 static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
@@ -296,6 +297,7 @@  static const struct argp_child bench_parsers[] = {
 	{ &bench_htab_mem_argp, 0, "hash map memory benchmark", 0 },
 	{ &bench_trigger_batch_argp, 0, "BPF triggering benchmark", 0 },
 	{ &bench_crypto_argp, 0, "bpf crypto benchmark", 0 },
+	{ &bench_private_stack_argp, 0, "bpf private stack benchmark", 0 },
 	{},
 };
 
@@ -542,6 +544,8 @@  extern const struct bench bench_local_storage_create;
 extern const struct bench bench_htab_mem;
 extern const struct bench bench_crypto_encrypt;
 extern const struct bench bench_crypto_decrypt;
+extern const struct bench bench_no_private_stack;
+extern const struct bench bench_private_stack;
 
 static const struct bench *benchs[] = {
 	&bench_count_global,
@@ -596,6 +600,8 @@  static const struct bench *benchs[] = {
 	&bench_htab_mem,
 	&bench_crypto_encrypt,
 	&bench_crypto_decrypt,
+	&bench_no_private_stack,
+	&bench_private_stack,
 };
 
 static void find_benchmark(void)
diff --git a/tools/testing/selftests/bpf/benchs/bench_private_stack.c b/tools/testing/selftests/bpf/benchs/bench_private_stack.c
new file mode 100644
index 000000000000..5ae1e9bfe706
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/bench_private_stack.c
@@ -0,0 +1,149 @@ 
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+
+#include <argp.h>
+#include "bench.h"
+#include "private_stack.skel.h"
+
+static struct ctx {
+	struct private_stack *skel;
+} ctx;
+
+static struct {
+	long nr_batch_iters;
+} args = {
+	.nr_batch_iters = 0,
+};
+
+enum {
+	ARG_NR_BATCH_ITERS = 3000,
+};
+
+static const struct argp_option opts[] = {
+        { "nr-batch-iters", ARG_NR_BATCH_ITERS, "NR_BATCH_ITERS",
+		0, "nr batch iters" },
+        {},
+};
+
+static error_t private_stack_parse_arg(int key, char *arg, struct argp_state *state)
+{
+	long ret;
+
+        switch (key) {
+        case ARG_NR_BATCH_ITERS:
+                ret = strtoul(arg, NULL, 10);
+		if (ret < 1)
+			argp_usage(state);
+		args.nr_batch_iters = ret;
+                break;
+        default:
+                return ARGP_ERR_UNKNOWN;
+        }
+
+        return 0;
+}
+
+const struct argp bench_private_stack_argp = {
+        .options = opts,
+        .parser = private_stack_parse_arg,
+};
+
+static void private_stack_validate(void)
+{
+	if (env.consumer_cnt != 0) {
+		fprintf(stderr,
+			"The private stack benchmarks do not support consumer\n");
+		exit(1);
+	}
+}
+
+static void common_setup(bool disable_private_stack)
+{
+	struct private_stack *skel;
+	struct bpf_link *link;
+	__u32 old_flags;
+	int err;
+
+	skel = private_stack__open();
+	if(!skel) {
+		fprintf(stderr, "failed to open skeleton\n");
+		exit(1);
+	}
+	ctx.skel = skel;
+
+	if (disable_private_stack) {
+		old_flags = bpf_program__flags(skel->progs.bench_trigger_fentry_batch);
+		bpf_program__set_flags(skel->progs.bench_trigger_fentry_batch, old_flags | BPF_F_DISABLE_PRIVATE_STACK);
+	}
+
+	skel->rodata->batch_iters = args.nr_batch_iters;
+
+	err = private_stack__load(skel);
+	if (err) {
+		fprintf(stderr, "failed to load program\n");
+		exit(1);
+	}
+
+	link = bpf_program__attach(skel->progs.bench_trigger_fentry_batch);
+	if (!link) {
+		fprintf(stderr, "failed to attach program bench_trigger_fentry_batch\n");
+		exit(1);
+	}
+}
+
+static void no_private_stack_setup(void)
+{
+	common_setup(true);
+}
+
+static void private_stack_setup(void)
+{
+	common_setup(false);
+}
+
+static void private_stack_measure(struct bench_res *res)
+{
+	struct private_stack *skel = ctx.skel;
+	unsigned long total_hits = 0;
+	static unsigned long last_hits;
+
+	total_hits = skel->bss->hits * skel->rodata->batch_iters;
+	res->hits = total_hits - last_hits;
+	res->drops = 0;
+	res->false_hits = 0;
+	last_hits = total_hits;
+}
+
+static void *private_stack_producer(void *unused)
+{
+	struct private_stack *skel = ctx.skel;
+	int fd;
+
+	fd  = bpf_program__fd(skel->progs.trigger_driver);
+	while (true)
+		bpf_prog_test_run_opts(fd, NULL);
+
+	return NULL;
+}
+
+const struct bench bench_no_private_stack = {
+	.name = "no-private-stack",
+	.argp = &bench_private_stack_argp,
+	.validate = private_stack_validate,
+	.setup = no_private_stack_setup,
+	.producer_thread = private_stack_producer,
+	.measure = private_stack_measure,
+	.report_progress = hits_drops_report_progress,
+	.report_final = hits_drops_report_final,
+};
+
+const struct bench bench_private_stack = {
+	.name = "private-stack",
+	.argp = &bench_private_stack_argp,
+	.validate = private_stack_validate,
+	.setup = private_stack_setup,
+	.producer_thread = private_stack_producer,
+	.measure = private_stack_measure,
+	.report_progress = hits_drops_report_progress,
+	.report_final = hits_drops_report_final,
+};
diff --git a/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh b/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh
new file mode 100755
index 000000000000..692a5f9676a7
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh
@@ -0,0 +1,11 @@ 
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for b in 1 8 64 512 2048 4096; do
+    summarize "no-private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} no-private-stack)"
+    summarize "private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} private-stack)"
+done
diff --git a/tools/testing/selftests/bpf/progs/private_stack.c b/tools/testing/selftests/bpf/progs/private_stack.c
new file mode 100644
index 000000000000..81d2efad5890
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/private_stack.c
@@ -0,0 +1,37 @@ 
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
+#include <linux/types.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+unsigned long hits = 0;
+const volatile int batch_iters = 0;
+
+SEC("raw_tp")
+int trigger_driver(void *ctx)
+{
+	int i;
+
+	for (i = 0; i < batch_iters; i++)
+		(void)bpf_get_numa_node_id(); /* attach point for benchmarking */
+
+	return 0;
+}
+
+__attribute__((weak)) int func1(void) {
+	return 0;
+}
+
+SEC("fentry/bpf_get_numa_node_id")
+int bench_trigger_fentry_batch(void *ctx)
+{
+	hits++;
+	(void)func1();
+	(void)func1();
+	(void)func1();
+	return 0;
+}
+