From patchwork Thu Jul 18 20:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonghong Song X-Patchwork-Id: 13736691 X-Patchwork-Delegate: bpf@iogearbox.net Received: from 69-171-232-181.mail-mxout.facebook.com (69-171-232-181.mail-mxout.facebook.com [69.171.232.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D50C78C8E for ; Thu, 18 Jul 2024 20:52:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=69.171.232.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721335938; cv=none; b=RC3nmlq356sB1AGQU75OrfCUiwK4pISrzDhyIhAMSkmbjzupBJu4knpEzCoH78QlklpoAlUJDXM5dJyyq88nRq5eM1trzJIcPC3gXuSWyHtTT069d3J5sDUT+Pdw8HoUAKmE9F3iR2LMaCNadKRe4loMgWqIaGbrh5Ym0PXC57U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721335938; c=relaxed/simple; bh=4RS7rIonO4mA802lCXvHhMb6tjaabzesbWVfWhwv2LY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=iiaAbVuhiFYDFNWsoJ4gEqEMw880yS8FOTCdSyLiHtujkPf96odQSFjYFgmPRyHMmwYlMEG0ZOrW/EMJA2d0YuxsDU3u1ZMGqbCiJlYv0xyWx5n7TUrgK5DeAc4JtExbTq8Tj+xsgYKjjhTZGAf9Zh6GIkC8DrGsBbnSsmlkYN8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev; spf=fail smtp.mailfrom=linux.dev; arc=none smtp.client-ip=69.171.232.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=linux.dev Received: by devbig309.ftw3.facebook.com (Postfix, from userid 128203) id C05796BC75EC; Thu, 18 Jul 2024 13:51:58 -0700 (PDT) From: Yonghong Song To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , kernel-team@fb.com, Martin KaFai Lau Subject: [PATCH bpf-next v2 1/2] bpf: Support private stack for bpf progs Date: Thu, 18 Jul 2024 13:51:58 -0700 Message-ID: <20240718205158.3651529-1-yonghong.song@linux.dev> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net The main motivation for private stack comes from nested scheduler in sched-ext from Tejun. The basic idea is that - each cgroup will its own associated bpf program, - bpf program with parent cgroup will call bpf programs in immediate child cgroups. Let us say we have the following cgroup hierarchy: root_cg (prog0): cg1 (prog1): cg11 (prog11): cg111 (prog111) cg112 (prog112) cg12 (prog12): cg121 (prog121) cg122 (prog122) cg2 (prog2): cg21 (prog21) cg22 (prog22) cg23 (prog23) In the above example, prog0 will call a kfunc which will call prog1 and prog2 to get sched info for cg1 and cg2 and then the information is summarized and sent back to prog0. Similarly, prog11 and prog12 will be invoked in the kfunc and the result will be summarized and sent back to prog1, etc. Currently, for each thread, the x86 kernel allocate 8KB stack. The each bpf program (including its subprograms) has maximum 512B stack size to avoid potential stack overflow. And nested bpf programs increase the risk of stack overflow. To avoid potential stack overflow caused by bpf programs, this patch implemented a private stack so bpf program stack space is allocated dynamically when the program is jited. Such private stack is applied to tracing programs like kprobe/uprobe, perf_event, tracepoint, raw tracepoint and tracing. But more than one instance of the same bpf program may run in the system. To make things simple, percpu private stack is allocated for each program, so if the same program is running on different cpus concurrently, we won't have any issue. Note that the kernel already have logic to prevent the recursion for the same bpf program on the same cpu (kprobe, fentry, etc.). The patch implemented a percpu private stack based approach for x86 arch. - The stack size will be 0 and any stack access is from jit-time allocated percpu storage. - In the beginning of jit, r9 is used to save percpu private stack pointer. - Each rbp in the bpf asm insn is replaced by r9. - For each call, push r9 before the call and pop r9 after the call to preserve r9 value. Compared to previous RFC patch [1], this patch added some conditions to enable private stack, e.g., verifier calculated stack size, prog type, etc. The new patch also added a performance test to compare private stack vs. no private stack. The following are some code example to illustrate the idea for selftest cgroup_skb_sk_lookup: the existing code the private-stack approach code endbr64 endbr64 nop DWORD PTR [rax+rax*1+0x0] nop DWORD PTR [rax+rax*1+0x0] xchg ax,ax xchg ax,ax push rbp push rbp mov rbp,rsp mov rbp,rsp endbr64 endbr64 sub rsp,0x68 push rbx push rbx ... ... ... mov r9d,0x8c1c860 ... add r9,QWORD PTR gs:0x21a00 ... ... mov rdx,rbp mov rdx, r9 add rdx,0xffffffffffffffb4 rdx,0xffffffffffffffb4 ... ... mov ecx,0x28 mov ecx,0x28 push r9 call 0xffffffffe305e474 call 0xffffffffe305e524 pop r9 mov rdi,rax mov rdi,rax ... ... movzx rdi,BYTE PTR [rbp-0x46] movzx rdi,BYTE PTR [r9-0x46] ... ... So the number of insns is increased by 1 + num_of_calls * 2. Here the number of calls are those calls in the final jited binary. Comparing function call itself, the push/pop overhead should be minimum in most common cases. Our original use case is for sched-ext nested scheduler. This will be done in the future. [1] https://lore.kernel.org/bpf/707970c5-6bba-450a-be08-adf24d8b9276@linux.dev/T/ Signed-off-by: Yonghong Song --- arch/x86/net/bpf_jit_comp.c | 63 ++++++++++++++++++++++++++++++++++--- include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 20 ++++++++++++ kernel/bpf/syscall.c | 1 + 4 files changed, 82 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d25d81c8ecc0..60f5d86fb6aa 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1309,6 +1309,22 @@ static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op) *pprog = prog; } +static void emit_private_frame_ptr(u8 **pprog, void *private_frame_ptr) +{ + u8 *prog = *pprog; + + /* movabs r9, private_frame_ptr */ + emit_mov_imm64(&prog, X86_REG_R9, (long) private_frame_ptr >> 32, + (u32) (long) private_frame_ptr); + + /* add , gs:[] */ + EMIT2(0x65, 0x4c); + EMIT3(0x03, 0x0c, 0x25); + EMIT((u32)(unsigned long)&this_cpu_off, 4); + + *pprog = prog; +} + #define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp))) /* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ @@ -1324,18 +1340,25 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image int insn_cnt = bpf_prog->len; bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; + u32 stack_depth = bpf_prog->aux->stack_depth; + void __percpu *private_frame_ptr = NULL; u64 arena_vm_start, user_vm_start; int i, excnt = 0; int ilen, proglen = 0; u8 *prog = temp; int err; + if (bpf_prog->private_stack_ptr) { + private_frame_ptr = bpf_prog->private_stack_ptr + round_up(stack_depth, 8); + stack_depth = 0; + } + arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); detect_reg_usage(insn, insn_cnt, callee_regs_used); - emit_prologue(&prog, bpf_prog->aux->stack_depth, + emit_prologue(&prog, stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); /* Exception callback will clobber callee regs for its own use, and @@ -1357,6 +1380,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image emit_mov_imm64(&prog, X86_REG_R12, arena_vm_start >> 32, (u32) arena_vm_start); + if (private_frame_ptr) + emit_private_frame_ptr(&prog, private_frame_ptr); + ilen = prog - temp; if (rw_image) memcpy(rw_image + proglen, temp, ilen); @@ -1376,6 +1402,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image u8 *func; int nops; + if (private_frame_ptr) { + if (src_reg == BPF_REG_FP) + src_reg = X86_REG_R9; + + if (dst_reg == BPF_REG_FP) + dst_reg = X86_REG_R9; + } + switch (insn->code) { /* ALU */ case BPF_ALU | BPF_ADD | BPF_X: @@ -2007,6 +2041,7 @@ st: if (is_imm8(insn->off)) emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0); /* Restore R0 after clobbering RAX */ emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX); + break; } @@ -2031,14 +2066,20 @@ st: if (is_imm8(insn->off)) func = (u8 *) __bpf_call_base + imm32; if (tail_call_reachable) { - RESTORE_TAIL_CALL_CNT(bpf_prog->aux->stack_depth); + RESTORE_TAIL_CALL_CNT(stack_depth); ip += 7; } if (!imm32) return -EINVAL; + if (private_frame_ptr) { + EMIT2(0x41, 0x51); /* push r9 */ + ip += 2; + } ip += x86_call_depth_emit_accounting(&prog, func, ip); if (emit_call(&prog, func, ip)) return -EINVAL; + if (private_frame_ptr) + EMIT2(0x41, 0x59); /* pop r9 */ break; } @@ -2048,13 +2089,13 @@ st: if (is_imm8(insn->off)) &bpf_prog->aux->poke_tab[imm32 - 1], &prog, image + addrs[i - 1], callee_regs_used, - bpf_prog->aux->stack_depth, + stack_depth, ctx); else emit_bpf_tail_call_indirect(bpf_prog, &prog, callee_regs_used, - bpf_prog->aux->stack_depth, + stack_depth, image + addrs[i - 1], ctx); break; @@ -3218,6 +3259,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_binary_header *rw_header = NULL; struct bpf_binary_header *header = NULL; + void __percpu *private_stack_ptr = NULL; struct bpf_prog *tmp, *orig_prog = prog; struct x64_jit_data *jit_data; int proglen, oldproglen = 0; @@ -3284,6 +3326,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.cleanup_addr = proglen; skip_init_addrs: + if (bpf_enable_private_stack(prog) && !prog->private_stack_ptr) { + private_stack_ptr = __alloc_percpu_gfp(prog->aux->stack_depth, 8, GFP_KERNEL); + if (!private_stack_ptr) { + prog = orig_prog; + goto out_addrs; + } + prog->private_stack_ptr = private_stack_ptr; + } + /* * JITed image shrinks with every pass and the loop iterates * until the image stops shrinking. Very large BPF programs @@ -3309,6 +3360,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog->jited = 0; prog->jited_len = 0; } + if (private_stack_ptr) { + free_percpu(private_stack_ptr); + prog->private_stack_ptr = NULL; + } goto out_addrs; } if (image) { diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4f1d4a97b9d1..19a3f5355363 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1563,6 +1563,7 @@ struct bpf_prog { const struct bpf_insn *insn); struct bpf_prog_aux *aux; /* Auxiliary fields */ struct sock_fprog_kern *orig_prog; /* Original BPF program */ + void __percpu *private_stack_ptr; /* Instructions for interpreter */ union { DECLARE_FLEX_ARRAY(struct sock_filter, insns); @@ -1819,6 +1820,7 @@ static inline void bpf_module_put(const void *data, struct module *owner) module_put(owner); } int bpf_struct_ops_link_create(union bpf_attr *attr); +bool bpf_enable_private_stack(struct bpf_prog *prog); #ifdef CONFIG_NET /* Define it here to avoid the use of forward declaration */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 7ee62e38faf0..f69eb0c5fe03 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2813,6 +2813,26 @@ void bpf_prog_free(struct bpf_prog *fp) } EXPORT_SYMBOL_GPL(bpf_prog_free); +bool bpf_enable_private_stack(struct bpf_prog *prog) +{ + if (prog->aux->stack_depth <= 64) + return false; + + switch (prog->aux->prog->type) { + case BPF_PROG_TYPE_KPROBE: + case BPF_PROG_TYPE_TRACEPOINT: + case BPF_PROG_TYPE_PERF_EVENT: + case BPF_PROG_TYPE_RAW_TRACEPOINT: + return true; + case BPF_PROG_TYPE_TRACING: + if (prog->expected_attach_type != BPF_TRACE_ITER) + return true; + fallthrough; + default: + return false; + } +} + /* RNG for unprivileged user space with separated state from prandom_u32(). */ static DEFINE_PER_CPU(struct rnd_state, bpf_user_rnd_state); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 869265852d51..89162ddb4747 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2244,6 +2244,7 @@ static void __bpf_prog_put_rcu(struct rcu_head *rcu) kvfree(aux->func_info); kfree(aux->func_info_aux); + free_percpu(aux->prog->private_stack_ptr); free_uid(aux->user); security_bpf_prog_free(aux->prog); bpf_prog_free(aux->prog); From patchwork Thu Jul 18 20:52:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yonghong Song X-Patchwork-Id: 13736692 X-Patchwork-Delegate: bpf@iogearbox.net Received: from 69-171-232-180.mail-mxout.facebook.com (69-171-232-180.mail-mxout.facebook.com [69.171.232.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D26978C9E for ; Thu, 18 Jul 2024 20:52:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=69.171.232.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721335938; cv=none; b=BlL1AfGuGxBxU16sas/u4iq6I0d6sHUQXgcrbP3973ci1Fy115XAOhW5iNRNxU3G549HLOfZMS4a04KhAQUsGfdzyPrwXJPFb1ERfdoM26gqL1uQwksxz7lXL3aH6x8VKAP4R8s2h37jM/T1AIt+qwNEeFhpYh1g73r/0UgZ0Ac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721335938; c=relaxed/simple; bh=6/SUzLXegkmyP/RMTC5T1I9nRN9VGYap1e1z857DSZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=c7sIFPnRINAYiP6+PDzzm8haGnjGit5+UGy111SRsyo2ZDHig+27eGgJdgNx9067LpqTC6iawm8lSMxuD/WICuxC1s32qLZ8p8CyxYCONz2c9RC6UIN3doOfmfW/FhvuiRbfaanc9Ig6Z/NiHt8nNRNIM4Z8W387G0/KAeaiBEc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev; spf=fail smtp.mailfrom=linux.dev; arc=none smtp.client-ip=69.171.232.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=linux.dev Received: by devbig309.ftw3.facebook.com (Postfix, from userid 128203) id DAF236BC7629; Thu, 18 Jul 2024 13:52:03 -0700 (PDT) From: Yonghong Song To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , kernel-team@fb.com, Martin KaFai Lau Subject: [PATCH bpf-next v2 2/2] [no_merge] selftests/bpf: Benchmark runtime performance with private stack Date: Thu, 18 Jul 2024 13:52:03 -0700 Message-ID: <20240718205203.3652080-1-yonghong.song@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240718205158.3651529-1-yonghong.song@linux.dev> References: <20240718205158.3651529-1-yonghong.song@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net This patch intends to show some benchmark results comparing a bpf program with vs. without private stack. The patch is not intended to land since it hacks existing kernel interface in order to do proper comparison. The bpf program is similar to 7df4e597ea2c ("selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks") where a raw_tp program is triggered with bpf_prog_test_run_opts() and the raw_tp program has a loop of helper bpf_get_numa_node_id() which will enable a fentry prog to run. The fentry prog calls three do-nothing functions to maximumly expose the cost of private stack. The following is the jited code for bpf prog in progs/private_stack.c without private stack. The number of batch iterations is 4096. subprog: 0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 31 c0 xor eax,eax 15: c9 leave 16: c3 ret main prog: 0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 48 bf 00 e0 57 00 00 movabs rdi,0xffffc9000057e000 1a: c9 ff ff 1d: 48 8b 77 00 mov rsi,QWORD PTR [rdi+0x0] 21: 48 83 c6 01 add rsi,0x1 25: 48 89 77 00 mov QWORD PTR [rdi+0x0],rsi 29: e8 6e 00 00 00 call 0x9c 2e: e8 69 00 00 00 call 0x9c 33: e8 64 00 00 00 call 0x9c 38: 31 c0 xor eax,eax 3a: c9 leave 3b: c3 ret The following are the jited progs with private stack: subprog: 0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 49 b9 70 a6 c1 08 7e movabs r9,0x607e08c1a670 1a: 60 00 00 1d: 65 4c 03 0c 25 00 1a add r9,QWORD PTR gs:0x21a00 24: 02 00 26: 31 c0 xor eax,eax 28: c9 leave 29: c3 ret main prog: 0: f3 0f 1e fa endbr64 4: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 9: 66 90 xchg ax,ax b: 55 push rbp c: 48 89 e5 mov rbp,rsp f: f3 0f 1e fa endbr64 13: 49 b9 88 a6 c1 08 7e movabs r9,0x607e08c1a688 1a: 60 00 00 1d: 65 4c 03 0c 25 00 1a add r9,QWORD PTR gs:0x21a00 24: 02 00 26: 48 bf 00 d0 5b 00 00 movabs rdi,0xffffc900005bd000 2d: c9 ff ff 30: 48 8b 77 00 mov rsi,QWORD PTR [rdi+0x0] 34: 48 83 c6 01 add rsi,0x1 38: 48 89 77 00 mov QWORD PTR [rdi+0x0],rsi 3c: 41 51 push r9 3e: e8 46 23 51 e1 call 0xffffffffe1512389 43: 41 59 pop r9 45: 41 51 push r9 47: e8 3d 23 51 e1 call 0xffffffffe1512389 4c: 41 59 pop r9 4e: 41 51 push r9 50: e8 34 23 51 e1 call 0xffffffffe1512389 55: 41 59 pop r9 57: 31 c0 xor eax,eax 59: c9 leave 5a: c3 ret From the above, it is clear for subprog and main prog, we have some r9 related overhead including retriving the stack in the jit prelog code: movabs r9,0x607e08c1a688 add r9,QWORD PTR gs:0x21a00 and 'push r9' and 'pop r9' around subprog calls. I did some benchmarking on an intel box (Intel(R) Xeon(R) D-2191A CPU @ 1.60GHz) which has 20 cores and 80 cpus. The number of hits are in the unit of loop iterations. The following are two benchmark results and a few other tries show similar results in terms of variation. $ ./benchs/run_bench_private_stack.sh no-private-stack-1: 2.152 ± 0.004M/s (drops 0.000 ± 0.000M/s) private-stack-1: 2.226 ± 0.003M/s (drops 0.000 ± 0.000M/s) no-private-stack-8: 89.086 ± 0.674M/s (drops 0.000 ± 0.000M/s) private-stack-8: 90.023 ± 0.117M/s (drops 0.000 ± 0.000M/s) no-private-stack-64: 1545.383 ± 3.574M/s (drops 0.000 ± 0.000M/s) private-stack-64: 1534.630 ± 2.063M/s (drops 0.000 ± 0.000M/s) no-private-stack-512: 14591.591 ± 15.202M/s (drops 0.000 ± 0.000M/s) private-stack-512: 14323.796 ± 13.165M/s (drops 0.000 ± 0.000M/s) no-private-stack-2048: 58680.977 ± 46.116M/s (drops 0.000 ± 0.000M/s) private-stack-2048: 58614.699 ± 22.031M/s (drops 0.000 ± 0.000M/s) no-private-stack-4096: 119974.497 ± 90.985M/s (drops 0.000 ± 0.000M/s) private-stack-4096: 114841.949 ± 59.514M/s (drops 0.000 ± 0.000M/s) $ ./benchs/run_bench_private_stack.sh no-private-stack-1: 2.246 ± 0.002M/s (drops 0.000 ± 0.000M/s) private-stack-1: 2.232 ± 0.005M/s (drops 0.000 ± 0.000M/s) no-private-stack-8: 91.446 ± 0.055M/s (drops 0.000 ± 0.000M/s) private-stack-8: 90.120 ± 0.069M/s (drops 0.000 ± 0.000M/s) no-private-stack-64: 1578.374 ± 1.508M/s (drops 0.000 ± 0.000M/s) private-stack-64: 1514.909 ± 3.898M/s (drops 0.000 ± 0.000M/s) no-private-stack-512: 14767.811 ± 22.399M/s (drops 0.000 ± 0.000M/s) private-stack-512: 14232.382 ± 227.217M/s (drops 0.000 ± 0.000M/s) no-private-stack-2048: 58342.372 ± 81.519M/s (drops 0.000 ± 0.000M/s) private-stack-2048: 54503.335 ± 160.199M/s (drops 0.000 ± 0.000M/s) no-private-stack-4096: 117262.975 ± 179.802M/s (drops 0.000 ± 0.000M/s) private-stack-4096: 114643.523 ± 146.956M/s (drops 0.000 ± 0.000M/s) It is is clear that private-stack is worse than non-private stack up to close 5 percents. This can be roughly estimated based on the above jit code with no-private-stack vs. private-stack. Although the benchmark shows up to 5% potential slowdown with private stack. In reality, the kernel enables private stack only after stack size 64 which means the bpf prog will do some useful things. If bpf prog uses any helper/kfunc, the push/pop r9 overhead should be minimum compared to the overhead of helper/kfunc. if the prog does not use a lot of helper/kfunc, there is no push/pop r9 and the performance should be reasonable too. With 4096 loop ierations per program run, I got $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 no-private-stack 18.47% bench [k] 17.29% bench bpf_trampoline_6442522961 [k] bpf_trampoline_6442522961 13.33% bench bpf_prog_bcf7977d3b93787c_func1 [k] bpf_prog_bcf7977d3b93787c_func1 11.86% bench [kernel.vmlinux] [k] migrate_enable 11.60% bench [kernel.vmlinux] [k] __bpf_prog_enter_recur 11.42% bench [kernel.vmlinux] [k] __bpf_prog_exit_recur 7.87% bench [kernel.vmlinux] [k] migrate_disable 3.71% bench [kernel.vmlinux] [k] bpf_get_numa_node_id 3.67% bench bpf_prog_d9703036495d54b0_trigger_driver [k] bpf_prog_d9703036495d54b0_trigger_driver 0.04% bench bench [.] btf_validate_type $ perf record -- ./bench -w3 -d10 -a --nr-batch-iters=4096 private-stack 18.94% bench [k] 16.88% bench bpf_prog_bcf7977d3b93787c_func1 [k] bpf_prog_bcf7977d3b93787c_func1 15.77% bench bpf_trampoline_6442522961 [k] bpf_trampoline_6442522961 11.70% bench [kernel.vmlinux] [k] __bpf_prog_enter_recur 11.48% bench [kernel.vmlinux] [k] migrate_enable 11.30% bench [kernel.vmlinux] [k] __bpf_prog_exit_recur 5.85% bench [kernel.vmlinux] [k] migrate_disable 3.69% bench bpf_prog_d9703036495d54b0_trigger_driver [k] bpf_prog_d9703036495d54b0_trigger_driver 3.56% bench [kernel.vmlinux] [k] bpf_get_numa_node_id 0.06% bench bench [.] bpf_prog_test_run_opts NOTE: I tried 6.4 perf and 6.10 perf, both of which have issues. I will investigate this further. I suspect top 18.47%/18.94% perf run probably due to fentry prog bench_trigger_fentry_batch, considering even subprog func1 takes 13.33%/16.88% time. Overall bpf prog include trampoline takes more than 50% of the time. Signed-off-by: Yonghong Song --- arch/x86/net/bpf_jit_comp.c | 5 +- include/linux/bpf.h | 3 +- include/uapi/linux/bpf.h | 3 + kernel/bpf/core.c | 3 +- kernel/bpf/syscall.c | 4 +- kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 3 + tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 6 + .../bpf/benchs/bench_private_stack.c | 149 ++++++++++++++++++ .../bpf/benchs/run_bench_private_stack.sh | 11 ++ .../selftests/bpf/progs/private_stack.c | 37 +++++ 12 files changed, 222 insertions(+), 5 deletions(-) create mode 100644 tools/testing/selftests/bpf/benchs/bench_private_stack.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh create mode 100644 tools/testing/selftests/bpf/progs/private_stack.c diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 60f5d86fb6aa..3f12f7a957ba 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3327,7 +3327,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) skip_init_addrs: if (bpf_enable_private_stack(prog) && !prog->private_stack_ptr) { - private_stack_ptr = __alloc_percpu_gfp(prog->aux->stack_depth, 8, GFP_KERNEL); + if (prog->aux->stack_depth == 0) + private_stack_ptr = __alloc_percpu_gfp(8, 8, GFP_KERNEL); + else + private_stack_ptr = __alloc_percpu_gfp(prog->aux->stack_depth, 8, GFP_KERNEL); if (!private_stack_ptr) { prog = orig_prog; goto out_addrs; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 19a3f5355363..2f8708465c19 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1551,7 +1551,8 @@ struct bpf_prog { call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */ call_get_func_ip:1, /* Do we call get_func_ip() */ tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */ - sleepable:1; /* BPF program is sleepable */ + sleepable:1, /* BPF program is sleepable */ + disable_private_stack:1; /* Disable private stack */ enum bpf_prog_type type; /* Type of BPF program */ enum bpf_attach_type expected_attach_type; /* For some prog types */ u32 len; /* Number of filter blocks */ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 35bcf52dbc65..98af8ea8a4d6 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1409,6 +1409,9 @@ enum { /* Do not translate kernel bpf_arena pointers to user pointers */ BPF_F_NO_USER_CONV = (1U << 18), + +/* Disable private stack */ + BPF_F_DISABLE_PRIVATE_STACK = (1U << 19), }; /* Flags for BPF_PROG_QUERY. */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index f69eb0c5fe03..b5d33cf87695 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2815,14 +2815,13 @@ EXPORT_SYMBOL_GPL(bpf_prog_free); bool bpf_enable_private_stack(struct bpf_prog *prog) { - if (prog->aux->stack_depth <= 64) + if (prog->disable_private_stack) return false; switch (prog->aux->prog->type) { case BPF_PROG_TYPE_KPROBE: case BPF_PROG_TYPE_TRACEPOINT: case BPF_PROG_TYPE_PERF_EVENT: - case BPF_PROG_TYPE_RAW_TRACEPOINT: return true; case BPF_PROG_TYPE_TRACING: if (prog->expected_attach_type != BPF_TRACE_ITER) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 89162ddb4747..bb2b632c9c2c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2715,7 +2715,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) BPF_F_XDP_HAS_FRAGS | BPF_F_XDP_DEV_BOUND_ONLY | BPF_F_TEST_REG_INVARIANTS | - BPF_F_TOKEN_FD)) + BPF_F_TOKEN_FD | + BPF_F_DISABLE_PRIVATE_STACK)) return -EINVAL; bpf_prog_load_fixup_attach_type(attr); @@ -2828,6 +2829,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) prog->expected_attach_type = attr->expected_attach_type; prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE); + prog->disable_private_stack = !!(attr->prog_flags & BPF_F_DISABLE_PRIVATE_STACK); prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = attr->attach_btf_id; prog->aux->dst_prog = dst_prog; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8da132a1ef28..397b2b9eed24 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19442,6 +19442,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) goto out_free; func[i]->is_func = 1; func[i]->sleepable = prog->sleepable; + func[i]->disable_private_stack = prog->disable_private_stack; func[i]->aux->func_idx = i; /* Below members will be freed only at prog->aux */ func[i]->aux->btf = prog->aux->btf; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 35bcf52dbc65..98af8ea8a4d6 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1409,6 +1409,9 @@ enum { /* Do not translate kernel bpf_arena pointers to user pointers */ BPF_F_NO_USER_CONV = (1U << 18), + +/* Disable private stack */ + BPF_F_DISABLE_PRIVATE_STACK = (1U << 19), }; /* Flags for BPF_PROG_QUERY. */ diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index dd49c1d23a60..44a6a43da71c 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -733,6 +733,7 @@ $(OUTPUT)/bench_local_storage_create.o: $(OUTPUT)/bench_local_storage_create.ske $(OUTPUT)/bench_bpf_hashmap_lookup.o: $(OUTPUT)/bpf_hashmap_lookup.skel.h $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h +$(OUTPUT)/bench_private_stack.o: $(OUTPUT)/private_stack.skel.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm $(OUTPUT)/bench: $(OUTPUT)/bench.o \ @@ -753,6 +754,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_local_storage_create.o \ $(OUTPUT)/bench_htab_mem.o \ $(OUTPUT)/bench_bpf_crypto.o \ + $(OUTPUT)/bench_private_stack.o \ # $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@ diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 627b74ae041b..4f4867cd80f9 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -282,6 +282,7 @@ extern struct argp bench_local_storage_create_argp; extern struct argp bench_htab_mem_argp; extern struct argp bench_trigger_batch_argp; extern struct argp bench_crypto_argp; +extern struct argp bench_private_stack_argp; static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 }, @@ -296,6 +297,7 @@ static const struct argp_child bench_parsers[] = { { &bench_htab_mem_argp, 0, "hash map memory benchmark", 0 }, { &bench_trigger_batch_argp, 0, "BPF triggering benchmark", 0 }, { &bench_crypto_argp, 0, "bpf crypto benchmark", 0 }, + { &bench_private_stack_argp, 0, "bpf private stack benchmark", 0 }, {}, }; @@ -542,6 +544,8 @@ extern const struct bench bench_local_storage_create; extern const struct bench bench_htab_mem; extern const struct bench bench_crypto_encrypt; extern const struct bench bench_crypto_decrypt; +extern const struct bench bench_no_private_stack; +extern const struct bench bench_private_stack; static const struct bench *benchs[] = { &bench_count_global, @@ -596,6 +600,8 @@ static const struct bench *benchs[] = { &bench_htab_mem, &bench_crypto_encrypt, &bench_crypto_decrypt, + &bench_no_private_stack, + &bench_private_stack, }; static void find_benchmark(void) diff --git a/tools/testing/selftests/bpf/benchs/bench_private_stack.c b/tools/testing/selftests/bpf/benchs/bench_private_stack.c new file mode 100644 index 000000000000..5ae1e9bfe706 --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/bench_private_stack.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include "bench.h" +#include "private_stack.skel.h" + +static struct ctx { + struct private_stack *skel; +} ctx; + +static struct { + long nr_batch_iters; +} args = { + .nr_batch_iters = 0, +}; + +enum { + ARG_NR_BATCH_ITERS = 3000, +}; + +static const struct argp_option opts[] = { + { "nr-batch-iters", ARG_NR_BATCH_ITERS, "NR_BATCH_ITERS", + 0, "nr batch iters" }, + {}, +}; + +static error_t private_stack_parse_arg(int key, char *arg, struct argp_state *state) +{ + long ret; + + switch (key) { + case ARG_NR_BATCH_ITERS: + ret = strtoul(arg, NULL, 10); + if (ret < 1) + argp_usage(state); + args.nr_batch_iters = ret; + break; + default: + return ARGP_ERR_UNKNOWN; + } + + return 0; +} + +const struct argp bench_private_stack_argp = { + .options = opts, + .parser = private_stack_parse_arg, +}; + +static void private_stack_validate(void) +{ + if (env.consumer_cnt != 0) { + fprintf(stderr, + "The private stack benchmarks do not support consumer\n"); + exit(1); + } +} + +static void common_setup(bool disable_private_stack) +{ + struct private_stack *skel; + struct bpf_link *link; + __u32 old_flags; + int err; + + skel = private_stack__open(); + if(!skel) { + fprintf(stderr, "failed to open skeleton\n"); + exit(1); + } + ctx.skel = skel; + + if (disable_private_stack) { + old_flags = bpf_program__flags(skel->progs.bench_trigger_fentry_batch); + bpf_program__set_flags(skel->progs.bench_trigger_fentry_batch, old_flags | BPF_F_DISABLE_PRIVATE_STACK); + } + + skel->rodata->batch_iters = args.nr_batch_iters; + + err = private_stack__load(skel); + if (err) { + fprintf(stderr, "failed to load program\n"); + exit(1); + } + + link = bpf_program__attach(skel->progs.bench_trigger_fentry_batch); + if (!link) { + fprintf(stderr, "failed to attach program bench_trigger_fentry_batch\n"); + exit(1); + } +} + +static void no_private_stack_setup(void) +{ + common_setup(true); +} + +static void private_stack_setup(void) +{ + common_setup(false); +} + +static void private_stack_measure(struct bench_res *res) +{ + struct private_stack *skel = ctx.skel; + unsigned long total_hits = 0; + static unsigned long last_hits; + + total_hits = skel->bss->hits * skel->rodata->batch_iters; + res->hits = total_hits - last_hits; + res->drops = 0; + res->false_hits = 0; + last_hits = total_hits; +} + +static void *private_stack_producer(void *unused) +{ + struct private_stack *skel = ctx.skel; + int fd; + + fd = bpf_program__fd(skel->progs.trigger_driver); + while (true) + bpf_prog_test_run_opts(fd, NULL); + + return NULL; +} + +const struct bench bench_no_private_stack = { + .name = "no-private-stack", + .argp = &bench_private_stack_argp, + .validate = private_stack_validate, + .setup = no_private_stack_setup, + .producer_thread = private_stack_producer, + .measure = private_stack_measure, + .report_progress = hits_drops_report_progress, + .report_final = hits_drops_report_final, +}; + +const struct bench bench_private_stack = { + .name = "private-stack", + .argp = &bench_private_stack_argp, + .validate = private_stack_validate, + .setup = private_stack_setup, + .producer_thread = private_stack_producer, + .measure = private_stack_measure, + .report_progress = hits_drops_report_progress, + .report_final = hits_drops_report_final, +}; diff --git a/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh b/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh new file mode 100755 index 000000000000..692a5f9676a7 --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/run_bench_private_stack.sh @@ -0,0 +1,11 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for b in 1 8 64 512 2048 4096; do + summarize "no-private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} no-private-stack)" + summarize "private-stack-${b}: " "$($RUN_BENCH --nr-batch-iters=${b} private-stack)" +done diff --git a/tools/testing/selftests/bpf/progs/private_stack.c b/tools/testing/selftests/bpf/progs/private_stack.c new file mode 100644 index 000000000000..81d2efad5890 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/private_stack.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +unsigned long hits = 0; +const volatile int batch_iters = 0; + +SEC("raw_tp") +int trigger_driver(void *ctx) +{ + int i; + + for (i = 0; i < batch_iters; i++) + (void)bpf_get_numa_node_id(); /* attach point for benchmarking */ + + return 0; +} + +__attribute__((weak)) int func1(void) { + return 0; +} + +SEC("fentry/bpf_get_numa_node_id") +int bench_trigger_fentry_batch(void *ctx) +{ + hits++; + (void)func1(); + (void)func1(); + (void)func1(); + return 0; +} +