From patchwork Tue Mar 11 15:48:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14012014 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EB8923C9 for ; Tue, 11 Mar 2025 15:49:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.145.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708190; cv=none; b=OiG/I8/1Uzj760OHZ4QI+/RL8ghFZXQerZf8P4Qwifm/wbfay5aB+GSvFZdFTGHdwMj7HT8/Cf5d5PC+nJABfoNzcAz744zpPdQ7HTskIt69R53/jB99XnVSUQEGQuACuep8K8tBL44NssnFyVSbL7V2RTuWrwySHapUk6lQ9CI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708190; c=relaxed/simple; bh=1tvpHMKyHl1gvVobtwl3HHfr0WbylUq5S7wJRA7Lafk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=P8EmeAkpbl2ixTYGboDrkmLPOK/Z48LCm2ALR/CV5jGRvxz/c5Nu22jN1XHNwNnWCfTeaW4bARw1XGMqaieuWFUDar+u5XMx6nvMgg9IEYYYD0g7fXz9Ec8tEoPgxjSg1/fAflLlpY3KrkB6WDu52wzoEx5Y4GCaMADnCUOybg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=jU1gElTS; arc=none smtp.client-ip=67.231.145.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="jU1gElTS" Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52B7wwd1004987; Tue, 11 Mar 2025 08:49:08 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=UXj6W9WxFkrCdxeZQ8TXgPMJn6PW3PJPr3on6ClweZs=; b=jU1gElTSSbQe iE+FO67NOkVRUIyrJNqeOQCEmq5F9rYFvInT9Vt0bxp8HuuEq7Qu1JI2v+T/63SI 09syxE85KxNzc+L96COuLAEaicLwrAaLEhn52j65SgV7qKtqy/e2uDn9+OvR+c9S ei59f9xN2UinTtxtlC+RmM3bNIpb/wnJC0q0xYnITA2di2yFska+jZGPjZ6lglPc 9GMRS5p284uk8uwpq+T0FcjsilkdNWqYbwvpMPHFaDeAfTREA6upZBD2chaxHLyU MMZlCDnOFPOAjDr1O5TG5ey60BNZUHf6ildmHgiiiquy8ePyYdffgUYzrmwkeh+k 7AApXTa3Vw== Received: from mail.thefacebook.com ([163.114.134.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 45ah8rttk0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 11 Mar 2025 08:49:07 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.14; Tue, 11 Mar 2025 15:49:04 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v10 1/4] bpf: add bpf_get_cpu_time_counter kfunc Date: Tue, 11 Mar 2025 08:48:47 -0700 Message-ID: <20250311154850.3616840-2-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250311154850.3616840-1-vadfed@meta.com> References: <20250311154850.3616840-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-GUID: OJM9DFYyCVp38BbgAxnkF8KxLaaJfXHt X-Proofpoint-ORIG-GUID: OJM9DFYyCVp38BbgAxnkF8KxLaaJfXHt X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-11_04,2025-03-11_02,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of bpf_ktime_get_ns() at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to get CLOCK_MONOTONIC_RAW value in ns. JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant. The simplest use-case is added in 4th patch, where we calculate the time spent by bpf_get_ns_current_pid_tgid() kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe. Acked-by: Eduard Zingerman Acked-by: Andrii Nakryiko Acked-by: Yonghong Song Signed-off-by: Vadim Fedorenko --- arch/x86/net/bpf_jit_comp.c | 47 +++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 33 ++++++++++++++++++++++++ include/linux/bpf.h | 3 +++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++++++++ kernel/bpf/helpers.c | 6 +++++ kernel/bpf/verifier.c | 41 +++++++++++++++++++++++++----- 7 files changed, 136 insertions(+), 6 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d3491cc0898b..92cd5945d630 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -2254,6 +2255,40 @@ st: if (is_imm8(insn->off)) case BPF_JMP | BPF_CALL: { u8 *ip = image + addrs[i - 1]; + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + IS_ENABLED(CONFIG_BPF_SYSCALL) && + imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { + /* The default implementation of this kfunc uses + * ktime_get_raw_ns() which effectively is implemented as + * `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip + * masking part because we assume it's not needed in BPF + * use case (two measurements close in time). + * Original code for rdtsc_ordered() uses sequence: + * 'rdtsc; nop; nop; nop' to patch it into + * 'lfence; rdtsc' or 'rdtscp' depending on CPU features. + * JIT uses 'lfence; rdtsc' variant because BPF program + * doesn't care about cookie provided by rdtscp in RCX. + * Save RDX because RDTSC will use EDX:EAX to return u64 + */ + emit_mov_reg(&prog, true, AUX_REG, BPF_REG_3); + if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC)) + EMIT_LFENCE(); + EMIT2(0x0F, 0x31); + + /* shl RDX, 32 */ + maybe_emit_1mod(&prog, BPF_REG_3, true); + EMIT3(0xC1, add_1reg(0xE0, BPF_REG_3), 32); + /* or RAX, RDX */ + maybe_emit_mod(&prog, BPF_REG_0, BPF_REG_3, true); + EMIT2(0x09, add_2reg(0xC0, BPF_REG_0, BPF_REG_3)); + /* restore RDX from R11 */ + emit_mov_reg(&prog, true, BPF_REG_3, AUX_REG); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -3865,3 +3900,15 @@ bool bpf_jit_supports_timed_may_goto(void) { return true; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call(s32 imm) +{ + if (!IS_ENABLED(CONFIG_BPF_SYSCALL)) + return false; + if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) + return true; + return false; +} diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index de0f9e5f9f73..7f13509c66db 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -16,6 +16,7 @@ #include #include #include +#include #include /* @@ -2094,6 +2095,27 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { int err; + if (IS_ENABLED(CONFIG_BPF_SYSCALL) && + imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { + /* The default implementation of this kfunc uses + * ktime_get_raw_ns() which effectively is implemented as + * `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip + * masking part because we assume it's not needed in BPF + * use case (two measurements close in time). + * Original code for rdtsc_ordered() uses sequence: + * 'rdtsc; nop; nop; nop' to patch it into + * 'lfence; rdtsc' or 'rdtscp' depending on CPU features. + * JIT uses 'lfence; rdtsc' variant because BPF program + * doesn't care about cookie provided by rdtscp in ECX. + */ + if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC)) + EMIT3(0x0F, 0xAE, 0xE8); + EMIT2(0x0F, 0x31); + break; + } + err = emit_kfunc_call(bpf_prog, image + addrs[i], insn, &prog); @@ -2621,3 +2643,14 @@ bool bpf_jit_supports_kfunc_call(void) { return true; } + +bool bpf_jit_inlines_kfunc_call(s32 imm) +{ + if (!IS_ENABLED(CONFIG_BPF_SYSCALL)) + return false; + if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) + return true; + return false; +} diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7d55553de3fc..599aaa854e4c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3387,6 +3387,9 @@ void bpf_user_rnd_init_once(void); u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); +/* Inlined kfuncs */ +u64 bpf_get_cpu_time_counter(void); + #if defined(CONFIG_NET) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type, diff --git a/include/linux/filter.h b/include/linux/filter.h index 590476743f7a..2fbfa1bc3f49 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1128,6 +1128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_inlines_helper_call(s32 imm); +bool bpf_jit_inlines_kfunc_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 62cb9557ad3b..1d811fc39eac 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3035,6 +3035,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm) return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to + * the imm. + * + * The verifier will not patch the insn->imm for the call to the helper if + * this returns true. + */ +bool __weak bpf_jit_inlines_kfunc_call(s32 imm) +{ + return false; +} + /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls(void) { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 5449756ba102..43bf35a15f78 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3193,6 +3193,11 @@ __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) local_irq_restore(*flags__irq_flag); } +__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ + return ktime_get_raw_fast_ns(); +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3293,6 +3298,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3303a3605ee8..0c4ea977973c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17035,6 +17035,24 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } } +/* True if fixup_kfunc_call() replaces calls to kfunc number 'imm', + * replacement patch is presumed to follow bpf_fastcall contract + * (see mark_fastcall_pattern_for_call() below). + */ +static bool verifier_inlines_kfunc_call(struct bpf_verifier_env *env, s32 imm) +{ + const struct bpf_kfunc_desc *desc = find_kfunc_desc(env->prog, imm, 0); + + if (!env->prog->jit_requested) + return false; + + if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) + return true; + + return false; +} + struct call_summary { u8 num_params; bool is_void; @@ -17077,7 +17095,10 @@ static bool get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call /* error would be reported later */ return false; cs->num_params = btf_type_vlen(meta.func_proto); - cs->fastcall = meta.kfunc_flags & KF_FASTCALL; + cs->fastcall = meta.kfunc_flags & KF_FASTCALL && + (verifier_inlines_kfunc_call(env, call->imm) || + (meta.btf == btf_vmlinux && + bpf_jit_inlines_kfunc_call(call->imm))); cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type)); return true; } @@ -21223,6 +21244,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; + s32 imm = insn->imm; if (!insn->imm) { verbose(env, "invalid kernel function call not eliminated in verifier pass\n"); @@ -21246,7 +21268,18 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn->imm = BPF_CALL_IMM(desc->addr); if (insn->off) return 0; - if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl] || + if (verifier_inlines_kfunc_call(env, imm)) { + if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { + insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); + *cnt = 1; + } else { + verbose(env, "verifier internal error: kfunc id %d has no inline code\n", + desc->func_id); + return -EFAULT; + } + + } else if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl] || desc->func_id == special_kfunc_list[KF_bpf_percpu_obj_new_impl]) { struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta; struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) }; @@ -21307,10 +21340,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, __fixup_collection_insert_kfunc(&env->insn_aux_data[insn_idx], struct_meta_reg, node_offset_reg, insn, insn_buf, cnt); - } else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || - desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { - insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); - *cnt = 1; } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { struct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(BPF_REG_4, (long)env->prog->aux) }; From patchwork Tue Mar 11 15:48:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14012018 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 974BB25EFBA for ; Tue, 11 Mar 2025 15:49:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708200; cv=none; b=eYYoTzI5M2pZzLEFwPDIgKHdUQhnEyH0lBhMv00QDytngi8eWSvxJZ6Yo7jbfabklrrYQaS9kRH4OH4RTmpejwyQqoPn3MxjIz4Lbr7DnsY7w6W8Yljvr3qnljagpdINAnOJbrfpWbanRnMcaKnDuX8J34/zK9deXthuYVXfio8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708200; c=relaxed/simple; bh=fLKkkBXfBYpe2I7wvhJIWoPKOgMYDyTHaY2UoU0fpwc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hC5Q3no8MinZKGVVsOwGGQW9hZu0Q46SOlYqQg9qj548xDOZ59LxA1pC2CtZ0AZGyP9APysxgU1gAzzCAppGRJSsfaypVqZozFDw+lwX0mV1HTWEbDLYgpMdTbCOZVd57AD156876FiPdv+8m5pa8GTVC4pwJ7fmXfROU+OmyQo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=RfWk/6qR; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="RfWk/6qR" Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52BBd2so022081; Tue, 11 Mar 2025 08:49:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=FGiRvDu5qMJvnFytUpRoQ6NMlmlPRC3o8wZMDSrWBx4=; b=RfWk/6qRyLYi teT95MHNYka8Q8FAjzt8bR4QcON2h2MdURAHYS9/jhTZFpF+KYT0yS8URO2sbo1w S1JJPiZKV0lsOfXJGB8xx01+6fyDnN5H3sxYDWKIdaHrFeD7xEDoLoaOPzjw5Vfs 0nnX2eak6smzziste+OSEi2xQotmCGKOxLi5IlLnKum94gd4chpLx4L6+qAknVZ2 guNZ5gNrKIJPOWRzqDx9ppVFVDsLQz0en+MnTbgUnLL2DMFrgqVzI7K9duTwGs0u gT4Iqa9JC5cNesVqiYqW7h8ezc+Js9N8gt0mjI0mktR4m0rz8EK5MHZz4HY+/oRY 2F/Iy0hBfQ== Received: from mail.thefacebook.com ([163.114.134.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 45amfx1ndy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 11 Mar 2025 08:49:10 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.14; Tue, 11 Mar 2025 15:49:07 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v10 2/4] bpf: add bpf_cpu_time_counter_to_ns helper Date: Tue, 11 Mar 2025 08:48:48 -0700 Message-ID: <20250311154850.3616840-3-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250311154850.3616840-1-vadfed@meta.com> References: <20250311154850.3616840-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-GUID: 1YjwnSh2eikBmS1fyzaT71TybA-XnTNL X-Proofpoint-ORIG-GUID: 1YjwnSh2eikBmS1fyzaT71TybA-XnTNL X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-11_04,2025-03-11_02,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net The new helper should be used to convert deltas of values received by bpf_get_cpu_time_counter() into nanoseconds. It is not designed to do full conversion of time counter values to CLOCK_MONOTONIC_RAW nanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds. This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases. When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds. Reviewed-by: Eduard Zingerman Acked-by: Andrii Nakryiko Signed-off-by: Vadim Fedorenko --- arch/x86/net/bpf_jit_comp.c | 27 ++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 27 ++++++++++++++++++++++++++- include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ 4 files changed, 59 insertions(+), 2 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 92cd5945d630..56f7557048d1 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -2289,6 +2290,29 @@ st: if (is_imm8(insn->off)) break; } + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { + struct cyc2ns_data data; + u32 mult, shift; + + cyc2ns_read_begin(&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; + cyc2ns_read_end(); + /* imul RAX, RDI, mult */ + maybe_emit_mod(&prog, BPF_REG_1, BPF_REG_0, true); + EMIT2_off32(0x69, add_2reg(0xC0, BPF_REG_1, BPF_REG_0), + mult); + + /* shr RAX, shift (which is less than 64) */ + maybe_emit_1mod(&prog, BPF_REG_0, true); + EMIT3(0xC1, add_1reg(0xE8, BPF_REG_0), shift); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -3906,7 +3930,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) { if (!IS_ENABLED(CONFIG_BPF_SYSCALL)) return false; - if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + if ((imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) || + imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns)) && cpu_feature_enabled(X86_FEATURE_TSC) && using_native_sched_clock() && sched_clock_stable()) return true; diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 7f13509c66db..9791a3fb9d69 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -2115,6 +2116,29 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT2(0x0F, 0x31); break; } + if (IS_ENABLED(CONFIG_BPF_SYSCALL) && + imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { + struct cyc2ns_data data; + u32 mult, shift; + + cyc2ns_read_begin(&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; + cyc2ns_read_end(); + + /* move parameter to BPF_REG_0 */ + emit_ia32_mov_r64(true, bpf2ia32[BPF_REG_0], + bpf2ia32[BPF_REG_1], true, true, + &prog, bpf_prog->aux); + /* multiply parameter by mut */ + emit_ia32_mul_i64(bpf2ia32[BPF_REG_0], + mult, true, &prog); + /* shift parameter by shift which is less than 64 */ + emit_ia32_rsh_i64(bpf2ia32[BPF_REG_0], + shift, true, &prog); + } err = emit_kfunc_call(bpf_prog, image + addrs[i], @@ -2648,7 +2672,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) { if (!IS_ENABLED(CONFIG_BPF_SYSCALL)) return false; - if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + if ((imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) || + imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns)) && cpu_feature_enabled(X86_FEATURE_TSC) && using_native_sched_clock() && sched_clock_stable()) return true; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 599aaa854e4c..5c4d35019b22 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3389,6 +3389,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); /* Inlined kfuncs */ u64 bpf_get_cpu_time_counter(void); +u64 bpf_cpu_time_counter_to_ns(u64 cycles); #if defined(CONFIG_NET) bool bpf_sock_common_is_valid_access(int off, int size, diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 43bf35a15f78..cc986d2048db 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3198,6 +3198,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void) return ktime_get_raw_fast_ns(); } +__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 cycles) +{ + return cycles; +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3299,6 +3304,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { From patchwork Tue Mar 11 15:48:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14012016 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AF551CAA87 for ; Tue, 11 Mar 2025 15:49:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.145.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708197; cv=none; b=lpSsiKO/944RGgnpNbQVUfFcaDkRxuxtpbcp+PpiJKh03LswSYCif1+0IYL3x6bJktMaBfxGrCLfmcMEDyb2lpa+Y3Mw8g+ma/tlt6pMmJCKvFi4mZ0yx5w/xhB1koWnPhtaKQnFnyi0cfsZK0nOK6AvV6IR6w3f76i9ih9p0Ok= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708197; c=relaxed/simple; bh=PzFqfoRtY4n2SOUoLNd8OHnIMa/d6SS3tquxpWs0kpA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Ld5c56lI85HKaptwZConDHagN566ED2D+XHwZD1byqrxoktHQGJFVVYBks0W/58V/tCnySqQJjqwXjedq75Mdcu+xf0rNBULjoeVJdCFvNdzIlFUXGu5ujZfpqPV7j5cVCW42oq5Zzf/GvM4cZ074nLMVxk3BxooUN48x+uwMlE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=kiEgAGRE; arc=none smtp.client-ip=67.231.145.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="kiEgAGRE" Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 52B7xUNZ006196; Tue, 11 Mar 2025 08:49:13 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=tAiN/cUXd5vSssqq46CTEACFwTd9FaSY1CA4OHp/kAo=; b=kiEgAGREhZwN RoAnhzFkNWC3lHVfMCJffgpeTWUhdiAvWUs7qOGPzxD5Hgg+/7h4HJjkeUYB7/Qo +pviD+p7CCqTolJhaTrAr5ReDnIx9OMY8LcYO932c61xFynfeCx3Uuwe7qdVoXok kp6im0Q5RhAHDCdUs0R0M4W/0eFaNeXow5qqGB6ohG7oHOTLy0i4Ye6YR2FlDmF9 4kjYaui+xTfrcguz8wHlBvjLmNE5fcpMgSC5LNURirlX969WnwXHTh/NqbFM7yVY 8oMbWQLfW4eUfFJ2/gyJJ8nC9ng43JKqzGy/O9iv/ckFTjnzoxWjcdxgeck0gtTJ hU1RPLN7ww== Received: from mail.thefacebook.com ([163.114.134.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 45ah8rttkv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 11 Mar 2025 08:49:13 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.14; Tue, 11 Mar 2025 15:49:09 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v10 3/4] selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit Date: Tue, 11 Mar 2025 08:48:49 -0700 Message-ID: <20250311154850.3616840-4-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250311154850.3616840-1-vadfed@meta.com> References: <20250311154850.3616840-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-GUID: uabo5GmSAtSK1yOW5GS6Zb67LRHgDu3u X-Proofpoint-ORIG-GUID: uabo5GmSAtSK1yOW5GS6Zb67LRHgDu3u X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-11_04,2025-03-11_02,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net bpf_get_cpu_time_counter() is replaced with rdtsc instruction on x86_64. Add tests to check that JIT works as expected. Acked-by: Eduard Zingerman Signed-off-by: Vadim Fedorenko --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index e66a57970d28..d5e7e302a344 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -102,6 +102,7 @@ #include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #include "irq.skel.h" #define MAX_ENTRIES 11 @@ -236,6 +237,7 @@ void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); } void test_verifier_lsm(void) { RUN(verifier_lsm); } void test_irq(void) { RUN(irq); } void test_verifier_mtu(void) { RUN(verifier_mtu); } +void test_verifier_cpu_cycles(void) { RUN(verifier_cpu_cycles); } static int init_test_val_map(struct bpf_object *obj, char *map_name) { diff --git a/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file mode 100644 index 000000000000..5b62e3690362 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Inc. */ +#include "vmlinux.h" +#include +#include +#include "bpf_misc.h" + +extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_time_counter(void) __weak __ksym; + +SEC("syscall") +__arch_x86_64 +__xlated("0: call kernel-function") +__naked int bpf_rdtsc(void) +{ + asm volatile( + "call %[bpf_get_cpu_time_counter];" + "exit" + : + : __imm(bpf_get_cpu_time_counter) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movq %rdx, %r11") +/* lfence may not be executed depending on cpu features */ +__jited(" {{(lfence|)}}") +__jited(" rdtsc") +/* combine EDX:EAX into RAX */ +__jited(" shlq ${{(32|0x20)}}, %rdx") +__jited(" orq %rdx, %rax") +/* restore RDX from R11 */ +__jited(" movq %r11, %rdx") +__jited(" leave") +__naked int bpf_rdtsc_jit_x86_64(void) +{ + asm volatile( + "call %[bpf_get_cpu_time_counter];" + "exit" + : + : __imm(bpf_get_cpu_time_counter) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +__xlated("0: r1 = 42") +__xlated("1: call kernel-function") +__naked int bpf_cyc2ns(void) +{ + asm volatile( + "r1=0x2a;" + "call %[bpf_cpu_time_counter_to_ns];" + "exit" + : + : __imm(bpf_cpu_time_counter_to_ns) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movabsq $0x2a2a2a2a2a, %rdi") +__jited(" imulq ${{.*}}, %rdi, %rax") +__jited(" shrq ${{.*}}, %rax") +__jited(" leave") +__naked int bpf_cyc2ns_jit_x86(void) +{ + asm volatile( + "r1=0x2a2a2a2a2a ll;" + "call %[bpf_cpu_time_counter_to_ns];" + "exit" + : + : __imm(bpf_cpu_time_counter_to_ns) + : __clobber_all + ); +} + +void rdtsc(void) +{ + bpf_get_cpu_time_counter(); + bpf_cpu_time_counter_to_ns(42); +} + +char _license[] SEC("license") = "GPL"; From patchwork Tue Mar 11 15:48:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14012017 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80B2025F787 for ; Tue, 11 Mar 2025 15:49:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708197; cv=none; b=UYcFs5blt+AIIJLgrOIazfi17HD0VjZfLU34RALaKcPgVrHRgikoVlJyo7+6XxtcGY8/g7+eMAIE3xNBkFAVlabd1jYP94/YLYjClVVIZnXFZj+p71EOwSR5knnF8vWXsU0hEbNRlSjZkFJzIYMIMpT9RtPOLGxoxESnT11pRpU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741708197; c=relaxed/simple; bh=UH7TVHDqiP1UuocCQTquKA8Es4asFiL/ebcSS/Ho8oQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=u31ESZ9ls6o3MQdhxW2faSfFxkjZFYlZZPokD+2SKxe7YMi/npifTbO4PVP7yBd+YbpnK4KrbRmICpweWS8kEZ5cuL8r85q61Fg/3zEJMoquMHnHk+TFlA8VkCYwDdH06O3adBWNZckaiyWRwn4wBxhDRR6Ww4A6cbCTrNx7l0w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=l5nW6NXp; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="l5nW6NXp" Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52B7NJK7005365; Tue, 11 Mar 2025 08:49:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=0CtSJOc8ThsJf8ORc2fFKFkruWqxLTJPmfNRsKPViZ4=; b=l5nW6NXpdIVB CUKYCDLiwbRg3fA/tpcUSlFrOf7+YhJVm/PT0CvGWP83MNQgtzfM+7owYnRJ/BbC sJwgoCnK3ZyyIIZZdhHYo9ltNnmBIwKuCh2h+e3AegC6oqmTk5x7TlQghWBgwCEN AoUWI2db8XK6iXhWk+NzpNjMFu31tDKFfv7ol03IFY4j+nAABLBqYFehxd97UZbZ ohHNGlyW9DQw2g2+ndf1WCty5gIS6EudmRnHQbYpJ/VJU8RIZ9t7pFNieu5zBlvC 0AwSFeZPlvnlyCw5O2z0f4NgZ2/9npkyvyRFIC8pwq495ztd4cS0Z1bLv34Andad 7ArD+DJkaQ== Received: from mail.thefacebook.com ([163.114.134.16]) by m0001303.ppops.net (PPS) with ESMTPS id 45a4880d8g-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 11 Mar 2025 08:49:16 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.14; Tue, 11 Mar 2025 15:49:12 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v10 4/4] selftests/bpf: add usage example for cpu time counter kfuncs Date: Tue, 11 Mar 2025 08:48:50 -0700 Message-ID: <20250311154850.3616840-5-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250311154850.3616840-1-vadfed@meta.com> References: <20250311154850.3616840-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-GUID: ms3HdZaRua4NY7GhrmlcDPx2JEUycESc X-Proofpoint-ORIG-GUID: ms3HdZaRua4NY7GhrmlcDPx2JEUycESc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-11_04,2025-03-11_02,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net The selftest provides an example of how to measure the latency of bpf kfunc/helper call using time stamp counter and how to convert measured value into nanoseconds. Signed-off-by: Vadim Fedorenko --- .../bpf/prog_tests/test_cpu_cycles.c | 35 +++++++++++++++++++ .../selftests/bpf/progs/test_cpu_cycles.c | 25 +++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file mode 100644 index 000000000000..d7f3b66594b3 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include +#include "test_cpu_cycles.skel.h" + +static void cpu_cycles(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct test_cpu_cycles *skel; + int err, pfd; + + skel = test_cpu_cycles__open_and_load(); + if (!ASSERT_OK_PTR(skel, "test_cpu_cycles open and load")) + return; + + pfd = bpf_program__fd(skel->progs.bpf_cpu_cycles); + if (!ASSERT_GT(pfd, 0, "test_cpu_cycles fd")) + goto fail; + + err = bpf_prog_test_run_opts(pfd, &opts); + if (!ASSERT_OK(err, "test_cpu_cycles test run")) + goto fail; + + ASSERT_NEQ(skel->bss->cycles, 0, "test_cpu_cycles 0 cycles"); + ASSERT_NEQ(skel->bss->ns, 0, "test_cpu_cycles 0 ns"); +fail: + test_cpu_cycles__destroy(skel); +} + +void test_cpu_cycles(void) +{ + if (test__start_subtest("cpu_cycles")) + cpu_cycles(); +} diff --git a/tools/testing/selftests/bpf/progs/test_cpu_cycles.c b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c new file mode 100644 index 000000000000..a7f8a4c6b854 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include "vmlinux.h" +#include + +extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_time_counter(void) __weak __ksym; + +__u64 cycles, ns; + +SEC("syscall") +int bpf_cpu_cycles(void) +{ + struct bpf_pidns_info pidns; + __u64 start; + + start = bpf_get_cpu_time_counter(); + bpf_get_ns_current_pid_tgid(0, 0, &pidns, sizeof(struct bpf_pidns_info)); + cycles = bpf_get_cpu_time_counter() - start; + ns = bpf_cpu_time_counter_to_ns(cycles); + return 0; +} + +char _license[] SEC("license") = "GPL";