From patchwork Wed Mar 19 16:36:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14022831 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52A221DF987 for ; Wed, 19 Mar 2025 16:38:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402286; cv=none; b=Yo9MjhLcHNg/7YSp16Vwopvy/8ZdyKUET0C/MkoJ/dO9P9retNXQxPjAqJM8mtYpth9hMD1cMehobcVK1oeWOA6le8h28/vSnvXyCi/SbPLYG4XPOX4FZALJw4iM3KmE5DjpKnoeT6STTtYpFo/fBwNngHZ4IujHSY4P7ubw3/M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402286; c=relaxed/simple; bh=uerTHu75CrYUa3ffYlB9U3gPlSyrz7/nEyaTjDB6yUk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DHnKiaz2zclDiR9zcPUGK/Wczt2dAPwe6Q3UBPwLxgc9SvctUvMuxjgK9d/T4DKXq3I6GTNAsALFAsOFlLxpRJYps+nmdDduDvwDP80NQtEMplmzS0wzNIiNNhOvBvDaIAW6cbt8BW/IoKH1sr3L+HFkf6Xp0BrZ7MUSDOHWT1w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=DOSZ7MLb; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="DOSZ7MLb" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52JGCVYt020382; Wed, 19 Mar 2025 09:37:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=zXGg4WeggFj0WdeURBWtWKb1JbwtTeK50Yyz1BgTvJ4=; b=DOSZ7MLbL1+V 4BUaBGP2GvNd8Y8Sqx68NZpPVGWapie/iq9hJmDgNYnxQ5kYbtgqvx0inp3lptkL htU2mvFuZDwTxWySq519xwPW1xdwIwcm56wLZ5fGcCG4aLiPsfq/J1NnPg5saIYN S8DVUwZqZ09DZIRdRmC6Xv2WZG2xtNqFFta/uzXs6H6mtRFWnbVuaoWkm/Q2zexH vLAW8mvi1BQJUjIwuXpCMf1e1tof1Xq5jTvGgHkgo5sBbmQAahlL43pvN7Xw0XY5 n8SgCC7Yqd0+a0+QkHcBiWd+GzjAOSEj8s/0eX+3SGX4YjcP5O19GQx0n/2LTLik m5ybdigizA== Received: from maileast.thefacebook.com ([163.114.135.16]) by m0089730.ppops.net (PPS) with ESMTPS id 45fn4mmb9n-12 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 19 Mar 2025 09:37:18 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a9:6f::237c) with Microsoft SMTP Server id 15.2.1544.14; Wed, 19 Mar 2025 16:36:47 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v12 1/5] bpf: adjust BPF JIT dependency to BPF_SYSCALL Date: Wed, 19 Mar 2025 09:36:34 -0700 Message-ID: <20250319163638.3607043-2-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250319163638.3607043-1-vadfed@meta.com> References: <20250319163638.3607043-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: _YJ5qyg1bKDhjlYM3-OwGkGTK87vrDb0 X-Proofpoint-GUID: _YJ5qyg1bKDhjlYM3-OwGkGTK87vrDb0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-19_06,2025-03-19_01,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net BPF JIT is moving towards optimizing kfuncs and it was long overdue to switch the dependency. Let's do it now to simplify other patches in the series. Signed-off-by: Vadim Fedorenko --- kernel/bpf/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/Kconfig b/kernel/bpf/Kconfig index 17067dcb4386..528d37819570 100644 --- a/kernel/bpf/Kconfig +++ b/kernel/bpf/Kconfig @@ -41,7 +41,7 @@ config BPF_SYSCALL config BPF_JIT bool "Enable BPF Just In Time compiler" - depends on BPF + depends on BPF_SYSCALL depends on HAVE_CBPF_JIT || HAVE_EBPF_JIT select EXECMEM help From patchwork Wed Mar 19 16:36:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14022828 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60B8A1D6199 for ; Wed, 19 Mar 2025 16:37:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402282; cv=none; b=XZcdax/wH3YDxzCt0/sEMvyRH7nupAAga79fOrF6f7gOCX5cSu817Hw4BcdlEKcTNBhJa8E+9+8wz5vQbchoZ71KtClKl9IXwU3mjuFdd4DYQusIzPB2WliU5BB1bMWzHUEGhpPDBO3Z0GRuGecte8nVtRUW32intleO8drCpnE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402282; c=relaxed/simple; bh=mlTDwHmmJnMg4iCz9K/a247Md3tPFmnlIYCjC4LQi9s=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ioLiXOMFkLrhHLRXtBrpTBelYH+j9XeYumtsbBZ6dXIberC303ksUuboAjx5SBX1s3MNJ92hZN16Aq5JqHojIfCYl1A9zkOg6VgFOrnFP4uaPQudPbsNSRGCCDCNmBHTsRMroIMd517+mRtJCt6HEoyi0mTWEJlF0gHmxI4FKeU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=bqHifISL; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="bqHifISL" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52JGCVYu020382; Wed, 19 Mar 2025 09:37:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=EJVaTwP3z/Rvmexq9wQ5AzI8bVrkToYk1Ty/ElEbfGg=; b=bqHifISLZu5w B+AJr2vBgWWlzgI2OCi+R3bOmpviN1rWiEb5bX1n8ZK7mrkfB3i69h6n/+JOT4VK 3Vay7QnQt0ub4KsfJikIJXs8reet5edltAn2I+gy0wp4cKz8vNoioiLQC0xW9AFO lf1TYISNZwVT/dRyzr2NDO082Zxjv1WwtrsK9ywOfELgMcRZ9SSjj1e9WHVrhaZT 1DpB7MGBKNVR16moqsL/wYHALuRH5hXt0mG+WqBrTqxaplBXoe9qOvADzTIWS3DY ekKcuTHkqXEHPURrmOzNLceUYqwWz20aPDUfEpJYXfMSflVjmNSHzVh3d8O2MP4r I28pOBzc0A== Received: from maileast.thefacebook.com ([163.114.135.16]) by m0089730.ppops.net (PPS) with ESMTPS id 45fn4mmb9n-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 19 Mar 2025 09:37:19 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a9:6f::237c) with Microsoft SMTP Server id 15.2.1544.14; Wed, 19 Mar 2025 16:36:49 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v12 2/5] bpf: add bpf_get_cpu_time_counter kfunc Date: Wed, 19 Mar 2025 09:36:35 -0700 Message-ID: <20250319163638.3607043-3-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250319163638.3607043-1-vadfed@meta.com> References: <20250319163638.3607043-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 2BCr5dZ8MtejBR4onWVEKgp2w1O_EHGE X-Proofpoint-GUID: 2BCr5dZ8MtejBR4onWVEKgp2w1O_EHGE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-19_06,2025-03-19_01,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of bpf_ktime_get_ns() at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86_64 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to get CLOCK_MONOTONIC_RAW value in ns. JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant. The simplest use-case is added in 5th patch, where we calculate the time spent by bpf_get_ns_current_pid_tgid() kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe. Acked-by: Eduard Zingerman Acked-by: Andrii Nakryiko Acked-by: Yonghong Song Signed-off-by: Vadim Fedorenko --- arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 1 + include/linux/bpf.h | 3 +++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 +++++++++ kernel/bpf/helpers.c | 11 +++++++++ kernel/bpf/verifier.c | 4 +++- 7 files changed, 73 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d3491cc0898b..284696d69df4 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -2254,6 +2255,38 @@ st: if (is_imm8(insn->off)) case BPF_JMP | BPF_CALL: { u8 *ip = image + addrs[i - 1]; + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + bpf_jit_inlines_kfunc_call(imm32)) { + /* The default implementation of this kfunc uses + * ktime_get_raw_ns() which effectively is implemented as + * `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip + * masking part because we assume it's not needed in BPF + * use case (two measurements close in time). + * Original code for rdtsc_ordered() uses sequence: + * 'rdtsc; nop; nop; nop' to patch it into + * 'lfence; rdtsc' or 'rdtscp' depending on CPU features. + * JIT uses 'lfence; rdtsc' variant because BPF program + * doesn't care about cookie provided by rdtscp in RCX. + * Save RDX because RDTSC will use EDX:EAX to return u64 + */ + emit_mov_reg(&prog, true, AUX_REG, BPF_REG_3); + if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC)) + EMIT_LFENCE(); + EMIT2(0x0F, 0x31); + + /* shl RDX, 32 */ + maybe_emit_1mod(&prog, BPF_REG_3, true); + EMIT3(0xC1, add_1reg(0xE0, BPF_REG_3), 32); + /* or RAX, RDX */ + maybe_emit_mod(&prog, BPF_REG_0, BPF_REG_3, true); + EMIT2(0x09, add_2reg(0xC0, BPF_REG_0, BPF_REG_3)); + /* restore RDX from R11 */ + emit_mov_reg(&prog, true, BPF_REG_3, AUX_REG); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -3865,3 +3898,13 @@ bool bpf_jit_supports_timed_may_goto(void) { return true; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call(s32 imm) +{ + if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) + return true; + return false; +} diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index de0f9e5f9f73..68511888eb27 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -16,6 +16,7 @@ #include #include #include +#include #include /* diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 973a88d9b52b..6cf9138b2437 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3389,6 +3389,9 @@ void bpf_user_rnd_init_once(void); u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); +/* Inlined kfuncs */ +u64 bpf_get_cpu_time_counter(void); + #if defined(CONFIG_NET) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type, diff --git a/include/linux/filter.h b/include/linux/filter.h index 590476743f7a..2fbfa1bc3f49 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1128,6 +1128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_inlines_helper_call(s32 imm); +bool bpf_jit_inlines_kfunc_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ba6b6118cf50..6ac61b9083ce 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3040,6 +3040,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm) return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to + * the imm. + * + * The verifier will not patch the insn->imm for the call to the helper if + * this returns true. + */ +bool __weak bpf_jit_inlines_kfunc_call(s32 imm) +{ + return false; +} + /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls(void) { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index ddaa41a70676..26f71e2438d2 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3195,6 +3195,16 @@ __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) local_irq_restore(*flags__irq_flag); } +__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ + /* CLOCK_MONOTONIC_RAW is the closest analogue to what is implemented + * in JIT. The access time is the same as for CLOCK_MONOTONIC, but the + * slope of 'raw' is not affected by NTP adjustments, and with stable + * TSC it can provide less jitter in short term measurements. + */ + return ktime_get_raw_fast_ns(); +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3295,6 +3305,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_get_cpu_time_counter) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9f8cbd5c61bc..aea1040b4462 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17077,7 +17077,9 @@ static bool get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call /* error would be reported later */ return false; cs->num_params = btf_type_vlen(meta.func_proto); - cs->fastcall = meta.kfunc_flags & KF_FASTCALL; + cs->fastcall = (meta.kfunc_flags & KF_FASTCALL) || + (meta.btf == btf_vmlinux && + bpf_jit_inlines_kfunc_call(call->imm)); cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type)); return true; } From patchwork Wed Mar 19 16:36:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14022833 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EB6F1DF996 for ; Wed, 19 Mar 2025 16:38:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402287; cv=none; b=QwOmgPKno0uqEqX2/LRaPwpVl3rDqYMljLoepUlND8/3QVd3yo5Sp5dTuN8yZqNXrvMEX0TN+nAddSOHb0lifMwj0veub1B0blTd+FEnkv//gNWGKu9wcMHRhWX++JsJX50cv4UlbB5iXk3wsG7FwE5+TZ+AMGXSvcDUE/o/8pY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402287; c=relaxed/simple; bh=bg3adV5mGxy4H31FCVfcOYW7nPNrXcTa7FLWNq8+Bck=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=aUlAy9T1nfibeDgVXt0aQXuhzuDllYUw6p+8HHB2bzp8HskmSPc4YPcf4AQkr5QZRYOwVNkzo12dYQdD3AttpP8Iopc4IxTaqZRAGC/bi3XywFAD6Iapp2Yrq4jBqy9YYVCEyMuToIe4vc7gfSFyI7U9BYcO/OGLFvzx109+hlw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=C7qQ4eKu; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="C7qQ4eKu" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52JGCVYv020382; Wed, 19 Mar 2025 09:37:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=Dvp1jwGe2CFsu1aBCwRwMzmZ7NbvDYio+ofvr57FOqE=; b=C7qQ4eKu7qvW gWoU8KgHKLnlZGO4V+TdVlSnj/Gu+omNbV31+AMvYdU3kYHStmLnsIN0RfqDbpeL yfk9hSlz8ghGiZ2Zs+Kj8WcG88625SzC8BaxrLU8FoPUsDP2csBH7r8Kb61fLqH0 1HGM150SMUzrIcW4EYPvNj/b0jirTxkdGdzHlgP/qjIIsIqZgtJ+d+Cv7qupnhsD SCaHqgYJz/Ynp/mq0LTRDzmFvkDjD1wWvcJvIX26N7NUfel9YwGd+WdpVOpPruT3 dWisGsSaI2Wx9oymIZGfcq5Jax2uAc9dRWz06n83M0qII1FZg9dKRxQZZL7cf3e7 wHJlKNiwUg== Received: from maileast.thefacebook.com ([163.114.135.16]) by m0089730.ppops.net (PPS) with ESMTPS id 45fn4mmb9n-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 19 Mar 2025 09:37:20 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a9:6f::237c) with Microsoft SMTP Server id 15.2.1544.14; Wed, 19 Mar 2025 16:36:50 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v12 3/5] bpf: add bpf_cpu_time_counter_to_ns helper Date: Wed, 19 Mar 2025 09:36:36 -0700 Message-ID: <20250319163638.3607043-4-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250319163638.3607043-1-vadfed@meta.com> References: <20250319163638.3607043-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 1tzwIaHPfVd_4Em2UJ7jP1yHxV8SD_2D X-Proofpoint-GUID: 1tzwIaHPfVd_4Em2UJ7jP1yHxV8SD_2D X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-19_06,2025-03-19_01,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net The new helper should be used to convert deltas of values received by bpf_get_cpu_time_counter() into nanoseconds. It is not designed to do full conversion of time counter values to CLOCK_MONOTONIC_RAW nanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds. This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases. When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds and can be optimized by verifier. Reviewed-by: Eduard Zingerman Acked-by: Andrii Nakryiko Signed-off-by: Vadim Fedorenko --- arch/x86/net/bpf_jit_comp.c | 29 ++++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 1 + include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ kernel/bpf/verifier.c | 9 ++++++++- 5 files changed, 44 insertions(+), 2 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 284696d69df4..8ff8d7436fc9 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -2287,6 +2288,31 @@ st: if (is_imm8(insn->off)) break; } + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + bpf_jit_inlines_kfunc_call(imm32)) { + struct cyc2ns_data data; + u32 mult, shift; + + /* stable TSC runs with fixed frequency and + * transformation coefficients are also fixed + */ + cyc2ns_read_begin(&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; + cyc2ns_read_end(); + /* imul RAX, RDI, mult */ + maybe_emit_mod(&prog, BPF_REG_1, BPF_REG_0, true); + EMIT2_off32(0x69, add_2reg(0xC0, BPF_REG_1, BPF_REG_0), + mult); + + /* shr RAX, shift (which is less than 64) */ + maybe_emit_1mod(&prog, BPF_REG_0, true); + EMIT3(0xC1, add_1reg(0xE8, BPF_REG_0), shift); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -3902,7 +3928,8 @@ bool bpf_jit_supports_timed_may_goto(void) /* x86-64 JIT can inline kfunc */ bool bpf_jit_inlines_kfunc_call(s32 imm) { - if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + if ((imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) || + imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns)) && cpu_feature_enabled(X86_FEATURE_TSC) && using_native_sched_clock() && sched_clock_stable()) return true; diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 68511888eb27..83176a07fc08 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6cf9138b2437..fc03a3805b36 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3391,6 +3391,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); /* Inlined kfuncs */ u64 bpf_get_cpu_time_counter(void); +u64 bpf_cpu_time_counter_to_ns(u64 counter); #if defined(CONFIG_NET) bool bpf_sock_common_is_valid_access(int off, int size, diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 26f71e2438d2..a176bd5a33d0 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3205,6 +3205,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void) return ktime_get_raw_fast_ns(); } +__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter) +{ + return counter; +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3306,6 +3311,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) BTF_ID_FLAGS(func, bpf_get_cpu_time_counter) +BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index aea1040b4462..3a908cf24e45 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12007,6 +12007,7 @@ enum special_kfunc_type { KF_bpf_iter_num_destroy, KF_bpf_set_dentry_xattr, KF_bpf_remove_dentry_xattr, + KF_bpf_cpu_time_counter_to_ns, }; BTF_SET_START(special_kfunc_set) @@ -12040,6 +12041,7 @@ BTF_ID(func, bpf_iter_css_task_new) BTF_ID(func, bpf_set_dentry_xattr) BTF_ID(func, bpf_remove_dentry_xattr) #endif +BTF_ID(func, bpf_cpu_time_counter_to_ns) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -12096,6 +12098,7 @@ BTF_ID(func, bpf_remove_dentry_xattr) BTF_ID_UNUSED BTF_ID_UNUSED #endif +BTF_ID(func, bpf_cpu_time_counter_to_ns) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -21246,6 +21249,9 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (!bpf_jit_supports_far_kfunc_call()) insn->imm = BPF_CALL_IMM(desc->addr); + /* if JIT will inline kfunc verifier shouldn't change the code */ + if (bpf_jit_inlines_kfunc_call(insn->imm)) + return 0; if (insn->off) return 0; if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl] || @@ -21310,7 +21316,8 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, __fixup_collection_insert_kfunc(&env->insn_aux_data[insn_idx], struct_meta_reg, node_offset_reg, insn, insn_buf, cnt); } else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || - desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast] || + desc->func_id == special_kfunc_list[KF_bpf_cpu_time_counter_to_ns]) { insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); *cnt = 1; } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { From patchwork Wed Mar 19 16:36:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14022829 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60BE61DE3A3 for ; Wed, 19 Mar 2025 16:38:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402282; cv=none; b=g3jEcJgW2m4HMWWI4GiQNFSHBeQnm2LIWTp5X34v5R0HeEs/UlsI382ty2945cvoVapZeFL4Jpm6vTS2uH8UVgUFn7JqAbPFdQ/or5OuwGO82PaS4S7csxjiGMH7qT8sM+IcAI9TFDs/rtPd8v1PjTAjYKhPvhrVUcwiilkQ8dk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402282; c=relaxed/simple; bh=dJnPrBaNWDajEGZjANm7VHYNZwzO2MZUg2YsF82+9I4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uMzebCjMx9ElUJyFdx0hmgtMUbglohtvyPsE4UwI4B9ftIhwUm/GATayGIPZAVcTy2/ZGRQYGqS3Tmrn+yjv3ISG36ErNKw8cBTSNp0HhjeDFJXVJo4pzrHENEvzVXDI5wvQ0TvKZ1ljSzeFSt0t637O7TXF1cqbRTi/YBy9v04= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=H30VXEHj; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="H30VXEHj" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52JGCVZ1020382; Wed, 19 Mar 2025 09:37:25 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=ZXqfrRQJLQ6WxpJ9Z049ew42YD1XhBNSedrcZPMBwBI=; b=H30VXEHj61La rLQne1SUbHIf2w/cyI2HhrMcKZBnVyorS/9hsswaTKxUw6K6tf5NbZ3AS8Wcz9tF gvgt7tG0g27U0SWngXFRsJsHp/RU8Y0mV78aeXFodpVgUJq9LZKmh1ZwH6I0K6Mq pG0JLtnFlmLAQWxmyzTnDnfgBGFDwkXptbHVDVVv4HMhUjk78UPvEIE/TqTKJ6cz aUgqN3iiLiDTBfBtbBvQewnOP9Z7RB6O4cKw8XENxzhhtbMjtqUs8Gx0VbDmb0zw 2tVVeUusu2s+BJhws+h1PcXwsCkKy5wy01VpomUSWGfSxReacCy+cFtkPy0blQQw DVFdHotHgg== Received: from maileast.thefacebook.com ([163.114.135.16]) by m0089730.ppops.net (PPS) with ESMTPS id 45fn4mmb9n-15 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 19 Mar 2025 09:37:25 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a9:6f::237c) with Microsoft SMTP Server id 15.2.1544.14; Wed, 19 Mar 2025 16:36:52 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v12 4/5] selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit Date: Wed, 19 Mar 2025 09:36:37 -0700 Message-ID: <20250319163638.3607043-5-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250319163638.3607043-1-vadfed@meta.com> References: <20250319163638.3607043-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: peQiA4MhotkHidMPfRzbRz4U_Rf2UT58 X-Proofpoint-GUID: peQiA4MhotkHidMPfRzbRz4U_Rf2UT58 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-19_06,2025-03-19_01,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net bpf_get_cpu_time_counter() is replaced with rdtsc instruction on x86_64. Add tests to check that JIT works as expected. When JIT is not supported, bpf_cpu_time_counter_to_ns() can be inlined by verifier. Acked-by: Eduard Zingerman Signed-off-by: Vadim Fedorenko --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 120 ++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index e66a57970d28..d5e7e302a344 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -102,6 +102,7 @@ #include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #include "irq.skel.h" #define MAX_ENTRIES 11 @@ -236,6 +237,7 @@ void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); } void test_verifier_lsm(void) { RUN(verifier_lsm); } void test_irq(void) { RUN(irq); } void test_verifier_mtu(void) { RUN(verifier_mtu); } +void test_verifier_cpu_cycles(void) { RUN(verifier_cpu_cycles); } static int init_test_val_map(struct bpf_object *obj, char *map_name) { diff --git a/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file mode 100644 index 000000000000..26c02010ccf1 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Inc. */ +#include "vmlinux.h" +#include +#include +#include "bpf_misc.h" + +extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_time_counter(void) __weak __ksym; + +SEC("syscall") +__arch_x86_64 +__xlated("0: call kernel-function") +__naked int bpf_rdtsc(void) +{ + asm volatile( + "call %[bpf_get_cpu_time_counter];" + "exit" + : + : __imm(bpf_get_cpu_time_counter) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movq %rdx, %r11") +/* lfence may not be executed depending on cpu features */ +__jited(" {{(lfence|)}}") +__jited(" rdtsc") +/* combine EDX:EAX into RAX */ +__jited(" shlq ${{(32|0x20)}}, %rdx") +__jited(" orq %rdx, %rax") +/* restore RDX from R11 */ +__jited(" movq %r11, %rdx") +__jited(" leave") +__naked int bpf_rdtsc_jit_x86_64(void) +{ + asm volatile( + "call %[bpf_get_cpu_time_counter];" + "exit" + : + : __imm(bpf_get_cpu_time_counter) + : __clobber_all + ); +} + +SEC("syscall") +__arch_arm64 +__xlated("0: r1 = 42") +__xlated("1: r0 = r1") +__naked int bpf_cyc2ns_arm(void) +{ + asm volatile( + "r1=0x2a;" + "call %[bpf_cpu_time_counter_to_ns];" + "exit" + : + : __imm(bpf_cpu_time_counter_to_ns) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +__xlated("0: r1 = 42") +__xlated("1: call kernel-function") +__naked int bpf_cyc2ns(void) +{ + asm volatile( + "r1=0x2a;" + "call %[bpf_cpu_time_counter_to_ns];" + "exit" + : + : __imm(bpf_cpu_time_counter_to_ns) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movabsq $0x2a2a2a2a2a, %rdi") +__jited(" imulq ${{.*}}, %rdi, %rax") +__jited(" shrq ${{.*}}, %rax") +__jited(" leave") +__naked int bpf_cyc2ns_jit_x86(void) +{ + asm volatile( + "r1=0x2a2a2a2a2a ll;" + "call %[bpf_cpu_time_counter_to_ns];" + "exit" + : + : __imm(bpf_cpu_time_counter_to_ns) + : __clobber_all + ); +} + +void rdtsc(void) +{ + bpf_get_cpu_time_counter(); + bpf_cpu_time_counter_to_ns(42); +} + +char _license[] SEC("license") = "GPL"; From patchwork Wed Mar 19 16:36:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 14022832 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 529C51DE881 for ; Wed, 19 Mar 2025 16:38:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402286; cv=none; b=q7vy9blV1wHYPtLiqSvulClMZKoUDceRk0t3Y+9hRDhdWUqPcgmzddEm7ByeJlU/3ofoen2aQQachRc+Ri24Oig00I4cLVCYKWk4PAwo9qk1bQhez/OC6UH4ZGu+zXD0FSK311wf3HWUyybhHNbySK2K4u6rfoAn/iT7YfmlYZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742402286; c=relaxed/simple; bh=lYYql7QWyfM/V2S2dDWpxbFvwK4evLRvMyIhPkcjKeg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ErQGLOXfTjCjCjRSHA7qy10+QzF/JmgEyJBExKFBLL7yFoSLlMX3zKSw/CilBAAQcoDs04hWa3nk5f4i/xxPXgNzUlI+4xPNqxRWhwaN/7HTrgN+g8jFFj0vvqcv2Bru7hn/3QQwYkXW84YyNu2iZPhdjgFC4LCBlmr3ShdiR3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=P4Ha0vnC; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="P4Ha0vnC" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 52JGCVZ2020382; Wed, 19 Mar 2025 09:37:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=pS46nJgPfSz7ph2B0rdWcYPLP3IYWmfeMsIQdAzzYRs=; b=P4Ha0vnCTu24 pRE8tLO/lukp/auw+os+xTd3Z+zWa60+sUZBQ5tLPtINAYUCcleZF5kDMj+Izd33 61WDxs595KbWKD77mSOoKgr64a7x5IsVdElOGUwcD3DeOcRspnIW76CLZ1EU2dUz ANsvEaQRTKZoi3+YkYXKWNVpfzKs4z9q6WwQOFTS7DJDRo48CjL48r+iQqa8SxCC WK9JF+iYqi9lGk3Vkr6DMbEAwPlCI1r678T5K5Tuz78nxRCKOI6HYhDZchG4Ayq/ ZFlwTafFdQfbAA6CfFdq5TuyXOjjXKTMX3cMf3AEiFkqO3P20FaWCCa/65jBWoWT PlDLbJ+e0Q== Received: from maileast.thefacebook.com ([163.114.135.16]) by m0089730.ppops.net (PPS) with ESMTPS id 45fn4mmb9n-16 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 19 Mar 2025 09:37:25 -0700 (PDT) Received: from devvm4158.cln0.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a9:6f::237c) with Microsoft SMTP Server id 15.2.1544.14; Wed, 19 Mar 2025 16:36:53 +0000 From: Vadim Fedorenko To: Borislav Petkov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Yonghong Song , Vadim Fedorenko , Mykola Lysenko CC: , , Peter Zijlstra , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v12 5/5] selftests/bpf: add usage example for cpu time counter kfuncs Date: Wed, 19 Mar 2025 09:36:38 -0700 Message-ID: <20250319163638.3607043-6-vadfed@meta.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250319163638.3607043-1-vadfed@meta.com> References: <20250319163638.3607043-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: LMzz39cR3H0Tphd4kvT7zKJvy9wQQXVi X-Proofpoint-GUID: LMzz39cR3H0Tphd4kvT7zKJvy9wQQXVi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-19_06,2025-03-19_01,2024-11-22_01 X-Patchwork-Delegate: bpf@iogearbox.net The selftest provides an example of how to measure the latency of bpf kfunc/helper call using time stamp counter and how to convert measured value into nanoseconds. Signed-off-by: Vadim Fedorenko --- .../bpf/prog_tests/test_cpu_cycles.c | 35 +++++++++++++++++++ .../selftests/bpf/progs/test_cpu_cycles.c | 25 +++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file mode 100644 index 000000000000..067307f0c4c2 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Inc. */ + +#include +#include "test_cpu_cycles.skel.h" + +static void cpu_cycles(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct test_cpu_cycles *skel; + int err, pfd; + + skel = test_cpu_cycles__open_and_load(); + if (!ASSERT_OK_PTR(skel, "test_cpu_cycles open and load")) + return; + + pfd = bpf_program__fd(skel->progs.bpf_cpu_cycles); + if (!ASSERT_GT(pfd, 0, "test_cpu_cycles fd")) + goto fail; + + err = bpf_prog_test_run_opts(pfd, &opts); + if (!ASSERT_OK(err, "test_cpu_cycles test run")) + goto fail; + + ASSERT_NEQ(skel->bss->cycles, 0, "test_cpu_cycles 0 cycles"); + ASSERT_NEQ(skel->bss->ns, 0, "test_cpu_cycles 0 ns"); +fail: + test_cpu_cycles__destroy(skel); +} + +void test_cpu_cycles(void) +{ + if (test__start_subtest("cpu_cycles")) + cpu_cycles(); +} diff --git a/tools/testing/selftests/bpf/progs/test_cpu_cycles.c b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c new file mode 100644 index 000000000000..3c428f3be831 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Meta Inc. */ + +#include "vmlinux.h" +#include + +extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_time_counter(void) __weak __ksym; + +__u64 cycles, ns; + +SEC("syscall") +int bpf_cpu_cycles(void) +{ + struct bpf_pidns_info pidns; + __u64 start; + + start = bpf_get_cpu_time_counter(); + bpf_get_ns_current_pid_tgid(0, 0, &pidns, sizeof(struct bpf_pidns_info)); + cycles = bpf_get_cpu_time_counter() - start; + ns = bpf_cpu_time_counter_to_ns(cycles); + return 0; +} + +char _license[] SEC("license") = "GPL";