[RESEND,bpf-next,v7,0/4] Add prog_kfunc feature probe

Message

Tao Chen Feb. 12, 2025, 3:39 p.m. UTC

More and more kfunc functions are being added to the kernel.
Different prog types have different restrictions when using kfunc.
Therefore, prog_kfunc probe is added to check whether it is supported,
and the use of this api will be added to bpftool later.

Change list:
- v6 -> v7:
  - wrap err with libbpf_err
  - comments fix
  - handle btf_fd < 0 as vmlinux
  - patchset Reviewed-by: Jiri Olsa <[e-mail]>
- v6
  https://lore.kernel.org/bpf/[e-mail]

- v5 -> v6:
  - remove fd_array_cnt
  - test case clean code
- v5
  https://lore.kernel.org/bpf/[e-mail]

- v4 -> v5:
  - use fd_array on stack
  - declare the scope of use of btf_fd
- v4
  https://lore.kernel.org/bpf/[e-mail]/

- v3 -> v4:
  - add fd_array init for kfunc in mod btf
  - add test case for kfunc in mod btf
  - refactor common part as prog load type check for
    libbpf_probe_bpf_{helper,kfunc}
- v3
  https://lore.kernel.org/bpf/[e-mail]

- v2 -> v3:
  - rename parameter off with btf_fd
  - extract the common part for libbpf_probe_bpf_{helper,kfunc}
- v2
  https://lore.kernel.org/bpf/[e-mail]

- v1 -> v2:
  - check unsupported prog type like probe_bpf_helper
  - add off parameter for module btf
  - check verifier info when kfunc id invalid
- v1
  https://lore.kernel.org/bpf/[e-mail]

Tao Chen (4):
  libbpf: Extract prog load type check from libbpf_probe_bpf_helper
  libbpf: Init fd_array when prog probe load
  libbpf: Add libbpf_probe_bpf_kfunc API
  selftests/bpf: Add libbpf_probe_bpf_kfunc API selftests

 tools/lib/bpf/libbpf.h                        |  19 ++-
 tools/lib/bpf/libbpf.map                      |   1 +
 tools/lib/bpf/libbpf_probes.c                 |  86 +++++++++++---
 .../selftests/bpf/prog_tests/libbpf_probes.c  | 111 ++++++++++++++++++
 4 files changed, 201 insertions(+), 16 deletions(-)

Comments

Tao Chen Feb. 17, 2025, 5:21 a.m. UTC | #1

在 2025/2/12 23:39, Tao Chen 写道:
> More and more kfunc functions are being added to the kernel.
> Different prog types have different restrictions when using kfunc.
> Therefore, prog_kfunc probe is added to check whether it is supported,
> and the use of this api will be added to bpftool later.
> 
> Change list:
> - v6 -> v7:
>    - wrap err with libbpf_err
>    - comments fix
>    - handle btf_fd < 0 as vmlinux
>    - patchset Reviewed-by: Jiri Olsa <[e-mail]>
> - v6
>    https://lore.kernel.org/bpf/[e-mail]
> 
> - v5 -> v6:
>    - remove fd_array_cnt
>    - test case clean code
> - v5
>    https://lore.kernel.org/bpf/[e-mail]
> 
> - v4 -> v5:
>    - use fd_array on stack
>    - declare the scope of use of btf_fd
> - v4
>    https://lore.kernel.org/bpf/[e-mail]/
> 
> - v3 -> v4:
>    - add fd_array init for kfunc in mod btf
>    - add test case for kfunc in mod btf
>    - refactor common part as prog load type check for
>      libbpf_probe_bpf_{helper,kfunc}
> - v3
>    https://lore.kernel.org/bpf/[e-mail]
> 
> - v2 -> v3:
>    - rename parameter off with btf_fd
>    - extract the common part for libbpf_probe_bpf_{helper,kfunc}
> - v2
>    https://lore.kernel.org/bpf/[e-mail]
> 
> - v1 -> v2:
>    - check unsupported prog type like probe_bpf_helper
>    - add off parameter for module btf
>    - check verifier info when kfunc id invalid
> - v1
>    https://lore.kernel.org/bpf/[e-mail]
> 
> Tao Chen (4):
>    libbpf: Extract prog load type check from libbpf_probe_bpf_helper
>    libbpf: Init fd_array when prog probe load
>    libbpf: Add libbpf_probe_bpf_kfunc API
>    selftests/bpf: Add libbpf_probe_bpf_kfunc API selftests
> 
>   tools/lib/bpf/libbpf.h                        |  19 ++-
>   tools/lib/bpf/libbpf.map                      |   1 +
>   tools/lib/bpf/libbpf_probes.c                 |  86 +++++++++++---
>   .../selftests/bpf/prog_tests/libbpf_probes.c  | 111 ++++++++++++++++++
>   4 files changed, 201 insertions(+), 16 deletions(-)
> 

Ping...

Hi Andrii, Eduard,

I've revised the previous suggestions. Please review it again. Thanks.

Eduard Zingerman Feb. 18, 2025, 10:51 p.m. UTC | #2

On Mon, 2025-02-17 at 13:21 +0800, Tao Chen wrote:
> 在 2025/2/12 23:39, Tao Chen 写道:
> > More and more kfunc functions are being added to the kernel.
> > Different prog types have different restrictions when using kfunc.
> > Therefore, prog_kfunc probe is added to check whether it is supported,
> > and the use of this api will be added to bpftool later.
> > 
> > Change list:
> > - v6 -> v7:
> >    - wrap err with libbpf_err
> >    - comments fix
> >    - handle btf_fd < 0 as vmlinux
> >    - patchset Reviewed-by: Jiri Olsa <[e-mail]>
> > - v6
> >    https://lore.kernel.org/bpf/[e-mail]
> > 
> > - v5 -> v6:
> >    - remove fd_array_cnt
> >    - test case clean code
> > - v5
> >    https://lore.kernel.org/bpf/[e-mail]
> > 
> > - v4 -> v5:
> >    - use fd_array on stack
> >    - declare the scope of use of btf_fd
> > - v4
> >    https://lore.kernel.org/bpf/[e-mail]/
> > 
> > - v3 -> v4:
> >    - add fd_array init for kfunc in mod btf
> >    - add test case for kfunc in mod btf
> >    - refactor common part as prog load type check for
> >      libbpf_probe_bpf_{helper,kfunc}
> > - v3
> >    https://lore.kernel.org/bpf/[e-mail]
> > 
> > - v2 -> v3:
> >    - rename parameter off with btf_fd
> >    - extract the common part for libbpf_probe_bpf_{helper,kfunc}
> > - v2
> >    https://lore.kernel.org/bpf/[e-mail]
> > 
> > - v1 -> v2:
> >    - check unsupported prog type like probe_bpf_helper
> >    - add off parameter for module btf
> >    - check verifier info when kfunc id invalid
> > - v1
> >    https://lore.kernel.org/bpf/[e-mail]
> > 
> > Tao Chen (4):
> >    libbpf: Extract prog load type check from libbpf_probe_bpf_helper
> >    libbpf: Init fd_array when prog probe load
> >    libbpf: Add libbpf_probe_bpf_kfunc API
> >    selftests/bpf: Add libbpf_probe_bpf_kfunc API selftests
> > 
> >   tools/lib/bpf/libbpf.h                        |  19 ++-
> >   tools/lib/bpf/libbpf.map                      |   1 +
> >   tools/lib/bpf/libbpf_probes.c                 |  86 +++++++++++---
> >   .../selftests/bpf/prog_tests/libbpf_probes.c  | 111 ++++++++++++++++++
> >   4 files changed, 201 insertions(+), 16 deletions(-)
> > 
> 
> Ping...
> 
> Hi Andrii, Eduard,
> 
> I've revised the previous suggestions. Please review it again. Thanks.
> 

I tried the test enumerating all kfuncs in BTF and doing
libbpf_probe_bpf_kfunc for BPF_PROG_TYPE_{KPROBE,XDP}.
(Source code at the end of the email).

The set of kfuncs returned for XDP looks correct.
The set of kfuncs returned for KPROBE contains a few incorrect entries:
- bpf_xdp_metadata_rx_hash
- bpf_xdp_metadata_rx_timestamp
- bpf_xdp_metadata_rx_vlan_tag

This is because of a different string reported by verifier for these
three functions.

Ideally, I'd write some script looking for
register_btf_kfunc_id_set(BPF_PROG_TYPE_***, kfunc_set)
calls in the kernel source code and extracting the prog type /
functions in the set, and comparing results of this script with
output of the test below for all program types.
But up to you if you'd like to do such rigorous verification or not.

Otherwise patch-set looks good to me, for all patch-set:

Reviewed-by: Eduard Zingerman <[e-mail]>

--- 8< -----------------------------------------------------

static const struct {
	const char *name;
	int code;
} program_types[] = {
#define _T(n) { #n, BPF_PROG_TYPE_ ## n }
	_T(KPROBE),
	_T(XDP),
#undef _T
};

void test_libbpf_probe_kfuncs_many(void)
{
	int i, kfunc_id, ret, id;
	const struct btf_type *t;
	struct btf *btf = NULL;
	const char *kfunc;
	const char *tag;

	btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
	if (!ASSERT_OK_PTR(btf, "btf_parse"))
		return;

	for (id = 0; id < btf__type_cnt(btf); ++id) {
		t = btf__type_by_id(btf, id);
		if (!btf_is_decl_tag(t))
			continue;
		tag = btf__name_by_offset(btf, t->name_off);
		if (strcmp(tag, "bpf_kfunc") != 0)
			continue;
		kfunc_id = t->type;
		t = btf__type_by_id(btf, kfunc_id);
		if (!btf_is_func(t))
			continue;
		kfunc = btf__name_by_offset(btf, t->name_off);
		printf("[%-6d] %-42s ", kfunc_id, kfunc);
		for (i = 0; i < ARRAY_SIZE(program_types); ++i) {
			ret = libbpf_probe_bpf_kfunc(program_types[i].code, kfunc_id, -1, NULL);
			if (ret < 0)
				printf("%-8d  ", ret);
			else if (ret == 0)
				printf("%8s  ", "");
			else
				printf("%8s  ", program_types[i].name);
		}
		printf("\n");
	}
	btf__free(btf);
}

----------------------------------------------------- >8 ---

Tao Chen Feb. 20, 2025, 6:09 p.m. UTC | #3

在 2025/2/19 06:51, Eduard Zingerman 写道:
> On Mon, 2025-02-17 at 13:21 +0800, Tao Chen wrote:
>> 在 2025/2/12 23:39, Tao Chen 写道:
>>> More and more kfunc functions are being added to the kernel.
>>> Different prog types have different restrictions when using kfunc.
>>> Therefore, prog_kfunc probe is added to check whether it is supported,
>>> and the use of this api will be added to bpftool later.
>>>
>>> Change list:
>>> - v6 -> v7:
>>>     - wrap err with libbpf_err
>>>     - comments fix
>>>     - handle btf_fd < 0 as vmlinux
>>>     - patchset Reviewed-by: Jiri Olsa <[e-mail]>
>>> - v6
>>>     https://lore.kernel.org/bpf/[e-mail]
>>>
>>> - v5 -> v6:
>>>     - remove fd_array_cnt
>>>     - test case clean code
>>> - v5
>>>     https://lore.kernel.org/bpf/[e-mail]
>>>
>>> - v4 -> v5:
>>>     - use fd_array on stack
>>>     - declare the scope of use of btf_fd
>>> - v4
>>>     https://lore.kernel.org/bpf/[e-mail]/
>>>
>>> - v3 -> v4:
>>>     - add fd_array init for kfunc in mod btf
>>>     - add test case for kfunc in mod btf
>>>     - refactor common part as prog load type check for
>>>       libbpf_probe_bpf_{helper,kfunc}
>>> - v3
>>>     https://lore.kernel.org/bpf/[e-mail]
>>>
>>> - v2 -> v3:
>>>     - rename parameter off with btf_fd
>>>     - extract the common part for libbpf_probe_bpf_{helper,kfunc}
>>> - v2
>>>     https://lore.kernel.org/bpf/[e-mail]
>>>
>>> - v1 -> v2:
>>>     - check unsupported prog type like probe_bpf_helper
>>>     - add off parameter for module btf
>>>     - check verifier info when kfunc id invalid
>>> - v1
>>>     https://lore.kernel.org/bpf/[e-mail]
>>>
>>> Tao Chen (4):
>>>     libbpf: Extract prog load type check from libbpf_probe_bpf_helper
>>>     libbpf: Init fd_array when prog probe load
>>>     libbpf: Add libbpf_probe_bpf_kfunc API
>>>     selftests/bpf: Add libbpf_probe_bpf_kfunc API selftests
>>>
>>>    tools/lib/bpf/libbpf.h                        |  19 ++-
>>>    tools/lib/bpf/libbpf.map                      |   1 +
>>>    tools/lib/bpf/libbpf_probes.c                 |  86 +++++++++++---
>>>    .../selftests/bpf/prog_tests/libbpf_probes.c  | 111 ++++++++++++++++++
>>>    4 files changed, 201 insertions(+), 16 deletions(-)
>>>
>>
>> Ping...
>>
>> Hi Andrii, Eduard,
>>
>> I've revised the previous suggestions. Please review it again. Thanks.
>>
> 
> I tried the test enumerating all kfuncs in BTF and doing
> libbpf_probe_bpf_kfunc for BPF_PROG_TYPE_{KPROBE,XDP}.
> (Source code at the end of the email).
> 
> The set of kfuncs returned for XDP looks correct.
> The set of kfuncs returned for KPROBE contains a few incorrect entries:
> - bpf_xdp_metadata_rx_hash
> - bpf_xdp_metadata_rx_timestamp
> - bpf_xdp_metadata_rx_vlan_tag
> 
> This is because of a different string reported by verifier for these
> three functions.
> 
> Ideally, I'd write some script looking for
> register_btf_kfunc_id_set(BPF_PROG_TYPE_***, kfunc_set)
> calls in the kernel source code and extracting the prog type /
> functions in the set, and comparing results of this script with
> output of the test below for all program types.
> But up to you if you'd like to do such rigorous verification or not.
> 
> Otherwise patch-set looks good to me, for all patch-set:
> 
> Reviewed-by: Eduard Zingerman <[e-mail]>
> 

Hi Eduard,

I try to run your test case, but it seems btf_is_decl_tag always return 
false, Are there any special restrictions for the tag feature of btf？

My compilation environment：

pahole --version
v1.29
clang --version
Ubuntu clang version 18.1.3 (1ubuntu1)

> --- 8< -----------------------------------------------------
> 
> static const struct {
> 	const char *name;
> 	int code;
> } program_types[] = {
> #define _T(n) { #n, BPF_PROG_TYPE_ ## n }
> 	_T(KPROBE),
> 	_T(XDP),
> #undef _T
> };
> 
> void test_libbpf_probe_kfuncs_many(void)
> {
> 	int i, kfunc_id, ret, id;
> 	const struct btf_type *t;
> 	struct btf *btf = NULL;
> 	const char *kfunc;
> 	const char *tag;
> 
> 	btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
> 	if (!ASSERT_OK_PTR(btf, "btf_parse"))
> 		return;
> 
> 	for (id = 0; id < btf__type_cnt(btf); ++id) {
> 		t = btf__type_by_id(btf, id);
> 		if (!btf_is_decl_tag(t))
> 			continue;
> 		tag = btf__name_by_offset(btf, t->name_off);
> 		if (strcmp(tag, "bpf_kfunc") != 0)
> 			continue;
> 		kfunc_id = t->type;
> 		t = btf__type_by_id(btf, kfunc_id);
> 		if (!btf_is_func(t))
> 			continue;
> 		kfunc = btf__name_by_offset(btf, t->name_off);
> 		printf("[%-6d] %-42s ", kfunc_id, kfunc);
> 		for (i = 0; i < ARRAY_SIZE(program_types); ++i) {
> 			ret = libbpf_probe_bpf_kfunc(program_types[i].code, kfunc_id, -1, NULL);
> 			if (ret < 0)
> 				printf("%-8d  ", ret);
> 			else if (ret == 0)
> 				printf("%8s  ", "");
> 			else
> 				printf("%8s  ", program_types[i].name);
> 		}
> 		printf("\n");
> 	}
> 	btf__free(btf);
> }
> 
> ----------------------------------------------------- >8 ---
>

Eduard Zingerman Feb. 20, 2025, 6:43 p.m. UTC | #4

On Fri, 2025-02-21 at 02:09 +0800, Tao Chen wrote:

[...]

> Hi Eduard,
> 
> I try to run your test case, but it seems btf_is_decl_tag always return 
> false, Are there any special restrictions for the tag feature of btf？

Hi Tao,
 
> My compilation environment：
> 
> pahole --version
> v1.29
> clang --version
> Ubuntu clang version 18.1.3 (1ubuntu1)

Hm, pahole should generate kfunc tags since 1.27.
I use pahole 'next' branch, but it is the same as 1.29 at the moment.
Do you see kfunc prototypes at the bottom of vmlinux.h?
They look like so:

  ...
  extern u32 tcp_reno_undo_cwnd(struct sock *sk) __weak __ksym;
  ...

These are generated by bpftool from decl tags I look for in the test case.
Decl tags are inserted by pahole, see btf_encoder.c:btf_encoder__tag_kfuncs().

Anyways, below is the list of all kfuncs from my config,
it is possible to adapt the test case with something like this:

        for (i = 0; i < ARRAY_SIZE(all_kfuncs); ++i) {
                kfunc = all_kfuncs[i];
                kfunc_id = btf__find_by_name_kind(vmlinux_btf, kfunc, BTF_KIND_FUNC);
                printf("%-42s ", kfunc);
                if (kfunc_id < 0) {
                        printf("<not found>\n");
                        continue;
                }
                ...
        }

--- 8< --------------------------------------

static const char *all_kfuncs[] = {
	"bbr_cwnd_event",
	"bbr_init",
	"bbr_main",
	"bbr_min_tso_segs",
	"bbr_set_state",
	"bbr_sndbuf_expand",
	"bbr_ssthresh",
	"bbr_undo_cwnd",
	"bpf_arena_alloc_pages",
	"bpf_arena_free_pages",
	"bpf_cast_to_kern_ctx",
	"bpf_cgroup_acquire",
	"bpf_cgroup_ancestor",
	"bpf_cgroup_from_id",
	"bpf_cgroup_release",
	"bpf_copy_from_user_str",
	"bpf_cpumask_acquire",
	"bpf_cpumask_and",
	"bpf_cpumask_any_and_distribute",
	"bpf_cpumask_any_distribute",
	"bpf_cpumask_clear",
	"bpf_cpumask_clear_cpu",
	"bpf_cpumask_copy",
	"bpf_cpumask_create",
	"bpf_cpumask_empty",
	"bpf_cpumask_equal",
	"bpf_cpumask_first",
	"bpf_cpumask_first_and",
	"bpf_cpumask_first_zero",
	"bpf_cpumask_full",
	"bpf_cpumask_intersects",
	"bpf_cpumask_or",
	"bpf_cpumask_release",
	"bpf_cpumask_set_cpu",
	"bpf_cpumask_setall",
	"bpf_cpumask_subset",
	"bpf_cpumask_test_and_clear_cpu",
	"bpf_cpumask_test_and_set_cpu",
	"bpf_cpumask_test_cpu",
	"bpf_cpumask_weight",
	"bpf_cpumask_xor",
	"bpf_crypto_ctx_acquire",
	"bpf_crypto_ctx_create",
	"bpf_crypto_ctx_release",
	"bpf_crypto_decrypt",
	"bpf_crypto_encrypt",
	"bpf_ct_change_status",
	"bpf_ct_change_timeout",
	"bpf_ct_insert_entry",
	"bpf_ct_release",
	"bpf_ct_set_nat_info",
	"bpf_ct_set_status",
	"bpf_ct_set_timeout",
	"bpf_dynptr_adjust",
	"bpf_dynptr_clone",
	"bpf_dynptr_from_skb",
	"bpf_dynptr_from_xdp",
	"bpf_dynptr_is_null",
	"bpf_dynptr_is_rdonly",
	"bpf_dynptr_size",
	"bpf_dynptr_slice",
	"bpf_dynptr_slice_rdwr",
	"bpf_fentry_test1",
	"bpf_get_dentry_xattr",
	"bpf_get_file_xattr",
	"bpf_get_fsverity_digest",
	"bpf_get_kmem_cache",
	"bpf_get_task_exe_file",
	"bpf_iter_bits_destroy",
	"bpf_iter_bits_new",
	"bpf_iter_bits_next",
	"bpf_iter_css_destroy",
	"bpf_iter_css_new",
	"bpf_iter_css_next",
	"bpf_iter_css_task_destroy",
	"bpf_iter_css_task_new",
	"bpf_iter_css_task_next",
	"bpf_iter_kmem_cache_destroy",
	"bpf_iter_kmem_cache_new",
	"bpf_iter_kmem_cache_next",
	"bpf_iter_num_destroy",
	"bpf_iter_num_new",
	"bpf_iter_num_next",
	"bpf_iter_scx_dsq_destroy",
	"bpf_iter_scx_dsq_new",
	"bpf_iter_scx_dsq_next",
	"bpf_iter_task_destroy",
	"bpf_iter_task_new",
	"bpf_iter_task_next",
	"bpf_iter_task_vma_destroy",
	"bpf_iter_task_vma_new",
	"bpf_iter_task_vma_next",
	"bpf_key_put",
	"bpf_kfunc_call_memb_release",
	"bpf_kfunc_call_test_release",
	"bpf_list_pop_back",
	"bpf_list_pop_front",
	"bpf_list_push_back_impl",
	"bpf_list_push_front_impl",
	"bpf_local_irq_restore",
	"bpf_local_irq_save",
	"bpf_lookup_system_key",
	"bpf_lookup_user_key",
	"bpf_map_sum_elem_count",
	"bpf_modify_return_test",
	"bpf_modify_return_test2",
	"bpf_modify_return_test_tp",
	"bpf_obj_drop_impl",
	"bpf_obj_new_impl",
	"bpf_path_d_path",
	"bpf_percpu_obj_drop_impl",
	"bpf_percpu_obj_new_impl",
	"bpf_preempt_disable",
	"bpf_preempt_enable",
	"bpf_put_file",
	"bpf_rbtree_add_impl",
	"bpf_rbtree_first",
	"bpf_rbtree_remove",
	"bpf_rcu_read_lock",
	"bpf_rcu_read_unlock",
	"bpf_rdonly_cast",
	"bpf_refcount_acquire_impl",
	"bpf_remove_dentry_xattr",
	"bpf_send_signal_task",
	"bpf_session_cookie",
	"bpf_session_is_return",
	"bpf_set_dentry_xattr",
	"bpf_sk_assign_tcp_reqsk",
	"bpf_skb_ct_alloc",
	"bpf_skb_ct_lookup",
	"bpf_skb_get_fou_encap",
	"bpf_skb_get_xfrm_info",
	"bpf_skb_set_fou_encap",
	"bpf_skb_set_xfrm_info",
	"bpf_sock_addr_set_sun_path",
	"bpf_sock_destroy",
	"bpf_task_acquire",
	"bpf_task_from_pid",
	"bpf_task_from_vpid",
	"bpf_task_get_cgroup1",
	"bpf_task_release",
	"bpf_task_under_cgroup",
	"bpf_throw",
	"bpf_verify_pkcs7_signature",
	"bpf_wq_init",
	"bpf_wq_set_callback_impl",
	"bpf_wq_start",
	"bpf_xdp_ct_alloc",
	"bpf_xdp_ct_lookup",
	"bpf_xdp_flow_lookup",
	"bpf_xdp_get_xfrm_state",
	"bpf_xdp_metadata_rx_hash",
	"bpf_xdp_metadata_rx_timestamp",
	"bpf_xdp_metadata_rx_vlan_tag",
	"bpf_xdp_xfrm_state_release",
	"cgroup_rstat_flush",
	"cgroup_rstat_updated",
	"crash_kexec",
	"cubictcp_acked",
	"cubictcp_cong_avoid",
	"cubictcp_cwnd_event",
	"cubictcp_init",
	"cubictcp_recalc_ssthresh",
	"cubictcp_state",
	"dctcp_cwnd_event",
	"dctcp_cwnd_undo",
	"dctcp_init",
	"dctcp_ssthresh",
	"dctcp_state",
	"dctcp_update_alpha",
	"scx_bpf_consume",
	"scx_bpf_cpu_rq",
	"scx_bpf_cpuperf_cap",
	"scx_bpf_cpuperf_cur",
	"scx_bpf_cpuperf_set",
	"scx_bpf_create_dsq",
	"scx_bpf_destroy_dsq",
	"scx_bpf_dispatch",
	"scx_bpf_dispatch_cancel",
	"scx_bpf_dispatch_from_dsq",
	"scx_bpf_dispatch_from_dsq_set_slice",
	"scx_bpf_dispatch_from_dsq_set_vtime",
	"scx_bpf_dispatch_nr_slots",
	"scx_bpf_dispatch_vtime",
	"scx_bpf_dispatch_vtime_from_dsq",
	"scx_bpf_dsq_insert",
	"scx_bpf_dsq_insert_vtime",
	"scx_bpf_dsq_move",
	"scx_bpf_dsq_move_set_slice",
	"scx_bpf_dsq_move_set_vtime",
	"scx_bpf_dsq_move_to_local",
	"scx_bpf_dsq_move_vtime",
	"scx_bpf_dsq_nr_queued",
	"scx_bpf_dump_bstr",
	"scx_bpf_error_bstr",
	"scx_bpf_exit_bstr",
	"scx_bpf_get_idle_cpumask",
	"scx_bpf_get_idle_smtmask",
	"scx_bpf_get_online_cpumask",
	"scx_bpf_get_possible_cpumask",
	"scx_bpf_kick_cpu",
	"scx_bpf_now",
	"scx_bpf_nr_cpu_ids",
	"scx_bpf_pick_any_cpu",
	"scx_bpf_pick_idle_cpu",
	"scx_bpf_put_cpumask",
	"scx_bpf_put_idle_cpumask",
	"scx_bpf_reenqueue_local",
	"scx_bpf_select_cpu_dfl",
	"scx_bpf_task_cgroup",
	"scx_bpf_task_cpu",
	"scx_bpf_task_running",
	"scx_bpf_test_and_clear_cpu_idle",
	"tcp_cong_avoid_ai",
	"tcp_reno_cong_avoid",
	"tcp_reno_ssthresh",
	"tcp_reno_undo_cwnd",
	"tcp_slow_start",
};

-------------------------------------- >8 ---

[...]

Andrii Nakryiko Feb. 21, 2025, 1:07 a.m. UTC | #5

On Tue, Feb 18, 2025 at 2:51 PM Eduard Zingerman <[e-mail]> wrote:
>
> On Mon, 2025-02-17 at 13:21 +0800, Tao Chen wrote:
> > 在 2025/2/12 23:39, Tao Chen 写道:
> > > More and more kfunc functions are being added to the kernel.
> > > Different prog types have different restrictions when using kfunc.
> > > Therefore, prog_kfunc probe is added to check whether it is supported,
> > > and the use of this api will be added to bpftool later.
> > >
> > > Change list:
> > > - v6 -> v7:
> > >    - wrap err with libbpf_err
> > >    - comments fix
> > >    - handle btf_fd < 0 as vmlinux
> > >    - patchset Reviewed-by: Jiri Olsa <[e-mail]>
> > > - v6
> > >    https://lore.kernel.org/bpf/[e-mail]
> > >
> > > - v5 -> v6:
> > >    - remove fd_array_cnt
> > >    - test case clean code
> > > - v5
> > >    https://lore.kernel.org/bpf/[e-mail]
> > >
> > > - v4 -> v5:
> > >    - use fd_array on stack
> > >    - declare the scope of use of btf_fd
> > > - v4
> > >    https://lore.kernel.org/bpf/[e-mail]/
> > >
> > > - v3 -> v4:
> > >    - add fd_array init for kfunc in mod btf
> > >    - add test case for kfunc in mod btf
> > >    - refactor common part as prog load type check for
> > >      libbpf_probe_bpf_{helper,kfunc}
> > > - v3
> > >    https://lore.kernel.org/bpf/[e-mail]
> > >
> > > - v2 -> v3:
> > >    - rename parameter off with btf_fd
> > >    - extract the common part for libbpf_probe_bpf_{helper,kfunc}
> > > - v2
> > >    https://lore.kernel.org/bpf/[e-mail]
> > >
> > > - v1 -> v2:
> > >    - check unsupported prog type like probe_bpf_helper
> > >    - add off parameter for module btf
> > >    - check verifier info when kfunc id invalid
> > > - v1
> > >    https://lore.kernel.org/bpf/[e-mail]
> > >
> > > Tao Chen (4):
> > >    libbpf: Extract prog load type check from libbpf_probe_bpf_helper
> > >    libbpf: Init fd_array when prog probe load
> > >    libbpf: Add libbpf_probe_bpf_kfunc API
> > >    selftests/bpf: Add libbpf_probe_bpf_kfunc API selftests
> > >
> > >   tools/lib/bpf/libbpf.h                        |  19 ++-
> > >   tools/lib/bpf/libbpf.map                      |   1 +
> > >   tools/lib/bpf/libbpf_probes.c                 |  86 +++++++++++---
> > >   .../selftests/bpf/prog_tests/libbpf_probes.c  | 111 ++++++++++++++++++
> > >   4 files changed, 201 insertions(+), 16 deletions(-)
> > >
> >
> > Ping...
> >
> > Hi Andrii, Eduard,
> >
> > I've revised the previous suggestions. Please review it again. Thanks.
> >
>
> I tried the test enumerating all kfuncs in BTF and doing
> libbpf_probe_bpf_kfunc for BPF_PROG_TYPE_{KPROBE,XDP}.
> (Source code at the end of the email).
>
> The set of kfuncs returned for XDP looks correct.
> The set of kfuncs returned for KPROBE contains a few incorrect entries:
> - bpf_xdp_metadata_rx_hash
> - bpf_xdp_metadata_rx_timestamp
> - bpf_xdp_metadata_rx_vlan_tag
>
> This is because of a different string reported by verifier for these
> three functions.
>
> Ideally, I'd write some script looking for
> register_btf_kfunc_id_set(BPF_PROG_TYPE_***, kfunc_set)
> calls in the kernel source code and extracting the prog type /
> functions in the set, and comparing results of this script with
> output of the test below for all program types.
> But up to you if you'd like to do such rigorous verification or not.
>
> Otherwise patch-set looks good to me, for all patch-set:
>
> Reviewed-by: Eduard Zingerman <[e-mail]>

Shouldn't we fix the issue with those bpf_xdp_metadata_* kfuncs? Do
you have details on what different string verifier reports?

>
> --- 8< -----------------------------------------------------
>
> static const struct {
>         const char *name;
>         int code;
> } program_types[] = {
> #define _T(n) { #n, BPF_PROG_TYPE_ ## n }
>         _T(KPROBE),
>         _T(XDP),
> #undef _T
> };
>
> void test_libbpf_probe_kfuncs_many(void)
> {
>         int i, kfunc_id, ret, id;
>         const struct btf_type *t;
>         struct btf *btf = NULL;
>         const char *kfunc;
>         const char *tag;
>
>         btf = btf__parse("/sys/kernel/btf/vmlinux", NULL);
>         if (!ASSERT_OK_PTR(btf, "btf_parse"))
>                 return;
>
>         for (id = 0; id < btf__type_cnt(btf); ++id) {
>                 t = btf__type_by_id(btf, id);
>                 if (!btf_is_decl_tag(t))
>                         continue;
>                 tag = btf__name_by_offset(btf, t->name_off);
>                 if (strcmp(tag, "bpf_kfunc") != 0)
>                         continue;
>                 kfunc_id = t->type;
>                 t = btf__type_by_id(btf, kfunc_id);
>                 if (!btf_is_func(t))
>                         continue;
>                 kfunc = btf__name_by_offset(btf, t->name_off);
>                 printf("[%-6d] %-42s ", kfunc_id, kfunc);
>                 for (i = 0; i < ARRAY_SIZE(program_types); ++i) {
>                         ret = libbpf_probe_bpf_kfunc(program_types[i].code, kfunc_id, -1, NULL);
>                         if (ret < 0)
>                                 printf("%-8d  ", ret);
>                         else if (ret == 0)
>                                 printf("%8s  ", "");
>                         else
>                                 printf("%8s  ", program_types[i].name);
>                 }
>                 printf("\n");
>         }
>         btf__free(btf);
> }
>
> ----------------------------------------------------- >8 ---
>

Eduard Zingerman Feb. 21, 2025, 1:11 a.m. UTC | #6

On Thu, 2025-02-20 at 17:07 -0800, Andrii Nakryiko wrote:
> On Tue, Feb 18, 2025 at 2:51 PM Eduard Zingerman <[e-mail]> wrote:
> > 
> > On Mon, 2025-02-17 at 13:21 +0800, Tao Chen wrote:

[...]

> > I tried the test enumerating all kfuncs in BTF and doing
> > libbpf_probe_bpf_kfunc for BPF_PROG_TYPE_{KPROBE,XDP}.
> > (Source code at the end of the email).
> > 
> > The set of kfuncs returned for XDP looks correct.
> > The set of kfuncs returned for KPROBE contains a few incorrect entries:
> > - bpf_xdp_metadata_rx_hash
> > - bpf_xdp_metadata_rx_timestamp
> > - bpf_xdp_metadata_rx_vlan_tag
> > 
> > This is because of a different string reported by verifier for these
> > three functions.
> > 
> > Ideally, I'd write some script looking for
> > register_btf_kfunc_id_set(BPF_PROG_TYPE_***, kfunc_set)
> > calls in the kernel source code and extracting the prog type /
> > functions in the set, and comparing results of this script with
> > output of the test below for all program types.
> > But up to you if you'd like to do such rigorous verification or not.
> > 
> > Otherwise patch-set looks good to me, for all patch-set:
> > 
> > Reviewed-by: Eduard Zingerman <[e-mail]>
> 
> Shouldn't we fix the issue with those bpf_xdp_metadata_* kfuncs? Do

I assume Tao would post a v8 with the fix.

> you have details on what different string verifier reports?

The string is "metadata kfuncs require device-bound program\n".

[...]

Tao Chen Feb. 21, 2025, 10:03 a.m. UTC | #7

在 2025/2/21 02:43, Eduard Zingerman 写道:
> On Fri, 2025-02-21 at 02:09 +0800, Tao Chen wrote:
> 
> [...]
> 
>> Hi Eduard,
>>
>> I try to run your test case, but it seems btf_is_decl_tag always return
>> false, Are there any special restrictions for the tag feature of btf？
> 
> Hi Tao,
>   
>> My compilation environment：
>>
>> pahole --version
>> v1.29
>> clang --version
>> Ubuntu clang version 18.1.3 (1ubuntu1)
> 
> Hm, pahole should generate kfunc tags since 1.27.
> I use pahole 'next' branch, but it is the same as 1.29 at the moment.
> Do you see kfunc prototypes at the bottom of vmlinux.h?
> They look like so:
> 
>    ...
>    extern u32 tcp_reno_undo_cwnd(struct sock *sk) __weak __ksym;
>    ...
> 
> These are generated by bpftool from decl tags I look for in the test case.
> Decl tags are inserted by pahole, see btf_encoder.c:btf_encoder__tag_kfuncs().
> 

It's all right now, when i use make 
PAHOLE=/home/dylane/sdb/dwarves/build/pahole -j4, thanks.

> Anyways, below is the list of all kfuncs from my config,
> it is possible to adapt the test case with something like this:
> 
>          for (i = 0; i < ARRAY_SIZE(all_kfuncs); ++i) {
>                  kfunc = all_kfuncs[i];
>                  kfunc_id = btf__find_by_name_kind(vmlinux_btf, kfunc, BTF_KIND_FUNC);
>                  printf("%-42s ", kfunc);
>                  if (kfunc_id < 0) {
>                          printf("<not found>\n");
>                          continue;
>                  }
>                  ...
>          }
> 

Well, i try it.

> --- 8< --------------------------------------
> 
> static const char *all_kfuncs[] = {
> 	"bbr_cwnd_event",
> 	"bbr_init",
> 	"bbr_main",
> 	"bbr_min_tso_segs",
> 	"bbr_set_state",
> 	"bbr_sndbuf_expand",
> 	"bbr_ssthresh",
> 	"bbr_undo_cwnd",
> 	"bpf_arena_alloc_pages",
> 	"bpf_arena_free_pages",
> 	"bpf_cast_to_kern_ctx",
> 	"bpf_cgroup_acquire",
> 	"bpf_cgroup_ancestor",
> 	"bpf_cgroup_from_id",
> 	"bpf_cgroup_release",
> 	"bpf_copy_from_user_str",
> 	"bpf_cpumask_acquire",
> 	"bpf_cpumask_and",
> 	"bpf_cpumask_any_and_distribute",
> 	"bpf_cpumask_any_distribute",
> 	"bpf_cpumask_clear",
> 	"bpf_cpumask_clear_cpu",
> 	"bpf_cpumask_copy",
> 	"bpf_cpumask_create",
> 	"bpf_cpumask_empty",
> 	"bpf_cpumask_equal",
> 	"bpf_cpumask_first",
> 	"bpf_cpumask_first_and",
> 	"bpf_cpumask_first_zero",
> 	"bpf_cpumask_full",
> 	"bpf_cpumask_intersects",
> 	"bpf_cpumask_or",
> 	"bpf_cpumask_release",
> 	"bpf_cpumask_set_cpu",
> 	"bpf_cpumask_setall",
> 	"bpf_cpumask_subset",
> 	"bpf_cpumask_test_and_clear_cpu",
> 	"bpf_cpumask_test_and_set_cpu",
> 	"bpf_cpumask_test_cpu",
> 	"bpf_cpumask_weight",
> 	"bpf_cpumask_xor",
> 	"bpf_crypto_ctx_acquire",
> 	"bpf_crypto_ctx_create",
> 	"bpf_crypto_ctx_release",
> 	"bpf_crypto_decrypt",
> 	"bpf_crypto_encrypt",
> 	"bpf_ct_change_status",
> 	"bpf_ct_change_timeout",
> 	"bpf_ct_insert_entry",
> 	"bpf_ct_release",
> 	"bpf_ct_set_nat_info",
> 	"bpf_ct_set_status",
> 	"bpf_ct_set_timeout",
> 	"bpf_dynptr_adjust",
> 	"bpf_dynptr_clone",
> 	"bpf_dynptr_from_skb",
> 	"bpf_dynptr_from_xdp",
> 	"bpf_dynptr_is_null",
> 	"bpf_dynptr_is_rdonly",
> 	"bpf_dynptr_size",
> 	"bpf_dynptr_slice",
> 	"bpf_dynptr_slice_rdwr",
> 	"bpf_fentry_test1",
> 	"bpf_get_dentry_xattr",
> 	"bpf_get_file_xattr",
> 	"bpf_get_fsverity_digest",
> 	"bpf_get_kmem_cache",
> 	"bpf_get_task_exe_file",
> 	"bpf_iter_bits_destroy",
> 	"bpf_iter_bits_new",
> 	"bpf_iter_bits_next",
> 	"bpf_iter_css_destroy",
> 	"bpf_iter_css_new",
> 	"bpf_iter_css_next",
> 	"bpf_iter_css_task_destroy",
> 	"bpf_iter_css_task_new",
> 	"bpf_iter_css_task_next",
> 	"bpf_iter_kmem_cache_destroy",
> 	"bpf_iter_kmem_cache_new",
> 	"bpf_iter_kmem_cache_next",
> 	"bpf_iter_num_destroy",
> 	"bpf_iter_num_new",
> 	"bpf_iter_num_next",
> 	"bpf_iter_scx_dsq_destroy",
> 	"bpf_iter_scx_dsq_new",
> 	"bpf_iter_scx_dsq_next",
> 	"bpf_iter_task_destroy",
> 	"bpf_iter_task_new",
> 	"bpf_iter_task_next",
> 	"bpf_iter_task_vma_destroy",
> 	"bpf_iter_task_vma_new",
> 	"bpf_iter_task_vma_next",
> 	"bpf_key_put",
> 	"bpf_kfunc_call_memb_release",
> 	"bpf_kfunc_call_test_release",
> 	"bpf_list_pop_back",
> 	"bpf_list_pop_front",
> 	"bpf_list_push_back_impl",
> 	"bpf_list_push_front_impl",
> 	"bpf_local_irq_restore",
> 	"bpf_local_irq_save",
> 	"bpf_lookup_system_key",
> 	"bpf_lookup_user_key",
> 	"bpf_map_sum_elem_count",
> 	"bpf_modify_return_test",
> 	"bpf_modify_return_test2",
> 	"bpf_modify_return_test_tp",
> 	"bpf_obj_drop_impl",
> 	"bpf_obj_new_impl",
> 	"bpf_path_d_path",
> 	"bpf_percpu_obj_drop_impl",
> 	"bpf_percpu_obj_new_impl",
> 	"bpf_preempt_disable",
> 	"bpf_preempt_enable",
> 	"bpf_put_file",
> 	"bpf_rbtree_add_impl",
> 	"bpf_rbtree_first",
> 	"bpf_rbtree_remove",
> 	"bpf_rcu_read_lock",
> 	"bpf_rcu_read_unlock",
> 	"bpf_rdonly_cast",
> 	"bpf_refcount_acquire_impl",
> 	"bpf_remove_dentry_xattr",
> 	"bpf_send_signal_task",
> 	"bpf_session_cookie",
> 	"bpf_session_is_return",
> 	"bpf_set_dentry_xattr",
> 	"bpf_sk_assign_tcp_reqsk",
> 	"bpf_skb_ct_alloc",
> 	"bpf_skb_ct_lookup",
> 	"bpf_skb_get_fou_encap",
> 	"bpf_skb_get_xfrm_info",
> 	"bpf_skb_set_fou_encap",
> 	"bpf_skb_set_xfrm_info",
> 	"bpf_sock_addr_set_sun_path",
> 	"bpf_sock_destroy",
> 	"bpf_task_acquire",
> 	"bpf_task_from_pid",
> 	"bpf_task_from_vpid",
> 	"bpf_task_get_cgroup1",
> 	"bpf_task_release",
> 	"bpf_task_under_cgroup",
> 	"bpf_throw",
> 	"bpf_verify_pkcs7_signature",
> 	"bpf_wq_init",
> 	"bpf_wq_set_callback_impl",
> 	"bpf_wq_start",
> 	"bpf_xdp_ct_alloc",
> 	"bpf_xdp_ct_lookup",
> 	"bpf_xdp_flow_lookup",
> 	"bpf_xdp_get_xfrm_state",
> 	"bpf_xdp_metadata_rx_hash",
> 	"bpf_xdp_metadata_rx_timestamp",
> 	"bpf_xdp_metadata_rx_vlan_tag",
> 	"bpf_xdp_xfrm_state_release",
> 	"cgroup_rstat_flush",
> 	"cgroup_rstat_updated",
> 	"crash_kexec",
> 	"cubictcp_acked",
> 	"cubictcp_cong_avoid",
> 	"cubictcp_cwnd_event",
> 	"cubictcp_init",
> 	"cubictcp_recalc_ssthresh",
> 	"cubictcp_state",
> 	"dctcp_cwnd_event",
> 	"dctcp_cwnd_undo",
> 	"dctcp_init",
> 	"dctcp_ssthresh",
> 	"dctcp_state",
> 	"dctcp_update_alpha",
> 	"scx_bpf_consume",
> 	"scx_bpf_cpu_rq",
> 	"scx_bpf_cpuperf_cap",
> 	"scx_bpf_cpuperf_cur",
> 	"scx_bpf_cpuperf_set",
> 	"scx_bpf_create_dsq",
> 	"scx_bpf_destroy_dsq",
> 	"scx_bpf_dispatch",
> 	"scx_bpf_dispatch_cancel",
> 	"scx_bpf_dispatch_from_dsq",
> 	"scx_bpf_dispatch_from_dsq_set_slice",
> 	"scx_bpf_dispatch_from_dsq_set_vtime",
> 	"scx_bpf_dispatch_nr_slots",
> 	"scx_bpf_dispatch_vtime",
> 	"scx_bpf_dispatch_vtime_from_dsq",
> 	"scx_bpf_dsq_insert",
> 	"scx_bpf_dsq_insert_vtime",
> 	"scx_bpf_dsq_move",
> 	"scx_bpf_dsq_move_set_slice",
> 	"scx_bpf_dsq_move_set_vtime",
> 	"scx_bpf_dsq_move_to_local",
> 	"scx_bpf_dsq_move_vtime",
> 	"scx_bpf_dsq_nr_queued",
> 	"scx_bpf_dump_bstr",
> 	"scx_bpf_error_bstr",
> 	"scx_bpf_exit_bstr",
> 	"scx_bpf_get_idle_cpumask",
> 	"scx_bpf_get_idle_smtmask",
> 	"scx_bpf_get_online_cpumask",
> 	"scx_bpf_get_possible_cpumask",
> 	"scx_bpf_kick_cpu",
> 	"scx_bpf_now",
> 	"scx_bpf_nr_cpu_ids",
> 	"scx_bpf_pick_any_cpu",
> 	"scx_bpf_pick_idle_cpu",
> 	"scx_bpf_put_cpumask",
> 	"scx_bpf_put_idle_cpumask",
> 	"scx_bpf_reenqueue_local",
> 	"scx_bpf_select_cpu_dfl",
> 	"scx_bpf_task_cgroup",
> 	"scx_bpf_task_cpu",
> 	"scx_bpf_task_running",
> 	"scx_bpf_test_and_clear_cpu_idle",
> 	"tcp_cong_avoid_ai",
> 	"tcp_reno_cong_avoid",
> 	"tcp_reno_ssthresh",
> 	"tcp_reno_undo_cwnd",
> 	"tcp_slow_start",
> };
> 
> -------------------------------------- >8 ---
> 
> [...]
>

Tao Chen Feb. 21, 2025, 10:03 a.m. UTC | #8

在 2025/2/21 09:11, Eduard Zingerman 写道:
> On Thu, 2025-02-20 at 17:07 -0800, Andrii Nakryiko wrote:
>> On Tue, Feb 18, 2025 at 2:51 PM Eduard Zingerman <[e-mail]> wrote:
>>>
>>> On Mon, 2025-02-17 at 13:21 +0800, Tao Chen wrote:
> 
> [...]
> 
>>> I tried the test enumerating all kfuncs in BTF and doing
>>> libbpf_probe_bpf_kfunc for BPF_PROG_TYPE_{KPROBE,XDP}.
>>> (Source code at the end of the email).
>>>
>>> The set of kfuncs returned for XDP looks correct.
>>> The set of kfuncs returned for KPROBE contains a few incorrect entries:
>>> - bpf_xdp_metadata_rx_hash
>>> - bpf_xdp_metadata_rx_timestamp
>>> - bpf_xdp_metadata_rx_vlan_tag
>>>
>>> This is because of a different string reported by verifier for these
>>> three functions.
>>>
>>> Ideally, I'd write some script looking for
>>> register_btf_kfunc_id_set(BPF_PROG_TYPE_***, kfunc_set)
>>> calls in the kernel source code and extracting the prog type /
>>> functions in the set, and comparing results of this script with
>>> output of the test below for all program types.
>>> But up to you if you'd like to do such rigorous verification or not.
>>>
>>> Otherwise patch-set looks good to me, for all patch-set:
>>>
>>> Reviewed-by: Eduard Zingerman <[e-mail]>
>>
>> Shouldn't we fix the issue with those bpf_xdp_metadata_* kfuncs? Do
> 
> I assume Tao would post a v8 with the fix.
> 

Sure, will fix it.

>> you have details on what different string verifier reports?
> 
> The string is "metadata kfuncs require device-bound program\n".
> 
> [...]
>