| Message ID | 20250105124403.991-2-laoar.shao@gmail.com (mailing list archive) |
|---|---|
| State | RFC |
| Delegated to: | BPF |
| Headers | show |
| Series | libbpf: Add support for dynamic tracepoint | expand |
On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > Dynamic tracepoints can be created using debugfs. For example: > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > This command creates a new tracepoint under debugfs: > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > enable filter format hist id trigger > > Although this dynamic tracepoint appears as a tracepoint, it is internally > implemented as a kprobe. However, it must be attached as a tracepoint to > function correctly in certain contexts. Nack. There are multiple mechanisms to create kprobe/tp via text interfaces. We're not going to mix them with the programmatic libbpf api.
On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > This command creates a new tracepoint under debugfs: > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > enable filter format hist id trigger > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > implemented as a kprobe. However, it must be attached as a tracepoint to > > function correctly in certain contexts. > > Nack. > There are multiple mechanisms to create kprobe/tp via text interfaces. > We're not going to mix them with the programmatic libbpf api. It appears that bpftrace still lacks support for adding a kprobe/tp and then attaching to it directly. Is that correct? What do you think about introducing this mechanism into bpftrace? With such a feature, we could easily attach to inlined kernel functions using bpftrace. -- Regards Yafang
On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote: > > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > This command creates a new tracepoint under debugfs: > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > enable filter format hist id trigger > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > function correctly in certain contexts. > > > > Nack. > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > We're not going to mix them with the programmatic libbpf api. > > It appears that bpftrace still lacks support for adding a kprobe/tp > and then attaching to it directly. Is that correct? what do you mean? bpftrace supports both kprobe attaching and tp too. > What do you think about introducing this mechanism into bpftrace? With > such a feature, we could easily attach to inlined kernel functions > using bpftrace. Attaching to inlined funcs also sort-of works. It relies on dwarf, and there is work in progress to add a special section to vmlinux to annotate inlined sites, so it can work without dwarf.
On Tue, Jan 7, 2025 at 6:33 AM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > > > This command creates a new tracepoint under debugfs: > > > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > > enable filter format hist id trigger > > > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > > function correctly in certain contexts. > > > > > > Nack. > > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > > We're not going to mix them with the programmatic libbpf api. > > > > It appears that bpftrace still lacks support for adding a kprobe/tp > > and then attaching to it directly. Is that correct? > > what do you mean? Take the inlined kernel function tcp_listendrop() as an example: $ perf probe -a 'tcp_listendrop sk' Added new events: probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) probe:tcp_listendrop (on tcp_listendrop with sk) You can now use it in all perf tools, such as: perf record -e probe:tcp_listendrop -aR sleep 1 Similarly, we can also use bpftrace to trace inlined kernel functions. For example: - add a dynamic tracepoint $ bpftrace probe -a 'tcp_listendrop sk' - trace the dynamic tracepoint $ bpftrace probe -e 'probe:tcp_listendrop {print(args->sk)}' > bpftrace supports both kprobe attaching and tp too. The dynamic tracepoint is not supported yet. > > > What do you think about introducing this mechanism into bpftrace? With > > such a feature, we could easily attach to inlined kernel functions > > using bpftrace. > > Attaching to inlined funcs also sort-of works. It relies on dwarf, > and there is work in progress to add a special section to vmlinux > to annotate inlined sites, so it can work without dwarf. What’s the benefit of doing this? Why not simply read the DWARF information directly from vmlinux? $ readelf -S /boot/vmlinux | grep debug_info [63] .debug_info PROGBITS 0000000000000000 03e2bc20 The DWARF information embedded in vmlinux makes it straightforward to trace inlined functions without requiring any kernel modifications. This approach allows all existing kernel releases to immediately take advantage of the functionality, eliminating the need for kernel recompilation or patching. -- Regards Yafang
On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote: > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > This command creates a new tracepoint under debugfs: > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > enable filter format hist id trigger > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > function correctly in certain contexts. > > > > Nack. > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > We're not going to mix them with the programmatic libbpf api. > > It appears that bpftrace still lacks support for adding a kprobe/tp > and then attaching to it directly. Is that correct? > What do you think about introducing this mechanism into bpftrace? With > such a feature, we could easily attach to inlined kernel functions > using bpftrace. so with the 'echo .. > kprobe_events' you create kprobe which will be exported through tracefs together with other tracepoints and bpftrace sees it as another tracepoint.. but it's a kprobe :-\ how about we add support for kprobe section like SEC("kprobe/SUBSYSTEM/PROBE"), so in your case above it'd be SEC("kprobe/kprobes/myprobe") then attach_kprobe would parse that out and use new new probe_attach_mode for bpf_program__attach_kprobe_opts to attach it correctly cc-ing Viktor jirka
On Tue, Jan 7, 2025 at 8:16 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote: > > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > > > This command creates a new tracepoint under debugfs: > > > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > > enable filter format hist id trigger > > > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > > function correctly in certain contexts. > > > > > > Nack. > > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > > We're not going to mix them with the programmatic libbpf api. > > > > It appears that bpftrace still lacks support for adding a kprobe/tp > > and then attaching to it directly. Is that correct? > > What do you think about introducing this mechanism into bpftrace? With > > such a feature, we could easily attach to inlined kernel functions > > using bpftrace. > > so with the 'echo .. > kprobe_events' you create kprobe which will be > exported through tracefs together with other tracepoints and bpftrace > sees it as another tracepoint.. but it's a kprobe :-\ exactly. > > how about we add support for kprobe section like SEC("kprobe/SUBSYSTEM/PROBE"), > so in your case above it'd be SEC("kprobe/kprobes/myprobe") This is similar to what I'm currently proposing: SEC("dynamic_tp/kprobes/my_dynamic_tp") My proposal requires only a 3-line change. In contrast, if we implement it as you suggested, it may require significantly more code changes. I prefer to introduce a new section, such as SEC("dynamic_tracepoint/"), SEC("kprobe_tracepoint/"), or something similar, for this special type of kprobe. However, if you believe SEC("kprobe/SUBSYSTEM/PROBE") is a better approach, I’m happy to implement it that way. > > then attach_kprobe would parse that out and use new new probe_attach_mode > for bpf_program__attach_kprobe_opts to attach it correctly Yes, that would be a great enhancement for tracing inlined kernel functions. -- Regards Yafang
Hi Yafang, On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote: > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > This command creates a new tracepoint under debugfs: > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > enable filter format hist id trigger > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > function correctly in certain contexts. > > > > Nack. > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > We're not going to mix them with the programmatic libbpf api. > > It appears that bpftrace still lacks support for adding a kprobe/tp > and then attaching to it directly. Is that correct? > What do you think about introducing this mechanism into bpftrace? With > such a feature, we could easily attach to inlined kernel functions > using bpftrace. Is the idea to have some other application create dynamic tracepoints based on kernel debuginfo? FWIW bpftrace has some initial support for probing inlined kernel functions w/ DWARF. I don't believe it's enabled by default yet, though - there's a few limitations. I'll comment in thread below with more details. Thanks, Daniel
On Tue, Jan 07, 2025 at 10:41:46AM +0800, Yafang Shao wrote: > On Tue, Jan 7, 2025 at 6:33 AM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote: > > > > > > > > > > Dynamic tracepoints can be created using debugfs. For example: > > > > > > > > > > echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events > > > > > > > > > > This command creates a new tracepoint under debugfs: > > > > > > > > > > $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ > > > > > enable filter format hist id trigger > > > > > > > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally > > > > > implemented as a kprobe. However, it must be attached as a tracepoint to > > > > > function correctly in certain contexts. > > > > > > > > Nack. > > > > There are multiple mechanisms to create kprobe/tp via text interfaces. > > > > We're not going to mix them with the programmatic libbpf api. > > > > > > It appears that bpftrace still lacks support for adding a kprobe/tp > > > and then attaching to it directly. Is that correct? > > > > what do you mean? > > Take the inlined kernel function tcp_listendrop() as an example: > > $ perf probe -a 'tcp_listendrop sk' > Added new events: > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > probe:tcp_listendrop (on tcp_listendrop with sk) > > You can now use it in all perf tools, such as: > > perf record -e probe:tcp_listendrop -aR sleep 1 Cool, I'm guessing perf-probe can speak DWARF and will parse all the inline information. > > Similarly, we can also use bpftrace to trace inlined kernel functions. > For example: > > - add a dynamic tracepoint > $ bpftrace probe -a 'tcp_listendrop sk' > > - trace the dynamic tracepoint > $ bpftrace probe -e 'probe:tcp_listendrop {print(args->sk)}' > > > bpftrace supports both kprobe attaching and tp too. > > The dynamic tracepoint is not supported yet. > > > > > > What do you think about introducing this mechanism into bpftrace? With > > > such a feature, we could easily attach to inlined kernel functions > > > using bpftrace. > > > > Attaching to inlined funcs also sort-of works. It relies on dwarf, > > and there is work in progress to add a special section to vmlinux > > to annotate inlined sites, so it can work without dwarf. > > What’s the benefit of doing this? Why not simply read the DWARF > information directly from vmlinux? > > $ readelf -S /boot/vmlinux | grep debug_info > [63] .debug_info PROGBITS 0000000000000000 03e2bc20 > > The DWARF information embedded in vmlinux makes it straightforward to > trace inlined functions without requiring any kernel modifications. > This approach allows all existing kernel releases to immediately take > advantage of the functionality, eliminating the need for kernel > recompilation or patching. I'd disagree that this approach works with all existing kernels. Kernel debuginfo is usually not available by default. On some distros, it's not available at all. This is particularly relevant for partial inlining - where compiler inlines some callsites but leaves the symbol in. In these cases, users trying to probe a symbol will succeed in attaching but then silently lose events. There is no obvious way for user to know to install debuginfo. Or to create dynamic tracepoints. This is the motivation for always-available metadata. Something small enough where distros can leave it on by default. Similar to motivation for BTF. There's also overhead involved w/ parsing DWARF. A more compact representation helps reduce overhead.
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 66173ddb5a2d..077bec761ebf 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -9504,6 +9504,7 @@ static const struct bpf_sec_def section_defs[] = { SEC_DEF("struct_ops.s+", STRUCT_OPS, 0, SEC_SLEEPABLE), SEC_DEF("sk_lookup", SK_LOOKUP, BPF_SK_LOOKUP, SEC_ATTACHABLE), SEC_DEF("netfilter", NETFILTER, BPF_NETFILTER, SEC_NONE), + SEC_DEF("dynamic_tp+", KPROBE, 0, SEC_NONE, attach_tp), }; int libbpf_register_prog_handler(const char *sec, @@ -12500,6 +12501,8 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin /* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */ if (str_has_pfx(prog->sec_name, "tp/")) tp_cat = sec_name + sizeof("tp/") - 1; + else if (str_has_pfx(prog->sec_name, "dynamic_tp/")) + tp_cat = sec_name + sizeof("dynamic_tp/") - 1; else tp_cat = sec_name + sizeof("tracepoint/") - 1; tp_name = strchr(tp_cat, '/');
Dynamic tracepoints can be created using debugfs. For example: echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events This command creates a new tracepoint under debugfs: $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/ enable filter format hist id trigger Although this dynamic tracepoint appears as a tracepoint, it is internally implemented as a kprobe. However, it must be attached as a tracepoint to function correctly in certain contexts. This update adds support in libbpf for handling such tracepoints, simplifying their usage and integration in BPF workflows. Signed-off-by: Yafang Shao <laoar.shao@gmail.com> --- tools/lib/bpf/libbpf.c | 3 +++ 1 file changed, 3 insertions(+)