[RFC,bpf-next,1/2] libbpf: Add support for dynamic tracepoint

Commit Message

Yafang Shao Jan. 5, 2025, 12:44 p.m. UTC

Dynamic tracepoints can be created using debugfs. For example:

   echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events

This command creates a new tracepoint under debugfs:

  $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
  enable  filter  format  hist  id  trigger

Although this dynamic tracepoint appears as a tracepoint, it is internally
implemented as a kprobe. However, it must be attached as a tracepoint to
function correctly in certain contexts.

This update adds support in libbpf for handling such tracepoints,
simplifying their usage and integration in BPF workflows.

Signed-off-by: Yafang Shao <[e-mail]>
---
 tools/lib/bpf/libbpf.c | 3 +++
 1 file changed, 3 insertions(+)

Comments

Alexei Starovoitov Jan. 6, 2025, 12:16 a.m. UTC | #1

On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <[e-mail]> wrote:
>
> Dynamic tracepoints can be created using debugfs. For example:
>
>    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
>
> This command creates a new tracepoint under debugfs:
>
>   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
>   enable  filter  format  hist  id  trigger
>
> Although this dynamic tracepoint appears as a tracepoint, it is internally
> implemented as a kprobe. However, it must be attached as a tracepoint to
> function correctly in certain contexts.

Nack.
There are multiple mechanisms to create kprobe/tp via text interfaces.
We're not going to mix them with the programmatic libbpf api.

Yafang Shao Jan. 6, 2025, 2:32 a.m. UTC | #2

On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
<[e-mail]> wrote:
>
> On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <[e-mail]> wrote:
> >
> > Dynamic tracepoints can be created using debugfs. For example:
> >
> >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> >
> > This command creates a new tracepoint under debugfs:
> >
> >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> >   enable  filter  format  hist  id  trigger
> >
> > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > implemented as a kprobe. However, it must be attached as a tracepoint to
> > function correctly in certain contexts.
>
> Nack.
> There are multiple mechanisms to create kprobe/tp via text interfaces.
> We're not going to mix them with the programmatic libbpf api.

It appears that bpftrace still lacks support for adding a kprobe/tp
and then attaching to it directly. Is that correct?
What do you think about introducing this mechanism into bpftrace? With
such a feature, we could easily attach to inlined kernel functions
using bpftrace.

--
Regards
Yafang

Alexei Starovoitov Jan. 6, 2025, 10:33 p.m. UTC | #3

On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <[e-mail]> wrote:
>
> On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> <[e-mail]> wrote:
> >
> > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <[e-mail]> wrote:
> > >
> > > Dynamic tracepoints can be created using debugfs. For example:
> > >
> > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > >
> > > This command creates a new tracepoint under debugfs:
> > >
> > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > >   enable  filter  format  hist  id  trigger
> > >
> > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > function correctly in certain contexts.
> >
> > Nack.
> > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > We're not going to mix them with the programmatic libbpf api.
>
> It appears that bpftrace still lacks support for adding a kprobe/tp
> and then attaching to it directly. Is that correct?

what do you mean?
bpftrace supports both kprobe attaching and tp too.

> What do you think about introducing this mechanism into bpftrace? With
> such a feature, we could easily attach to inlined kernel functions
> using bpftrace.

Attaching to inlined funcs also sort-of works. It relies on dwarf,
and there is work in progress to add a special section to vmlinux
to annotate inlined sites, so it can work without dwarf.

Yafang Shao Jan. 7, 2025, 2:41 a.m. UTC | #4

On Tue, Jan 7, 2025 at 6:33 AM Alexei Starovoitov
<[e-mail]> wrote:
>
> On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <[e-mail]> wrote:
> >
> > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> > <[e-mail]> wrote:
> > >
> > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <[e-mail]> wrote:
> > > >
> > > > Dynamic tracepoints can be created using debugfs. For example:
> > > >
> > > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > > >
> > > > This command creates a new tracepoint under debugfs:
> > > >
> > > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > > >   enable  filter  format  hist  id  trigger
> > > >
> > > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > > function correctly in certain contexts.
> > >
> > > Nack.
> > > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > > We're not going to mix them with the programmatic libbpf api.
> >
> > It appears that bpftrace still lacks support for adding a kprobe/tp
> > and then attaching to it directly. Is that correct?
>
> what do you mean?

Take the inlined kernel function tcp_listendrop() as an example:

$ perf probe -a 'tcp_listendrop sk'
Added new events:
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)

You can now use it in all perf tools, such as:

        perf record -e probe:tcp_listendrop -aR sleep 1

Similarly, we can also use bpftrace to trace inlined kernel functions.
For example:

- add a dynamic tracepoint
  $ bpftrace probe -a 'tcp_listendrop sk'

- trace the dynamic tracepoint
  $ bpftrace probe -e 'probe:tcp_listendrop {print(args->sk)}'

> bpftrace supports both kprobe attaching and tp too.

The dynamic tracepoint is not supported yet.

>
> > What do you think about introducing this mechanism into bpftrace? With
> > such a feature, we could easily attach to inlined kernel functions
> > using bpftrace.
>
> Attaching to inlined funcs also sort-of works. It relies on dwarf,
> and there is work in progress to add a special section to vmlinux
> to annotate inlined sites, so it can work without dwarf.

What’s the benefit of doing this? Why not simply read the DWARF
information directly from vmlinux?

$ readelf -S /boot/vmlinux  | grep debug_info
  [63] .debug_info       PROGBITS         0000000000000000  03e2bc20

The DWARF information embedded in vmlinux makes it straightforward to
trace inlined functions without requiring any kernel modifications.
This approach allows all existing kernel releases to immediately take
advantage of the functionality, eliminating the need for kernel
recompilation or patching.


--
Regards
Yafang

Patch

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 66173ddb5a2d..077bec761ebf 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -9504,6 +9504,7 @@  static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("struct_ops.s+",	STRUCT_OPS, 0, SEC_SLEEPABLE),
 	SEC_DEF("sk_lookup",		SK_LOOKUP, BPF_SK_LOOKUP, SEC_ATTACHABLE),
 	SEC_DEF("netfilter",		NETFILTER, BPF_NETFILTER, SEC_NONE),
+	SEC_DEF("dynamic_tp+",          KPROBE, 0, SEC_NONE, attach_tp),
 };
 
 int libbpf_register_prog_handler(const char *sec,
@@ -12500,6 +12501,8 @@  static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 	/* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */
 	if (str_has_pfx(prog->sec_name, "tp/"))
 		tp_cat = sec_name + sizeof("tp/") - 1;
+	else if (str_has_pfx(prog->sec_name, "dynamic_tp/"))
+		tp_cat = sec_name + sizeof("dynamic_tp/") - 1;
 	else
 		tp_cat = sec_name + sizeof("tracepoint/") - 1;
 	tp_name = strchr(tp_cat, '/');