[RFC,0/3] tracing: Support poll on event hist file

Message

Masami Hiramatsu (Google) June 25, 2024, 3:16 p.m. UTC

Hi,

Here is an RFC patch to support polling on event 'hist' file.

There has been interest in allowing user programs to monitor kernel
events in real time. Ftrace provides `trace_pipe` interface to wait
on events in the ring buffer, but it is needed to wait until filling
up a page with events in the ring buffer. We can also peek the
`trace` file periodically, but that is inefficient way to monitor
a randomely happening event.

This patch set allows user to `poll`(or `select`, `epoll`) on event
histogram interface. As you know each event has its own `hist` file
which shows histograms generated by trigger action. So user can set
a new hist trigger on any event you want to monitor, and poll on the
`hist` file until it is updated.

There are 2 poll events are supported, POLLIN and POLLPRI. POLLIN
means that there are any readable update on `hist` file and this
event will be flashed only when you call read(). So, this is
useful if you want to read the histogram periodically.
The other POLLPRI event is for monitoring trace event. Like the
POLLIN, this will be returned when the histogram is updated, but
you don't need to read() the file and use poll() again.

Note that this waits for histogram update (not event arrival), thus
you must set a histogram on the event at first.

Here is an example usage:

----
TRACEFS=/sys/kernel/tracing
EVENT=$TRACEFS/events/sched/sched_process_free

# setup histogram trigger and enable event
echo "hist:key=comm" >> $EVENT/trigger
echo 1 > $EVENT/enable

# Wait for update
poll $EVENT/hist

# Event arrived.
echo "process free event is comming"
tail $TRACEFS/trace
----

The 'poll' command is in the selftest patch.

You can take this series also from here;

https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/event-hist-poll

Thank you,

---

Masami Hiramatsu (Google) (3):
      tracing/hist: Add poll(POLLIN) support on hist file
      tracing/hist: Support POLLPRI event for poll on histogram
      selftests/tracing: Add hist poll() support test


 include/linux/trace_events.h                       |    5 +
 kernel/trace/trace_events.c                        |   18 ++++
 kernel/trace/trace_events_hist.c                   |  101 +++++++++++++++++++-
 tools/testing/selftests/ftrace/Makefile            |    3 +
 tools/testing/selftests/ftrace/poll.c              |   34 +++++++
 .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   46 +++++++++
 6 files changed, 204 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/poll.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc

--
Masami Hiramatsu (Google) <[e-mail]>

Comments

Tom Zanussi June 30, 2024, 9:07 p.m. UTC | #1

Hi Masami,

On Wed, 2024-06-26 at 00:16 +0900, Masami Hiramatsu (Google) wrote:
> Hi,
> 
> Here is an RFC patch to support polling on event 'hist' file.
> 
> There has been interest in allowing user programs to monitor kernel
> events in real time. Ftrace provides `trace_pipe` interface to wait
> on events in the ring buffer, but it is needed to wait until filling
> up a page with events in the ring buffer. We can also peek the
> `trace` file periodically, but that is inefficient way to monitor
> a randomely happening event.
> 
> This patch set allows user to `poll`(or `select`, `epoll`) on event
> histogram interface. As you know each event has its own `hist` file
> which shows histograms generated by trigger action. So user can set
> a new hist trigger on any event you want to monitor, and poll on the
> `hist` file until it is updated.
> 
> There are 2 poll events are supported, POLLIN and POLLPRI. POLLIN
> means that there are any readable update on `hist` file and this
> event will be flashed only when you call read(). So, this is
> useful if you want to read the histogram periodically.
> The other POLLPRI event is for monitoring trace event. Like the
> POLLIN, this will be returned when the histogram is updated, but
> you don't need to read() the file and use poll() again.
> 
> Note that this waits for histogram update (not event arrival), thus
> you must set a histogram on the event at first.
> 
> Here is an example usage:
> 
> ----
> TRACEFS=/sys/kernel/tracing
> EVENT=$TRACEFS/events/sched/sched_process_free
> 
> # setup histogram trigger and enable event
> echo "hist:key=comm" >> $EVENT/trigger
> echo 1 > $EVENT/enable
> 
> # Wait for update
> poll $EVENT/hist
> 
> # Event arrived.
> echo "process free event is comming"
> tail $TRACEFS/trace
> ----
> 
> The 'poll' command is in the selftest patch.
> 
> You can take this series also from here;
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/event-hist-poll
> 
> Thank you,

I think this is a clever use of the histogram files, and will be very
useful for real-time monitoring apps. I'm looking forward to using it
myself - thanks for doing this.

For the whole series,

Reviewed-by: Tom Zanussi <[e-mail]>

> 
> ---
> 
> Masami Hiramatsu (Google) (3):
>       tracing/hist: Add poll(POLLIN) support on hist file
>       tracing/hist: Support POLLPRI event for poll on histogram
>       selftests/tracing: Add hist poll() support test
> 
> 
>  include/linux/trace_events.h                       |    5 +
>  kernel/trace/trace_events.c                        |   18 ++++
>  kernel/trace/trace_events_hist.c                   |  101
> +++++++++++++++++++-
>  tools/testing/selftests/ftrace/Makefile            |    3 +
>  tools/testing/selftests/ftrace/poll.c              |   34 +++++++
>  .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   46 +++++++++
>  6 files changed, 204 insertions(+), 3 deletions(-)
>  create mode 100644 tools/testing/selftests/ftrace/poll.c
>  create mode 100644
> tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc
> 
> --
> Masami Hiramatsu (Google) <[e-mail]>