mbox series

[V3,00/11] rtla improvements

Message ID cover.1686066600.git.bristot@kernel.org (mailing list archive)
Headers show
Series rtla improvements | expand

Message

Daniel Bristot de Oliveira June 6, 2023, 4:12 p.m. UTC
This is a series of improvements for rtla, mainly as a result of our
daily usage of the tool debugging problems at red hat.

The cgroup support and house keeping options are from our usage
of the tool debugging containers.

The auto-analysis overhead reduction is needed when we go to
large boxes - but it is really hand in practice, as it gives an idea
of the problem without having to look at the trace.

Running hwnoise 100 % of CPU time might cause some systems
to slow down too much. Reduce its utilization to 75% by default to
avoid problems for people using it for the first time.

Finally, it adds support for running timerlat user-space threads,
and to collect the additional field via rtla timerlat top/hist.

Changes from V2:
  - Add timerlat hist -u option
  - Link: https://lore.kernel.org/lkml/cover.1684863094.git.bristot@kernel.org/
Changes from V1:
  - Add the user-space thread support to rtla timerlat top
  - Link: https://lore.kernel.org/lkml/cover.1683827510.git.bristot@kernel.org/

Daniel Bristot de Oliveira (11):
  rtla: Add -C cgroup support
  rtla: Add --house-keeping option
  rtla: Change monitored_cpus from char * to cpu_set_t
  rtla: Automatically move rtla to a house-keeping cpu
  rtla/timerlat: Give timerlat auto analysis its own instance
  rtla/timerlat_hist: Add auto-analysis support
  rtla: Start the tracers after creating all instances
  rtla/hwnoise: Reduce runtime to 75%
  rtla: Add timerlat user-space support for timerlat top
  rtla: Add timerlat user-space support for
  Documentation: Add tools/rtla timerlat -u option documentation

 Documentation/tools/rtla/common_options.rst   |   8 +
 .../tools/rtla/common_timerlat_aa.rst         |   7 -
 .../tools/rtla/common_timerlat_options.rst    |   7 +
 .../tools/rtla/rtla-timerlat-hist.rst         |   7 +-
 .../tools/rtla/rtla-timerlat-top.rst          |   7 +
 tools/tracing/rtla/src/osnoise.c              |  65 ++++
 tools/tracing/rtla/src/osnoise.h              |   5 +
 tools/tracing/rtla/src/osnoise_hist.c         |  90 ++++-
 tools/tracing/rtla/src/osnoise_top.c          |  83 ++++-
 tools/tracing/rtla/src/timerlat_aa.c          |  35 +-
 tools/tracing/rtla/src/timerlat_aa.h          |   5 +-
 tools/tracing/rtla/src/timerlat_hist.c        | 262 ++++++++++++--
 tools/tracing/rtla/src/timerlat_top.c         | 229 +++++++++++--
 tools/tracing/rtla/src/timerlat_u.c           | 224 ++++++++++++
 tools/tracing/rtla/src/timerlat_u.h           |  18 +
 tools/tracing/rtla/src/utils.c                | 324 +++++++++++++++++-
 tools/tracing/rtla/src/utils.h                |   7 +
 17 files changed, 1277 insertions(+), 106 deletions(-)
 create mode 100644 tools/tracing/rtla/src/timerlat_u.c
 create mode 100644 tools/tracing/rtla/src/timerlat_u.h

Comments

Juri Lelli June 12, 2023, 9:41 a.m. UTC | #1
Hi,

On 06/06/23 18:12, Daniel Bristot de Oliveira wrote:
> This is a series of improvements for rtla, mainly as a result of our
> daily usage of the tool debugging problems at red hat.
> 
> The cgroup support and house keeping options are from our usage
> of the tool debugging containers.
> 
> The auto-analysis overhead reduction is needed when we go to
> large boxes - but it is really hand in practice, as it gives an idea
> of the problem without having to look at the trace.
> 
> Running hwnoise 100 % of CPU time might cause some systems
> to slow down too much. Reduce its utilization to 75% by default to
> avoid problems for people using it for the first time.
> 
> Finally, it adds support for running timerlat user-space threads,
> and to collect the additional field via rtla timerlat top/hist.
> 
> Changes from V2:
>   - Add timerlat hist -u option
>   - Link: https://lore.kernel.org/lkml/cover.1684863094.git.bristot@kernel.org/
> Changes from V1:
>   - Add the user-space thread support to rtla timerlat top
>   - Link: https://lore.kernel.org/lkml/cover.1683827510.git.bristot@kernel.org/
> 
> Daniel Bristot de Oliveira (11):
>   rtla: Add -C cgroup support
>   rtla: Add --house-keeping option
>   rtla: Change monitored_cpus from char * to cpu_set_t
>   rtla: Automatically move rtla to a house-keeping cpu
>   rtla/timerlat: Give timerlat auto analysis its own instance
>   rtla/timerlat_hist: Add auto-analysis support
>   rtla: Start the tracers after creating all instances
>   rtla/hwnoise: Reduce runtime to 75%
>   rtla: Add timerlat user-space support for timerlat top
>   rtla: Add timerlat user-space support for
>   Documentation: Add tools/rtla timerlat -u option documentation
> 
>  Documentation/tools/rtla/common_options.rst   |   8 +
>  .../tools/rtla/common_timerlat_aa.rst         |   7 -
>  .../tools/rtla/common_timerlat_options.rst    |   7 +
>  .../tools/rtla/rtla-timerlat-hist.rst         |   7 +-
>  .../tools/rtla/rtla-timerlat-top.rst          |   7 +
>  tools/tracing/rtla/src/osnoise.c              |  65 ++++
>  tools/tracing/rtla/src/osnoise.h              |   5 +
>  tools/tracing/rtla/src/osnoise_hist.c         |  90 ++++-
>  tools/tracing/rtla/src/osnoise_top.c          |  83 ++++-
>  tools/tracing/rtla/src/timerlat_aa.c          |  35 +-
>  tools/tracing/rtla/src/timerlat_aa.h          |   5 +-
>  tools/tracing/rtla/src/timerlat_hist.c        | 262 ++++++++++++--
>  tools/tracing/rtla/src/timerlat_top.c         | 229 +++++++++++--
>  tools/tracing/rtla/src/timerlat_u.c           | 224 ++++++++++++
>  tools/tracing/rtla/src/timerlat_u.h           |  18 +
>  tools/tracing/rtla/src/utils.c                | 324 +++++++++++++++++-
>  tools/tracing/rtla/src/utils.h                |   7 +
>  17 files changed, 1277 insertions(+), 106 deletions(-)
>  create mode 100644 tools/tracing/rtla/src/timerlat_u.c
>  create mode 100644 tools/tracing/rtla/src/timerlat_u.h
> 
> -- 

I've been heavily relying on these for debugging various issues with
latency sensitive workloads and they work like a charm.

Tested-by: Juri Lelli <juri.lelli@redhat.com>

Best,
Juri