[v6,00/45] Add trace file compression

Message

Tzvetomir Stoyanov (VMware) June 14, 2021, 7:49 a.m. UTC

New implementation, adding a compression to trace file. A basic infrastructure
for compression support is added into the trace-cmd library. The zlib is used
for compression, but more libraries and algorithms can be added. Trace data is
commpressed and part of the trace file metadata: 
 - ftrace events format
 - format of recorded events
 - information about the mapping of function addresses to the function names
 - trace_printk() format strings
 - information of the mapping a PID to a process name
 - options
Note: not all trace-cmd commands are tested with these changes. These are
verified to work with compressed trace files:
 trace-cmd record
 trace-cmd report
 trace-cmd dump
 trace-cmd agent
 trace-cmd profile
 trace-cmd split
 trace-cmd convert

Todo list, in order to have complete trace file compression:
 - Add support for more compression libraries.
 - Test all trace-cmd commands with trace file v7.

v6 changes:
 - Documentation upadate.
 - Bug fixes.
 - Coding style fixes.
v5 changes:
 - Comress the latency trace data.
 - Add new "trace-cmd convert" subcommand.
 - Fixed bug with empty CPUs in the trace file.
 - A lot of minor bug fixes.
v4 changes:
 - Tested and fixed profile and split subcommands with compression files.
 - Bug fixes.
v3 changes:
 - Compress the trace data.
 - Added documentation to all compression APIs.
 - A few minor fixes.
v2 changes:
 - Refactored compression APIs.
 - Moved the trace buffers description out of the trace options section.
 - Added compression of "options" section of the trace file.
 - Updated "trace-cmd list" to show available compression algorithms.
 - Tested with host-guest tracing.
 - Merged with the patchset that bumps the trace file version.

Tzvetomir Stoyanov (VMware) (45):
  trace-cmd library: Remove unused private APIs for creating trace files
  trace-cmd library: Remove unused API tracecmd_update_option
  trace-cmd: Check if file version is supported
  trace-cmd library: Add new API to get file version of input handler
  trace-cmd library: Select the file version when writing trace file
  trace-cmd: Add APIs for library initialization and free
  trace-cmd library: Add support for compression algorithms
  trace-cmd list: Show supported compression algorithms
  trace-cmd library: Bump the trace file version to 7
  trace-cmd library: Compress part of the trace file
  trace-cmd library: Read compressed trace file
  trace-cmd library: Add new API to get compression of input handler
  trace-cmd library: Inherit compression algorithm from input file
  trace-cmd library: Extend the create file APIs to support different
    compression
  trace-cmd record: Add new parameter --compression
  trace-cmd dump: Add support for trace files version 7
  trace-cmd library: Add support for zlib compression library
  trace-cmd library: Hide the logic for updating buffer offset
  trace-cmd: Move buffers description outside of options
  trace-cmd library: Track the offset in the option section in the trace
    file
  trace-cmd library: Add compression of the option section of the trace
    file
  trace-cmd library: Refactor the logic for writing trace data in the
    file
  trace-cmd library: Add APIs for read and write compressed data in
    chunks
  trace-cmd: Compress trace data
  trace-cmd: Read compressed trace data
  trace-cmd library: Compress latency trace data
  trace-cmd: Read compressed latency trace data
  trace-cmd library: Reuse within the library the function that checks
    file state.
  trace-cmd library: Make tracecmd_copy_headers() to work with output
    handler
  trace-cmd: Do not use trace file compression with streams
  trace-cmd library: Add new API to get file version of output handler
  trace-cmd: Add file state parameter to tracecmd_copy
  trace-cmd: Copy CPU count in tracecmd_copy
  trace-cmd: Copy buffers description in tracecmd_copy
  trace-cmd: Copy options in tracecmd_copy
  trace-cmd library: Refactor the logic for writing CPU trace data
  trace-cmd library: Refactor the logic for writing CPU instance trace
    data
  trace-cmd: Copy trace data in tracecmd_copy
  trace-cmd: Add compression parameter to tracecmd_copy
  trace-cmd: Add new command "trace-cmd convert"
  trace-cmd record: Update man page
  trace-cmd: Add convert man page
  trace-cmd: Update bash completion
  trace-cmd list: Update the man page
  trace-cmd: Update trace.dat man page

 .../trace-cmd/trace-cmd-convert.1.txt         |  60 ++
 Documentation/trace-cmd/trace-cmd-list.1.txt  |   3 +
 .../trace-cmd/trace-cmd-record.1.txt          |  10 +
 Documentation/trace-cmd/trace-cmd.1.txt       |   4 +-
 Documentation/trace-cmd/trace-cmd.dat.5.txt   |  56 +-
 lib/trace-cmd/Makefile                        |  11 +
 .../include/private/trace-cmd-private.h       |  92 +-
 lib/trace-cmd/include/trace-cmd-local.h       |  44 +-
 lib/trace-cmd/trace-compress-zlib.c           | 172 +++
 lib/trace-cmd/trace-compress.c                | 797 ++++++++++++++
 lib/trace-cmd/trace-input.c                   | 999 ++++++++++++++----
 lib/trace-cmd/trace-output.c                  | 877 ++++++++++-----
 lib/trace-cmd/trace-util.c                    |  60 ++
 tracecmd/Makefile                             |   1 +
 tracecmd/include/trace-local.h                |   2 +
 tracecmd/trace-cmd.bash                       |  37 +
 tracecmd/trace-cmd.c                          |  12 +-
 tracecmd/trace-convert.c                      |  94 ++
 tracecmd/trace-dump.c                         | 162 ++-
 tracecmd/trace-list.c                         |  26 +
 tracecmd/trace-listen.c                       |   3 +
 tracecmd/trace-read.c                         |  14 +-
 tracecmd/trace-record.c                       |  48 +-
 tracecmd/trace-restore.c                      |   6 +-
 tracecmd/trace-split.c                        |   2 +-
 tracecmd/trace-stream.c                       |   2 +-
 tracecmd/trace-usage.c                        |  18 +
 27 files changed, 3076 insertions(+), 536 deletions(-)
 create mode 100644 Documentation/trace-cmd/trace-cmd-convert.1.txt
 create mode 100644 lib/trace-cmd/trace-compress-zlib.c
 create mode 100644 lib/trace-cmd/trace-compress.c
 create mode 100644 tracecmd/trace-convert.c

Comments

Steven Rostedt June 22, 2021, 2:22 a.m. UTC | #1

Please rebase the series on top of upstream. I pushed up 6 of your
patches from this series. And one bug fix that will affect the series
as well.

Also, we need to come up with a better option name than
"--compression". One that also can handle an optional parameter.
Perhaps just have it be "-z". We can have that be default zlib?

Maybe, zlib is enough, and we don't need to have different compression
types.

-- Steve