[v2,00/87] Trace file version 7

Message

Tzvetomir Stoyanov (VMware) July 29, 2021, 5:08 a.m. UTC

Introduce the next version of the trace-cmd trace file - 7. Major changes
since version 6:
 - New, flexible and extendable structure of the file.
 - Optional compression of file metadata and tracing data.
 
By default, trace-cmd still uses the old version 6 when recording trace
file. There are new arguments to force using the new version.
A new "trace-cmd convert" sub-command is introduced, to convert files
between different versions.

Known issues:
 - "trace-cmd split" does not work with trace file version 7, if there are
   CPUs with empty trace data.

v2 changes:
 - Fixed trace-cmd agent to work with v7 compressed files.
 - Fixed converting files from v6 to v7 with empty top buffer.
 - Minor coding style fixes.

Tzvetomir Stoyanov (VMware) (87):
  trace-cmd library: Read option id with correct endian
  trace-cmd report: Fix typos in error messages
  tarce-cmd library: Fix version string memory leak
  trace-cmd library: Fixed a memory leak on input handler close
  trace-cmd library: Fix possible memory corruption on processing a
    trace buffer
  trace-cmd library: Add constructor and destructor
  trace-cmd library: Add cache functionality to network message handler
  trace-cmd library: Add support for compression algorithms
  trace-cmd list: Show supported compression algorithms
  trace-cmd library: Internal helpers for compressing data
  trace-cmd library: Internal helpers for uncompressing data
  trace-cmd library: Define trace file version 7
  trace-cmd library: Refactor APIs for creating output handler
  trace-cmd library: Reuse within the library the function that checks
    file state.
  trace-cmd library: New API to get the version of output handler
  trace-cmd library: Inherit compression algorithm from input file
  trace-cmd library: New API to configure compression on an output
    handler
  trace-cmd record: Add compression to the trace context
  trace-cmd library: Write compression header in the trace file
  trace-cmd library: Compress part of the trace file
  trace-cmd library: Add internal helper functon for writing headers
    before file sections
  trace-cmd library: Write header before file sections
  trace-cmd library: Refactor the logic for writing trace data in the
    file
  trace-cmd library: Add local helper function for data compression
  trace-cmd library: Compress the trace data
  tarce-cmd library: Add multiple options sections in trace file version
    7
  trace-cmd library: Do not write CPU count section in trace files
    version 7
  trace-cmd library: Move CPU flyrecord trace metadata into the buffer
    option, for trace file version 7
  trace-cmd record: Append trace options after the trace data are
    written
  trace-cmd library: Add section header before flyrecord trace data
  trace-cmd library: Fit CPU latency trace data in the new trace file
    version 7 format
  trace-cmd library: Do not write CPUs with empty trace data
  trace-cmd library: Add macro to check file state on reading
  trace-cmd library: Introduce sections in trace file reading logic
  trace-cmd library: Initialize internal sections database on file read
  trace-cmd library: Use sections database when reading parts of the
    trace file
  trace-cmd library: Set log size to the input tep handler when it is
    read from the file
  trace-cmd library: Fix possible memory leak in read_ftrace_files()
  trace-cmd library: Fix possible memory leak in read_event_files()
  trace-cmd library: Fix possible memory leak in read_proc_kallsyms()
  trace-cmd library: Fix possible memory leak in read_ftrace_printk()
  trace-cmd library: Fix possible memory leak in
    read_and_parse_cmdlines()
  trace-cmd library: Track maximum CPUs count in input handler
  trace-cmd library: Set input handler default values in allocation
    function
  trace-cmd library: Read headers from trace file version 7
  tarce-cmd library: Do not use local variables when reading CPU stat
    option
  trace-cmd library: Read handle header and compression of the option
    section
  trace-cmd library: Read extended BUFFER option
  trace-cmd library: Handle the extended DONE option
  trace-cmd library: Read compression header
  trace-cmd library: Extend the input handler with trace data
    decompression context
  trace-cmd library: Initialize CPU data decompression logic
  trace-cmd library: Initialize CPU data for reading from version 7
    trace files
  trace-cmd library: Add logic for in-memory decompression
  trace-cmd library: Handle latency trace in version 7 files
  trace-cmd library: Handle buffer trace data init for version 7 files
  trace-cmd report: Use the new latency API to read data
  trace-cmd report: Close input file handlers on exit
  trace-cmd report: Do not print empty buffer name
  trace-cmd report: Init the top trace instance earlier
  trace-cmd: Call additional APIs when creating trace file
  trace-cmd dump: Add helpers for processing trace file version 7
  trace-cmd dump: Print compression header
  trace-cmd dump: Add helpers for processing trace file sections
  trace-cmd dump: Read recursively all options sections
  trace-cmd dump: Read extended BUFFER option
  trace-cmd dump: Dump sections
  trace-cmd dump: Dump trace file version 7
  trace-cmd dump: Dump sections content
  trace-cmd dump: Add new argument --sections
  trace-cmd dump: Align better the output of flyrecord dump
  trace-cmd library: Add zlib compression algorithm
  trace-cmd library: Reuse local function that writes to output handler
  trace-cmd library: Use output handler when copying data from input
    file
  trace-cmd library: Handle version 7 files when copying headers between
    files
  tarce-cmd library: Copy CPU count between trace files
  tarce-cmd library: New API to copy buffer description between trace
    files
  tarce-cmd library: New API to copy options between trace files
  tarce-cmd library: New API to copy trace data between trace files
  trace-cmd library: Extend tracecmd_copy() API
  trace-cmd library: Set correct CPU to the record, retrieved with
    tracecmd_peek_data
  trace-cmd: Add new subcommand "convert"
  trace-cmd report: Add new parameters for version 7 trace files
  trace-cmd: Update bash completion
  tarce-cmd: Man page for "trace-cmd convert"
  tarce-cmd: Update record man page
  trace-cmd: Document trace file version 7

 .../trace-cmd/trace-cmd-convert.1.txt         |   65 +
 Documentation/trace-cmd/trace-cmd-list.1.txt  |    3 +
 .../trace-cmd/trace-cmd-record.1.txt          |   13 +
 Documentation/trace-cmd/trace-cmd.1.txt       |    4 +-
 ...e-cmd.dat.5.txt => trace-cmd.dat.v6.5.txt} |    8 +-
 .../trace-cmd/trace-cmd.dat.v7.5.txt          |  442 ++++
 Makefile                                      |    7 +
 include/trace-cmd/trace-cmd.h                 |    5 +
 lib/trace-cmd/Makefile                        |    8 +
 .../include/private/trace-cmd-private.h       |  115 +-
 lib/trace-cmd/include/trace-cmd-local.h       |   63 +-
 lib/trace-cmd/trace-compress-zlib.c           |  109 +
 lib/trace-cmd/trace-compress.c                |  910 +++++++
 lib/trace-cmd/trace-input.c                   | 2134 ++++++++++++++---
 lib/trace-cmd/trace-msg.c                     |  131 +-
 lib/trace-cmd/trace-output.c                  | 1685 ++++++++++---
 lib/trace-cmd/trace-util.c                    |   45 +-
 tracecmd/Makefile                             |    5 +
 tracecmd/include/trace-local.h                |    2 +
 tracecmd/trace-cmd.bash                       |   37 +
 tracecmd/trace-cmd.c                          |    1 +
 tracecmd/trace-convert.c                      |  106 +
 tracecmd/trace-dump.c                         |  458 +++-
 tracecmd/trace-list.c                         |   26 +
 tracecmd/trace-listen.c                       |    8 +-
 tracecmd/trace-read.c                         |   39 +-
 tracecmd/trace-record.c                       |  141 +-
 tracecmd/trace-restore.c                      |   37 +-
 tracecmd/trace-split.c                        |    5 +-
 tracecmd/trace-usage.c                        |   21 +-
 30 files changed, 5725 insertions(+), 908 deletions(-)
 create mode 100644 Documentation/trace-cmd/trace-cmd-convert.1.txt
 rename Documentation/trace-cmd/{trace-cmd.dat.5.txt => trace-cmd.dat.v6.5.txt} (98%)
 create mode 100644 Documentation/trace-cmd/trace-cmd.dat.v7.5.txt
 create mode 100644 lib/trace-cmd/trace-compress-zlib.c
 create mode 100644 lib/trace-cmd/trace-compress.c
 create mode 100644 tracecmd/trace-convert.c