From patchwork Wed Jan 19 08:26:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95AA2C433EF for ; Wed, 19 Jan 2022 08:27:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352436AbiASI1V (ORCPT ); Wed, 19 Jan 2022 03:27:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352435AbiASI1U (ORCPT ); Wed, 19 Jan 2022 03:27:20 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3371DC061574 for ; Wed, 19 Jan 2022 00:27:20 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id j23so2451433edp.5 for ; Wed, 19 Jan 2022 00:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vbnwboL+wPBkZFVpm25VuFsBPc7AQhoBNS8VDwvEa4c=; b=GwXHVkp3Qml6x3gDTyYhznLFSCuZBR86WVt/K5Eprlm63o7HJX+CDqa4o6FpN6WKAW SXrd3Sm6eBcrlkek616TSMX2t4/vlFnMu/AfzWlvjMFKzlLrebXzok2gfEMg0cvJEvMC js/XggHkO9rBf18RXThBwUyp3Mtg0Jlit0h8odni7NGRy89cZLlZ1N/7MWKNql+0Q/pZ uGaG66bbJ4x8tR5voPr6OL9i/bCd6Arlz4qd1yvPHqvPrsgAGfK/TeyJkRiaZ+tY0BmY B+ziZ1FSu/zNnoboRusmfrJbXdHBSTpwB3LQK0BTwLUOZFQgqPcM9devWSrsk1TT6c/w 3ROQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vbnwboL+wPBkZFVpm25VuFsBPc7AQhoBNS8VDwvEa4c=; b=oT0vLZwXFs+bSnNeBqFw6rrA9OMHaanpF99EYMbgEUyJk6TFKouM0Jsjhgh7J5N63y zksE8o9T7is0aL7D67ydjASZWD3zlJ74WmAPMFo0rTInf8tqR60DB8miP8bTR8nAqkoq iXNYWXirb8LbeZLbxGy5ebAvNZ+afgtez74f1wA759LB8er4MoxkAuZTO+vPhxLNbKvp terwq4mcFBlySTGlYKyP+ngVxhJsUhQvdCuQBDAVfz+8jQMeq+Jj/VUoKwOEAN5eIafm yHSiYr4uvbdwgiNmQFu/WRZJvhfdByOKPfZj9Mxw6sTMA36rAJcwHaeBMVCfp4pvKyqC 8lBQ== X-Gm-Message-State: AOAM531x0ze7zVwEVfVYEovVYg9x/XUeIxhxhiwViRoq+N8S70rATIsb ULKzjWQdex7n3Vg95tJxH2XgWxGj9o0= X-Google-Smtp-Source: ABdhPJxCj4uDvnNCHBZAdvD/Y8J1zZPfOd4+hPCvP1lkXckPWx/d3cl/PkEIkk6QpmaD6+ik4hBhMg== X-Received: by 2002:a17:907:8a0b:: with SMTP id sc11mr16234122ejc.376.1642580838611; Wed, 19 Jan 2022 00:27:18 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:18 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 01/20] trace-cmd library: Add support for compression algorithms Date: Wed, 19 Jan 2022 10:26:56 +0200 Message-Id: <20220119082715.245846-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 910 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 965 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 1efafba1..f8f0ba15 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -35,6 +35,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -160,6 +161,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -492,6 +494,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b4f3d8c8..d4047429 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -36,6 +36,9 @@ struct data_file_write { unsigned long long file_data_offset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..883cf669 --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,910 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handle + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handle + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handle + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int size; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + size = s_uncompressed > s_compressed ? s_uncompressed : s_compressed; + handle->buffer = malloc(size); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + s_uncompressed = size; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handle + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handle + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handle, used to encode the data + * @msg_handle: message handle, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handle. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handle + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handle + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index 21f1b065..06071f6c 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -635,6 +635,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Wed Jan 19 08:26:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C57E4C433FE for ; Wed, 19 Jan 2022 08:27:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352435AbiASI1V (ORCPT ); Wed, 19 Jan 2022 03:27:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI1V (ORCPT ); Wed, 19 Jan 2022 03:27:21 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB3BEC06161C for ; Wed, 19 Jan 2022 00:27:20 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id f21so7480921eds.11 for ; Wed, 19 Jan 2022 00:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3bc2yDUheJlaiksjyLj44PAiO5rak/8S3mqzs2HiHtA=; b=fXd/rtFAFfQcKZqXBLHZdyleQr97neMy3vk9rqIi47xk4ahmKXMdegDR2PU3GLfTMM H75lKVknMyQhSXq4SnXaFevrrHOsNMRYNohvUNMqZLAvLzsedT+z7pChpgsJnVB5WS4G QFYr6Ss/Z+YJvTegDZSF1mqFeDJt9oBjBIOkOdoP44d4yUnjL2MhC3//KCsha5S1KmaD LwTJvKTFeTG/2DslezQ/j6J9xBYGOOTShZtWq7dUKfpfCfSKZ4bpqIYIvhnzZKqTw19W A9+m8bl0aWfw4mnOMwCA9pibmFavJx8590fFDMsTIKjvfd8dutpWz4Ielh5yf1lHdfPi EMdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3bc2yDUheJlaiksjyLj44PAiO5rak/8S3mqzs2HiHtA=; b=5jPDn6mOROtveLCFCsF8y+Dvp+ise+d3FHohnkTgigJSyLpluG/FhhNjBfOcmESiS/ ouklvZiY5Tf8sTj9Sf91X9Rry8hFUVNOOzw/OrDietjk+Xb3fP+kPixCSUmuYdpGEmL0 YZEMZnCrSB1q5fJDqTejlvu/Y79wgYur2XwaIqYK7WlMnMs2LR1wqEtWW1M4eoi4rzic JkQjDXhtAgtgJTFACUpR5FTp9P8E21sSjv6mdTSOv0C9LmTsuCrNmyD9go99lTajywlg p22/ieEBDEbBAG8zlv+rWdEirr/BNurOZYN3C0081SdVr3hnN/BXbzlTK5dAzC9f4ysL p9vg== X-Gm-Message-State: AOAM533x+4+ITKj1QHahAm/DaF4W3sZeytFhj6fLnxHEdYaK0Rx0Zkb7 gFp/BA8KdcMYxV8+L42yOzd+DyhTUVE= X-Google-Smtp-Source: ABdhPJzRQaRVLkzYkVukaOsFUBDndC6n5TAKdCsIxfGoYPC8nL71pbrxKTvt2F4FQiUOt+TKPbCj3A== X-Received: by 2002:a17:906:66cb:: with SMTP id k11mr3264629ejp.85.1642580839425; Wed, 19 Jan 2022 00:27:19 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:19 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 02/20] trace-cmd library: Internal helpers for compressing data Date: Wed, 19 Jan 2022 10:26:57 +0200 Message-Id: <20220119082715.245846-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index d4047429..b848514e 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -42,6 +42,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 15baef8f..2b944913 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -70,6 +70,8 @@ struct tracecmd_output { unsigned long long options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -100,18 +102,27 @@ static int save_string_section(struct tracecmd_output *handle); static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -139,6 +150,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + static long add_string(struct tracecmd_output *handle, const char *string) { int size = strlen(string) + 1; @@ -1645,7 +1693,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Wed Jan 19 08:26:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E72CAC433F5 for ; Wed, 19 Jan 2022 08:27:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352438AbiASI1W (ORCPT ); Wed, 19 Jan 2022 03:27:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI1V (ORCPT ); Wed, 19 Jan 2022 03:27:21 -0500 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98564C061574 for ; Wed, 19 Jan 2022 00:27:21 -0800 (PST) Received: by mail-ed1-x535.google.com with SMTP id p12so7521305edq.9 for ; Wed, 19 Jan 2022 00:27:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zomYlE8NAQmflyOPsjJiUCp0aPpWWVdf21/5YjVh3D0=; b=b8UZUPEfnPmpuQA5+PtAEQsFeV2rFuH9M568S+ThA2yucmw48+bVOaPr8B5JuhPJRK ohz6RanmB+H6Eay7EVvY3CbqPC22FuPT7IKA3aExqh8QbuSnSv3hCW52G9H341DhJKxc YuyRTDX8WAyDMUv7vqOw7v4R354AkA/dgL5qON7A8fON/ph4LEV21LL6rnCdrahkn186 SH+zEkkynIYJbAGsXlrXCpPhuhspkR0RrJSGbEGaJcmzH2KF0YRN/fdGbl6ikVI9A934 gh0UCgSTT07reD5se4OC5Gm7ECFCh84eVfwOi8mo5xX43IwShCM7MINmRKjas5at3zJ7 ZZrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zomYlE8NAQmflyOPsjJiUCp0aPpWWVdf21/5YjVh3D0=; b=kf1CupYUPcFjwkSNBdDE3XvgcILL1g0HycX5Vj3X0FoLzbhlhdaGQEw41Ri1p7jYVZ dSaHsdOAwcWh/+dkYL4Iy72G4+d+W5f8kfEGx9DS6XjXDVHRLLD2dF5L0Xpku4QMq1qC u617xDGpvCDCtcnp1q8TZdIPvFX5cdytFB3w3uiPHy2XF2ZwgFejS4BH2KWDpS1eKWfe CFiJolXdxk+zH37oJaMLY0i5Pr9886t+qiEbyB8J8GmQIXLIZdP8G8kR5N4HSuci3eWT /gviO0S2j96UaarhgU8p8Okceh1kSQXKMskTrtQP0ZGA1bZN6Fmws8ZwE2w3eqGUWw5I /rdQ== X-Gm-Message-State: AOAM530iS5wsG3Zo+8aBPpFG7kv53UQkMS2gIU2KqRkvdETN8wTLFceo 17CA7G5yRvbsm+XSPDH75/bMvUd2Mwk= X-Google-Smtp-Source: ABdhPJxCm8X98YYMu7yMu/X4Y1hPlGteBDVIXrHq0lSk/f8ak6h+HRZRfssfiS+QtT860G+9gUjXKg== X-Received: by 2002:a05:6402:4382:: with SMTP id o2mr29379997edc.38.1642580840276; Wed, 19 Jan 2022 00:27:20 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:19 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Wed, 19 Jan 2022 10:26:58 +0200 Message-Id: <20220119082715.245846-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b848514e..ca682ed9 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -49,6 +49,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index ecb56826..49ada92f 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -160,6 +160,9 @@ struct tracecmd_input { unsigned int strings_size; /* size of the metadata strings */ char *strings; /* metadata strings */ + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; char * cpustats; @@ -266,13 +269,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -284,6 +287,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -308,9 +327,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -336,7 +353,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -400,6 +417,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Wed Jan 19 08:26:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CECBC4332F for ; Wed, 19 Jan 2022 08:27:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352439AbiASI1X (ORCPT ); Wed, 19 Jan 2022 03:27:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI1W (ORCPT ); Wed, 19 Jan 2022 03:27:22 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91142C061574 for ; Wed, 19 Jan 2022 00:27:22 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id cx27so7701664edb.1 for ; Wed, 19 Jan 2022 00:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VDDOy2QmvFB+qN76D8uPWFs6HR7m9UDxa5BcE8UItZs=; b=iK2gmkWmnWXKc5K+m907bL+1qu2W6LD3W2Ot4qrxR6fgxOvDchpSE9zQhUxCejC4Tm iteciP/J3SRcRS79UPZqpndMXh/An5KRaZ/fnBGffM7Am6EZbCDg50o9011d3ee0y8ii utXvx6nnryhzYrzThZb7odxL1iLrLVkAg4O3AZMsF1jnJIVeOObJHwa41Aj7sYLLHcf+ hffzd/YdoGhCaYi3kXMj8KmwIspztwVMoBuwlv3lVXECZ0Du5grujEiOilg2aG849ccY uU5oxSg9hw3ZEB1ja2CKP8F4J9nnS0vgZfWiy944q0p+UeP1OuQxAF8f0rmfwMBTror/ bz6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VDDOy2QmvFB+qN76D8uPWFs6HR7m9UDxa5BcE8UItZs=; b=1HhbpemhmOAqUPV5J9Amc+cN8YOUpAnIY2+BtgqNDevJGKsHKxeHCV/YOtD1plEaFy WlWN/IpplaLv/ly+h+8FQ6EMPfGNERYM4fZPmc81BWqd6Fml6RjfFvygk2h+MaXlHtQz W7waLZ2j7M7I8O0Pfjfbayycbpi5gS892GRWqVk+UrdtSI0JkEEUEzSkpRPIdfForlZk s/CmbSlePGrrGlT2k4FTYkZf5u4C5nuR9/p1SFDpEiHsxPcLAG38ckCdu1sGqniFbAE8 VSVqxCjjfrblR/xZmpNxXEywkb4K1/bjNDE80fY5EfOqWNl38kMIqc20zcPecIIrbdVo iF8w== X-Gm-Message-State: AOAM533z9Jiqg9L6b2HtPF9hxuHRgDc0UXtoU+kLPcbyVx2VeQkeq1lz 3i45hMHnaAj/Zy+XVUye+jKnkVvnQuo= X-Google-Smtp-Source: ABdhPJziaSYE8Qk0q963yV6K/DQfjF+FxigImVtl4ZG00VQhPaDkXyBO4sO7qMdLdRcBcjZmZB5m0w== X-Received: by 2002:a17:906:4786:: with SMTP id cw6mr18330245ejc.600.1642580841214; Wed, 19 Jan 2022 00:27:21 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:20 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Wed, 19 Jan 2022 10:26:59 +0200 Message-Id: <20220119082715.245846-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 +++++++++++++++ lib/trace-cmd/trace-output.c | 20 +++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index f8f0ba15..bcc4c9f3 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -261,6 +261,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 49ada92f..a81112f0 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4661,6 +4661,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 2b944913..c37fee1a 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1299,6 +1299,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kal */ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handle || !ihandle || handle->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1310,6 +1313,15 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracec handle->file_version = tracecmd_get_in_file_version(ihandle); handle->big_endian = tep_is_file_bigendian(handle->pevent); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + return -1; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->file_version = FILE_VERSION_COMPRESSION; + } + return 0; } @@ -2270,6 +2282,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2310,6 +2324,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Wed Jan 19 08:27:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2801FC433F5 for ; Wed, 19 Jan 2022 08:27:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352442AbiASI1Y (ORCPT ); Wed, 19 Jan 2022 03:27:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI1Y (ORCPT ); Wed, 19 Jan 2022 03:27:24 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1FD7C061574 for ; Wed, 19 Jan 2022 00:27:23 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id a18so7571405edj.7 for ; Wed, 19 Jan 2022 00:27:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CMls2xTtvpRTRGCAZD2HybvQcNbb5m92n8AFdzk2vPo=; b=N4YvfIoKmgvp7DkQCrE4fa9R+GKaDKHTejw/HeCUrWl1On9yg2XjUQDVTvG5ZTZhCD UAl7c7pr9nDy0O6jl/yD7kuMSewnVqPnTrh1B1V9mDWjMeGBGpYd/0fAUV3tqKLQKjMI sxYLEWU6pXk0MEAzV8oCrzwFFlzjox+k5DfLH4IE9LGDAcNDzr+fNhx14lOApmgO4r4L jrFsxQmSRipMIxwauMUNDS+HwFlgiTEiMrEb+ADbNNgq3cj5qVyBl5KXzLn2uT9eT/aa vOaui1o8C7KhVnyLgUH5keVex6J1/SoJc14gqhI/pXV44CRwBanK/GuRQBC8L0li7vdS gUkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CMls2xTtvpRTRGCAZD2HybvQcNbb5m92n8AFdzk2vPo=; b=6QuDYCjhErFzl9NIr8Avj1m6wOlrT3AJTGB5tQD9m1eipj39pAr3Zhn2Ij/xRWyOVZ RylT47zF5/89jlj6UCEuYivb1OvZ3lz1tqtKd9U4HvadkV8fpEqcFJu+LiwtxOmvW/Dm ktz3XszFAGZfNbtP5ZWEhmw6WhSmWSpIMQRhJx6tQBB2SAMM8dOX/ALq5lnSaKaPOHYm RPm56bHaE65itCWlc6o6fTUZo1ejdsd8PSKfeTK2Zrw8pCQKyLUobEucVXyOCuGqPwPK Ng8Oy4Ud5ZNzMSxB+uHEGE4uMas00c1RWE7WVQ6Ogewjjg/pgmxvBVs25xWQ8xMKBKRO Tmxg== X-Gm-Message-State: AOAM531gw1q1uMq4+yjoxCLIJ5d81pD/OcYv2Bw9kE76yledo86QuNOJ xCdXczACddN4gLUp1xuXStfqqQgPNa4= X-Google-Smtp-Source: ABdhPJwhDkf2+Kwxr0Yc2IT0ASqUCfcodwaXeih3DCuoyjFE4bjSO+OjbUwUppX9MkTCIjpRY5Rp1A== X-Received: by 2002:a17:907:60d6:: with SMTP id hv22mr24847802ejc.438.1642580842262; Wed, 19 Jan 2022 00:27:22 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:21 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 05/20] trace-cmd library: New API to configure compression on an output handler Date: Wed, 19 Jan 2022 10:27:00 +0200 Message-Id: <20220119082715.245846-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handle to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 58 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 60 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index bcc4c9f3..e22c3b7e 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -302,13 +302,14 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handle, const char *tr int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression); int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_output_create(const char *output_file); struct tracecmd_output *tracecmd_output_create_fd(int fd); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle, unsigned short id, int size, diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index c37fee1a..f13d2b8e 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -263,6 +263,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) free(handle->strings); free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1341,6 +1342,55 @@ int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handle->file_version = file_version; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->compress = NULL; + return 0; +} + +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handle + * @handle: output handle to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_headers(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handle is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression) +{ + if (!handle || handle->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handle->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handle->compress = tracecmd_compress_alloc(NULL, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handle->compress = tracecmd_compress_alloc(compression, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handle->compress && handle->file_version < FILE_VERSION_COMPRESSION) { + handle->file_version = FILE_VERSION_COMPRESSION; + if (handle->msg_handle) + tracecmd_msg_handle_cache(handle->msg_handle); + } + return 0; } @@ -1951,7 +2001,7 @@ out_add_buffer_option(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1964,6 +2014,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_headers(handle, NULL)) goto out_free; /* diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 4cf48c86..ece5a0c2 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4511,7 +4511,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Wed Jan 19 08:27:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B407CC433EF for ; Wed, 19 Jan 2022 08:27:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352443AbiASI1Z (ORCPT ); Wed, 19 Jan 2022 03:27:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI1Z (ORCPT ); Wed, 19 Jan 2022 03:27:25 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8DD2C061574 for ; Wed, 19 Jan 2022 00:27:24 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id cx27so7702161edb.1 for ; Wed, 19 Jan 2022 00:27:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Zpoh1ToHrvkrUdXW8bnduCOSTQYPppeOMpPDPoLOzFw=; b=NOBSFokFonXXC4Nkj1QyXAgJb2eNtnOw2yKvc59+UDA0h7FlYKomhzDh6EWcUFpL2+ 2fLIETbOTwG7r0Xho2V0+NtfWAso0uXERc91CdePqmpu87wLB1b03yrjAYFDzL1ikf08 TiU/4pIEfvsFo2PTFDTXNzWRAMixJX+tvDCvmO6VuQYvWYk5m9W0p5SYQW8YqO03/Q1p Q/L2zrymFhkPGcJhsfajGr7WfkZBwUoR55DKx2LMYw/zF+VP1osI+M5Hpcdpz8T8cW4A 3KMSk+XbmZoSKYGkTqf1DVSe5NeHCZ6YMt37tCQQ4Gdu/P1EyHb7jqC//UWK+uZVTOS4 FuFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zpoh1ToHrvkrUdXW8bnduCOSTQYPppeOMpPDPoLOzFw=; b=JkMCHiBT0P8f1w/gVct4cXtX2Zb+6lrT6vV9U4SmH4eIN8p+7OMSwNus5XhVvcj0mA SFzOewLX3YLbUIfGEblWME5UCLiXlwIKbRLnPjAOvP/6gYEfb8tchHCQkYz7xNLrPi7F ClR5VhF55Xu7IZQM7lFpexlNi6n7jQBXdfurJYD0TIZlBxvXXwCyujURbSBgoZHVO9Ar mVdj52PFz+AH3Vz+hPigZJeEZ5F4kRVtMGf7o086zFjJVuuRNkirueQb/uWQrEs74u3L QJzStN6s0dOHaLn1HOv912YfMGM9gmz/y1dgw3UFHBR3YnXweDpclmBGlbeRQuWhkz40 yPOw== X-Gm-Message-State: AOAM530VXMk2EClyZBHHWjByaoXu9aeqrbcc2CC2IB4UDrhmTbS94CsM UP1i3IsF0o3OqCz+CdC/uFm3OuW66dU= X-Google-Smtp-Source: ABdhPJzD/bNuzOtyR640JccbFaapS0MN8Q6/T0OnqPx47dcIZjDXv0S6sdOBc0vdGFfvm8MlkvzGzw== X-Received: by 2002:a17:907:3e82:: with SMTP id hs2mr24623323ejc.30.1642580843341; Wed, 19 Jan 2022 00:27:23 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:22 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 06/20] trace-cmd library: Write compression header in the trace file Date: Wed, 19 Jan 2022 10:27:01 +0200 Message-Id: <20220119082715.245846-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index f13d2b8e..e8a02bf9 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1156,6 +1156,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_create_fd - allocate new output handle to a trace file * @fd: File descriptor for the handle to write to. @@ -1448,6 +1466,10 @@ static int output_write_init(struct tracecmd_output *handle) endian4 = convert_endian_4(handle, handle->page_size); if (do_write_check(handle, &endian4, 4)) return -1; + if (handle->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handle)) + return -1; + } if (HAS_SECTIONS(handle)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Wed Jan 19 08:27:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CD12C433EF for ; Wed, 19 Jan 2022 08:27:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352445AbiASI10 (ORCPT ); Wed, 19 Jan 2022 03:27:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI10 (ORCPT ); Wed, 19 Jan 2022 03:27:26 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7AF0C061574 for ; Wed, 19 Jan 2022 00:27:25 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id c71so7612135edf.6 for ; Wed, 19 Jan 2022 00:27:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nXPjESN+IZYvavt56bFb3mWFnbygaB7vIUGWsR0B8yc=; b=hNdWDjYocO5TD/11pg5yLpHwTrf4GEEIswpfQuShU7PS5eZuo3vd92MFD0GjEXBEmS WPsUgWZqNwlRyHFXDpmFJnW+3/fQdeSO+xQ9/jTHWzZeq47uNqq/8LUbQwcjmpMhIey6 1aLR3coJZnJ5K8x9jXRDAGy4kgDRx7g8CnInI9CtYVCmK8Ny0jG7upgIsrNS1tszJI9t rnQkGFNFOFXBTQPRkmScNDenW7DZabpshbtHpUni76jxO8FGy6KBVWxkwFhrTAQFBQFh EGNvydxGKfWLZU4+pkd7VhwNfAxu2R9rGQB6NNLyeLqM4LY5SnorTHI9Te/A1giUx5aD gszw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nXPjESN+IZYvavt56bFb3mWFnbygaB7vIUGWsR0B8yc=; b=Q2xSPzqLktMa3EUN81OIMc09AWzTkPWpU1DQbGGtSbIqKdext8vN9Y2HfafcNlWbN/ fuO9z12JHDx0DDpfbkRYHYFPdxn+SFgeRwFVOious3gdz8xxX37STvcfK9NAWeE4IqOa wOivvvOcdPHeM8R1v99HSq9zQsNWv+GHLA1iPsmQrtUG7U9iWLuQ5JHB+FtndwFJLuTo erq6928AVoRF9NX3TDRBR0tkm7H7hOAPPeI6F00C9btgo8CNwKuvxM/AijgHstN9VTe7 rSkj5r6SQsoYmrsFAwaC3PftbBhCxNgooOKizi0O9Ow88Nn18d06p2hFkF9u92VCAGN3 KsdA== X-Gm-Message-State: AOAM533fPCBau9MvvuTGe1nEtnAgdJsITdnlTIZ2XGxSdlrfTnajLD8/ B78ATJMwAZXkfCSad0Nuljw/9sJMjX4= X-Google-Smtp-Source: ABdhPJy6ylqRA1e+EE9j12u8hKlGEP1X1528qG5oKO09NchV8hkxi31g7PKzZ2IJoJ2firDa0bJDWw== X-Received: by 2002:a17:907:6218:: with SMTP id ms24mr11748379ejc.677.1642580844281; Wed, 19 Jan 2022 00:27:24 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:23 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 07/20] trace-cmd library: Compress part of the trace file Date: Wed, 19 Jan 2022 10:27:02 +0200 Message-Id: <20220119082715.245846-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name - strings Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 97 +++++++++++++++++++++++++++++------- 1 file changed, 79 insertions(+), 18 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index e8a02bf9..df65d02c 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -97,7 +97,7 @@ struct list_event_system { #define HAS_SECTIONS(H) ((H)->file_version >= FILE_VERSION_SECTIONS) static int write_options(struct tracecmd_output *handle); -static int save_string_section(struct tracecmd_output *handle); +static int save_string_section(struct tracecmd_output *handle, bool compress); static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) @@ -277,7 +277,7 @@ void tracecmd_output_close(struct tracecmd_output *handle) write_options(handle); /* write strings section */ - save_string_section(handle); + save_string_section(handle, true); } if (handle->fd >= 0) { @@ -433,6 +433,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -494,7 +496,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, tsize_t o return 0; } -static int save_string_section(struct tracecmd_output *handle) +static int save_string_section(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t offset; @@ -508,13 +510,20 @@ static int save_string_section(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_STRINGS, "strings", flags, false); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + if (do_write_check(handle, handle->strings, handle->strings_p)) goto error; + if (out_compression_end(handle, compress)) + goto error; + if (out_update_section_header(handle, offset)) return -1; @@ -526,10 +535,11 @@ static int save_string_section(struct tracecmd_output *handle) return 0; error: + out_compression_reset(handle, compress); return -1; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -549,11 +559,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -567,6 +580,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -619,6 +634,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -626,6 +643,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -854,7 +872,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -868,15 +886,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -902,7 +925,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -920,6 +943,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -940,7 +965,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -955,6 +980,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -962,6 +990,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -1008,7 +1038,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1026,11 +1056,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1056,14 +1089,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1082,10 +1120,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1108,12 +1149,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1500,21 +1544,25 @@ static int output_write_init(struct tracecmd_output *handle) int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list) { + bool compress = false; + if (!handle || handle->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handle->file_state < TRACECMD_FILE_INIT && output_write_init(handle)) return -1; - if (read_header_files(handle)) + if (handle->compress) + compress = true; + if (read_header_files(handle, compress)) return -1; - if (read_ftrace_files(handle)) + if (read_ftrace_files(handle, compress)) return -1; - if (read_event_files(handle, list)) + if (read_event_files(handle, list, compress)) return -1; - if (read_proc_kallsyms(handle)) + if (read_proc_kallsyms(handle, compress)) return -1; - if (read_ftrace_printk(handle)) + if (read_ftrace_printk(handle, compress)) return -1; return 0; } @@ -1745,7 +1793,7 @@ int tracecmd_write_meta_strings(struct tracecmd_output *handle) if (!HAS_SECTIONS(handle)) return 0; - return save_string_section(handle); + return save_string_section(handle, true); } int tracecmd_write_options(struct tracecmd_output *handle) @@ -1894,6 +1942,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1903,14 +1952,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Wed Jan 19 08:27:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39A2CC433F5 for ; Wed, 19 Jan 2022 08:27:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352449AbiASI12 (ORCPT ); Wed, 19 Jan 2022 03:27:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352446AbiASI11 (ORCPT ); Wed, 19 Jan 2022 03:27:27 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD092C061574 for ; Wed, 19 Jan 2022 00:27:26 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id 30so7635315edv.3 for ; Wed, 19 Jan 2022 00:27:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=413OjZJM1Z7F2rj3MwiyKK8vli4GllgDEOol2+g1xto=; b=OUrjBRmOWgNPhGGcqGF62IeKv9dLSH3b4A3RAXGADce3yLrhG1ii8rlJYSeEXuNaAH GH7PmzfPjBxwMvkN+JDrX1EbNjOVwDO+0jQnmXugFKQRimZ8uAOlE6DnIl89oIIA8qyV 5ZWmJkRjX+idvbleHH+sGnczQuNOFljZOFmCdT4wNdMBBa4HXOqjTiL4ABO1Wz1R+b3v TLxsZMTZ+KEy1bhRfqZkd9RDjATvoBOKnDOZte+mEgH+6vau7FZe5BZ7NhLsnnGXQx3K sM8Wy6O8VH2AOi8js+RTpEHTSMJAmNb87mQU8PziYHZ/Wefg1+2sz48twYOi522YuDyC vRpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=413OjZJM1Z7F2rj3MwiyKK8vli4GllgDEOol2+g1xto=; b=QNAn/vL31byYyyvyWMGxlEVLdYpQi73FrtYyKJpQhkcKA2bu+43OeTAuaZrRik0eus Bb7nStS4VhqXvTwJyL+W+3AFl+U1i2SyTSIZlky25UiQmNCoCU+pAqqEgfeJ1FmMp82u tWh/2LHqICCCuHmmYEhfX22PvlNH9S++ny3fBQBup95J1GZwX0aXAVH8balGYOQ7WImb tpq5m9ZR0bd77fAelGj9Q4tgYSxGvQZXLClZ/FJeL/S9yR5x1L44O6OW6yquayw6ivrN byOH7XTARl+faTbhCklcJQtegQP8i2ecDFkMJffcxFMJkrRHYpPMIlHQf1Ne4UXRHTQr LREQ== X-Gm-Message-State: AOAM530yc06ngzcTXoO6rovTjK7qtwNJtJc0nNM0aTEYsL4/rsqO4qh4 XpeFqq1H4KBZ2FipVwZ+Wxsq6fApojI= X-Google-Smtp-Source: ABdhPJzTengmD/Awr9du8LOQzrBoTiFCXF4UX36x5o8dRyWsknRUKcJEIHVVw+SwBzxjGorIveOjiw== X-Received: by 2002:a17:907:7b99:: with SMTP id ne25mr19919555ejc.769.1642580845211; Wed, 19 Jan 2022 00:27:25 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:24 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 08/20] trace-cmd library: Add local helper function for data compression Date: Wed, 19 Jan 2022 10:27:03 +0200 Message-Id: <20220119082715.245846-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index df65d02c..e2e03cf9 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -320,18 +320,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -349,12 +358,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -601,7 +660,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -627,7 +686,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2292,7 +2351,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Wed Jan 19 08:27:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FEF2C433FE for ; Wed, 19 Jan 2022 08:27:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352451AbiASI13 (ORCPT ); Wed, 19 Jan 2022 03:27:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343541AbiASI12 (ORCPT ); Wed, 19 Jan 2022 03:27:28 -0500 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 714BDC06161C for ; Wed, 19 Jan 2022 00:27:27 -0800 (PST) Received: by mail-ed1-x529.google.com with SMTP id p12so7522427edq.9 for ; Wed, 19 Jan 2022 00:27:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l4VpDDGnpeFEAKGWzW1MYwObtP8anbC24KAg/QZZZ/U=; b=Jn1Mnemf4iVKz8VrtDI6/bBV/BX+GoXa9RNaj5YUITaYDV1miB2T6XU1L6VQ9yTtUF mgR8enecYnJMSTk1bMiO9vH2gDsuRzM5hx4rQ1bVhiKUz2owviT2a0jSSJLvvKjIRVFB oEqnqtVXDANG8siHcjCh88M6Z7hzgH4gqP/PjJ+nDtGI875ziZXLYhKQDJ1P8biHPnFL IkOtCuqc14bsmswVV9S4tFNhK2Q7bjtGcRQvOfAYv0i0626bFcqmhzcHUC4r7ta4dFcr 2k09gpag2GVPjtL5x7bHC6Kr+LyOvY2nMqchg1omVawA4Oj2TW5HnjT0JmZ36ZKFgBDI hMdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l4VpDDGnpeFEAKGWzW1MYwObtP8anbC24KAg/QZZZ/U=; b=gTzdTkKoawclqO6s6jNgu6+/mNcIJHOIwAJJHbVpKu8eO91hTUt5rysyqqVBqGTTo3 2AaLILmzW1JOx2HyxEwPXO0Bz+g9t4WelPnvA35AEMeQdTtT/XYFtl+MR/zrzP4fua0O 2OPWiWP8PaKXNQ2AAsW24K/OaX7FkdBzSIbQn0zHZqISwgpgc6fmKAqNy1pXRLoyeMOM F/hnyyivG/bDaAZ6pQ2RfZWoC93oKoLQpgFNjdT3jfRvqMQ+QzZj3gmsy6b1hzVUpZfd HfxDxbVAdAMZq7BaPL7yz2XiZ9V64FvCBi2KMjTZ/oipYgTmsYK4qe6movDI2v76vcxC 5ETg== X-Gm-Message-State: AOAM5303Uj8rrPXtpvnJYmWA6M+FK/WSdBQWIS65EThKDFBfnEEodSAW a5wahtp72J6dy9lUcequGanD+4VlOI8= X-Google-Smtp-Source: ABdhPJx9cQyIuPQiy+eyS2vpQVmFlPXsgE6hzvopE1cJS9kx41hLGlil2kGIi9VsG74SwvsKG+pp5g== X-Received: by 2002:a17:906:c1d8:: with SMTP id bw24mr13243649ejb.286.1642580846100; Wed, 19 Jan 2022 00:27:26 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:25 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 09/20] trace-cmd library: Compress the trace data Date: Wed, 19 Jan 2022 10:27:04 +0200 Message-Id: <20220119082715.245846-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index e2e03cf9..09146a0a 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2194,11 +2194,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2299,6 +2301,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2351,14 +2355,15 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); + if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Wed Jan 19 08:27:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2887BC4332F for ; Wed, 19 Jan 2022 08:27:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352454AbiASI13 (ORCPT ); Wed, 19 Jan 2022 03:27:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352448AbiASI12 (ORCPT ); Wed, 19 Jan 2022 03:27:28 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 688EDC061574 for ; Wed, 19 Jan 2022 00:27:28 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id f21so7482327eds.11 for ; Wed, 19 Jan 2022 00:27:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OdwH4YxJAeE+ENYL/y4J+2dMWHNLN8/7e0aIdw/5UEs=; b=Pg8Kd/6zm84Ky88k6xkiXZZH8QH2d+ggcy7TFjHWXBrA+C+XxIIsrESumcweQVGCVr wmy32CLLbJc1sQnqfRVqgl6ugSq6O8RtOBsbaxHb4YGxLCoVeo4KocXMmPakDhtfgo2J r2VPs+wF66IKBSzJp4fPtrkRWtlTz2PRGbBZrAqPC+46cXDV7hLSHYriOq4dxAjIPzgM B6lokGzfS49BxeDlcGMKd+OyqoZDepTZReiDoKNsZ5oaw8kipjffOlWU1+Q8JCg4q15n SnflVNZiYVTGSPHTXVSEz+GFo6jxw6A2YFHlPCs9XjstxzweWbhKe9oMz+b5peZn2Kee fS/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OdwH4YxJAeE+ENYL/y4J+2dMWHNLN8/7e0aIdw/5UEs=; b=KufvaKln7f/8ZL4d+Fk8CPevYpG51+XUIsDm6T6NnBKIgeAWDUp12MmV2UIM7X731J UjKGvbQDAWR7F+ex/z7TLCLpIjeXvxGkeDXdMy3wpS+Ci+BkxsOJZkAY9WpqN9z28tf0 pCKIzUyUN0qqOoxXp5eyFWGY02ftmRWDDB9Lr5aYYAF23BPWmwGGoMzyPIpFFkD+RtAz P1V8DfRrPJA8zUwIKUEKIYkRho3a9SVQr+IiYob5d6849e4S9fVHY6MfinlA3pBZH48D GS2wy75DxyTAU7SKmYtzvlE2O6kW92P4S6jbh78bzg+iSbAE3PWeUyt1/8LTvH9gOW6s RttQ== X-Gm-Message-State: AOAM531jHI+tPsQ2sGRBTenZUlhKDcGUJjevojoEiehYqczOSnGPHz8l e7R6O2oad6KfGIn9WRpUU7aV9eYJR24= X-Google-Smtp-Source: ABdhPJzDpKUU2mZQPmWepgJhtmSkiGwNJeujjQZfX2Xl9I1XOeKJuPKl+qSe9gCUEvVoOntZtaUm6A== X-Received: by 2002:a17:906:4a4d:: with SMTP id a13mr1437214ejv.744.1642580847042; Wed, 19 Jan 2022 00:27:27 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:26 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Wed, 19 Jan 2022 10:27:05 +0200 Message-Id: <20220119082715.245846-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 45 ++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index a81112f0..d5295c98 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3045,6 +3045,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3056,23 +3057,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3126,15 +3136,17 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: buf[size-1] = '\n'; cpustats = realloc(handle->cpustats, handle->cpustats_size + size + 1); - if (!cpustats) - return -ENOMEM; + if (!cpustats) { + ret = -ENOMEM; + goto out; + } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; cpustats[handle->cpustats_size] = 0; @@ -3144,7 +3156,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3203,6 +3215,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3216,7 +3230,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Wed Jan 19 08:27:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C824C433EF for ; Wed, 19 Jan 2022 08:27:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352460AbiASI1b (ORCPT ); Wed, 19 Jan 2022 03:27:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352458AbiASI1a (ORCPT ); Wed, 19 Jan 2022 03:27:30 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B424C061574 for ; Wed, 19 Jan 2022 00:27:29 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id 30so7635783edv.3 for ; Wed, 19 Jan 2022 00:27:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Rhbc61nx2547b5/LCa11SZYtwlB5pblDv8z6EA9D7So=; b=NCD4+OQoyh97YZdEOaxBSMFc62HlrEsce9gU08f8ymFpJk6KwmiPMPfg/2IrjLoF6N n3jA/CVjThOd0rNeIJsl0YsQLNGtRGhib3Sa5NoGLHKBoK1fK2RHql/b+CrKZHrdDsEx Mj+rTzJMO/ZmNgLsXx0sXGzWP9c8tT0QJkwpWuoVH0qoMhUqXr386Fr+VR7JNVvCmRIj Ja7tpfmQAy+48BeRiEnnUAe98AI8F2/tgPjr2lgKTs17DZMZXjzOy4ufk8RAKO5hMpJ6 oRqHro5bs3pEGSWJiNnReRvAd4Vh4Duwz2HUl9ggX/1muZ+jsPcEiMxIka0EFMAY394s gq5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Rhbc61nx2547b5/LCa11SZYtwlB5pblDv8z6EA9D7So=; b=SG2+/HGvkIPjRIUCfmI658buikgYRWnihMfG2x1fFp+e3LyEZ9bz4428tNxhFFOdWt wA/UCbcpzw4tnSWe1X441VH/HUTmEZkJOFFBxvTDNe2dy3ait8tvsabigvhu+hqTbmbD xoAh22t0RaOhxfueYRplUdeK+y+BeEL7Xca4cAZNRk54/w5OdAVXESWEJOZoImkemHlx iVlsnhaSj+PrVO0ivlc8toYw69RWZ+YwT5/RAb21Lkzv7/CE6+5S4mlEpa5sY2Z+TECm uTDKFFTZDHmbfIBqs4o/MQCwZo7XEge/eoyMwEGIQhtW+PUmXdtA0TdrW00MvUKvrjst Lc3g== X-Gm-Message-State: AOAM531Kr7pvOgYtMbmlnMRrM/McmbIt+5mLZw0sDJWVLAjs81xipTmn 1Hn77h9qyOVFd2l1Z2NlX3EH3iJ7NHg= X-Google-Smtp-Source: ABdhPJz16bbpHnlnnXyDt57Y5d064F75iOLt79Xt+h+6O4arErn3KvIS4CVMLJt2m0tRgG/fJi1Xjg== X-Received: by 2002:a17:906:954f:: with SMTP id g15mr18051973ejy.755.1642580848007; Wed, 19 Jan 2022 00:27:28 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:27 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 11/20] trace-cmd library: Read compression header Date: Wed, 19 Jan 2022 10:27:06 +0200 Message-Id: <20220119082715.245846-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index d5295c98..ac3234f4 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -193,6 +193,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3849,7 +3850,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3887,9 +3890,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3919,6 +3925,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3932,6 +3958,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4131,7 +4160,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Wed Jan 19 08:27:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14763C433FE for ; Wed, 19 Jan 2022 08:27:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352463AbiASI1c (ORCPT ); Wed, 19 Jan 2022 03:27:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352447AbiASI1a (ORCPT ); Wed, 19 Jan 2022 03:27:30 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38470C06161C for ; Wed, 19 Jan 2022 00:27:30 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id 30so7635917edv.3 for ; Wed, 19 Jan 2022 00:27:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OIi8QagZLh+y9dxAxkuYy9YD6d/O1jpJL+cICmJBoEE=; b=iWqzMGJAoJpQR7Evs8rDMftvS5U07zEred8aynCqyFSG4syb2hc3tNwCf/wKNIcCtn eRaNpZksM5e76dAvonIp9ueWnpi5sBH4hoWNBXPPUMpIu1MtH4feGT4Kvjd7UU9YJ1g2 MU/xfjQG5pbzPxUG/z97ZhW9MPmWpib7TduGHnvUkXMDX25YQL5fkrXlzmMqKiglPGN5 UQaFhaQfk4xGekEjBJUbDJUzDDAKq0VJEc8TYTXbwMoJLUtNRbj8+lgMH2uBti+cpdsr CkKAPOSUR9Z6fFsLcfyFk1I4sUs+JXkkTbA9J9SMxMtXwfgMDxdiPAX6c/hsgPaYtc73 iOOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OIi8QagZLh+y9dxAxkuYy9YD6d/O1jpJL+cICmJBoEE=; b=pc5PlPOBgzW7v1vNW8js0AGrcD+Yy8mvhWAViPvGNmK0TgO0awP0E7ZbiobOTtIUAQ V5BlmX7YRwbx69TuElESR2HP3KDtPO4yqQk9d9ZSOLBvOWj80RHLLc9NUikihX2rcw4o auhyIxT7Ge2b4TWx9Dg4tI4moMbLLlFlZ/J+we9MGQ5JpkUXXY0C3ItHRQXPOzMdy+lb Qxm8/GTaRYfEj4rMsu0u8THGtTcUa3a6mtqvglaSA6gniOzxlaJgN/XR7IZXgFsTAEkl B3PRyflxs3slPVmEAHU8gnuWCKvkngbAr+7MVl1zoDwz4gdx9rCnCXIaA+oD8MkdMY00 s2Dw== X-Gm-Message-State: AOAM531wSjAMXbwtC83te7vZhsTD/6Vmhq9iWTjmeYUM5CQfTMgdoA3J JrvoYRmpmGLIo0z3E+zdpfxF7MzUgOU= X-Google-Smtp-Source: ABdhPJz+gq32/IihVV2hBUR3in2pHpMSJ+uBRkWJ29STAVR/G7OP2XTsfAY5hl2/TCplTDTjhjGmBg== X-Received: by 2002:a17:907:3e8b:: with SMTP id hs11mr23554410ejc.461.1642580848851; Wed, 19 Jan 2022 00:27:28 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:28 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Wed, 19 Jan 2022 10:27:07 +0200 Message-Id: <20220119082715.245846-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index ac3234f4..65b8111a 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -151,6 +170,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; unsigned int cpustats_size; struct cpu_data *cpu_data; @@ -3315,6 +3336,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4096,6 +4118,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4115,17 +4138,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Wed Jan 19 08:27:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC9DEC433F5 for ; Wed, 19 Jan 2022 08:27:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352447AbiASI1d (ORCPT ); Wed, 19 Jan 2022 03:27:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352462AbiASI1b (ORCPT ); Wed, 19 Jan 2022 03:27:31 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D811C061574 for ; Wed, 19 Jan 2022 00:27:31 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id c71so7613078edf.6 for ; Wed, 19 Jan 2022 00:27:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WyquO/8iYychIoeQjr5xHCttbTELREcklo5NEIKijfw=; b=oaonJRzCfKoKTZF968+VCiLs9zlwBaVOKUaj87TEyTE3HELPiOJV7qc6uM2IyRdBF5 xx9RKCSiBe+dWMw6/2i9E5SWXLbZG13Ab2GimQKx8cIb1a5D9NFQERKucSIlHG0MDbkP INAPIMj/8NPXshwDf9sIvL+SvCPqLSd273AAZIpGkH3yMocYkIC8DJDxWj1FU9lRgR8q 97cB8CdvVkpTMX3nD1yePTcDnYpOdas29SNc8TDgVhyFuAybOZTDCPfvz4vTqogtzVgM aFxMHKpk10Bqoi8vSPv1xhjEmAbxkF+dqpqncOSY4/jJJH5ktfYDpoZ6YHVpFXnWVgWk WUvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WyquO/8iYychIoeQjr5xHCttbTELREcklo5NEIKijfw=; b=Ch/KNQywy3gqsLXnmvBtG7Cc+kcdhd4JaS/AV/7f/7P0Dxzb219ceMJ7lQ2TWHIFf4 DGoer/X8Mv7I7Mil839mJo5XVj5evXPEMszS9yAfL7GGFzq29gqRzy5H6A/GEMnKtpq9 YyYgJFSdcu8W7lHidN9M4T+jU06WE+bNUt712eQ4heaa5SFKkOvGxvo1RfBCHuv5DQRk 8kSmo2aZ2M3LCyLaNTUGCq6AzzHoEAU2VGJmsYtrizlt4jh07Rvc6j9wfi1Wb5WPui90 vq0TDIicf3o3sDgXEkRH8kZfZ6diRY6aoTAcV/pOSkXuNd/Fgo282q57vIaD4YhFgVYk MVzg== X-Gm-Message-State: AOAM530on1Qr2bAVvUCK0bJydQpdE1C9J6L1ooe4bKu3EZcEhzFp853J g90Z79cYvc43ljLz58277kx2EHFLk4Q= X-Google-Smtp-Source: ABdhPJzITeTIJay7mrT3/ObV8sPADHVlQQ3kHrTe/svrj9xvtiLA4kyWPz6+A/stFlmIa1/YkBSKkw== X-Received: by 2002:a17:906:4790:: with SMTP id cw16mr23568727ejc.701.1642580849867; Wed, 19 Jan 2022 00:27:29 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:29 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Wed, 19 Jan 2022 10:27:08 +0200 Message-Id: <20220119082715.245846-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 65b8111a..45a87a63 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1266,6 +1266,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1305,12 +1306,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2498,16 +2502,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3393,6 +3457,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Wed Jan 19 08:27:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD81C433EF for ; Wed, 19 Jan 2022 08:27:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352462AbiASI1d (ORCPT ); Wed, 19 Jan 2022 03:27:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352473AbiASI1c (ORCPT ); Wed, 19 Jan 2022 03:27:32 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20832C06173F for ; Wed, 19 Jan 2022 00:27:32 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id m11so7427001edi.13 for ; Wed, 19 Jan 2022 00:27:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6SGw2HVDaHzqWOD6TYkmGMJuM3zBd6yLfGquv4wWDRc=; b=ZNTQaacDY5l9pFxWxt2YjRbYJr0eYeQ7mAtIp5f85w4QryoQV+adMt5bB07oMiB9Js Vu+l4pdvxiroB3iEUa7kV85lOXFus7I0BHK0NcQJxDd5yU/OeOm4sZYZ4snvARU4p3yg SBmA4KaRSs265Yw+kbjD0OUKW33KUp2pLqxUYHOlCol7LPmcPMOSA6euhzcMxO9TT6pc em0LJyH/wGt8wlnbSxYtCjBPWTtr8WrhzqyjJ50RUOril1qNZmOmTy2qWmSAexorY41Z 8DpDNXxLApztRqeK1inD9DYb2NUPYl8dia2YFoNEbkQwZkIQtPariOB4w2Nk65C50/iT roSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6SGw2HVDaHzqWOD6TYkmGMJuM3zBd6yLfGquv4wWDRc=; b=eWObZoKARFkfOfuOb8ep7P2ketwZJnwhYCuCT5qLrHM1ZjNx4n5W0aJUnYe97Y13Kw QBBUJpwEYmYc8zGJ4ggsuQtpwlMvLjf6AWV30TTmkxZUCcRj//N7/CpseyraMKxQN8wr LgxZGbGEX+bagqnj4kSd2Uwxel/cz8pIAL7qMLgUgPJwUV0QHIboy17qs7xWTLaiHPRG 6ADwjB9rcRSUXXWeKbKHe7MYkAqDmuzGFApLjC7wVcf6yIIOQzenu7X/WMu3QGG4Gd2S 6ulgSiLzmt8QXae8pU3mGLhBvE+ZOA6hedYQ6L/uOHwTI+RTd3jA19fJw8aJ4tWadquA C8Pw== X-Gm-Message-State: AOAM530Uo5+nu6KMPXhcQEPmh5Ivt+I78CvqTCGswCo9B2B+5FNzMCQ8 lFzcHtILhIlh1bGnZDnCwRHzuI4uqFM= X-Google-Smtp-Source: ABdhPJwVTRaWg5vRX4m3G4BVADpmPcjCrfeXhJVMu+6Kr2PQlJMX8WymtetFThDFzYemBk1KWA7PWw== X-Received: by 2002:a17:907:3f93:: with SMTP id hr19mr1080186ejc.57.1642580850741; Wed, 19 Jan 2022 00:27:30 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:30 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 14/20] trace-cmd library: Add logic for in-memory decompression Date: Wed, 19 Jan 2022 10:27:09 +0200 Message-Id: <20220119082715.245846-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 45a87a63..f5241e4b 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1257,6 +1260,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1268,6 +1370,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1410,6 +1515,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3954,6 +4061,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Wed Jan 19 08:27:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96ACAC433F5 for ; Wed, 19 Jan 2022 08:27:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352479AbiASI1h (ORCPT ); Wed, 19 Jan 2022 03:27:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352465AbiASI1d (ORCPT ); Wed, 19 Jan 2022 03:27:33 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AE18C061574 for ; Wed, 19 Jan 2022 00:27:33 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id j2so7540395edj.8 for ; Wed, 19 Jan 2022 00:27:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bJy8UUzL2DhI3wDSTtX7iRU3BncYzFKDyu55GqD5EMY=; b=Zru0kviUvMd8tQZYHRchlfXTv1DlivLPBobsug2kE8tROI91Z6G2dg3cZEPItvDVQH kRpmE4r87J80wQGMZNiXHBUie09kFPMugQfrK3pQQ8Dt1j2SD399p7mSlrtA9NfnlQz+ zXhOwx8KgM2T0egnHRe3ntindvrC/pRmJdDg4spWhFJxqZU9hChYqjfZx3V2K1rOdZQR EkbCuT8fUifC3OSGsxKjoYlunhI7invMtNMKSFYe46BOtKIq/D33kkl3zJF/ghIYueU2 wr4raple4fBIej+jJZodM9M7S76pfmA5QIM8c7phZOxxlxmqJL1YDMPUGXDDAo9Ocj+Z 7uDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bJy8UUzL2DhI3wDSTtX7iRU3BncYzFKDyu55GqD5EMY=; b=QviBOHPyZfDn/wKvQd+0fXrleXW3xt3sq/zCkJfhlr9JhrpBBmN7F5djAhuy+NxQBX GNm6u93AtHR7XLI+rulk45LYMjpYlLmsI0sbaNad2Ts3gTueGf8HOmXn7B6sZBGAzv09 aq35KVSvZY7ADYXQZLUARzr8vCytDmdjnu48kppfd1hCH2XzPe2qVdTt40FFZx2ryUxS yF73pXhPfBvx47Uw/tXgPQ0psBGquMXWrRT7pupsdLCKOpJHMnJQHzGMVNxS72MNtnMf uqZY1ZBqqTv4j2alLjxun/GBYfky2wRfVL/VTuKxL61gbEmWFZw2DGVmEmbEdnbjiIo4 h7DQ== X-Gm-Message-State: AOAM5334mROeV+kGmQEj7L9blS31Bs+ufTZC7Chi58bSvjersZP9c2Ds qQDYfAvgbnI9v/K5AIM4iVFYM4nY0dM= X-Google-Smtp-Source: ABdhPJzKXYnfB++kcQx2l7tr75FFYvc+YuqEqexl+XCkyMXPTJK/TB1JRUQZh71AyKYGuZrqnfBmIg== X-Received: by 2002:a17:907:7292:: with SMTP id dt18mr3922250ejc.446.1642580851830; Wed, 19 Jan 2022 00:27:31 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:31 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 15/20] trace-cmd library: Read compressed latency data Date: Wed, 19 Jan 2022 10:27:10 +0200 Message-Id: <20220119082715.245846-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index f5241e4b..e772b463 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -177,6 +177,7 @@ struct tracecmd_input { bool cpu_compressed; int file_version; unsigned int cpustats_size; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3465,20 +3466,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3546,7 +3579,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -4058,6 +4111,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4349,7 +4403,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->strings); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Wed Jan 19 08:27:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81D00C433EF for ; Wed, 19 Jan 2022 08:27:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352470AbiASI1i (ORCPT ); Wed, 19 Jan 2022 03:27:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352458AbiASI1e (ORCPT ); Wed, 19 Jan 2022 03:27:34 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FD98C06161C for ; Wed, 19 Jan 2022 00:27:34 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id j23so2454106edp.5 for ; Wed, 19 Jan 2022 00:27:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vpewCo1rrp1XIy486ODOklGV+ZGpCBJZq5i/DlMZglU=; b=JM184s1+NH/p+6uZJ9dgUkSQxT0zKLbadaB4UjP4d05r5v1NGm/I2qP/4AaN831f4B iTZM4stqaFVjhgebBeEEhXF0Y0FjuLiWmhlyQ6gcNLG+MieGVuQEHu5RkfFM6Pt+kuPG r49cN1zTL/LQnrnFl/9JHgePCuHJhM2jNwiyNTuAUPaAY9lWTK/ys3zAgErHpkzl+MxT iDeSyQsQ2+I/zXVGKQisd8+8EeERsPog0e/kabpFg4FSOKwKXq7rySPUkSNg9kil2NNM iFsKN75has8rtRG9Df23Td5mLkskEnNfa7ZtT9iCJlTAAzmIpjdJ3KyP66JNtMxyDMmT TNBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vpewCo1rrp1XIy486ODOklGV+ZGpCBJZq5i/DlMZglU=; b=sMhZNd08AiLdANFsLZeasLC4qgrWSTBp3IsGA/lf8n89eZaKbq5oFMRfSFAKCDJ99B 97kbpwP6BcD35Welt4wkvT3MS6YWPyzXZYZXiYgtDGBkl9GB0ig64YutyocNNhp+KhCF yvVDXMOkL1OONBP8w/HpkQvI81G/Pvsj+Hu5DEelRl8g2Go6+feQhB2dc04U57Vrr1kY V/6BuFUFmLJ2OVaKwhCWiCva53Qug7KbVdhodh1q8YESq3h9Oq89NcDbcZPYNOyryVtH Kh08l7ZYuw0vIV0vhkDUn70fMXHcphghpQWSbKwLct/VS+xEP0+RGDc5TRMAcZY6Z9Ht fFjw== X-Gm-Message-State: AOAM530AbLPN5geKAyY4cEp6yQvoY/kTLB+eWxro5hVOXSJ3taMN8x7k Puo2MR5o2dPp07ZqSGymadD+jy8ue3c= X-Google-Smtp-Source: ABdhPJwWsegwfoOOQMZnpBYoZh3Awp7PgCsciZdRYm6ZadT+jodiyB5X2PCi2o6X4AOqOLXqLF8+1Q== X-Received: by 2002:a17:907:92e:: with SMTP id au14mr21457914ejc.547.1642580852682; Wed, 19 Jan 2022 00:27:32 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:32 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 16/20] trace-cmd library: Decompress file sections on reading Date: Wed, 19 Jan 2022 10:27:11 +0200 Message-Id: <20220119082715.245846-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index e772b463..cac1d554 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -484,12 +484,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1116,6 +1119,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1141,6 +1146,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } @@ -4055,6 +4063,7 @@ static int read_metadata_strings(struct tracecmd_input *handle) unsigned short flags; int found = 0; unsigned short id; + unsigned int csize, rsize; unsigned long long size; off64_t offset; @@ -4064,7 +4073,18 @@ static int read_metadata_strings(struct tracecmd_input *handle) break; if (id == TRACECMD_OPTION_STRINGS) { found++; - init_metadata_strings(handle, size); + if ((flags & TRACECMD_SEC_FL_COMPRESS)) { + read4(handle, &csize); + read4(handle, &rsize); + do_lseek(handle, -8, SEEK_CUR); + if (in_uncompress_block(handle)) + break; + } else { + rsize = size; + } + init_metadata_strings(handle, rsize); + if (flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } else { if (lseek64(handle->fd, size, SEEK_CUR) == (off_t)-1) break; From patchwork Wed Jan 19 08:27:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ADACC433FE for ; Wed, 19 Jan 2022 08:27:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352473AbiASI1l (ORCPT ); Wed, 19 Jan 2022 03:27:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352476AbiASI1f (ORCPT ); Wed, 19 Jan 2022 03:27:35 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DD19C06173E for ; Wed, 19 Jan 2022 00:27:35 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id z22so7449763edd.12 for ; Wed, 19 Jan 2022 00:27:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kdybLkXisjJOl1Uti90EsSuHbqy3Z/tOcfeXyKUJH24=; b=dF05hJWlzI/I4soZpp6VfXtL/+Ro9DtbOwKgJ/lSbUOB1MB/Oh96GPg4zbD+k8ACjS rA8g8cvzsi451r8uks7fzuq91witiqCnMRGOa8RPjhPukKhpYzo3v7rXItxCq5qXctiK HCHCIaZ8GVYYdzmrXRdAh4ndUnzLUnxCXcX2SlYSMuXWzUBl7MjRvhnmB7H1Y0jmfqYo ExlO3fZyj5UVasCEWNd07QgZ59+ZKORhy1dXMQqpG5YzKpD5qWGoHzzgKEAANTiH+05T gI7lO7sUOC6dpVdFO7mglegIxpysFMJJJr4T0FunVisg4A/fwS7e4L0yZr57wmzYN0Fp O/MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kdybLkXisjJOl1Uti90EsSuHbqy3Z/tOcfeXyKUJH24=; b=No+CqhiJdF5FTOCX/XKHVXZ2iKU2cwVY6RlK0LS7rIDL3tihgYnwQ4k5XKDL+AUIy1 5CGNGtf74EOHNVIK6j+qxzi0WObwpAFv3ELVGZfAKjUxsSWATZUv166tVUuj8+OzGfoq NEKdwdkJ0DG61aoj7Bbfn78DYBgyR4gcm4GD6y64Ktk9Y7WaGmzJIqkX9qXc+y1R3po/ QYT2paHbJEOa9LRYzyFLHBjgdoxVx63oK071Af5GDye/Sytjaax4sCk3pacSYylahbXG VS8Ts2cgIyR231HeIFHAuzFJvX+Be1E46tKDOrqQAar2yU4DUbcemRfI0ZG+pO3whxPh GBPA== X-Gm-Message-State: AOAM531DCrvM8zDx62i1cCZ2QsYZ5UlJ8i2N/DkBjw/R/a4e6igG5mJ+ UEwHi6f2Mfu/3fEALAs68PtJqxgkcw4= X-Google-Smtp-Source: ABdhPJzEPPCHp0EJiPMn9RP4+Hx/xNhUF6Z2x4yOdJ5V9X2xHHSDO5pWuGwV1V5oDKi24h8Fd7dT4Q== X-Received: by 2002:a17:907:8a0b:: with SMTP id sc11mr16234637ejc.376.1642580853656; Wed, 19 Jan 2022 00:27:33 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:33 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 17/20] trace-cmd library: Add zlib compression algorithm Date: Wed, 19 Jan 2022 10:27:12 +0200 Message-Id: <20220119082715.245846-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index 883cf669..3d8f31bf 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -359,6 +359,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Wed Jan 19 08:27:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1A37C43217 for ; Wed, 19 Jan 2022 08:27:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352477AbiASI1m (ORCPT ); Wed, 19 Jan 2022 03:27:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352478AbiASI1g (ORCPT ); Wed, 19 Jan 2022 03:27:36 -0500 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E50BDC061746 for ; Wed, 19 Jan 2022 00:27:35 -0800 (PST) Received: by mail-ed1-x529.google.com with SMTP id f21so7483629eds.11 for ; Wed, 19 Jan 2022 00:27:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v4D+2uCZ7+pjLxy4nudPTs1AVxRDbpF5rZR3iadjTKE=; b=l5AGYkTOcNSySrknDOLwy4D0Cgaq/CotYoCvG4qWyaj7O6DAvq1rpvEQbQotVmbHFB DOt1pFwaqq1Nil2kmbz80LdGYMBwXdYQLVsbKt1Sye2mtGgDMc9S2c+qv4yXvsOk0lME XJ+pN86xKmGo+GN8WAk7Zsu1MesAdESOLyRvdVwgHoCABr5T9jnxn/cjcTzmAjbCJHMV v2Jj2oTlA+Bnp1ylz7sg2wtaFqn4XhY/ECPpr9lNJbe2a9oWEdXxsO84mgz7H98gZ2+i DfAYg6C6mOhSCi2gPARnx36B3Q2qRwxxapFJ2DgvkKxDnjR9V3w4TETt6k6c5g4guS5v 5KhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v4D+2uCZ7+pjLxy4nudPTs1AVxRDbpF5rZR3iadjTKE=; b=KieR/3GdBnvlbnCV5tyeIGj5bK28PsOiUQjky3+bDaOUHnzWCWgfq/9Gd03sjlLZFh sZv39HF/Ct86DkxQpWBR8mPeI8ZGum9qWAazFto/zvj+tUbOumdjBXdE6N98rm/+Zugg EhuaY3vRAkpRRj+INyx60T275y3HHE5r6B3+UJkOCDwayHOI4FwrEGc8EY9C6EWxcuNV 5fcBhr2NABoRqMnf/GfZfPkjs2ia8qXV9SKy8itzKKPXMstIQYAlL0b8rkP1nxIznkr3 XmehJK2JeDvfN62s9mNX5VEhH5OBWmq2sfe1cfh2JbWPU8ICpCASVeWjpsNKX2o/LyLq Q6ZQ== X-Gm-Message-State: AOAM5311dzETeQ/wQzpLukdByR/92130oHpvoK6D3J/fcj/jIy/Udb58 pj3IlIF4LqxoRgI8O7VbprIioaVzL2I= X-Google-Smtp-Source: ABdhPJym+UPodiT+eghjzaSxze9HzyEn3NHdL1RrvC+AhTgi0cxM9dy2NXtbqVnB5IrOZu8j27KrFg== X-Received: by 2002:a17:906:66cb:: with SMTP id k11mr3265149ejp.85.1642580854562; Wed, 19 Jan 2022 00:27:34 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:34 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 18/20] trace-cmd list: Show supported compression algorithms Date: Wed, 19 Jan 2022 10:27:13 +0200 Message-Id: <20220119082715.245846-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Wed Jan 19 08:27:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EC70C433EF for ; Wed, 19 Jan 2022 08:27:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352458AbiASI1m (ORCPT ); Wed, 19 Jan 2022 03:27:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352480AbiASI1h (ORCPT ); Wed, 19 Jan 2022 03:27:37 -0500 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01EFFC06161C for ; Wed, 19 Jan 2022 00:27:37 -0800 (PST) Received: by mail-ed1-x535.google.com with SMTP id q25so7671715edb.2 for ; Wed, 19 Jan 2022 00:27:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YCky87BbhWShbH/aduoBm+n7yT9b6LFV+2ZPGGNgaQo=; b=L9seZpxqMR+AEnFqKxqNVt8RY/l22MtEyxo6E9op6s6wplTqZp7xrBA1IF5fQvTkj/ Rrz7eFkBScHWLaHIyq7xfPbPcYVtiYc8Kh9NGMJXFXGh9A7lRiAwS9XyKk+4UVdd3Uql kKNHUq2U7BxpRC0yFC3O1TJwgo8ig7zgKM0w0E/nrFMi9pw7ibTZByen0G+b6TPyuTmU UeC+F7wvxu4q7pVPbHHHyfAho36vFZoILpO7FOd4K7nIroj2SenxgK9JeDdfeYEzkN79 CzpiwLDPjMtq/fh9NzsTWSYGa7laebXmpKLhWYu80MuX/FYLlQYSc6s+AS+2VYbqCIsv MVwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YCky87BbhWShbH/aduoBm+n7yT9b6LFV+2ZPGGNgaQo=; b=HLllBLTMaxbE23ySYT/cdCI4pAD104LMjgOYfNdg1K5VfpPd6TawhenkzLyZ6GUt/1 MoGynOGmPpcDSOauEMFF5oizYP7QN2ezaVd3jmT9zK9qudRo1WBG4CHAo0UmkUNgl33D tztRlSBZ4td2mTjZHNUqjH/TpxTfyYGj5gk0lNWpSKPJY4hBWwn2qca1kNQEC1bkYXql 3kX02J0qkX4EbgcIaTYG9RHwxMdivzF+D4DSiEpfolnRWYAcizyxD72SX29YnFJH6II6 SAHFnpSkDkdmNGia+BV2UKBpyeWEpwwqeFb7PvsXxQYcV+4zmn5zo5Az/zrT3ZxVDBpA K98w== X-Gm-Message-State: AOAM53195KFDvulxGBhTADK4VTmB1quACl7A1TzeiLSixgzx/vTFrCvj MyAArP4hocyQVFNDzuTkoryfvz3IYrM= X-Google-Smtp-Source: ABdhPJxQPZze4SVzS1Q0UPibBI6j4lBtp++b+6QZOoNmUYqdjMM3WMN7AMDdmtpWepmcT1O4ZKq4WA== X-Received: by 2002:a17:906:dc8a:: with SMTP id cs10mr7814463ejc.598.1642580855602; Wed, 19 Jan 2022 00:27:35 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:35 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 19/20] trace-cmd record: Add compression to the trace context Date: Wed, 19 Jan 2022 10:27:14 +0200 Message-Id: <20220119082715.245846-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index ece5a0c2..f1dfe458 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4477,6 +4490,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -4511,7 +4530,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Wed Jan 19 08:27:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12717249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF13C4332F for ; Wed, 19 Jan 2022 08:27:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352472AbiASI1m (ORCPT ); Wed, 19 Jan 2022 03:27:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352483AbiASI1i (ORCPT ); Wed, 19 Jan 2022 03:27:38 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18D17C061747 for ; Wed, 19 Jan 2022 00:27:38 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id j2so7541282edj.8 for ; Wed, 19 Jan 2022 00:27:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=g9Z+lPTqfjjx1/lJ4Ho6epL2jahPAOSSVxlt4Bx/vek=; b=HgiXzHhQqH5guah8aaMCBG2lqWJGdUt7240w0lX7/ZkjEfvpmp+8XnM2k/qKF1LeA4 s2IAnrizNtRSNhLZIaEeIK29a3WisnjHhLsEg931MO+Y6DaZA5zM7VHJg2wlDfjz1Kxe ysyyx99u7vsmV1uxFDlmdYZKVA2hbynOIGIBNVc0nxM2/G+vOMn2Qp+jAWGOMPSoKNip tSfTsytsv7d6G7kt4yGIPT0Sb6EWrAxbqXOsDf10p/hoQdPiLTTaiHXOtM5Rfd64Uti2 ECMFcHOEVxSkZBzOO3o0cyJd3vQWDGqExJS7/HybFurCY0XX1YtWMjFRcTBSQihJ5CYB r1GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g9Z+lPTqfjjx1/lJ4Ho6epL2jahPAOSSVxlt4Bx/vek=; b=TxFGoY1El1o+l0LyjWhzRWPKdMr/HMjELK9ww+DSkrWBnEC/YOj83FR3QcC9reyWW4 zCsQ4FshU9zOA/5UnCJ6TpwEys85KzUX4Cebrfxu/inxnoBcABw9zUFA8RTzl3FUmIuq WZwXwfI8M338X8jjxWSzmx3fGMRBzjkqeG7vpB/HTY6zIzw92GMBkyT3iqoPs5qVM4Kg 9yNPDcrbnyUSFE3ryqMc/IdgCkaKHSG1PEsIDsN8EdytMdLAj/Vvu6yfGNCHR6EFNOAN XW04nZmOtC98u7EhenqSRK+myiuYLmYMCeY0Sc3ubnj7yniHCt/eN2D/Y63IvL2WEfkX QmLg== X-Gm-Message-State: AOAM530ixt7NtEV7lzsa9C/MfmBXJd2BQYVjRx3123tY3YZNAMoEnOp6 oGV/NtNgsrFV7OIJfutB5uBN09c5Rbs= X-Google-Smtp-Source: ABdhPJziB9EcpZkY9QnGIKwtSULnln8YF5hUj5VzYrMSArwtDafkJSqwYzGkm5SyXbU0FK8/9d40aQ== X-Received: by 2002:a17:906:3e8a:: with SMTP id a10mr23195359ejj.612.1642580856641; Wed, 19 Jan 2022 00:27:36 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id d14sm848703edu.57.2022.01.19.00.27.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jan 2022 00:27:36 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v7 20/20] trace-cmd report: Add new parameter for trace file compression Date: Wed, 19 Jan 2022 10:27:15 +0200 Message-Id: <20220119082715.245846-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220119082715.245846-1-tz.stoyanov@gmail.com> References: <20220119082715.245846-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index f1dfe458..d862c7fe 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5804,6 +5804,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6244,6 +6245,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6670,6 +6672,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: if (ctx->curr_cmd != CMD_record && ctx->curr_cmd != CMD_record_agent) die("--file_version has no effect with the command %s\n", diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",