From patchwork Thu Dec 2 12:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 108DEC433EF for ; Thu, 2 Dec 2021 12:25:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231488AbhLBM2e (ORCPT ); Thu, 2 Dec 2021 07:28:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231482AbhLBM2e (ORCPT ); Thu, 2 Dec 2021 07:28:34 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 821F9C06174A for ; Thu, 2 Dec 2021 04:25:11 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id w1so115506360edc.6 for ; Thu, 02 Dec 2021 04:25:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QF4oTcXZsc+PQgY09YkJ2eWArj0M466PZLZlUEnyJWU=; b=gfu5AINnNzV/OaJgRRD84q6cIGGHbMJrf5lGYao2EII589zi8CxjU/GQWPC5JsEJ+c csEcJnOCUCIW26iZto+XNMMkX8jD1MJPn9bhTz0HRLChmNDIFrrCD3mGc3rh3yuOa60p hSxy2NUNM8o4rNVsL8siqrbMyo2bpRcoyHsit2vu3+CdL1Phm6z8oGZnNk+YonnKkg5D O5DNiWyQX1T/SHEkuY510jlTeed7onka9OUHyTf6/3ezSurJtVNHHqDQN4i7QjQKHrBm vJvYvzrhKRm9nP4YWxhKwpQU1Nn+39VAYF4J3aVwgHIPkyQdfrDRieDJ1usGKG3nZO8p XXmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QF4oTcXZsc+PQgY09YkJ2eWArj0M466PZLZlUEnyJWU=; b=IhFIr1zrfsjVUmRLsH5EsL6sAm3hOn/OTsh3x57qeEf8WycarylzbkMxVF8J9dvKiL mI47TDSMqYbiw8Jb+Jx9oQDQ9HzA2sRTWUrbRouxsWaEEflxUUQwMj99GE+9o4LlxPZr qwrMuMrhkI+JkYIzKUOOB54bZP+XKri79J8SuxcB9tFNnww6sXMyTcj18D6UPWewGPkA p4HB0KPoOQich/Bhxd1s7yV4F5AnFbm0z9c4Nbl7C78eRg0QK6WOqL/jZBA6OYoYmWsr cMBshSGICl0Pq6ga+nXHiMNoaIrCaiBCJhSNhZnJPfWC5FIKASQ5NLQDOP7oUC4vwldS gdRw== X-Gm-Message-State: AOAM532E4WwKdvK5Oq9U+M7mPb+hm6GlhwdaVRTgfFde3k2VS3sTU4oM 9rSepsVD4akuZhd8nj3AeI8KFEGHOSOuuQ== X-Google-Smtp-Source: ABdhPJxTS+Iio57ScUe2hjHQOm20MXyUw81FrCIO/EmlrQkBRO+8fHoBkRHgFAOSpYx9SNFckZd1hQ== X-Received: by 2002:a17:906:3a9b:: with SMTP id y27mr15064369ejd.563.1638447909729; Thu, 02 Dec 2021 04:25:09 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:09 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 01/20] trace-cmd library: Add support for compression algorithms Date: Thu, 2 Dec 2021 14:24:48 +0200 Message-Id: <20211202122507.43572-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 907 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 962 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 0feb4b30..fea75f54 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -35,6 +35,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -154,6 +155,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -485,6 +487,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index f4b68352..23970341 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -36,6 +36,9 @@ struct data_file_write { unsigned long long file_data_offset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..133d1308 --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,907 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handle + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handle + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handle + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + handle->buffer = malloc(s_uncompressed); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handle + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handle + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handle, used to encode the data + * @msg_handle: message handle, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handle. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handle + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handle + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index 21f1b065..06071f6c 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -635,6 +635,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Thu Dec 2 12:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4CFCC433F5 for ; Thu, 2 Dec 2021 12:25:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231482AbhLBM2f (ORCPT ); Thu, 2 Dec 2021 07:28:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36542 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231743AbhLBM2e (ORCPT ); Thu, 2 Dec 2021 07:28:34 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 360C7C061757 for ; Thu, 2 Dec 2021 04:25:12 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id t5so115449315edd.0 for ; Thu, 02 Dec 2021 04:25:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KpTIxeAACJKs5iFeyI0KTMzRKHfHKLzDZVcXaDc7wqo=; b=jVqb4P0Dxxcutt6tNPqlnwggsOhzKrz9OPRWVhSn2vFuS8wbmgJ7A/Bzym8b1eaNyC JklGgVFsrUfJbaAEWVRdwpM41WsMP3u/NuqDbdPNLEVJwjF3MCM11K2HeD2rSvFtPfwu +5IpyL+HfZ7QoL0zi3Q3DMRPW02p77JfMFdo4gNcAouUmZ3Y3yWdviR5DG4UtBArd1IU LrnoFa41ymdQLZCz5Akr1hZCeWcMkUh524GMTvqboPHfaIRwEci6aJkZ39j58N55+zqG Qqm7raX9G24HwbfVoof0fYnia6CKtUub75/nfLiub1PyPXhDwTtzDbmGg3C1DdgAfWdp pLmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KpTIxeAACJKs5iFeyI0KTMzRKHfHKLzDZVcXaDc7wqo=; b=dTMbKiFUD4PRIUMj9SSCiNEE2DeEX8L1qOx+5UV9wMR1clYwW5GNfD2OoUZx+V5Zto E1KnHotmf0zD5AjM/QAnZt/zDz7RDpOgJcdQlPaibsacl93Q988TIkAH9z4FAk1S2SWv 1tgwOMocN1P0iBJgKd2HEsT+2DKn8EnLBWHref1fampn+t7J82vWPhMzADhopD0DqcJL 42Txb8UK/uCg1bMes8GFgHZr2k1IhxXQA+qkwjisMDcvkdoQdR1gFMJ0PUnKp5s6C31V bAeguWiijUw44kpU9oGcl56Frx3ZdZyAMbn7pc6VPV+UnVeOG61AuRDbKKFu/CMBpYlE VzPw== X-Gm-Message-State: AOAM530CMgU1ojKHxIQbTPxylhmWcMZsixKwVZA7MF+PlLbI0YKc2gtC CCywbdp1NpYvSkDsXPXNi13i9LiCrVnGNg== X-Google-Smtp-Source: ABdhPJxXN54N36eXdE/mePIZhLvNbObXBDEOKtqBwctTS2J5/NUJhUF631CIfkFbkXNP0OcneSQ0pw== X-Received: by 2002:a17:906:12db:: with SMTP id l27mr15130312ejb.244.1638447910598; Thu, 02 Dec 2021 04:25:10 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:10 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 02/20] trace-cmd library: Internal helpers for compressing data Date: Thu, 2 Dec 2021 14:24:49 +0200 Message-Id: <20211202122507.43572-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 23970341..4d58438b 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -42,6 +42,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index e57d32ba..455fa93b 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -64,6 +64,8 @@ struct tracecmd_output { unsigned long file_version; tsize_t options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -90,18 +92,27 @@ static int write_options_v7(struct tracecmd_output *handle); static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -129,6 +140,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + /** * tracecmd_set_quiet - Set if to print output to the screen * @quiet: If non zero, print no output to the screen @@ -1562,7 +1610,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Thu Dec 2 12:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A14D2C4332F for ; Thu, 2 Dec 2021 12:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231743AbhLBM2h (ORCPT ); Thu, 2 Dec 2021 07:28:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231820AbhLBM2f (ORCPT ); Thu, 2 Dec 2021 07:28:35 -0500 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F552C06174A for ; Thu, 2 Dec 2021 04:25:13 -0800 (PST) Received: by mail-ed1-x52f.google.com with SMTP id t5so115449499edd.0 for ; Thu, 02 Dec 2021 04:25:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bVSawsv30ZVfg5NoTn5D9FwmtdRF9aNmZ70voQlRL6w=; b=oE6MNkIqBrtsob/OlfSSmOu+jlVN+7pkTBQOtF4w9JwU9JNuU5YbzEZgLj20ur4QIb UyGysribet+YSToVRIb00YEYX5nFVO+XyTMJmBFkLD2bBeN5aEd73Ta8uXkd4jZg/ERX pq5Up6ttMQbPzQiaJ/7/z2BAGLXiMwEKd6uUOmOKVtspRJcPaV0hzyAEMZZHKP6z7XCm 3wJzIxtZLpSR1UstPoJtCMEL+a5z/gotcq4h34qQAtqKxkGuUYxjTmjWqM/llxgJxV70 QcBYVS1WbRzL/+sYgahhmUo15UNX/jxDuN47jfYyE9lMeWyPOyswwPrvkFnilrzoRGA4 oF8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bVSawsv30ZVfg5NoTn5D9FwmtdRF9aNmZ70voQlRL6w=; b=BweYA8iJYMDopY4xjvNlmnL8pfSM8hLBMX6Bg3PLP5oe2tL2qAZTRtmkIRGjTBJUx5 v1flLi8bdTgPVPb9m9tUJrSBC3cKltf50ty8iRnBEiulvdsra+p9/9YvKuOSd8c2It7W pn5Pzwr77uQahN0XJm8tmwwkF9lNlt3O+JVj04FwgBXaE7tyAqkurrul4NP8S4dCOosV Pb1QnVkufiICKAlgW72OoaIJR312Y5wklElfX3Gzj7gHhIRAcZrXGoLt+JZg5+KofRhO 7ftxo6L28wBMUUe5AkUxSEImDjRzEpxcGD7XNWHJUPKT1K+MzlzXx9dTLnQIhU/Qs547 7syA== X-Gm-Message-State: AOAM532NsJ37kMhsCWAsrxxqypxuHeII4zEdkSFniF2UYntxZiNwq8k+ txBo1W4zutTn6MbsbyrfPq8ua4RzzypBtQ== X-Google-Smtp-Source: ABdhPJztdlyJ5O1sVbQx42IQ8kfxPLGJdpezqm04HDX3TumOX8+yhx/X0Wu55D5LeTKzFSPgplpkbw== X-Received: by 2002:a05:6402:27cd:: with SMTP id c13mr17431240ede.236.1638447911714; Thu, 02 Dec 2021 04:25:11 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:11 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Thu, 2 Dec 2021 14:24:50 +0200 Message-Id: <20211202122507.43572-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 4d58438b..fb479d8d 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -49,6 +49,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 5b8f5c93..f6d60aea 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -157,6 +157,9 @@ struct tracecmd_input { long long ts_offset; struct tsc2nsec tsc_calc; + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; char * cpustats; @@ -263,13 +266,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -281,6 +284,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -305,9 +324,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -333,7 +350,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -397,6 +414,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Thu Dec 2 12:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78441C433F5 for ; Thu, 2 Dec 2021 12:25:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232118AbhLBM2j (ORCPT ); Thu, 2 Dec 2021 07:28:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231820AbhLBM2i (ORCPT ); Thu, 2 Dec 2021 07:28:38 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 865D8C06174A for ; Thu, 2 Dec 2021 04:25:15 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id y12so115202851eda.12 for ; Thu, 02 Dec 2021 04:25:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uqS6FFyiUt0C7g7rtUtam5Y2FPLT2botWa/gIDeNprg=; b=Z3soMD3HZBN3V4ycdor3uU9xDwlSEyGB8TUlvpbCuDEPFcg9qv0d0hGI+vgky8Xln3 BXxwCagBN9wIWr011k5ntSuBGCosyNeKJ1fyiWBRVA5zTE7w8goIyUKufta0qn0gYQRR BGBZpTe7RNq88HTPrvEDW1MNOPxSxIj41Noh+MNwRxQudBBLiAX+n3/UslXDAOyXinKP 9junYNAUSbsIsvGwx7apBUgRy69EAu6I1KMW/jrsUPYjhQzEzKZixkc4lHZ6xIa6Vrap bF18M4TD2bSZ+YB8jEZqaS6ust96i7UQsCzhms7NR9kM2glSTCJ2117k9FkoVmq2Axtk iWCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uqS6FFyiUt0C7g7rtUtam5Y2FPLT2botWa/gIDeNprg=; b=nx/KwkFM+YHNc2EjpZZkHx7GIbQK08xL9nHMlZbujVh7CErlt4TjDZM7+uWGb77mu5 JIX/CjS7S9RFOslOWxbEk3A6l2OBS9mI1U6M4LQDAVNVi2ymnytcLVUU/u5LHM1fBtht OhZm6n0PuJ80SYhwyZjhYP0qbjOZmhm57JD6+OTv/8ZBDUlUTnFUAlQEWavqHRanp50C +VV9XH3ZEc/R71brEo9ydbVQ0Dl3NtMd+FTrImOBSwDcTs/Pf5S34iyZgqpZibVYDwhg hqXvn24ha6kbruZgJStulxWZtMA3oClLeSzFDAAu5xENHkPM8bJ1WZt/VAgL5MZHb+Og IBOg== X-Gm-Message-State: AOAM533Bf0m3CXxW53hIXr8zyKeCxaHaDF3/XzJnwN0GtdMhinuBTEOF I+pkL3VrSnzymBXPmADzkPXm3s1YYrQArQ== X-Google-Smtp-Source: ABdhPJzgvp9VbPmYxl9HcqBznTWwYA1Jo+dnG5ZkK5dodjXKOSEb7g5h76XshHfWkUlzl/C//JwkOA== X-Received: by 2002:a17:907:9488:: with SMTP id dm8mr15198581ejc.36.1638447912523; Thu, 02 Dec 2021 04:25:12 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:12 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Thu, 2 Dec 2021 14:24:51 +0200 Message-Id: <20211202122507.43572-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 +++++++++++++++ lib/trace-cmd/trace-output.c | 20 +++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index fea75f54..9cdd56a9 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -255,6 +255,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index f6d60aea..8db43708 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4593,6 +4593,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 455fa93b..7906fd92 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1225,6 +1225,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kal */ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handle || !ihandle || handle->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1236,6 +1239,15 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracec handle->file_version = tracecmd_get_in_file_version(ihandle); handle->big_endian = tep_is_file_bigendian(handle->pevent); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + return -1; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->file_version = FILE_VERSION_COMPRESSION; + } + return 0; } @@ -2178,6 +2190,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2217,6 +2231,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Thu Dec 2 12:24:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CCE7C433EF for ; Thu, 2 Dec 2021 12:25:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231820AbhLBM2j (ORCPT ); Thu, 2 Dec 2021 07:28:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231949AbhLBM2i (ORCPT ); Thu, 2 Dec 2021 07:28:38 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D69AC061757 for ; Thu, 2 Dec 2021 04:25:16 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id r25so49162693edq.7 for ; Thu, 02 Dec 2021 04:25:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B+n5yh+LhF2w2p0i5FPFfVknoDtA6B4pxWVcKJU3WhU=; b=d7uJVJuzFqPiUDhe9AGm9UCSk4MQW4m6cQy6yjh0jRMhBmqhk4ePVw7G8rqrojtNLz 74IhI7lGgr6gZjj+j4FDcGYQ8UgkmxRdpW7L1fIIerYG4LFDBtAX9S57cYaKYPhK83fF dgH9sU9gRZGXC2hS2YzoasrJubKXt6r+x/4Ajb4leySc7CWAi1a7Nnha/g39k9o8alCe SOewZ2Nww8ppQ+y8n10w4HZ7Wo5tnoNi87CLyirlDmVTb8zg9rA9VEMLd4P36UwSXwWb HDI730c0QGUyWd5oaskSW1A48TOuUjVEr/Sl8xpTBT3q5oIkPH0E3epcNL1Ha7PynUxs vGuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B+n5yh+LhF2w2p0i5FPFfVknoDtA6B4pxWVcKJU3WhU=; b=2DlK+ReaF0swL6mrLJVB6bVsv7Np5InW46hiG+2RJTYxHsaCQrDmRvV+lixQ9VDIq/ SpST6vh7lACxbfpN7ETCrF0L+zyStKuZFjcrUuxCJ5OIK9LdBNQ0XUWfGU0GpJMTWU3M 0aX1QHtHr+WiipsfT9kv2VEcouNOTTudRNFRovtALBrNTAeeXXRJ8IoC8MirEA5P56NJ A8BoPYF+55J+hL6wqoGk7CJ4s3V71BSzWxoYlPhYrJZby0Ety0+LEO1LcWQzqbMzm8N3 r5znvdgaoEHjI7ZwyG96hhUR4VDwIK+0TjkZOT5Sqm5s0iKCPENfQAE7zkzTSrosCclw kBtQ== X-Gm-Message-State: AOAM532ZfUqAxlQI2/SBpuux2HmaTJo0vSt3wOlmXcydvQSAQZAVAPOB zNJKXChGTXep5DXV04aqiE/AI2N00zgIcw== X-Google-Smtp-Source: ABdhPJxE7UV/VsGp946b8rRQdLHLDvyrCeM03hglr8oAIvrqNYDElZrbWcr66bX5WpUgDkZc6atS6A== X-Received: by 2002:a17:907:72d4:: with SMTP id du20mr15660807ejc.419.1638447913473; Thu, 02 Dec 2021 04:25:13 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:13 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 05/20] trace-cmd library: New API to configure compression on an output handler Date: Thu, 2 Dec 2021 14:24:52 +0200 Message-Id: <20211202122507.43572-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handle to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 58 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 60 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 9cdd56a9..6cc6ce46 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -296,13 +296,14 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handle, const char *tr int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression); int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_output_create(const char *output_file); struct tracecmd_output *tracecmd_output_create_fd(int fd); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle, unsigned short id, int size, diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 7906fd92..bfd6e385 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -236,6 +236,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) } free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1267,6 +1268,55 @@ int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handle->file_version = file_version; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->compress = NULL; + return 0; +} + +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handle + * @handle: output handle to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_headers(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handle is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression) +{ + if (!handle || handle->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handle->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handle->compress = tracecmd_compress_alloc(NULL, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handle->compress = tracecmd_compress_alloc(compression, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handle->compress && handle->file_version < FILE_VERSION_COMPRESSION) { + handle->file_version = FILE_VERSION_COMPRESSION; + if (handle->msg_handle) + tracecmd_msg_handle_cache(handle->msg_handle); + } + return 0; } @@ -1857,7 +1907,7 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1870,6 +1920,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_headers(handle, NULL)) goto out_free; /* diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index ead2c107..5f6fcc72 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4510,7 +4510,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Thu Dec 2 12:24:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 970A1C433FE for ; Thu, 2 Dec 2021 12:25:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232134AbhLBM2l (ORCPT ); Thu, 2 Dec 2021 07:28:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234072AbhLBM2j (ORCPT ); Thu, 2 Dec 2021 07:28:39 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F84DC061757 for ; Thu, 2 Dec 2021 04:25:17 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id r25so49162873edq.7 for ; Thu, 02 Dec 2021 04:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wYktZ+MsZzYFknwer+nO8UYK0p0Tvf+pEC5aGqGFh14=; b=FNyvHRcLn1yl6If6ldtsOTW5LoOFQxsePi1dPBEstpzT6wa514uPd4kezr3CUjGof2 a99h7Vkaj8a6pGbYTufV0bG+K+w64x7F26OIz/95C7zyG1cAj2T+iRbnzy9QH/wr49rc QQaKuejRGQ2r0X3SEdB5SepxaeXaZiaCcvjLeQkAcLOhIX2rdKrOt+nmYYLMhna+V9Gu o22cbcWORy3s0qrRTvFoKi0Fe027B23WyrSYX1Xt4cjS6NWfgqLXr2G+e0jka2tk5JZj DraMLyiD8uILMumvJ0R0cht0DPa3WMx2b3ej7niJ1bf9vvefdtgVWFKrQdLSLTUN6vW9 XLHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wYktZ+MsZzYFknwer+nO8UYK0p0Tvf+pEC5aGqGFh14=; b=G8idjVeVXI+qQYoqUlXJhNKTxYozOE5Ci4HZLMMBwilYvU6ATlc7omv3IqbuWGnja7 VqKZAsF0z4ViVtpE1OZ1Wkfo3ZL4ShwkpfTUevplhiiA4XK5wn6tPD4fVC7hJSITrbPl X/z05OhPOWfOo+eX4X19yN1BT/olNWq+RCb53H225d4styJT7PYXXyMrZMw2pVoGG8JV T7b5AI03hXZOpP72AbFoAE/ZB8NS4wmQmtboVqgeRtZSnOD0W627oLjJBU5uZecBoFuk +Z/gWFMKonvZBIzsqwuASfxH+lXAgt+qFVyUWg33cQNcwSfzpMORim+3fq3sOB51zJ6w l7FQ== X-Gm-Message-State: AOAM533IxDTLCbgHzgWNUcyBGqK0V5CPsxqS4OEcNKrr3WsVk/iwtd/d wFEW8UDM1NNsAJqBxjr2tUK7tuUBgP+UTQ== X-Google-Smtp-Source: ABdhPJwrw61JeY6OdpL/m9YDLRU3bbOhYP92TqMybw0MHg3m/iR0q3YcuUf0e3Arshm217Zk7xTL3g== X-Received: by 2002:a17:907:d89:: with SMTP id go9mr15131537ejc.330.1638447914342; Thu, 02 Dec 2021 04:25:14 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:13 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 06/20] trace-cmd library: Write compression header in the trace file Date: Thu, 2 Dec 2021 14:24:53 +0200 Message-Id: <20211202122507.43572-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index bfd6e385..0dafa2d2 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1082,6 +1082,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_create_fd - allocate new output handle to a trace file * @fd: File descriptor for the handle to write to. @@ -1374,6 +1392,10 @@ static int output_write_init(struct tracecmd_output *handle) endian4 = convert_endian_4(handle, handle->page_size); if (do_write_check(handle, &endian4, 4)) return -1; + if (handle->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handle)) + return -1; + } if (HAS_SECTIONS(handle)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Thu Dec 2 12:24:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81DAAC43219 for ; Thu, 2 Dec 2021 12:25:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231949AbhLBM2l (ORCPT ); Thu, 2 Dec 2021 07:28:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234603AbhLBM2k (ORCPT ); Thu, 2 Dec 2021 07:28:40 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C76DC06175A for ; Thu, 2 Dec 2021 04:25:17 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id v1so115094543edx.2 for ; Thu, 02 Dec 2021 04:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S9u7c0rh/PE/OQ2MH1QjoK+RqGLIuPkcbKbrcSX41rY=; b=CgSca0jaaLeTHT+yGN7Uxu8WoD7UZ8dXD6/6RPLzf6DR25dI0zV53fQt1ruY7Vxiyk u4fV3YTqGYTmyX2lD8B2gaCIBSJ5bpdcNW/vEhqvRpxB//o+oURfsMb2Dy21objwFmAY IO489EAVw3E1oxa23FZa3lx7Co/2v4J9EyTPmVmzcYQoiEMy3SNeOEjAZhJ1SLPC4mi0 QZ3sQlTf8rj9Jnwdpbp5aPJkXRxMUjK0t24fbVRQPWucWkWueZEpzJlFvK3wG+UeNuSQ PV0HBHlPJx+vV/fnebRzCidtrD8rECr2erWCROIgYT7QbGn041nKvfQzkORjrPxBj/jQ QgtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S9u7c0rh/PE/OQ2MH1QjoK+RqGLIuPkcbKbrcSX41rY=; b=6FO+t146QPiZgtziPUMQYfkUIFf/C2VCL8hCkHmTR+6roa8xZ3X7Z82vQTZeJzN2zR MPa6QM7J19FFqq6VKRi5p0/uJlCY6cvOd5s4ilRl8J4rSi1rimLubb4ALQCiaoQTQQlw UYucd9LIg5MtM1hv15OPQX4FhBYjzqvM+ZAtmTJmlAgXSWz+37+NugVjzu9/fH9E83TA Rc3UYY8NeA05xz+cTK1YADYdI0zEqYXwflEiGOTiQaj7BXuts2cLzuhzWqdWvAz6JAXf qxLNUW+F8+ytFT8L2iIjiQyizYblJuBRrIOqpvXvInLtrNWKZw8yL8FnoMtRM0Gen1cx MJIg== X-Gm-Message-State: AOAM532BW0lJuEPlE0OSgAAbVNiWzQsMezLrDee9HD4mM/Txy2FN6si2 gdty98kvz2IsVbWkG16aI++D0jo4InwozA== X-Google-Smtp-Source: ABdhPJzAsFi6teL/CvSp2FoHzrfDfkrLTCKQCAsgrZ5IPxkCt4ZK4Ty/Ex6tIRqPABY7xYJrT9Xa1g== X-Received: by 2002:a17:906:9153:: with SMTP id y19mr14934627ejw.516.1638447915419; Thu, 02 Dec 2021 04:25:15 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:14 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 07/20] trace-cmd library: Compress part of the trace file Date: Thu, 2 Dec 2021 14:24:54 +0200 Message-Id: <20211202122507.43572-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 81 +++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 14 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 0dafa2d2..876be404 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -400,6 +400,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -455,7 +457,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, unsigned return 0; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -475,11 +477,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -493,6 +498,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -545,6 +552,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -552,6 +561,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -780,7 +790,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -794,15 +804,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -828,7 +843,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -846,6 +861,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -866,7 +883,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -881,6 +898,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -888,6 +908,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -934,7 +956,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -952,11 +974,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -982,14 +1007,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1008,10 +1038,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1034,12 +1067,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1426,21 +1462,25 @@ static int output_write_init(struct tracecmd_output *handle) int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list) { + bool compress = false; + if (!handle || handle->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handle->file_state < TRACECMD_FILE_INIT && output_write_init(handle)) return -1; - if (read_header_files(handle)) + if (handle->compress) + compress = true; + if (read_header_files(handle, compress)) return -1; - if (read_ftrace_files(handle)) + if (read_ftrace_files(handle, compress)) return -1; - if (read_event_files(handle, list)) + if (read_event_files(handle, list, compress)) return -1; - if (read_proc_kallsyms(handle)) + if (read_proc_kallsyms(handle, compress)) return -1; - if (read_ftrace_printk(handle)) + if (read_ftrace_printk(handle, compress)) return -1; return 0; } @@ -1811,6 +1851,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1820,14 +1861,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Thu Dec 2 12:24:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 156E6C4332F for ; Thu, 2 Dec 2021 12:25:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232969AbhLBM2l (ORCPT ); Thu, 2 Dec 2021 07:28:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231949AbhLBM2k (ORCPT ); Thu, 2 Dec 2021 07:28:40 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B75E8C06175B for ; Thu, 2 Dec 2021 04:25:17 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id w1so115507655edc.6 for ; Thu, 02 Dec 2021 04:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LSgYRmDfZp81HNGOmHabjzlTc32cEBgh9Ei/Ctt8H/Y=; b=XAaZUGBpElVdvwpnReqftscY4d/MF94uU31SP2wz5TgQp2mUfLoqXAXYS+SzNwXSCO ETUberhWxZeyK2eg0GtXCiqI5/MvIjvoSJo7rHdKVymJQuEI0V0JTj4qvnkOl/ccYtbm 0Qad94N00iCYtPmMOUV38GPMrma7eJ1rtDMiSTvJfWVCnrp6v5FkL4YggXDDsFxyxi0O 0Qz3VxIc33fIVYk1gzsgpsnjy8Yrc4AyRLW1RUuXaSQ9FRQ35WLrPfLNuLeU4adU5N8Q CzeBoHsnrz59ufj/cP8vsoK/XefQm+/BmWaO1cNT0aYXWVeQS2pI6Fk8Xs+u44fnXW9p gs1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LSgYRmDfZp81HNGOmHabjzlTc32cEBgh9Ei/Ctt8H/Y=; b=f4U8Spd+c0wyCXXgZk+jP8oc2G383nj5MEFkAimkJ+W6K1d6kJv0rsigqb7q0MFulM CwGio/ta65+55ifnD58L/51Mx0a+jC9ewVoSWG47IMU0/EQk5nYGyAvNM3cyX9ijwiS9 M3DBgASEa18wUe23dEM146IVFuUMOb3pokzDlCVNQ9ymB2iGBrKzsM4POE1wzCtG25Rk +t56ijIi/O1fpoY8SaP5FRuhKYqx+jYKIdt9x+NGWopyFZvg7kFwmG/Ogo/b/u3O0SF5 Y1G41gMWOXBkOaLTKYoPb6OSeQVTXs1/qJPO8ihhXpLAAIbpTmQ4XH0Cid4+0Z4ddyWH t8/g== X-Gm-Message-State: AOAM533ptuI4Ztilw5p6HgtMZEQYtPqLdEJZpzwfLQ0hnxF7PnyBQHPm K9Ycy2dK2t2/G6hWVLnZG+yES8em+OeO4g== X-Google-Smtp-Source: ABdhPJxJvKjQ9xrm5FS/ypr+7HD4QpKSSm72RpeCyr7t/YGYP5b4/kw1RFyEKUQ2wA++K+Hrb64koQ== X-Received: by 2002:a17:906:1c56:: with SMTP id l22mr14791431ejg.208.1638447916231; Thu, 02 Dec 2021 04:25:16 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:15 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 08/20] trace-cmd library: Add local helper function for data compression Date: Thu, 2 Dec 2021 14:24:55 +0200 Message-Id: <20211202122507.43572-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 876be404..bfd99b8a 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -289,18 +289,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -318,12 +327,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -519,7 +578,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -545,7 +604,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2192,7 +2251,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Thu Dec 2 12:24:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91C68C433EF for ; Thu, 2 Dec 2021 12:25:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234603AbhLBM2m (ORCPT ); Thu, 2 Dec 2021 07:28:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232977AbhLBM2m (ORCPT ); Thu, 2 Dec 2021 07:28:42 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C684C061757 for ; Thu, 2 Dec 2021 04:25:19 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id g14so115268542edb.8 for ; Thu, 02 Dec 2021 04:25:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h89mIIiJTT0/Hk8nW5znXdSqyLULJPfcrTfMFa+RWh4=; b=gTXGJ0eOwdnfFxoyUeAYt7+Ol/eaA5Kv/qX8zkJRh3WK8b3bnIE+ztJC3vz8KuGcab vao3maRzcGjL/swUuomUtY9LkhQoJRTnm7nox4nnsGfPhEmRAm/WjSXQIpajZyskDY0g mYg3gBmVAsUROlnFQvbG1h7DGykatDy07fUGf3GJf5zcCB90hAOc4YwHAbqL+VeDAx1T ZueNOpU37M9LvkT5ALVuLxW5DAw6X3F/Z1UxN+FbRG5/CqhO5fu1Sx3CvRfVZoerm0/q wp/Ugh6/jiJj3BD/mNCWjEdUGqLyZvBRPo9cPiVq7vlwMzoZMlaNrLenVQ+4ztTaz7B0 l9Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h89mIIiJTT0/Hk8nW5znXdSqyLULJPfcrTfMFa+RWh4=; b=fZDtlNg41LBZAiElD/0FfVIX/aEEcP/VCHuAa/kO1ofkOCZ1vSHX+lcnXiU/0wZkmW FKIuv7sEaP98himoBLV8olz6I8iJDk6huzGS9xz2QGeG25RMKbgzCOBMfvdj9RtjV5Om cNykoUdvfSMbkw7mYkaB7fWatEK2tZYwgZbz1Ol7nq0wwxZymAyisjeANr33m79GaYp4 j7hoEjdZS6CNJEGfe+aW2iNWTCIACm6LY4cSVJtNoJoaQE2m/tbPTZ4SjWpXJ6NoiTGX jRd2ObCkqP+bJrSqLnYsZOAzoB8gEQAp6ma1bu7Ip/R/FNBlaTmMXT25ZWYh+3/qJrW3 YNBw== X-Gm-Message-State: AOAM532V7yC1GrzRKx4fjN39HaV/tOAbFsVSTZVZY1/J5lD53/9Oiu83 viD4B/ea09vDQmhOMpLBQRi6zFcpEfhy6Q== X-Google-Smtp-Source: ABdhPJyDGoch9Z3e51/8fKTl3uwPGtOY2wbwAOM589qwWsyG5rKfGRJpmrN/er+FzSt7wSifkQ9KwA== X-Received: by 2002:a17:907:8a1b:: with SMTP id sc27mr14976624ejc.572.1638447917267; Thu, 02 Dec 2021 04:25:17 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:16 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 09/20] trace-cmd library: Compress the trace data Date: Thu, 2 Dec 2021 14:24:56 +0200 Message-Id: <20211202122507.43572-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index bfd99b8a..bd838d1d 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2092,11 +2092,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2199,6 +2201,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2251,14 +2255,15 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); + if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Thu Dec 2 12:24:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36D4FC433FE for ; Thu, 2 Dec 2021 12:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232977AbhLBM2n (ORCPT ); Thu, 2 Dec 2021 07:28:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234072AbhLBM2m (ORCPT ); Thu, 2 Dec 2021 07:28:42 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C531C06174A for ; Thu, 2 Dec 2021 04:25:19 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id z5so50278331edd.3 for ; Thu, 02 Dec 2021 04:25:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ztvDFnMs8qoOx4qtDt7YdpO/uQKxoQoibBCS6/uCJFU=; b=d8Agr4jHEKf9LQaWES9AJvVkoP/rvmc/je7+stSA9t/muacz3XHeZWM9Nqar1Gxwgm 5uviLyd7r+53XvrA/gC+e/695uMjRaHuIXH1ndIrW/1mdJx8ODQnHn4fkXheS+As10Ft 7trxdcJ+bIvLLbNXVfouq/CaKbDBS+JqLLTkjDlGt7xR460++fbQ1/xHVSxAhW6X9kzW gQrIuxiezlk2aQKc2CTMo3SXyS+sSFgAk3AtlS/zpG09SU5Uk0W+aeckj6KljEAQYUmC vpQUEzmXl2riFMQFGhPeVp576PdZtaUNo/PhSDZoJjb1g8fbmP7TbDnz9/llMM5tnPtj B3Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ztvDFnMs8qoOx4qtDt7YdpO/uQKxoQoibBCS6/uCJFU=; b=DKjt7HZhB3xEg/5hzlo0nhofeICf13z6w+Kaehe7vGPTCEZoVE5kj3uoiBAp9BAL9a dTvDR5fc9eDP9nIpPmP00MGJm9LzOjzesgndBfmgGro0MneP/g8hbG19fFhrlFice2x5 4o2XppcL8+xubzAlxmyXeSS40CcZNjSKgdRVXJjhRk4Epj1FvMWfBemqgBEENOahQDGm mxbgY2qBipr9uovvphWu5vDj/svpK6nhKZGzSgR1G+98HY0xM2Ta4pLwd/XSMwWnue8j 4R7jFb9sz2xgnGdBO5uLSDheRjeFn6Cr2FKVsL55yZWIIquXrjkRTysRFLsz9YAtvDEQ 7hxw== X-Gm-Message-State: AOAM5326nT6wswfCD9GpQiShtq6Xl+iD/ShgLnvaiR9BWjzqlDetWjHF RBwqs53HsMrN/2y7EoHCWWk1PW8JPTc+pw== X-Google-Smtp-Source: ABdhPJwY3sCm+zX3Q1ESIhtiPW6G+nxxpYygua2faIb4iVUvZs+BKyCFB/yPj4TPPeqyy8GWU4LMfQ== X-Received: by 2002:a05:6402:34cd:: with SMTP id w13mr17058284edc.112.1638447918223; Thu, 02 Dec 2021 04:25:18 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:17 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Thu, 2 Dec 2021 14:24:57 +0200 Message-Id: <20211202122507.43572-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 45 ++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 8db43708..bd655d0c 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3025,6 +3025,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3036,23 +3037,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3106,15 +3116,17 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: buf[size-1] = '\n'; cpustats = realloc(handle->cpustats, handle->cpustats_size + size + 1); - if (!cpustats) - return -ENOMEM; + if (!cpustats) { + ret = -ENOMEM; + goto out; + } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; cpustats[handle->cpustats_size] = 0; @@ -3124,7 +3136,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3183,6 +3195,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3195,7 +3209,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Thu Dec 2 12:24:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04E4CC433FE for ; Thu, 2 Dec 2021 12:25:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234072AbhLBM2p (ORCPT ); Thu, 2 Dec 2021 07:28:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234914AbhLBM2o (ORCPT ); Thu, 2 Dec 2021 07:28:44 -0500 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A613C06174A for ; Thu, 2 Dec 2021 04:25:22 -0800 (PST) Received: by mail-ed1-x52f.google.com with SMTP id r25so49163706edq.7 for ; Thu, 02 Dec 2021 04:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N3oqLot0YlcbRcNAc07Y6XrPI/fvmTo4H4nilHDSxes=; b=SCqdnSNMTkFe0oBjZKCxM3e6Bxh8QP16PSH2ePVD7wYBGwrUWrqbpjSLNCpKcDBuO8 SFXAUnX966zn00nVyFcbVEwKTszItMFxqz4i4dcmfwWfrZjtwb8E51QKnWit08ldH6Vt gBbH1OwtGvFYGTwOT4GnOHc7drk/WtaWIjhOQ1UUpO6tRk1FtOKl/pyS17YzTQ+1P7ZD JLqFhz/2D8v/RP1haIhBgS4mByfLNxhEYCY+3VlMhLRbNOydEynJ+JaaOyyYBL+jXpIl 0eKLdIC5tQE4xDTLAkmAJ/uFBSWetxm7Ea24RYcsX5cfwz1zmVd/gRCmz6TsRxoxJ74E yPJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N3oqLot0YlcbRcNAc07Y6XrPI/fvmTo4H4nilHDSxes=; b=PPRJkEyaLmCO9zyFWjXPunm/fCcGIMjOZXLeinR7fMbARl/LvAeHXMNIxIlEPrJ3j9 Ra+HRpmtt/Ea534LcVxKT+dX3MH6WaXqzoEjD+t16Q/KId5kdn+JR807u8uUwouf3Uob oe81SetOi/xO9dv1HmSXHlkW1FeO+uZswKQHJfytCopZHKJ4kYcaoZjlR6L/NAWkM12s GiITHNPPsOyGzBtlH04mnFZuvtda6x4sr4aA/Z8XqT1sUjDoTyN3mxDveWvL238jbpdZ irdTw1JXcisnjmQ/1S9AP8ygg7k7L6Ho54sOzUzO+/5hfpfwurFMSwFUv3orYz5LKuXX 0VeA== X-Gm-Message-State: AOAM533q3mU6lPIZyLoAnX9qeBioS+8QIeqCMQ4ZOGQdRBnqJ5JVuIga XZoqfKm8xDxYbGx9u4+Uv+iWfN8Gx86VgQ== X-Google-Smtp-Source: ABdhPJw5nIq0U7WDMKDAyrhtujXgC0srPn/xSw7MC8YtsfPOV7GoYnUpJGX0RoOQOtMhzIwKK+aaOg== X-Received: by 2002:a17:907:7664:: with SMTP id kk4mr14550477ejc.319.1638447919059; Thu, 02 Dec 2021 04:25:19 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:18 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 11/20] trace-cmd library: Read compression header Date: Thu, 2 Dec 2021 14:24:58 +0200 Message-Id: <20211202122507.43572-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index bd655d0c..3d67fb97 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -190,6 +190,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3784,7 +3785,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3822,9 +3825,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3854,6 +3860,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3866,6 +3892,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4064,7 +4093,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Thu Dec 2 12:24:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D1FBC433EF for ; Thu, 2 Dec 2021 12:25:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234654AbhLBM2o (ORCPT ); Thu, 2 Dec 2021 07:28:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234072AbhLBM2o (ORCPT ); Thu, 2 Dec 2021 07:28:44 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FB3BC06174A for ; Thu, 2 Dec 2021 04:25:21 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id v1so115095360edx.2 for ; Thu, 02 Dec 2021 04:25:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H5r0DvR30Am7AFkraurPbx/cosS3KErdUZfOYM7PGzo=; b=MkRthLYFZOfaU6pyS4s6n8933RWLgLRvxBOlC5Er3k8+MLb01IUK9dMCOAFE0IU47j xC5E8JNg7nJypJZeKvIeQN4ErAqVpm+Jzs3EXoZimHWMKnaoaw6XcQFUs4zQjg1KjdBs +ly+FCZc2eHHU+1tbYiVfJ9fH+HkcHnCfOvB68F/wyQ1HObuj4TWcOXadF32qM7SYW19 OpZ+pVUl3MTihuvTdSxLD14XB2Y+7rCLXFOzItOVdb55cqjFxGoqR1CF54bDR7ieTDfq ZVAWLCuZWsNPWPLisv261TPOH/ib/s7KddhkCbH/GK542ULBsp7RWg4ZbJL+qpHp30/e iN8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H5r0DvR30Am7AFkraurPbx/cosS3KErdUZfOYM7PGzo=; b=N+NAnnK900Y5tMDSxZ9zrmXTM/fMPs6UfDpqbcBKRDrUHYPWSKooMaiYFX5X8Zqzmn jGLnJEektggVtrrw8UJ/UMiz/2OyIsSK+Y6uV0MkxI7kqGUr/Df6H3MuXfUO4FlEZd0+ fDwBs5r/AQwgyVp8l4duW5St0HCu9Yci1mGh57Q7OnLhfgai8qisnZ3DUn5hShcZiVGs kicyZfMmtNRnHnW9Re23TPAdRbiTuLpi2uKAhEWmpqBTmnU/nVnbCD4yYprj3DLOnrnp G5ekcrMOuTSAkxC83NDnzdNWhgdlFFCKI3j+WQRfVNO7TgMF5gxqI/Uat483/6GHYrgi wcGw== X-Gm-Message-State: AOAM532vQK1K+ACLRMhL9w2zSmQ6yJGzJzHsbujwC3gaBdJCX/IwfpiU iV4HRrC+VDeKxu2fcY2dbsddBGzRhtbyIg== X-Google-Smtp-Source: ABdhPJxPOGG14xb5W5Ns/V3doozJ/c6PIU+xorB4Sgw3VdORgaE1ZFgUIaKeLRiPU0XxA/ZhuCu7NQ== X-Received: by 2002:a17:906:388c:: with SMTP id q12mr14715076ejd.281.1638447920059; Thu, 02 Dec 2021 04:25:20 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:19 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Thu, 2 Dec 2021 14:24:59 +0200 Message-Id: <20211202122507.43572-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 3d67fb97..bfe9d333 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -151,6 +170,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; unsigned int cpustats_size; struct cpu_data *cpu_data; @@ -3294,6 +3315,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4030,6 +4052,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4049,17 +4072,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Thu Dec 2 12:25:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9203C4332F for ; Thu, 2 Dec 2021 12:25:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235534AbhLBM2p (ORCPT ); Thu, 2 Dec 2021 07:28:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235035AbhLBM2p (ORCPT ); Thu, 2 Dec 2021 07:28:45 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 922C2C061757 for ; Thu, 2 Dec 2021 04:25:22 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id w1so115508612edc.6 for ; Thu, 02 Dec 2021 04:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mcqzKScOqZrmTyjH5Sw2H7H39vZqIREw7NGGP7Qnf4o=; b=FjfeVgi6xG/Iswhv1QkhLdpGlEnBfnwN+ZXwZzjkVxPkm4V3zk/RS8UEL2mykPcO/N nsBZyEj6fM3Rgxko834g6yrusmhpTdQI9C58FLTwEa3ce2zw8VvxLefFYNtq3QhU0RYv 8qg5HchS6fKR15lTWEnG+IidhAeXok6652ctmHA0W8SuXNIoMu5AZ+UCwIR0Q1Ll+hVH lS5jxaN3SLgsJQXJsPOf6ggMZXX01Bhj9LaFkLbCxKJ6jAFnms1G2Xrk/yX1S3/X/BKB 0wUq7IV99oN1f6jRinFjkq0JE5EhNKJzDKHuGklIlP2CjYA2Jm6OuVKurStmThmCCaiu 4Rgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mcqzKScOqZrmTyjH5Sw2H7H39vZqIREw7NGGP7Qnf4o=; b=1IIxgpXCoj58FQlvSIwJiX9V4UMqd01edKCz3zomF3KcU15/xDgM3VoMZFJ4d1g6lM eJqUEVqSrtBqS4GkcCJZblgusO6z85I4S6wRhRiRYTHBwPFApg45893Zo4Kb915PAHua EirbwMzL2MDAdSZTYDHUbKuD3Zfxsvyi+3qrFYwBLvEItNNVfaOqPMmSOJAwWCFgANJy Ugg/b2uJno8r/UfAcKwd19rXbUDgYe6TFWR7xRrJUhgO29BRAB85Ub0EEt5C1Tz+L4j4 xFrGEqs0VNG6eZMEbztWVdjHHA4r9F6MHQaVpjsxKQaQJk5pJbK1s71CV0NjhjluJ1wa jcLQ== X-Gm-Message-State: AOAM533owuTiXz9Vj1UNI4kv0p74xm7neZWnfLrx1IUcjDzD9EMUPk3j +94q3QKI5ePxedE2XTN1DG44uYk2f32+8Q== X-Google-Smtp-Source: ABdhPJyV71LgJ96LeHkknyO4etcL0lFwhDX3DlJZCO47VQh+Bl9h5wAWTqYjlZ5VvnGRyPDkUIsA0g== X-Received: by 2002:a17:907:60cf:: with SMTP id hv15mr14344023ejc.561.1638447921174; Thu, 02 Dec 2021 04:25:21 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:20 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Thu, 2 Dec 2021 14:25:00 +0200 Message-Id: <20211202122507.43572-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index bfe9d333..7bbd5485 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1259,6 +1259,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1298,12 +1299,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2491,16 +2495,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3372,6 +3436,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Thu Dec 2 12:25:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE41EC433F5 for ; Thu, 2 Dec 2021 12:25:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235762AbhLBM2t (ORCPT ); Thu, 2 Dec 2021 07:28:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234914AbhLBM2r (ORCPT ); Thu, 2 Dec 2021 07:28:47 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BEFFC06174A for ; Thu, 2 Dec 2021 04:25:24 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id y12so115204397eda.12 for ; Thu, 02 Dec 2021 04:25:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vd+5I0JDkU0IGofdmfAZ9EoBCP92QoZYMavCimfRqvU=; b=IamQ9FDezJAzklJu4ii1CNCYjW8GFLVjCc1fwYiwl/gQRi7huOrhe6Gq69WW3+zEmG mIfrg5Mv/yCxEF/1BxP6IjFO+iMQfpabClJfGMWqyDCAGy7uHSY0Rtvu/dLDffif+wfk idk40p1oY3SMQTmkl1ZK6VquqR3JfUww6Y7AHjARqt1gKPGy0XOaIgpbZ7HCFfS4SSeJ X6Oq064+RRs3QBzweaE+HrpjxpEbp2zGPsU4TTo6YmqWQ54AAV7Y5pSP9xoLeeMziWD1 5mWVPTHtuVt1PMJsrxJeVRa2+dEjLtfPW0+AUZPoCo9xrfYYatQ2D4ITbhs/YCb5jE9+ Nfgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vd+5I0JDkU0IGofdmfAZ9EoBCP92QoZYMavCimfRqvU=; b=6gU7yGQXoUHtiG1BUZhgwI/XARplIuuONjdkyTE4PWQmzG17v5ufRpG3LAgDWKaB9h 4/DpGKfskYnJP1Hr1JjARFEOxxWrlGvletSWQRpv3FF4wjNYRw+oMZO+SlWdp5V60QGA 1kcNoPKyIAnu1I5fQ5n71+aqpGuqaF+9Aqi+YGBT8u/balhDomzO+hfQc1UjyeneATiL Pq+erU59Hm1hlxO8AOok2N41UvPnKRqALcscnUWoHEk06EG2Rn5UNPvFW15ew0GP3HgK VPN6rqh6dzXzXEog8IGCug95lRL1qGdf6AGH3RhGS1qmtd74ISuXEBZ4KJsbYIMlcmu5 c5hw== X-Gm-Message-State: AOAM532a5mcMwEfe16XRRfWHN0eljXlj8dcaAfJqSF2l++12/8gVYhhw f8Cd6g+JS5Vh27kZXI+VfjZtzQL8Q8u1zA== X-Google-Smtp-Source: ABdhPJwW2Mftt+wlFcxdisylwxaK47nuu89NjPmoVNYQ6g5OuvNJoqKc4ZTXJctwizGo600+DXepkg== X-Received: by 2002:a05:6402:26d4:: with SMTP id x20mr17459312edd.119.1638447922214; Thu, 02 Dec 2021 04:25:22 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:21 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 14/20] trace-cmd library: Add logic for in-memory decompression Date: Thu, 2 Dec 2021 14:25:01 +0200 Message-Id: <20211202122507.43572-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 7bbd5485..baad5994 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1250,6 +1253,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1261,6 +1363,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1403,6 +1508,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3889,6 +3996,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Thu Dec 2 12:25:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2809AC433FE for ; Thu, 2 Dec 2021 12:25:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235548AbhLBM2u (ORCPT ); Thu, 2 Dec 2021 07:28:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235035AbhLBM2r (ORCPT ); Thu, 2 Dec 2021 07:28:47 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8044FC061757 for ; Thu, 2 Dec 2021 04:25:24 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id z5so50279130edd.3 for ; Thu, 02 Dec 2021 04:25:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pUl0FrQ08+8JJ48IxcOYd6UdUBPSrP/d+5xADJCF6DE=; b=S0Qk5MWEVwMSyUzBH6DfKMArjDRceyt/OOeAL/rTXyPzkPZzC9HJ31ak3Fjknq+E0x js8Y3dHklgd80nHWVOgjLpPLS9mXh5sIe34zp1imHOTNuNm7TVSH7FfFtHCoElNMuLe5 jQiFsXnp5RxkxODoSogqVeve4FUxc6GFz0ljFgsiclTLdtrMeRASrwbnb8LYp+kIpsL9 +LyJh4RJsnFmkXa6rP0WLu1Dk1GwoHOlRbZ3bGE1hnx7UNoTblGdTyidzIM7w5/1BpN4 F8MI0aOKLWbd2UXjJgob0cwoesUswpIS9vCT8MfnVZ/M44K3q3tYLrvoBlH0+mAxf9wJ uwtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pUl0FrQ08+8JJ48IxcOYd6UdUBPSrP/d+5xADJCF6DE=; b=yESvlgVYBbEhkKH2ndLkbLKMnBSdeCe/yKPR31ljZmjpZ9KokUXuwgAzo2D/YLdIe0 rYNTL77UxopOINS7mAfwHpNog5oMqgvsrSQOS4l60G8o23APdFiXiVTsg/9253BWKiDh gYw2l44L826qEz85vONuSLkm14hNi2F/3Yu/yya902/OJJxzbZydcLxGnhCpuqnN6Iyj CdN19X2XBqQs603N0iyLFClIqjS9pSeS1pYos3XJgz6dwBK2ihofwKgLKyf90KxjwR2p WSZ4zCx+vwf/+6P7MXAUmBHLaIqe6eHdcxeLCdf6U7MhF873DHsPcyJczkGbKzUHdhCH y4cw== X-Gm-Message-State: AOAM532aeN49PhNWkOvqDu57jpwJ6wzz7rH3pbXXVEo7MnFATd/UM0rt oDap640h6PJwYJxFn8TFfgyCcLKOGGpHXw== X-Google-Smtp-Source: ABdhPJxZrxZNo2QZUNlg1ZRPMQ8r36GiZd1Yddh77mQXY3icDbOI7/cHZt9neIQbWYqKEMQFbZ8y3A== X-Received: by 2002:a05:6402:2d9:: with SMTP id b25mr16582079edx.383.1638447923091; Thu, 02 Dec 2021 04:25:23 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:22 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 15/20] trace-cmd library: Read compressed latency data Date: Thu, 2 Dec 2021 14:25:02 +0200 Message-Id: <20211202122507.43572-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index baad5994..9bf5cf7a 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -177,6 +177,7 @@ struct tracecmd_input { bool cpu_compressed; int file_version; unsigned int cpustats_size; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3444,20 +3445,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3525,7 +3558,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -3993,6 +4046,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4282,7 +4336,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->trace_clock); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Thu Dec 2 12:25:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84082C433EF for ; Thu, 2 Dec 2021 12:25:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234914AbhLBM2t (ORCPT ); Thu, 2 Dec 2021 07:28:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235548AbhLBM2s (ORCPT ); Thu, 2 Dec 2021 07:28:48 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98877C061758 for ; Thu, 2 Dec 2021 04:25:25 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id e3so115373499edu.4 for ; Thu, 02 Dec 2021 04:25:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=V4T/nqu5BcdfnctLll0nHcIWzjBkei1sx3rSAABEUjE=; b=BjJufgRM1oCZsvqkkc6IXj3wEDkVarjut71gwkKReo6fEaQeH/WiTGXfF4ISLUQN2h IDE04AtIXAcP/18HIkcVtlv6MVPR0szbJ9A4Mv/pkCAX4Vqd+Kzs1yDRogONd61NKPAH +rA8fsKm2MoHkdQzGWDYbQdgzCwiY0HdZaJh2gh0WPquaKB6XSrE+HmPHZvlNrD2hKlX tF6vl7UWB/9OxvAxl6tJ6TtNq1/lkOJdBZypMvTW27O05dY6nBv7TuaGPwUkQsD0MJsf gVSc5Un1ABmI4nF97PE4XlOakY8AoEKHZe5YdH+QwImt+n+2JYzPjJQELU31Q40i11ON up3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=V4T/nqu5BcdfnctLll0nHcIWzjBkei1sx3rSAABEUjE=; b=imYR4SE5LjNdmkP6MRCei95JU5htYMmj6UinwIMd3l0v8RGBzsk0dm9M9VbAcOabIZ pIX0ZjiIuIASoZeHLecyz+PVysEa7ZX2zQEh7TjLkJBpTnvcT/vAhqYGarMsMQGQGuDe +hXV3V19W/CqVbPk4CZVrs6iSc9YTz3VSpjOJiVZYz2RCGPcVzPpBCb1xZS3EGNR9isU i+B2Ov5xFeTJGxu7O1Eswe/yLPR/9ElqSNsGK0TWJu0TKXc7uihrVMfMALM0TTntkWAu Wn5mjtQouuWcQi2IeYzrLugSDw4IJH0rGL0WIT+2T+sG8dIXFVk9X2IQYvyfnxGYVcdw ex/w== X-Gm-Message-State: AOAM533J56WS6YWsWPGSUNxz3QwMc8ROvX1P9ex8ezDzv7Mmvxl+MX6i Tl5KnFLst4s0bRQ59rfXurm3+26UABaw2A== X-Google-Smtp-Source: ABdhPJwRtKvwT1vWR2TJE39NW5Bk4nvU0GHRE/vEnDlzZ6WABO2D6wOJ1ezCZImGK800saA2fCiE1g== X-Received: by 2002:a05:6402:350b:: with SMTP id b11mr17545878edd.212.1638447924184; Thu, 02 Dec 2021 04:25:24 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:23 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 16/20] trace-cmd library: Decompress file sections on reading Date: Thu, 2 Dec 2021 14:25:03 +0200 Message-Id: <20211202122507.43572-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 9bf5cf7a..69c7c55d 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -481,12 +481,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1109,6 +1112,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1134,6 +1139,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } From patchwork Thu Dec 2 12:25:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73ECBC4332F for ; Thu, 2 Dec 2021 12:25:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236092AbhLBM2u (ORCPT ); Thu, 2 Dec 2021 07:28:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235759AbhLBM2t (ORCPT ); Thu, 2 Dec 2021 07:28:49 -0500 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89D0DC061759 for ; Thu, 2 Dec 2021 04:25:26 -0800 (PST) Received: by mail-ed1-x529.google.com with SMTP id y13so115218004edd.13 for ; Thu, 02 Dec 2021 04:25:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rRRaXmobcWSLIN9RqPvdMqevONWo7Khh7/TGyV5KCZY=; b=KtwgRL7ecewlewfjbxG12jMvaoE3d6QaMi3A5aQjy8cOAlWQ5jbTrEXTf1y1LXnVw3 DXFUn0Vbi0ZJulT0eTjf7nBn75k/B/ydczJRHpvW7zANWYw/oNXcahgztwXw8t23fG7/ BZGDcezSAkk9J5rqnUrfC66bMyot/1mFyoqiG0kkYLpnkEVMnF958y658FFOwj1B/XC+ o16zgFp3q8xdWcVs5JYy7WOvwTf2KM/FXRM6j4S+EPtLH4l2kh7WViFY5jCxmNe0+o45 b03Cm8NSyglbneERHd4Kx9/rqSqLMtlHtkf6Av24zVz42u7R8hV4vbMW7el65Vmdb/dc Pdyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rRRaXmobcWSLIN9RqPvdMqevONWo7Khh7/TGyV5KCZY=; b=cVYsUoYZ0nU6iCqEg9WEAEcxBpIKloHrLiF+w2FyL1GFsNomo3w4N+aVa6CWq8ECCO uO7vpO690gFyLXnOWBnlH+Wx8b7aCQ3sMe+4RCoXCQVAH/QxeUJ6dno/UIGe+LUX4Hyz X4AIO9Eyazr5ptRZ6YSn98qC1eps6IvzJDJbmxmSJTe2ZihcOMdWtdVnYPI6szD59vRk wXgI/Iw5szrneOI2o996S8kK57b9thNhBPGC/mlHF3yHhTsUTd+HKwT+Zckz0hzdeWxH 9pVZ3B9wv0DBGcoodswW0t2yGFbcEnEDLOc3p4PF+hJP0zJI+tCTUU2uoWChMzArEVHY BC2Q== X-Gm-Message-State: AOAM531wDge8RkkL1ghUozcpBA6CECZSqUYlpUazbOBuf0KLmy30Gto8 wb2V841AOJmwJwdrRqyHzhd9I7x1PCUUJg== X-Google-Smtp-Source: ABdhPJx2IK5z0vf3SFaJ+KVNXRJHac/fETgLRI5AK220bq/iS3fcrFmKU4vdSvhLObgW7XdFy5njnA== X-Received: by 2002:a17:906:3a4a:: with SMTP id a10mr14793372ejf.253.1638447925162; Thu, 02 Dec 2021 04:25:25 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:24 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 17/20] trace-cmd library: Add zlib compression algorithm Date: Thu, 2 Dec 2021 14:25:04 +0200 Message-Id: <20211202122507.43572-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index 133d1308..fc0724c6 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -356,6 +356,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Thu Dec 2 12:25:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC720C43217 for ; Thu, 2 Dec 2021 12:25:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235759AbhLBM2v (ORCPT ); Thu, 2 Dec 2021 07:28:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236179AbhLBM2u (ORCPT ); Thu, 2 Dec 2021 07:28:50 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A481C061757 for ; Thu, 2 Dec 2021 04:25:27 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id e3so115373796edu.4 for ; Thu, 02 Dec 2021 04:25:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0vF4/kyI9N2FIR8o+/109ZpzkBALfqAJzj8PQ0/ofkw=; b=I6dnLkN8VJnk12F+IXz2Bkoq4y5k3rt5be/pIGkAnhHTKw8Yrwr0S5tzc3kcYOFn1l +WkGjZkutNerF5ii+BXJcymZpoU9aVc080vDS3oS7IWaWhsJxhVs+zY8paU8E+mipVt0 m0fjv1aIUuRqxnh30WB9oAi3evVnEq6jG0+j1nsCnpxBuAAXD2VS5t/klbPvaHHqW7jY 8iIKrrobHqFo7pnU4tG7CveZTZe2XwKVbJC+k7er5Bt/SfjGKlB99U3Ot1yFYI4Lvfmb DhqJvTSny0uR7xecjweyBwNyu3eRhyQZBIMuGj7e6qf27+1D3s7bu1e+iUBuIw/ACnRi SddA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vF4/kyI9N2FIR8o+/109ZpzkBALfqAJzj8PQ0/ofkw=; b=4Kk5D2CcQDpgbPFQd4Iy/aPXuikY3jWmqnSy6tq+TWw2xRuPq6PDebsXdNXc8OK5uE P1pJlH+ZmE83R2snw+DsLeBV3uIesuC3o4wpapbeKtc6mGRyH9FCTos9761+Y1lY6m56 HtPXdlZmk9krnjiiALVc4L3fvBY7sVrmpyEe0FiGu/2CwUBazgwjosiw6j5LH1AzUUjL V9HDMfQBC/strPCENbmbwqvsfQjW9xNoKLYSOut0eaX5dKEew8BI0Vaq4BasS/EUM+ud YdkhIr0Ojga1QuO82eowrwUp8Q+eugzUfqZYLSzZmT3t4R1Anq9HVaJ2skItU2r0O9Oz HolQ== X-Gm-Message-State: AOAM530yKfjir2wHoEEnNH7C2BOqEMYENHP/YfnAuBXKdPU0+mCZBrqb p04G0KWMkeQKL2+/dYqNFfuAMQW4Z40PjA== X-Google-Smtp-Source: ABdhPJwZWpuH0YUEQIZZkvRQ56QMOb+xAjzOwMviNn4tZOGsc8COewrcca4oNeNyqGs/xrEzl8I3wg== X-Received: by 2002:aa7:c406:: with SMTP id j6mr17883156edq.76.1638447926115; Thu, 02 Dec 2021 04:25:26 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:25 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 18/20] trace-cmd list: Show supported compression algorithms Date: Thu, 2 Dec 2021 14:25:05 +0200 Message-Id: <20211202122507.43572-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Thu Dec 2 12:25:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9601EC433F5 for ; Thu, 2 Dec 2021 12:25:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236179AbhLBM2v (ORCPT ); Thu, 2 Dec 2021 07:28:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235035AbhLBM2u (ORCPT ); Thu, 2 Dec 2021 07:28:50 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AD56C06174A for ; Thu, 2 Dec 2021 04:25:28 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id t5so115452333edd.0 for ; Thu, 02 Dec 2021 04:25:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TDxUFwepNTTvF3M60OKnx4d8nzRSjUwdPXzqsUzLtXw=; b=Lu73oZ/4j8/7A9d1ag11a4dTSl6ovYgpxDo7Xw+9LTscu4r637kKoJdws+uzK0/HzX UxENjVtfAvkIY51ZuG1gQrPCa4FklEKXvSr0WanLsUAWw9MIF4Gb3qQhE8GFXcb1QFVe LXfj40mBRXMvwV6Me8V1NvlVyuiibYsP/pJDMAI7i6q56LzDnoKbSMaqWDj1d9ETyuYp xvyh9UMZ4oAZwBrlfmorUzUrlKHxBdcHUgPLF8dpVc11X6Jlz/ePULxT2BBZYlz3ZVFx 6p641pBo9YHO9oE2DCQ+pU82OLO5EQtgBhgpvri9ju4UFIHR5FtJFaSx2i20XtC5+f4Y fp4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TDxUFwepNTTvF3M60OKnx4d8nzRSjUwdPXzqsUzLtXw=; b=IUlQOX2xQI2aj9rJfXw5oMZBWOrECbExWEFqaxxdKUnzzTDgUEKPvkIXIY8Womer+x nhbyNpTyoMQbq3kQkiTwvkUmW9MxpmHaPM6h1m48u64BsJ/HZck9Z5tbKepPx2At/Ktn qnpaWM7aI24Bp9zNtrfxp5Qt7cWVKiM8kPxu0Fmsty8S14ej/o1UlL7rs7dhCm0IW0j/ Dh+vzMSh+FuHV63Xd8Ix0J6OG/1KL7/Wzf5Ob9kpLgT5oJE18KOIC9XeBL6IQAEEL6Gr N6Pl0VIi1Etsd6zRPKED+q2wRtrSg3JapaF4Sb6eZeBv6K06zQiodY0W+RnnjrDB71bu 2z4g== X-Gm-Message-State: AOAM532Aic+n8va7+xNo+zam0mR1Jt819L6bF0QLT2hDzTQ6AaxPN4qq iegbrEhFMqNZ2A/+a108nZ4rim/4PH654g== X-Google-Smtp-Source: ABdhPJwB20egTo/fZk6eQ9eMXI/Ib1oKILM3zASfzKJItquj76+wtn03AbqzbUxI9BuPnMmk5ws7Sg== X-Received: by 2002:a17:907:6d20:: with SMTP id sa32mr15149190ejc.108.1638447926942; Thu, 02 Dec 2021 04:25:26 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:26 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 19/20] trace-cmd record: Add compression to the trace context Date: Thu, 2 Dec 2021 14:25:06 +0200 Message-Id: <20211202122507.43572-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 5f6fcc72..9c799197 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4476,6 +4489,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -4510,7 +4529,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Thu Dec 2 12:25:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12652481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A2B5C433EF for ; Thu, 2 Dec 2021 12:25:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235035AbhLBM2w (ORCPT ); Thu, 2 Dec 2021 07:28:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237026AbhLBM2v (ORCPT ); Thu, 2 Dec 2021 07:28:51 -0500 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52960C06174A for ; Thu, 2 Dec 2021 04:25:29 -0800 (PST) Received: by mail-ed1-x52d.google.com with SMTP id o20so114585151eds.10 for ; Thu, 02 Dec 2021 04:25:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qi9ZOCkkrI67lvo79BJ1FoEwbE7E/NHK8slqqwg02jo=; b=Xy5ECM3AbItrKVrpsEMNlAX+b5R3ai66t1OxnTECOJ+rYdlpiKwFdYE7EabFHeud/C 0ZSavHfND/Fn6vwk2PQGAuimDC6T5t5EZ+twlqwLU83tw5VYsoms9JxpQbGY6xSvoHSa zc1ZPH6ZmsRUbYQdqNkOW3fLIss//42JcV8m4B7NAHJiCUEjXodpRxrJYN5Mnj08bLgY 7rzkBvy62jXLe2JkjXi3TtxXjCbsgRO+RkHCpE5ODToB+aem3rGS/aTzObW0I/cW2TCO kTyyifLx8J6/+E9NCtssCWzLIT2WT0RLpbJiVD9bJMWji5cPZfm8wB1fUUx1VO/D3r0u auUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qi9ZOCkkrI67lvo79BJ1FoEwbE7E/NHK8slqqwg02jo=; b=s/fXQCEITGNNeTuvPBI2gjYoXZsOw/AL4XCOaNMpJhYHKYwHuNcIvdFnIuP+eAAMuv k7TRGhUUPdctzrj6k6d9Z+HGMUY+fBqEqYGAkAsNK8v+H2C9HNqc16gDJWTIFWvPaH7H ZOYBc1B4ZHIbKpyY0V0/4eUQi73PmDVe86np2UO3dxWiLOB1HSfbresJtG3Eo1zEK+8l I0rKeS6JBPy42p8ePRh8gbaBlZquLkS6jMQRQ+Y2zeS6DoI9t12D/z8dqlVt9ieNKkvw 4ijmFCsDRVMPUf9YoCypATud5o4UkmppoMiWLnefcqee0dsupzcVpD3dsm6wZ6iUms/G /svw== X-Gm-Message-State: AOAM533j1Zf1pBMh5RH96YylWeBgKyQ/CcoHcckC/u/8BaCLhqohbIcV TGsuymDk7/XUpSnhbnc7XUymq0k2216p5Q== X-Google-Smtp-Source: ABdhPJwz+FMsMU1cWWBDxpq3tm2qDX+aC27vOBhD8QapKpEkd+03JKSEDrM9UFuHWyhCV8iMxHabOg== X-Received: by 2002:a05:6402:278e:: with SMTP id b14mr17370279ede.354.1638447927873; Thu, 02 Dec 2021 04:25:27 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id eg8sm1998507edb.75.2021.12.02.04.25.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Dec 2021 04:25:27 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v5 20/20] trace-cmd report: Add new parameter for trace file compression Date: Thu, 2 Dec 2021 14:25:07 +0200 Message-Id: <20211202122507.43572-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211202122507.43572-1-tz.stoyanov@gmail.com> References: <20211202122507.43572-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 9c799197..379b92d4 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5803,6 +5803,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6243,6 +6244,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6669,6 +6671,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version"); cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version"); diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",