From patchwork Fri Oct 8 04:21:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5319C433EF for ; Fri, 8 Oct 2021 04:22:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C360761073 for ; Fri, 8 Oct 2021 04:22:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237296AbhJHEYP (ORCPT ); Fri, 8 Oct 2021 00:24:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbhJHEYP (ORCPT ); Fri, 8 Oct 2021 00:24:15 -0400 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BD73C061570 for ; Thu, 7 Oct 2021 21:22:19 -0700 (PDT) Received: by mail-ed1-x52c.google.com with SMTP id y12so17998750eda.4 for ; Thu, 07 Oct 2021 21:22:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HNCXG0TPNxv6lE5pLmmVK/5MHmqU5WAIqkoiWFl6e6w=; b=K77WD9bHYFLP22ok+AziC1d0AaLizA89O8X52I1NoN//R5482GL30WwAZ1IsSHBxqW vOo0SaJkpTY9hWHZQ+SHXBfRBoT2hfK7eybZnrzhifuxZSb2qqmt6wwiBInvdfGuW+Lp w+YgYlyVkIwnMnzAzLUCN4JrbZ4xMWN2cax+2mNaJaeRwESQcnaLKMuHquJiMCz22ppB 573Cg19/KCE8kTAs7bs1aiDvC+8LXv8g+2IWqSojsnb68S1BzuOB1zLpPYWOZKRVkuDT AsH5wq3d+Yj79SWuDsZYYRyO+MjlgPSg+FeN+BvJEjI6Nc+iXwvQi8DYFw5qTYZN1Rks ZuBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HNCXG0TPNxv6lE5pLmmVK/5MHmqU5WAIqkoiWFl6e6w=; b=4Z9CRewMajOtoiyf7T2ehSPuTeQ0LVb/tgyaPmflyuIYyzSZnT+Hr+diwZUjWJgvV4 2f3W+KA5CgQ3ySBfAahOW4dba2fAcsp8pKIEVTbWBxLkEoMw86tU/Wc8tM5pZxEkMUxt iMWSubs0e7KNGxUOvcvpzWRtwUTr/Mia+jipphaDz3uSK4QhCKko0irni3sN3xH70z7S QDV7zrz2bHM5mfaCAU+Jttfd/PCQ1oj2hUoqalEdzgUNw7lqvCY4FuHWEnWCNvPnMdkP UZNdUN7ZYn28r7vwFXK8/QSU+voBF3hMu6tPYCmUyAVI3Oa9XlzCBdxEb/ZS2vgY7OjZ 9wpQ== X-Gm-Message-State: AOAM533qLrTh6F8SAZIUsgN4rTFxuZEAAEQTwqsMk3ndKQ/jzQMlhEyI aOgimPKm8H9xVn1Wdm7H9TN3hU7wcacTdA== X-Google-Smtp-Source: ABdhPJwOABCrjyhSSXdkB31/KmEySSKwyxkfZvO7cv6kH59LA8k76WG5vRHgEW6ZZcjtqvUE47cJWw== X-Received: by 2002:a05:6402:410:: with SMTP id q16mr11843028edv.286.1633666937943; Thu, 07 Oct 2021 21:22:17 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:17 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 01/20] trace-cmd library: Add support for compression algorithms Date: Fri, 8 Oct 2021 07:21:55 +0300 Message-Id: <20211008042214.977193-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 907 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 962 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 045fd5a1..04b3cb41 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -36,6 +36,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -154,6 +155,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -487,6 +489,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 4b26960f..fb008678 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -34,6 +34,9 @@ struct data_file_write { unsigned long long file_data_offset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..b198b8dc --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,907 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handler + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handler + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handler + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + handle->buffer = malloc(s_uncompressed); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handler + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handler + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handler, used to encode the data + * @msg_handle: message handler, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handler. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handler + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handler + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index 9c0f047b..dd6ce06e 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -625,6 +625,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Fri Oct 8 04:21:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E354C433FE for ; Fri, 8 Oct 2021 04:22:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 810C960F6E for ; Fri, 8 Oct 2021 04:22:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237300AbhJHEYQ (ORCPT ); Fri, 8 Oct 2021 00:24:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYP (ORCPT ); Fri, 8 Oct 2021 00:24:15 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31E09C061755 for ; Thu, 7 Oct 2021 21:22:20 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id a25so15479556edx.8 for ; Thu, 07 Oct 2021 21:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ulj39QlGWIj2iZbN2TtCGOnuyhw30EOyLNVn/hHxpj0=; b=GHOdyvncxIbHqSNvxqc5B59QmD15gz2OCuhjkVVaT7wmCZjamIrmyaDJy2iSAlRpHZ u3YiKo8gO6RY/iGdYMOSXMDH6ZzeD5GZcpw5ofDjFp6NNUfDxUIkH/qo/6IOc7njgWu9 QgxGHHNf6Is6820Az6hIi//S1F+JMYzTUjMmM/v8C0eSsOEf5KCWM9HF/trWa6Cg3o53 E062jqi1W1IYfZk/9u/Kh9bxNkZn7Yw7+BFhvbBwJGyi8fLWW3lyw5bgTh74mLk7BvqL s5kYWH6jZyG55KeO++0dTJpomgjrteq9YNpeGh8s4ERoDhXX63iiFwHZwM96Cm08zn43 pSBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ulj39QlGWIj2iZbN2TtCGOnuyhw30EOyLNVn/hHxpj0=; b=NCPJmWNfgaq+eASOSKYevM/nxd5vCUQ7rDHieQzg6N3xPjSx76WXlyLuByhVbS21zv QMmFg73UWSqUJcuf8w8PTuf18vZvtDeZHOgiDmOay62d/bJDwU2cjzVQzO0vASz0tbyU SFNzhHL3L6z+rPjwE9Zjc2cCbFovnzqPnJVjxq5fQwCwqf4ePVXt+RyaRVS4POZeovrw STsyA9iREJR7aQKC4QkMCAdAbUBUIaWMI+grmUmsLvMbFSugPbgaqW/aGjxIr/vLrEfg bmC5S9MzNAwOL7pE7T2kMZ2uBPMceCEnJVYeU0yeKQdbmc/vGUr+441tzzpL37fXC3ID Pwqg== X-Gm-Message-State: AOAM53112xOsEOfbG8ZGqgerfXj4M8toqp8RAq1jwJmRDOMzlCz3OW+h 3gJp6Qhl/Asax5AOsXXxpzFymBoaMjN8Wg== X-Google-Smtp-Source: ABdhPJy0m2vESGIZmF3vHTjJ4rqJ/sG1Ukw49nrjxHl2caAzwVcQFh1jf/16gbng2Yak5/9gNDAIFA== X-Received: by 2002:aa7:c78f:: with SMTP id n15mr12089074eds.338.1633666938826; Thu, 07 Oct 2021 21:22:18 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:18 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 02/20] trace-cmd library: Internal helpers for compressing data Date: Fri, 8 Oct 2021 07:21:56 +0300 Message-Id: <20211008042214.977193-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index fb008678..393d05e8 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -40,6 +40,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index a3b75c08..54eec93c 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -64,6 +64,8 @@ struct tracecmd_output { unsigned long file_version; tsize_t options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -88,18 +90,27 @@ struct list_event_system { static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -127,6 +138,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + /** * tracecmd_set_quiet - Set if to print output to the screen * @quiet: If non zero, print no output to the screen @@ -1544,7 +1592,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Fri Oct 8 04:21:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FAD7C433F5 for ; Fri, 8 Oct 2021 04:22:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB3EE610CE for ; Fri, 8 Oct 2021 04:22:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237331AbhJHEYR (ORCPT ); Fri, 8 Oct 2021 00:24:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbhJHEYP (ORCPT ); Fri, 8 Oct 2021 00:24:15 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 369EBC061570 for ; Thu, 7 Oct 2021 21:22:21 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id a25so15479719edx.8 for ; Thu, 07 Oct 2021 21:22:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jZlOUq11WfGf2SYeJWLjV9onGqB7pK+xi6WjnShSs2Q=; b=MLAKU9PDB0PYq4uJ8JR1//Jxt/ByvsF9btFvdIXNx+L095HjefmajBXwIoFLzk+aug zToYTCvyWBSTZv8HQIaxosO+LagyJRB+vydnnzbP1hwmn7zypNpooi4+i7Wfyg5Ji3T5 Z29wX1AmHami5wFxSeGwB5u8zoxOOf8FlitSD7ObEEW8c3r+CvB/kXzPdceAEXbdgKoX 8ftkPKQe9LpfArt/nD+ZfHbNWWGMVpOcjWcBRmb8rkxYzeVJWcGCdNBAbcZ/R7PPlDvx CGpAvBO8rXlhQuBp5mnBmoIdWZiBd6crSTZ0SUxpQO4ShEkdl0J4jvxBfAqQXkA8GCfY vkOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jZlOUq11WfGf2SYeJWLjV9onGqB7pK+xi6WjnShSs2Q=; b=oDj/OOvw2g2zzd0PHJmvC1IuZmxsxNGI3xQX2dvpT4QqY1alcftauXxk92kLTbRKIO tfjMTcUdGZ/3pAZNTZqkxxSbGPhrvm+6u3rkrODI5jwLmEiC3gwid81kQBvhneSz8uUS h0BCGPSRkQVhR5K+ufvJG0YT2T9zz+W8FouOmU2fc85/aP5wlNj6MAvP9csQd3gp5NJx HzGPC1iSiogw0pZKh7JN8EQ+nq/6WonrKB8EzMUPzP8K0wOl2hg24is2TDxDSDPra7P8 Cbt5e0gcQbeG1dxRvp5r1gQhzvARTwo6U3dYsDUj62ZtSoMWotoVT63cNTMtmfREZ8Pd Wplw== X-Gm-Message-State: AOAM5306HILkAgoMKJ+I9zm1yrBpiHEZlMraHDpct+6UE0pclT+ttYXQ zn0e6SH584bHJurMKL4TQ/vgQPAAPI7ncQ== X-Google-Smtp-Source: ABdhPJzLA1vrA34m9vcAXlECvweiN7qwNdqkmbYw7TKUS7baGP3YQgjR0FC9n51Qt/0uCbKKhoR7/w== X-Received: by 2002:a05:6402:1778:: with SMTP id da24mr11927844edb.398.1633666939828; Thu, 07 Oct 2021 21:22:19 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:19 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Fri, 8 Oct 2021 07:21:57 +0300 Message-Id: <20211008042214.977193-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 393d05e8..fd61150d 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -47,6 +47,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index bc1f4a87..71d5ea38 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -156,6 +156,9 @@ struct tracecmd_input { long long ts_offset; struct tsc2nsec tsc_calc; + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; char * cpustats; @@ -262,13 +265,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -280,6 +283,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -304,9 +323,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -332,7 +349,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -396,6 +413,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Fri Oct 8 04:21:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FAD2C4332F for ; Fri, 8 Oct 2021 04:22:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3946C61090 for ; Fri, 8 Oct 2021 04:22:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229932AbhJHEYS (ORCPT ); Fri, 8 Oct 2021 00:24:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYR (ORCPT ); Fri, 8 Oct 2021 00:24:17 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A622C061570 for ; Thu, 7 Oct 2021 21:22:22 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id z20so30929098edc.13 for ; Thu, 07 Oct 2021 21:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BzHSBWiqmJVBlrsu8Hh3kihiaFTfORc2i4irlq5+T+Q=; b=f55510kGjKsrM96FOSVIfZhKFvQMu90Z1ZqeV1NtmiQPDBbKw6IEuaQptTPWhDhemP u68Onqq4U1Gn/iCZP9jbku0Rvyqd+EGUGLvw7PMJWslSoyRd6bbdZbO8EkMiz3GQeBRP gEFaW3uCdGrZzNk8j09xcQAlL7+UYY9o0DsjOtcJQe9T4xO+b5H8ood0/ppfHTgG8Lyy cYZaDQejPBVx2aVt6zzf/Fj9P3r3TNTwEIIIP0ZVHiXsKashTCC2Iki1oLYwsM8V6Sp4 u3HlUM9BR5l4h+IyQTFmw0aVEqrTrILuN5Gbs5Zu4VgfJMwPrwzJ6K1VQcFcGKQprg60 BCGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BzHSBWiqmJVBlrsu8Hh3kihiaFTfORc2i4irlq5+T+Q=; b=FiZuGydroIsm3JfCsJx3KEhd2jLqXeEIan0KGz3pcZyD964B+fdXSyDZvnv2X6+kjl JSdHC9ScGTPUyTkIXCUAyiPPiOh1/CpNT/7z0QPagN53rtt9JklbJhBhsv4IW/4iCzt7 UQhpbqt6/AKhT7QVnQ6tnryuDeUSgsstCf2hdntZoYalGLXhGFHB3ljaUO6KfWPyZ8Uw Sh6PGogZF1TGOvUzFz169q/dFey9Pd7V7dd+8wSUhUMXnVF3Nt92ZIgLmHAyFbXZlAVd u1bVN5sDK9y+w4MLE3YASKWLmdAkskzOt5BvQ5zH8z8MGJZnvcTmbTe/KrWpmRTzfho7 mHIA== X-Gm-Message-State: AOAM531f2P14KPidcKldtf7J5cHU9uuOTQDfLMRBzbXzVJdCbxZ7YVhF h/9Cn697iJxnvCKSCvu5fkLKOJre+Kjimg== X-Google-Smtp-Source: ABdhPJxH3VNYW2YcUuLATNpH9UU3uU6U4SmcyhsoQCtnYEUYhwum+VZgPLHl8nwihLQktyNq01+9Jg== X-Received: by 2002:a50:9dc4:: with SMTP id l4mr11575218edk.208.1633666940939; Thu, 07 Oct 2021 21:22:20 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:20 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Fri, 8 Oct 2021 07:21:58 +0300 Message-Id: <20211008042214.977193-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 +++++++++++++++ lib/trace-cmd/trace-output.c | 20 +++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 04b3cb41..30315c00 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -255,6 +255,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 71d5ea38..35b49ac6 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4594,6 +4594,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 54eec93c..816db13d 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1209,6 +1209,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *ka */ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handler || !ihandle || handler->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1220,6 +1223,15 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct trace handler->file_version = tracecmd_get_in_file_version(ihandle); handler->big_endian = tep_is_file_bigendian(handler->pevent); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handler->compress = tracecmd_compress_alloc(cname, cver, handler->fd, + handler->pevent, handler->msg_handle); + if (!handler->compress) + return -1; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->file_version = FILE_VERSION_COMPRESSION; + } + return 0; } @@ -2167,6 +2179,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2206,6 +2220,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Fri Oct 8 04:21:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DD29C43217 for ; Fri, 8 Oct 2021 04:22:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 08D3061090 for ; Fri, 8 Oct 2021 04:22:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237325AbhJHEYT (ORCPT ); Fri, 8 Oct 2021 00:24:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYS (ORCPT ); Fri, 8 Oct 2021 00:24:18 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8846C061570 for ; Thu, 7 Oct 2021 21:22:23 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id r18so31297197edv.12 for ; Thu, 07 Oct 2021 21:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e5RPV7v8rnimg5v9gzfZIlUzgwnYJDHlYLlGtbei0HE=; b=DRAqh5uE5l3y+xMbPfDGfcn6t1dCb3cp2wTMMMqziGCX03zAMW9+oMk7GIjPaRjD6C OD6YQuacujmEqSfoxl5jETUwvldTu0HlFZNIT9e5S+toI/VhoOBhYNqxXt3mbJgYXj8n nIb3aKRM6G0P42iomqFEeCsAtWc4yvBl/b0u/V1MFQId34FxbgDaZEW6jnOYj9iPQ2es GBGD4xyKAvmZN1kzsYIjANaxxREUiFp5e/OFz22mLcJ9GQJJCYP3p+r8s6BBii9jowd8 Ohi98spmxncnHUgF29t34OpWXKQfqiu1isKo/HFkKIZpKDCO/ztNKVfIQVZmVIRmH9XL Uh0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e5RPV7v8rnimg5v9gzfZIlUzgwnYJDHlYLlGtbei0HE=; b=ZIyL9Tw0dFAgtAUyTgF/cRLBanmRmXYsdPaZr5XGpvvR7dSUTrIBPuM2cu6d4vBWd5 5UmtCkblkN6KmGrd4+hDDZHuDACLqShNcbCRSxm6XHtMT8hJPJlRsNpZYEMVtWjEo0sq IBOn8435q5LwiQvFeMdNyoPluD0/cpWJY9Jxg//RMvSHHmOHm8dE6IYnfGhS9Bow3jkH R/4LiQf2ksEfpg8JlUGUPtTDLXxSx4ellUjfMShdxjL0r0SOG1uhu06CvaDQMlRYagbA msux2c1pwE/dD6Afz+2virCx8h8C9aeQlYVFcwk7QiBALEv+FF0Mm2loiBPzuDsxIPLM 57GA== X-Gm-Message-State: AOAM533TldVtSDDJaeJnwnpSArRtWS1WzhpiNZrzTuY07Tb6FWicxlnn gy18gb0dJvzGydE1/3VHSOqRR/peyTXb/A== X-Google-Smtp-Source: ABdhPJylqiwrswmTJFIt5icgKRK6SQab4Bm5rdO3gxVy5JnMVqbYlyvQQzdiCTzcja1+s8Cp4b7UFw== X-Received: by 2002:a17:906:f243:: with SMTP id gy3mr1316161ejb.327.1633666942224; Thu, 07 Oct 2021 21:22:22 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:21 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 05/20] trace-cmd library: New API to configure compression on an output handler Date: Fri, 8 Oct 2021 07:21:59 +0300 Message-Id: <20211008042214.977193-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handler to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 57 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 30315c00..8f5408ff 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -297,12 +297,13 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *t int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression); int tracecmd_output_write_init(struct tracecmd_output *handler); int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_output *tracecmd_create_init_fd(int fd); struct tracecmd_output *tracecmd_create_init_file(const char *output_file); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 816db13d..f3afa7e6 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -234,6 +234,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) } free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1251,9 +1252,57 @@ int tracecmd_output_set_version(struct tracecmd_output *handler, int file_versio if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handler->file_version = file_version; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->compress = NULL; return 0; } +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handler + * @handle: output handler to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_init(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handler is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression) +{ + if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handler->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handler->compress = tracecmd_compress_alloc(NULL, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handler->compress = tracecmd_compress_alloc(compression, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handler->compress && handler->file_version < FILE_VERSION_COMPRESSION) { + handler->file_version = FILE_VERSION_COMPRESSION; + if (handler->msg_handle) + tracecmd_msg_handle_cache(handler->msg_handle); + } + + return 0; +} /** * tracecmd_output_write_init - Write the initial magics in the trace file @@ -1839,7 +1888,7 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1856,6 +1905,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in goto out_free; if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_init(handle)) goto out_free; if (tracecmd_output_write_headers(handle, NULL)) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index fab34361..699fa511 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4516,7 +4516,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Fri Oct 8 04:22:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A30C433FE for ; Fri, 8 Oct 2021 04:22:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C4E161073 for ; Fri, 8 Oct 2021 04:22:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237403AbhJHEYT (ORCPT ); Fri, 8 Oct 2021 00:24:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYT (ORCPT ); Fri, 8 Oct 2021 00:24:19 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FD0FC061570 for ; Thu, 7 Oct 2021 21:22:24 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id d3so3893358edp.3 for ; Thu, 07 Oct 2021 21:22:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kLY994riiAtfXyV0uZnSFb+ehr5ETVupLTBx32ojddg=; b=NKOlEiV7npiDwT+VUulpRCrs8MCtxb3LAXEq1rBznk/L5E5ihUsG2VN55fZTFACVnm 3TN52qyDqDzScCejdppJPUtu7og8gn30G0ubBY3o47ckL1byOTXQ5LAevG7gkRZANqgm x+n4bXtdyun01VWLgxDdhyiGFPjtTXq/3nGaCm+kkw8NZlsqB+p6VGSF92ojsZ/AU8K3 Mn6yuvMpCRI72WknTI9qCeLn11oNT4T1YuR49dQ3/HC/yfUWJ28m4sv1kk9PbRva3PJc CQGPfMb65HQ242xncKc6YYlT+5fkA5lmEbTukjvZrfeUAJM7HIaOycYpTmgNZw6/Ryfu d59Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kLY994riiAtfXyV0uZnSFb+ehr5ETVupLTBx32ojddg=; b=PRWj+P6AHbGn2uTKuvYpZz3A/aLAbyaFqJurDSROWcmedNsPQctHtmc0FmBBBUKjp8 TO2dCll5PpM2GuxFTX2dZxqedNDEp8jUH5+4CBgYs3HlpeaTi2N6i9rr070+fYSxQiWT q5rUx7vORVkfhGdj2TUM+L2nHVTsVivwTwYfVVc1g4OWWP05ZBGMSWHKE2L1j1MU2ShB X9Zx8aFq2dfG764dlPL8jYuKnKAU2YbGfcmGoME2/l1UBCa9wMcXfUQnvsPSA30GT0mc fspkmTedOLdGmlOIAM3qRm3Isv0995ExBgqXDPMBWMJCQl9sp/WKaO8V0BALoTKxjT0Y LOmA== X-Gm-Message-State: AOAM530+xpHdfCOQPoO6tb1gaEB54dO8S1OP2C+tqVgSLy976nfS5jq6 HjK7kYyjZHVt8PeHi5Cruts= X-Google-Smtp-Source: ABdhPJwX5Or5IX/FA6Jgo1QSfCNE6U1428HSa9bx6VXsSBaea+tIB3qkFR2dEAzKw/QiKq969l2gOw== X-Received: by 2002:a17:906:1e0c:: with SMTP id g12mr1268858ejj.155.1633666943137; Thu, 07 Oct 2021 21:22:23 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:22 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 06/20] trace-cmd library: Write compression header in the trace file Date: Fri, 8 Oct 2021 07:22:00 +0300 Message-Id: <20211008042214.977193-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index f3afa7e6..ee978409 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1076,6 +1076,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_allocate - allocate new output handler to a trace file * @handle: file descriptor to an empty file, it can be -1 if the handler @@ -1355,6 +1373,10 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) endian4 = convert_endian_4(handler, handler->page_size); if (do_write_check(handler, &endian4, 4)) return -1; + if (handler->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handler)) + return -1; + } if (HAS_SECTIONS(handler)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Fri Oct 8 04:22:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C345C433EF for ; Fri, 8 Oct 2021 04:22:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3DC4E60F6E for ; Fri, 8 Oct 2021 04:22:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237434AbhJHEYV (ORCPT ); Fri, 8 Oct 2021 00:24:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYU (ORCPT ); Fri, 8 Oct 2021 00:24:20 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96FF3C061570 for ; Thu, 7 Oct 2021 21:22:25 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id r18so31297343edv.12 for ; Thu, 07 Oct 2021 21:22:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4mkCTRKMwSpXL7qMAK0zSmx8Npt2l2sm/JfSDNeFJsY=; b=kLoDErA9sATOsnRObdyGJQbDHcg59zkLZIEiPKTYtOcF9f83Xpq/4tRHRk5qX+ASux PNhCnbRGpG545UcNxnAcyk13ACOOC4r4tG7ws+QaTpN9B0hJ+LkUTyXZcERCgVUDhvVm 4OTp0BLnQAiZbu/CdOI7NtUpq+EvpMJx5EQMMZ/edCVTr+j6B9YyuQZGxdUEHSnP0pmq XlZZQkz7NPpFlueYPmPEdNNO7nsMIFXWM3UiqVR/JaGbDrvivPXU9jIdiq9rp1StQskY UdAszawY29WHVsT7ny0cs0HrIfjzqoAgBFYsHCu83bUCyPFN/St8ELr2T8GHJPyG4im0 o76g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4mkCTRKMwSpXL7qMAK0zSmx8Npt2l2sm/JfSDNeFJsY=; b=UWYyIwkBjE9qAsCQDGrvRSfb9XNYZjhkK7Ox3F7v/z2+elKThVf+FBVmt2frNX+wX+ dwY/OyB6OlpZ4JrGnXP/9qG0+qN+cgqpN0ovNq58QrT+1Ky/Snkm9D2u8O9VmIhTCfWb CjEXAzQ/myRDLWAXc9KgV4xhGy1++5dkxhcz59hED2DQzZvRnTCpKwnkqZTU7fvycu3q SNW7wX7a4EE2ko5ujwM3E5xneFQo8428PZrFxPtoeG68nM0W9YTx0qnH1ijxdZICMbJK rJulx1Ij1fP13CVs9f3ZT9QqSayTwePtovlbzHtsd8IRa+D3QNZ9MtxkjZs5QnrE4Qbd 7aBQ== X-Gm-Message-State: AOAM531zrz00HIA93d5wckzLTtySuLGeMD9DYabgSV1U5Fj+zFvGvh14 91TmxzOV1VWmlocZhZxtW5XVbXJmSrIihg== X-Google-Smtp-Source: ABdhPJzbLQ/AyaDmIVDJJLLhe2ttr0X26oN/BuQbVsmwDrWQW3Xuh5cdHD0sc113Z+aYBVXfYN2Yfw== X-Received: by 2002:a17:906:54c3:: with SMTP id c3mr1269148ejp.536.1633666944229; Thu, 07 Oct 2021 21:22:24 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:23 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 07/20] trace-cmd library: Compress part of the trace file Date: Fri, 8 Oct 2021 07:22:01 +0300 Message-Id: <20211008042214.977193-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 81 +++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 14 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index ee978409..d15a5986 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -394,6 +394,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -449,7 +451,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, unsigned return 0; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -469,11 +471,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -487,6 +492,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -539,6 +546,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -546,6 +555,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -774,7 +784,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -788,15 +798,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -822,7 +837,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -840,6 +855,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -860,7 +877,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -875,6 +892,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -882,6 +902,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -928,7 +950,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -946,11 +968,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -976,14 +1001,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1002,10 +1032,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1028,12 +1061,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1407,21 +1443,25 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list) { + bool compress = false; + if (!handler || handler->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handler->file_state < TRACECMD_FILE_INIT && tracecmd_output_write_init(handler)) return -1; - if (read_header_files(handler)) + if (handler->compress) + compress = true; + if (read_header_files(handler, compress)) return -1; - if (read_ftrace_files(handler)) + if (read_ftrace_files(handler, compress)) return -1; - if (read_event_files(handler, list)) + if (read_event_files(handler, list, compress)) return -1; - if (read_proc_kallsyms(handler)) + if (read_proc_kallsyms(handler, compress)) return -1; - if (read_ftrace_printk(handler)) + if (read_ftrace_printk(handler, compress)) return -1; return 0; } @@ -1792,6 +1832,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1801,14 +1842,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Fri Oct 8 04:22:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B7DEC433F5 for ; Fri, 8 Oct 2021 04:22:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EECA2610CE for ; Fri, 8 Oct 2021 04:22:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237470AbhJHEYW (ORCPT ); Fri, 8 Oct 2021 00:24:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYV (ORCPT ); Fri, 8 Oct 2021 00:24:21 -0400 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 851D4C061570 for ; Thu, 7 Oct 2021 21:22:26 -0700 (PDT) Received: by mail-ed1-x52c.google.com with SMTP id g10so30385998edj.1 for ; Thu, 07 Oct 2021 21:22:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FDbY+Q//wqmNyJEUVf6j9jClB8JKizMICldYbJnoFTw=; b=Z7/uW7mVfDY1aVxgQ6lbCsybJOgjcAC/jPsg5HXj/6qdsYtsQrxsKT7MNNVLYD0di1 pOQRffnC2O+4MMU6zWbafBykyK5UQLQYioi9rY68VcUaz/a6PMlAq5vdOB9uicBiEJcc BXhf3AmPcesiN6c9QShPMuGIajcCkgn3aFxY57MuDN0bLCblNKbFp1ifVOXGMQg9kVny DG9hMY8ioSZ3ysqd5GAWCrbAbU2yGWXt9PC8U29qYpGpYJRz3oMjOq8j9Y5B/dSbHsre mLXS5FjFmUOsh3exnyRiNMv0gXTXgKmYVNj6UlGg0H5L+Z79xHUR/mvm2gTzFz9dhhJx L4Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FDbY+Q//wqmNyJEUVf6j9jClB8JKizMICldYbJnoFTw=; b=jU8ruIyWZ49cmr1pq/lF3cXwk414SC2jzHDaG8GjXVqmJvJzeqcxmmZJw+tzoA5DBa lcfhJyKFbtHKY5AKAJd+TqrNOQnG0CeNVZzT0CNqfX02qWkcLnW3+XKvcH7PX66XOeL5 caj/HlGCg2kSqpr3wYat3KhiS+J+olcz7JlGPuVdB/ra4kmnAijQY9bx/zKVp6PYdyT6 mBteMBIxQSG3INWir/HOe7gjqPEEKmwV2Uxxew9sEBGACbjrv+hM/jf3o1g/TDzSVPHf ngYgaCzJZBGy+eVYW+2hhPSfmIBpGtvz7OpuLoGED7XgCtP8T3lpepNEQKwgTJOvJBLs Z3zA== X-Gm-Message-State: AOAM533ORPfKBIFqkkdNfv/X47oSVtWRKPwt6W1RY0n+xgfOyC6IPFko aU5td+fFvimZVNSF9SkKDRKiFrmsOXcPKg== X-Google-Smtp-Source: ABdhPJwIxZz6ynq7TgsrgCLyEKAfOHA+NHoVR4T+UQNirUcMKSmDAhEmSaOmDkQSh4vB6O9hGRBFEg== X-Received: by 2002:a17:906:b1d5:: with SMTP id bv21mr1298790ejb.346.1633666945164; Thu, 07 Oct 2021 21:22:25 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:24 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 08/20] trace-cmd library: Add local helper function for data compression Date: Fri, 8 Oct 2021 07:22:02 +0300 Message-Id: <20211008042214.977193-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index d15a5986..c46643f5 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -283,18 +283,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -312,12 +321,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -513,7 +572,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -539,7 +598,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2181,7 +2240,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Fri Oct 8 04:22:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E752CC433EF for ; Fri, 8 Oct 2021 04:22:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CB10160F6E for ; Fri, 8 Oct 2021 04:22:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237588AbhJHEY2 (ORCPT ); Fri, 8 Oct 2021 00:24:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEYW (ORCPT ); Fri, 8 Oct 2021 00:24:22 -0400 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81355C061570 for ; Thu, 7 Oct 2021 21:22:27 -0700 (PDT) Received: by mail-ed1-x531.google.com with SMTP id p13so31789110edw.0 for ; Thu, 07 Oct 2021 21:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mbkWVYX6bZnepbH8tvfpiRdflloaOMFMD+LcyGxN4jE=; b=YwwRm2tkJUo2vLA2pa983oIR0wFiORt4Lw4NL6eaWIa1L3F9mgXYHICcu0J2YoNX9l EbXX5C333+SYc3BrW8Fc8duRIjxCQDvKvAqzqfAtjGcX0Dix+Y7SpxcEh8feA53Y6w1r igLFoe4kxVAH5kcmvEAKXiVYC/x4yr6QTy0pWk5APKp2l6cpCnr/SZ51B900w2Oerwcs g5S6aFMt0KIwsQylfMFw3cLaxA56UzvKNa0wJ1oAF895LEQYfbMmI9xcuAhtLMTbfAHQ YeJmY1KhOd1/+s3TWp3V6aQ2nT62rgwudJuQJktg7gX0Gog7CYMkZF96psb01yaVlKn4 U8fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mbkWVYX6bZnepbH8tvfpiRdflloaOMFMD+LcyGxN4jE=; b=rmesFi7o+mKF9HvMIyh83x4BvALXD9kYsI34fVuKUQs527xRkcSMqK5zALPDnfMgbn tz7poP6OMAAZn0BhZckiQHEKhtTBSt2jOJqZA2zUPwYEWgK4YmNjbBDgGbsycr0xjW/B qKZCHsU3I9OXWJqy88g8sCTnPZIA2pk/PxBkvjG9fOGlijcX55Er9sgucy4OOYCt3sRA XpeOyd0MXFfkcaoxRJ6z28w1cV7lus9iuYWZ3lJ4SX2U4iDouHtLuYHoxR8gCVAoKTiT NPSKy3FbA93hscS/0sa1jBtbREnep7OfTYfZGgo/hNTrfrCfN6bu49LDgccjon0wR3jX SVdg== X-Gm-Message-State: AOAM531/KurNvClA8oVF0xUXdrQ16l2xWvI1QuxTN7575MuQwQOtHbx+ wnGVZ9Gy5lBWLe5JHN7aLV1GQZrXDlGoGQ== X-Google-Smtp-Source: ABdhPJz0GEAAF9YqjHgqjVGYEFBZgjSKHmiSGyApDZFZndtzeoJ3pFhhdZuX/D4yntoxuAOvwtGq8Q== X-Received: by 2002:a17:906:c7d0:: with SMTP id dc16mr1245817ejb.555.1633666946193; Thu, 07 Oct 2021 21:22:26 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:25 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 09/20] trace-cmd library: Compress the trace data Date: Fri, 8 Oct 2021 07:22:03 +0300 Message-Id: <20211008042214.977193-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index c46643f5..84fecabf 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2079,11 +2079,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2188,6 +2190,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2240,14 +2244,15 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); + if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Fri Oct 8 04:22:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14BF6C433F5 for ; Fri, 8 Oct 2021 04:22:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ED7F561073 for ; Fri, 8 Oct 2021 04:22:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237676AbhJHEY2 (ORCPT ); Fri, 8 Oct 2021 00:24:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229990AbhJHEY1 (ORCPT ); Fri, 8 Oct 2021 00:24:27 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9697EC061755 for ; Thu, 7 Oct 2021 21:22:28 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id x7so30440353edd.6 for ; Thu, 07 Oct 2021 21:22:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OplLznRiYNZoJoYX1kaMLf/ADzl46nZzui0VyRKjHlI=; b=dqwc/XTYhcPTD/utk3NscaUt/Go+VcrF8yl6XNWrztYz6wVqdBOmk78wBFIYgjfQ1I 1WR4YIKde5wcY4DVgcw8A/2yciuQZFASDGlvyPJfKtr4Rw6MC6/5ukYXq3Udj4F2Araf BM9hLfNF9vja6ZlAHpHR55GqJfMPjkDBSx3wUbcWfRmKMxDm/aGfE8WRrJczABUwSohw wOLJGt5Y/CmU1I47KbW8+bLMO8JLizyBFS5DeYw7p69enOygxiTYn9ew8MT4N1v2JtDf 1zjE1sFFJlNClmWZgkd5l3zbEEB/3s34p574pO6LByYV3ZK+iKsrwgFoUsypGK9V6qR9 OpQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OplLznRiYNZoJoYX1kaMLf/ADzl46nZzui0VyRKjHlI=; b=DU3z4g1+8+Yq9zX3LIBzLSoy/XIjOH8Ug+S10OaC7A9hTIvXHdcq0o5qnvLpl6D0lt eeonbbtA33fcpF3aQMqwTAf+SKgaTozf1YW4upVdif9jDid0BMP1Wi/jOouNOTwjcpIL CQlguacWMUqzzu5u0xP40xlB0tbyVlsrVvI82Sg8kj2svz5SM0pDdh6ll9mbqE5tNr4L HtRDMXnJz0Z00VBmjTLi/0OazTmJFSFP3s5IpzJ+boCaebCerhPQQye243pU/LiaJDb3 Dc6aOlKlmXKlJ98kF6GDH44J4JnpfwYKxYPCtdWlf+AkKGvf6l9N05wA/3IzF/FtC+lA 1kOA== X-Gm-Message-State: AOAM533ETIi0T84kaTCQi00Ir3b1CMd8f8DCKYxJuPMrymLieWvKlfSQ u+IdgpHEtyhmUeBYeXRCFZZ9oHu2aV7PhA== X-Google-Smtp-Source: ABdhPJxR5G/ZuGzHRSZcKLVDJBCh+Uq1YAqRAHU/0GVyDgLplvMdZ+bTGW9fD4OjXSUqMv091hmLjg== X-Received: by 2002:a17:906:c0d7:: with SMTP id bn23mr1362250ejb.426.1633666947218; Thu, 07 Oct 2021 21:22:27 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:26 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Fri, 8 Oct 2021 07:22:04 +0300 Message-Id: <20211008042214.977193-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 45 ++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 35b49ac6..2bac5d2c 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3030,6 +3030,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3041,23 +3042,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3107,15 +3117,17 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: buf[size-1] = '\n'; cpustats = realloc(handle->cpustats, handle->cpustats_size + size + 1); - if (!cpustats) - return -ENOMEM; + if (!cpustats) { + ret = -ENOMEM; + goto out; + } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; cpustats[handle->cpustats_size] = 0; @@ -3125,7 +3137,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3184,6 +3196,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3196,7 +3210,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Fri Oct 8 04:22:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5DE5C4332F for ; Fri, 8 Oct 2021 04:22:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9EC7661073 for ; Fri, 8 Oct 2021 04:22:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237483AbhJHEY3 (ORCPT ); Fri, 8 Oct 2021 00:24:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237473AbhJHEY1 (ORCPT ); Fri, 8 Oct 2021 00:24:27 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 688F4C061760 for ; Thu, 7 Oct 2021 21:22:29 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id i20so15207327edj.10 for ; Thu, 07 Oct 2021 21:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T0K7GwnJLmI+4qfTUOWRjmpoq7S0xJk9BVAQkvoraVQ=; b=FbNwmnxeuWMeju2GlNIAARSnn8UutJfWuK8hvQzYKpxWN2paxS56mTgKT3QVcWrbrD sjhDGs4uIlj9Im4t/HrpHfYBTRqz12+ZZg2vQkfO8r8YJ02Ot/EQtIJsES0fa1f1JNCU pc0M1BjjikhMhHhtFlAh+AlpHKwE6BaIhnDY6n4b1VGVRa5VK2/maSjWKCQ/FWvOyrVv 0Nrj3sE/66mI+GO2dsu7THqJvTA2Gdz+uoWGAxL0Br57wNykYDT/iYPrG5476J2k9dD/ PLu7tDwjPK3ziCRIYcb85OcqSJ3b9nQDz5TfAdPDzQkjXOYL2gDFL9vYeCbyMQeweb6B 8rwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T0K7GwnJLmI+4qfTUOWRjmpoq7S0xJk9BVAQkvoraVQ=; b=3A+P9+LVgf1TgfT0l4b96LW6IB7eH0pH3zdaPeVvf4IwHeN7tcsZv7GypmeNpkIRMu iz7/iPtOI+XxRFdnHrSZ6NcQ+QH0w6tdLs4vqiVuZEaDJPCRVJphJpSgtTomO3I1yra+ FDwAKk6K25dWi7vd46loW2id53LLiXl8+3ho36HJU9hmeQvPqaFhzQZM8VkSBcFHlFMB HAP3zeIJwx2BhGAGXOlwn/MbGmg/CRaG8AJ1L37mi+B8jXeQB/cmCDMGCvE1750AL7JQ 5nnZUWmATzJI+2QZsUsiqLm6bLY4H5Tu9g3Dp8tkp608ZygdYnAGHm164Fv1xZ90lY3D BFlQ== X-Gm-Message-State: AOAM533nAWOfts9yDl7Ax+4gtYSwaxhPETUPURlU+KW07lvAbpcsuXe9 zGTtTBFFDw0v4KNmwAYvyggL3D3lA0MAtQ== X-Google-Smtp-Source: ABdhPJzi49lJrauvf+dXldym/8cVIX+0KVXXYCHzF7h4GvnOWSLXGksiPxNopO7SYt6+C9zIvBF0Vw== X-Received: by 2002:a17:906:1757:: with SMTP id d23mr1394111eje.102.1633666948070; Thu, 07 Oct 2021 21:22:28 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:27 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 11/20] trace-cmd library: Read compression header Date: Fri, 8 Oct 2021 07:22:05 +0300 Message-Id: <20211008042214.977193-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 2bac5d2c..5bab81c7 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -189,6 +189,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3785,7 +3786,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3823,9 +3826,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3855,6 +3861,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3867,6 +3893,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4065,7 +4094,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Fri Oct 8 04:22:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57E26C433FE for ; Fri, 8 Oct 2021 04:22:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E808610D1 for ; Fri, 8 Oct 2021 04:22:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229990AbhJHEY3 (ORCPT ); Fri, 8 Oct 2021 00:24:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237483AbhJHEY1 (ORCPT ); Fri, 8 Oct 2021 00:24:27 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96124C061762 for ; Thu, 7 Oct 2021 21:22:30 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id a25so15480526edx.8 for ; Thu, 07 Oct 2021 21:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+HttJMT1mBdtCkRnWh8CR8m3YCUIELsyTpMeyuTgX+Q=; b=bX5cOo5YDt82Y/86a/5IFGKKYHrrfc2Kp7aRF2oIJe/9OQT2uhSFxQPyXhJA60cAfD mXq3oA6qMOk9PsY7YmE9zI8czlSmGCcfkUECttzfz6YAbJvooEZIRb7zmePKq7WA/DXs O07rFU8j/7FBQ1blQy+l9TStoZURsLaeeyim+DeSuxXidhf3VjoOJjdV+a2ehxlRlHVn ZX+OW4BQ0e89KstWu4aLpVkSMwajmGciebLVhDabISAZtul5ZzHsnoAd/hiRwEDArKdZ HfXMLMyfccgZsj7QsrZEfjUz54uCMGhtVAl8ZXg8LbB5lYithC+fDEcE7U/W8VdXs7wn VTqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+HttJMT1mBdtCkRnWh8CR8m3YCUIELsyTpMeyuTgX+Q=; b=GfpsCoPy/SXnEOXpM59HCdM2Xp2UjB/2JdlktB3/o7kmSApkg3yW0b6qqk3ciMD8S8 qP0Iy8xnRxK7SPtb2lr9DMOYy6zuia5u6nXllBfr/Sr0KKF9D1JTEv8rLbXs25F/VDMl diJrAodvJ+4itCgKwVvOkP2rKFgTgOsyn+3lRpwqG1hWS98XCI3bGjbmoqdF/Q+JZkNw GjpqZHax5xbHUTPJTA4JcqY6F/uw7cJrb2UqDL58mepE2vE3oH1s4gFsY0rErf8vD8KD 7NHBaPAJ1PAaB8Sz8e9KzJzYyIBNzhMXCUWwDWp5fCvVbY3j43bno10D74KGmrctc+6o +Zrg== X-Gm-Message-State: AOAM532MS6wbuaOrQ4R7KaP2dPb9QXxPHDSjm0YTOGr3+GDT4LFYVy5I wuhGdX683U3AOiTk+BrB3TMkKFhAdIKxmQ== X-Google-Smtp-Source: ABdhPJwvBJeP/op/wBC0Q0Eme9J+ntz8wZwek4QDsOBi15ORVWh/jUZF5SqPFk/M4l1569/uKfHwBQ== X-Received: by 2002:a17:906:af49:: with SMTP id ly9mr1246647ejb.479.1633666949135; Thu, 07 Oct 2021 21:22:29 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:28 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Fri, 8 Oct 2021 07:22:06 +0300 Message-Id: <20211008042214.977193-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 5bab81c7..05245b01 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -150,6 +169,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; unsigned int cpustats_size; struct cpu_data *cpu_data; @@ -3295,6 +3316,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4031,6 +4053,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4050,17 +4073,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Fri Oct 8 04:22:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A917C43217 for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 519A2610CE for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237528AbhJHEYa (ORCPT ); Fri, 8 Oct 2021 00:24:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237506AbhJHEY2 (ORCPT ); Fri, 8 Oct 2021 00:24:28 -0400 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F28C6C061764 for ; Thu, 7 Oct 2021 21:22:31 -0700 (PDT) Received: by mail-ed1-x533.google.com with SMTP id d3so3894102edp.3 for ; Thu, 07 Oct 2021 21:22:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nmuDKwjQ0gmZrGZMOUsAsxupVw+MyW0hRiiylwgfjwc=; b=FgQZY30TSpOOEve+JUIVOKsJVKfRGoYanswPeWNCjzUZJExGxWCgeewd9i8OAtmdJd ssNE0GgUVahewxoG4TGgOSmmNiwN+U+myP5eQNeYVT44W3dgglo7lbhrgzQ6YqQPnlRM sTkRz28G3iu1VDZ7eQjZ4uzHPyNVySq1TlCQDrZGg6Nzscbg2E/ZQYsLXoozYLCBWgyM XWrEHTyIhTmDyk7rdDXOVFfpnJsTUnzZgyz7ZXYaG3jwhryXiAZaNfNKyroJkoCOo5b3 ToObdqqKeYouuIP3psxrzHveFYTIGMD6zbNgJTV45Kkdj1MN2nz/MqvH1ewolIRHsp/D llyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nmuDKwjQ0gmZrGZMOUsAsxupVw+MyW0hRiiylwgfjwc=; b=5ZJYCMr3bgWzYjR3XG2947ykmvbpE5Mo1G+YoYqhjw3pmDDvSa+IP0TXzgvJ2Ku3Li l6r4aPzYqwbT5jKr+KeFz4TuBpRGTA27AR3tgqvrUjLJbB/9RTUG0+yJvdlJ1+KXlSgo TyU2MGZXvonVUSxaEwzYghayvvLrs7FKCQCT8LjjIDQVxJ/bA65ROTIm+p0it2aBraX5 hiOgrN0noWGSXbwe+YHdpsQmUu1+I97NdwPki3MkoAW0ku6nxJZJAG64RNL4xZyfRW3u HpsbPAnodLqzHK2NrjQAMWqa6KmQ5Lpg+eJKrf1KhdcClE8dWVRSt9NbV1jvQ+r/ntW3 HBVg== X-Gm-Message-State: AOAM533ZkyJTFMst+c2cMIFyXudbJPA8dfGt8vffPsxM0Ni5Rz0B3ReM PMzm09AyXmIvACoS793VWYrmop/jWejmlA== X-Google-Smtp-Source: ABdhPJxqkiAlTZwrXiaBUifHQT3hPDX4D8OfvdYAGZHnGL/Z3fNtwJekBwwvMP+lekAqJEQ18Zml1A== X-Received: by 2002:a17:906:1146:: with SMTP id i6mr1341550eja.12.1633666950266; Thu, 07 Oct 2021 21:22:30 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:29 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Fri, 8 Oct 2021 07:22:07 +0300 Message-Id: <20211008042214.977193-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 05245b01..91114483 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1258,6 +1258,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1297,12 +1298,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2494,16 +2498,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3373,6 +3437,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Fri Oct 8 04:22:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A705C43219 for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1038161090 for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237473AbhJHEY3 (ORCPT ); Fri, 8 Oct 2021 00:24:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237528AbhJHEY2 (ORCPT ); Fri, 8 Oct 2021 00:24:28 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F29BCC061765 for ; Thu, 7 Oct 2021 21:22:32 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id b8so31229227edk.2 for ; Thu, 07 Oct 2021 21:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o+Fxm31SyfeqGreYI9kYGBHxACCNUlbbnZmJapszDTk=; b=E7sRB0xCD9yKtwPkCWQNH15CHXE9VD4wmp1tCh3x/lskhbpC4KISg6leCyjDw5uUYn 3G2ZRUZnxvPMsPq8VrdtX0yQxVzONEVoGCI5eWA50lKjZYr/NNusYPfVGe5Bisa/hUiq Kz/2/ZSmJsn10/eqZIqlbjWCMAFGVMf+S0VdO8iP2wBr0FqW0/P+gJgztbga4d8wSIP4 W8Ly21EV59sgGeh5r9znOmRkNy3H4rIXhHmiM4eaYui8pNrRfZJaFDTe1RfBG77nIM+9 RT80ZMSKgrRPfgLrgDN5aA3cksxxnJtDDV1KQZguQAk4C9BcpGK2X5wMcjgTEGovYZhf FDAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o+Fxm31SyfeqGreYI9kYGBHxACCNUlbbnZmJapszDTk=; b=AlmSGLgXfAn0l0et5Nt4Rb8hPZcpdFQI14WNYcsRjD9euq+ED/0l4GG5uhx/sBPziM ZJpjulblrYsNT2ChfxHVfyM5l9IcFUuYWtQNidXniY+pOY9TVGl5rDco0wToln0JgLxG qBnScfSNrXQ/3JMYsebe+NoJ+1sLnWMdFPHzSjKbf0TAykiY5oWUP3rThIVnvMKIpuQJ kLOa3fEVZ6Xw11OxYyByGXO+6trxeD0ziaWt6ZHD9vaSkTjAesiV8MO07xH++S9Cx2// F5kWQ9q3AwKq/Ppb7fto5pIsea7cgDxHhu5PGbg7+RXEL+4+qnsXOYVEYpDlAqqq7LE7 43HQ== X-Gm-Message-State: AOAM532KYdlAg0c8h/1xRuxw9W9FM3Uxb1bYfF8hetUoPu8YOK6N4hoe PzvbpmtR/zkb///59xzGZGDij/x4f3045A== X-Google-Smtp-Source: ABdhPJw/cwo9eJquMfbRNwEYMXTTg8Nr/dh8oFgu3Sz7gR0Br0LyQFPKZA1WvGx+X27pEvH+bio4pg== X-Received: by 2002:a17:906:1e43:: with SMTP id i3mr1251510ejj.313.1633666951510; Thu, 07 Oct 2021 21:22:31 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:30 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 14/20] trace-cmd library: Add logic for in-memory decompression Date: Fri, 8 Oct 2021 07:22:08 +0300 Message-Id: <20211008042214.977193-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 91114483..5b9ff041 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1249,6 +1252,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1260,6 +1362,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1402,6 +1507,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3890,6 +3997,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Fri Oct 8 04:22:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D47D9C4321E for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BEB9C61073 for ; Fri, 8 Oct 2021 04:22:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237506AbhJHEYa (ORCPT ); Fri, 8 Oct 2021 00:24:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237272AbhJHEY2 (ORCPT ); Fri, 8 Oct 2021 00:24:28 -0400 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D53C1C061570 for ; Thu, 7 Oct 2021 21:22:33 -0700 (PDT) Received: by mail-ed1-x52b.google.com with SMTP id p13so31789602edw.0 for ; Thu, 07 Oct 2021 21:22:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P996cjf3GrFUMmj4HFkJqxrKoqFh12ht3l56R5JeA1E=; b=DiNoyKODDyTL5Od3cK8qmnIFzo8sx2ABujbpKTWOhywX7v8qAnW+qoupJDV+nXqCAs 2oIouXl8ULE61d9jslL5pOc7ofCkhT11yl9hV/6QWR12CgU4v7f/u0JawCawQ7Agn0bZ BE4Q1QKuBt4siriNjjTqji8BJwWoVJZQGK+jmWZOelYNoy0tMads0Etc8IvNAmK1FpcC gmKgreFCsfoR90/Zo9PVYwSqSmPcoraxH3MgnsF9fbQUWu3YBuXcdYnXTWqsHOGGHaQ3 4nhAByROb2Q9oexzNwbIPDUR6PP4M8e8/5E70iNlLqSpUBTpvhjn4pp0Y+/bl5V/xYul ZaUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P996cjf3GrFUMmj4HFkJqxrKoqFh12ht3l56R5JeA1E=; b=pV5Rhjhj1PHelilxvynjLbxP8j45FbGd7thlqDewDUF64gwpiK9Vh6lC6yLAcwA4fb i/x0q6AEqYu3yMTh9XEAB4ggMX8Q+qYo/kR284tpSquvGeYS1+2o8ehFw0Kx7COxB0im 7Qic2r/NsVBBFbTmheurerZOmuBUD+98f3E7vB8+Ri2lHRZG2puDA2K3I2hXJ7ARkfGZ sHHoQDoTynjJ2AkD1xRxB7wJS/VXCjU9MNa2DWy+Wx0epZXsgmEpylPhYYfP6jyWxlmN xDzVpvsAlEbOpog/N4Oz1OJkJtCfAdfPY/JeGCNU17MZdehEKFaJHjMFaUO4hhM5s1Xx P5Mg== X-Gm-Message-State: AOAM530Io2hnUe7rn1u8LNkES/fu6MBlE5GczkpbdhqnirLXX1edWDVi BYuGQxsCzj/UnwfqI1E/zFifU6Fi9aIgrQ== X-Google-Smtp-Source: ABdhPJyco6qq+CmuBnE6mBLG5YNF/VD/1lQrtpiYTDlmDKxqtaH7RbJEjS1MAj3aOeziynPSgHueqA== X-Received: by 2002:a17:906:5ac8:: with SMTP id x8mr1339240ejs.132.1633666952385; Thu, 07 Oct 2021 21:22:32 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:31 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 15/20] trace-cmd library: Read compressed latency data Date: Fri, 8 Oct 2021 07:22:09 +0300 Message-Id: <20211008042214.977193-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 5b9ff041..fac19bc3 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -176,6 +176,7 @@ struct tracecmd_input { bool cpu_compressed; int file_version; unsigned int cpustats_size; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3445,20 +3446,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3526,7 +3559,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -3994,6 +4047,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4283,7 +4337,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->trace_clock); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Fri Oct 8 04:22:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A9B9C433EF for ; Fri, 8 Oct 2021 04:22:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 16639610CE for ; Fri, 8 Oct 2021 04:22:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237272AbhJHEYb (ORCPT ); Fri, 8 Oct 2021 00:24:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237548AbhJHEY3 (ORCPT ); Fri, 8 Oct 2021 00:24:29 -0400 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E941BC061570 for ; Thu, 7 Oct 2021 21:22:34 -0700 (PDT) Received: by mail-ed1-x532.google.com with SMTP id t16so9453914eds.9 for ; Thu, 07 Oct 2021 21:22:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NhNKCt42LjGqWZzqsilFGm9PzXjpdDKYXJwMnrgf8S8=; b=US58dpwVeRd5nREiQ3CLIPT6aRbv9bEExI9R4qrp7pQStfU0QqTdsHVVAlS4NQQgK7 fPCMErm/AzneHwRk+ZgbTb6FV/iSblOdvTs1D+hqtG9k6ViTnXJRZvhyBt5SSakoDops ksqGY0ueJgjGZ1V/9eZAUnS49DaxMMHZ4dlsxrJFaW/e260xljemkNjboRAfItl0XF0P dOUbkkEQhzwBzwR9FUy/aljMoy7o4obVVC462sCG5vyu7fA1r56WWgcTbNz0Egwhb2v5 y9NURQlDjG2eEc+DhwQ1+h6pIbehXrcL0qJkDchyUn19GAmHgClUugMdZ9ORDM377UkB 6R1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NhNKCt42LjGqWZzqsilFGm9PzXjpdDKYXJwMnrgf8S8=; b=V0sssCQxFw5bcd/q4gcdIN6Lm5tUuGGmsuP+eM6qhU48go1SdUTvqV8ILqY08YqrcH 2600vViaAltR//vDiR4U8ufh8aq1DTVru5dUU1kU/fu/JYRzo+Sl6WQWnoqX3c/rA3X5 7LfIoeE90lG1xIjwQ+i3hrd5RHmSHlZV4aD5Q8FzL/+hr7K4abIb/gCTL8mIaoWhYFfN ZMTWiiEMCkXsuUVHnV/5m8iQWNSdjjrl2r5EW7rmycxvXzbYUgdz9rpvKC4L15npXl3f BxaRiQKKGQpaz6KLmR4muRZgXi8zGxn4rEDtXhpVbZbyOwI+JueC6YMomO2uSBFShBl/ 0abg== X-Gm-Message-State: AOAM531tO3yCOML5P0FWV/jeU9723UrUXGm5TCYd748qtH/+zHsjS8IX R5ALwmM7taoM+KI8dVEN+8q7XH02+ZDvDQ== X-Google-Smtp-Source: ABdhPJzOhASNr9zU5cIwtRGSrX9TtbC28G5nOrjFqlRqZxG+wt/XVziHZ49WyyeabD5R0kMCT3vmPw== X-Received: by 2002:a50:d9c9:: with SMTP id x9mr11879458edj.179.1633666953612; Thu, 07 Oct 2021 21:22:33 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:32 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 16/20] trace-cmd library: Decompress file sections on reading Date: Fri, 8 Oct 2021 07:22:10 +0300 Message-Id: <20211008042214.977193-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index fac19bc3..a70bb0e6 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -480,12 +480,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1108,6 +1111,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1133,6 +1138,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } From patchwork Fri Oct 8 04:22:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95394C4167B for ; Fri, 8 Oct 2021 04:22:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80845610CE for ; Fri, 8 Oct 2021 04:22:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237548AbhJHEYb (ORCPT ); Fri, 8 Oct 2021 00:24:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230409AbhJHEYa (ORCPT ); Fri, 8 Oct 2021 00:24:30 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 049E1C061755 for ; Thu, 7 Oct 2021 21:22:36 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id a25so15481069edx.8 for ; Thu, 07 Oct 2021 21:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iLadVhxsdhc2oKpVldExKhouCWK2WtumXPezfpMgf2o=; b=jK9/dGaxQDfCzksZsMMfVKcsak2UZF1IXZBB5WIHdYMcWTjxgoaFCLDvnfEa3AkY1x VjBY3WZ30jxKkm8Ggqoa3Dea208t3Uk1VoB/ZvoHo7YQjuImTi6EPyTosNSXTuwzB/x2 iuGR5Q8TgXYtko4zOrJ/JX4y/YvH0eFldBIyYltVdd1Wq0ZX7GLE5ItgNKKbsCLSHey3 9J20Abe7Uj7NgC5yho+bXuokuCiK/NYwRq3U52T2FDuKWwMjyKtyLwbE8N4nxZsGHUGo TO0ZgiNKg+Q7miQwDBL0GjuEkwqwNk6c3mnspARbirvo/8B7NPJ0y6O3Ovo3uMHB4KV4 s9vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iLadVhxsdhc2oKpVldExKhouCWK2WtumXPezfpMgf2o=; b=E9wICkWb4EFboquA0/wsaEnzbxpID+JZsHVcMr7KC9NKdpvFQSOS8FuPza62xiTPsu POuKjkdexbUUmX/dbucVNdApIcRTNZtpgvuQAP0/wQZQZgPHVOmMdDYBjqr9vfwGHvwa h1nnHUAYgb0Q9+fVwiyyy9OO2eLXMJaaIwJBKv2vHZopSKDwtj/FvX9tSFbzbjYpLF7c QCzr4xBen10at2fjtntg9OjUlp8VlHiAAtENL9xQwdEF9sMQY0bCk8LjmBbDdbm9P6s8 /69LTYq9t6esBjfKRNZCi15VqGDEf6vyon0zBUZyq8ewrjnBP0U4hsOIGanStZHrYunN eHbw== X-Gm-Message-State: AOAM530S24M4+lRx7HqMAg0i9iIYK65E3ACTAW94ani91p8lhyML/zl2 YhUX2IdULXlyPJ4x3Fcb7e/xB6SjDtXmYw== X-Google-Smtp-Source: ABdhPJz3Fw0jdjkHGbIAf/DwS7vRQzGF/rj02PcDULKs8x/T6f/2x8HMezSivMEwo0acD66WBBGfkQ== X-Received: by 2002:a50:da48:: with SMTP id a8mr11545196edk.146.1633666954609; Thu, 07 Oct 2021 21:22:34 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:34 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 17/20] trace-cmd library: Add zlib compression algorithm Date: Fri, 8 Oct 2021 07:22:11 +0300 Message-Id: <20211008042214.977193-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index b198b8dc..c1f37cdf 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -356,6 +356,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Fri Oct 8 04:22:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28C8AC433F5 for ; Fri, 8 Oct 2021 04:22:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 14CA9610CE for ; Fri, 8 Oct 2021 04:22:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237743AbhJHEYc (ORCPT ); Fri, 8 Oct 2021 00:24:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230409AbhJHEYb (ORCPT ); Fri, 8 Oct 2021 00:24:31 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4EBEC061570 for ; Thu, 7 Oct 2021 21:22:36 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id d9so7380431edh.5 for ; Thu, 07 Oct 2021 21:22:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=S9kNfKdEtFktYBRt7XYN2HCvseOpo0b2E9fjbHNGVRElxzn+QafoUlnwXlkCotsUmp 5jTP51UasxrvMUI/jiYifV/7hEJCazJpcBW2lBX/rIMGbztgQhj1N0XQ/WWlYZ8BAMHK CJLTa3zJpC/frmHnH0glDmOk5+SnPYdtOetLtDsEfo2/xLl1ySAv/ZpF9/JOmUHm6g0y PdRRJCuAKoIojhT0pyFkwxb0IgEHx5XGFKO0wy3vYXeGmU9JBIhscgCaQ6fpiQETDckB mlxeVGketcNcNqQHOXdB764UI+Ni60ziyQjs41TFNiupMRoxfDcHt5Alkcbcfr/amFSI ea7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=J8NvGYIavmrN1+bw1ILeRetn2GUo4mkbW2vgsw47Nj4QvUxTIRZmEfGLUke7++Wioe tKHzriVG3z1+faEyZC7q1dyyZf6Z9bf8x+bREFK3Cs0MCSmr+SOOd4nNYHjt1PiDSQ7m /62hmB7qJqF3xXWl/wWPRAzkfgW433sGnTsbxtIowiH58+3JtE5wMsB7x8g/QV0eTlwo HM1QuZM+GbCrjyomvX/KFSqA3GEIhH42jZ8XzkKyBWeu1pwfOUmxpL/+piCKh09tTzCT 3VXGgORkXHelQ//ET+BR7YBSTrbO+uKns+3BPP+i264ptoMT/OFBEdgFL56S12RsCi9W 4FYA== X-Gm-Message-State: AOAM531pWEaZ0EBeQcDywaIU7lHwy5k2bhxjHs+Icj9f5PflJSj2v7Sq z178G8qc9b2krMmeg9+Z5+lv/bkdF53uNg== X-Google-Smtp-Source: ABdhPJy1G2Ma5dUxiNpdJ+u+fGEMCCu4f2bviPZiwTgV/7G3mWeiAlBVLw5Qd4qXlginGaAR1EA2IQ== X-Received: by 2002:a17:906:16d0:: with SMTP id t16mr1268252ejd.199.1633666955492; Thu, 07 Oct 2021 21:22:35 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:35 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 18/20] trace-cmd list: Show supported compression algorithms Date: Fri, 8 Oct 2021 07:22:12 +0300 Message-Id: <20211008042214.977193-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Fri Oct 8 04:22:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 702BDC433FE for ; Fri, 8 Oct 2021 04:22:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5CB22610D1 for ; Fri, 8 Oct 2021 04:22:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237563AbhJHEYd (ORCPT ); Fri, 8 Oct 2021 00:24:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230409AbhJHEYc (ORCPT ); Fri, 8 Oct 2021 00:24:32 -0400 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F30D1C061570 for ; Thu, 7 Oct 2021 21:22:37 -0700 (PDT) Received: by mail-ed1-x533.google.com with SMTP id d9so7380526edh.5 for ; Thu, 07 Oct 2021 21:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H/kfknDnaK/8HeyoD4vfja4iBq75jUdeNSesjqMXEnc=; b=iJ8BahdvLsW7URfqCyg00wjv3ipg8q2EG/zNuHju1KiWKy0ih/8Z8zl6+kFXKkWv0D 2m2ERsUiAWUzDO7lbf1STUbctUuvLUjBgNnwqkvjZFWZu1hOqlCiEV8b+a0cGc33qwA7 dDrVDWMO/4FkHtow7LTiV0uqqHCVe+nxvvPdbQBj7e7CtsXCJnFP+a9EgpmMo6FrvIOx VLuWzjl8it53GDqBMRYdl7v+xxGCNCojJGX6gLr8tHPb8IlEp9CWXe5imlm08GHyn2/V Vi8BZUmALGhESmeGrs8kn1qUHI7fl6fLV5oqtLM+wp/xfhmvg6bnsUde4Mu8ve+9vWma iu5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H/kfknDnaK/8HeyoD4vfja4iBq75jUdeNSesjqMXEnc=; b=E0Z9cvNPTPBYyC4fZj9g2sYehieYwNySW5vk3pbDomRqzsYtjqA5zAZ3ebDTkgRWcx S12dBO4WS+fnj+SHIRiVSwOatgGPULuMifFZkHKbfn35l9KuFodPgtNdQSKPIpV5kTrH w36nBRy6ZSgvh6vhuQURoFnlvYNzdvAK37G9tqPGPdAFi3iYTHqN/hHoM5ODU9zzAvXH K60ZNPlb9j8sWC4TPSes9kX4fQP+otSODGNsS34AQLYOhrjhNc+VUC39wMh2xNmL/5IT Zgve43UZUY0Poq2VEwk5hJyEFoHIdbQfI8ddQVS4dG5qLFQIW+jyps+U68SnocxY/g+7 eZVw== X-Gm-Message-State: AOAM533n/VqC787IWgrU+9NrRITrfuVlCtM8R1iabgGX0EapBJCi8XIx UhvQIF086nZ6i8i81gGcHaGuCGofHgfxsQ== X-Google-Smtp-Source: ABdhPJzzyNFeTllLy8F6vxELmVI2Q6pXVKtkFQ3LkHcrecDFr4WKRrAaVXFYkmQy1UD29KlLmROSFw== X-Received: by 2002:a17:906:3c51:: with SMTP id i17mr1346595ejg.86.1633666956642; Thu, 07 Oct 2021 21:22:36 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:36 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 19/20] trace-cmd record: Add compression to the trace context Date: Fri, 8 Oct 2021 07:22:13 +0300 Message-Id: <20211008042214.977193-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 699fa511..98d663f1 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4481,6 +4494,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; return out; @@ -4516,7 +4535,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Fri Oct 8 04:22:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12544335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDE0EC433F5 for ; Fri, 8 Oct 2021 04:22:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A409060F6E for ; Fri, 8 Oct 2021 04:22:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238054AbhJHEYe (ORCPT ); Fri, 8 Oct 2021 00:24:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230409AbhJHEYd (ORCPT ); Fri, 8 Oct 2021 00:24:33 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33376C061570 for ; Thu, 7 Oct 2021 21:22:39 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id g8so31262127edt.7 for ; Thu, 07 Oct 2021 21:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DmZHkuLjsgT5eWVyHug1kQZkTbOA0cWVB3dmUNlEydc=; b=moW4N2QWQMRq1Kstpxc89Yrm0HzmaCbKxk6slB7kPCnvFPwDIHlsNdGSChUOJOGD8K KhnLc+zTEVRSNKTpVR1rUjIlvbuIGHPOoI6L7vS6ktQRxydOw6aRzDLVuiIRBsXPFVqG qmzUmN/C1Fq8dSq/HJYrSO39EiV0CcBXdFFGgIbnycrEDqLKYp0THAJ13q7g/P11ZLUS xUJoaNQ+bl/E2VyuBxgaoG3Ks58tIdaek/APEdxeK1oJ+2FE0At/fH8EgwBQmBG64oFV 18s6NKsDN1hM03fG3M0miarqzxxm89TfoBYzQIXQjfFsvijeUXJ7EETCaxNA7PYKfQwy 60/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DmZHkuLjsgT5eWVyHug1kQZkTbOA0cWVB3dmUNlEydc=; b=zkQ4eGGf3zSbSVmcIjhCNTcUhpQPX52I3r6HaaKoWzBbpOd181UmtHO7e+YMULMv6T jxTpIkdfY3tbf3AYREI05T4B4ZPzg1zASQbbX+3Nq7Gi+v77szyoZAxV9KyhYtxy1ZFn 3ilQvK39am+9o7Lahu//HNN7DDLK8dkBJGrjyH5vBFSGusWn7usZQ2QSbE9H9fYyT2qs lIVG2jH03i8IoMjpZz9+SVN18s6JXjhN17jUzPu4JPQ9+ZjsHmh0CIYPDjEtGUvUhFds pEW2KOg1igWJnXuXG8+FoWZVRWu2JHNJQsz8B4LOL92sTLQHn5s6uNCQLahku4Hb/2v6 avxg== X-Gm-Message-State: AOAM530gM3ZVbUwndYZnTY458TTjWzus6tBGFMPeTcmdZLLD0ODfETAO 9cs8+s75oXBepV1A9FNALg+zgx7CDrh9Kg== X-Google-Smtp-Source: ABdhPJyO+nOfCOgbKdOMYPYFh+ZOhj/lhjfqi4PU1r0myZEvVx9+NzpV8juGaF5Fllhbhf0eWJm7Lg== X-Received: by 2002:a17:906:2c57:: with SMTP id f23mr1344988ejh.474.1633666957821; Thu, 07 Oct 2021 21:22:37 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id g23sm517429edt.1.2021.10.07.21.22.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 21:22:37 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v3 20/20] trace-cmd report: Add new parameter for trace file compression Date: Fri, 8 Oct 2021 07:22:14 +0300 Message-Id: <20211008042214.977193-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211008042214.977193-1-tz.stoyanov@gmail.com> References: <20211008042214.977193-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 98d663f1..cbf51418 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5810,6 +5810,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6250,6 +6251,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6676,6 +6678,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version"); cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version"); diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",