From patchwork Mon Sep 13 12:41:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D302C433FE for ; Mon, 13 Sep 2021 12:42:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 14EC560FC0 for ; Mon, 13 Sep 2021 12:42:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239052AbhIMMnY (ORCPT ); Mon, 13 Sep 2021 08:43:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239048AbhIMMnX (ORCPT ); Mon, 13 Sep 2021 08:43:23 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C4C3C061574 for ; Mon, 13 Sep 2021 05:42:07 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id x6so14451924wrv.13 for ; Mon, 13 Sep 2021 05:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dCwJEKOeshLtdpXMCJCbKaY22zxydQd9xaL4ctaCW2c=; b=X99/IYtKmFoksv8xgj9dArHXZbY0gvyG37snFERd/Okurgzgu9xQfy8r7ZvtqeTUmm Tqqmhk3Q489OVK4wQ16FL1ft0Qe4YOZrlR3+QfYgY1USNbrFpxAjpmeMipMwpDcbB18m NUGGGdcGtu2sN1f2b/BpGed5mNp3bjT0Tp/UJNBW9asR0hOCVJrpSr7Gst0sG3TymVcK PucveOMe+1RxEvbgUIbFuOHaVK/iVExkTYo3sSzi4spr2NVorDJi0lnPg4uwAyyDoksm lOkU3JzhHY1pc32ABj8OXe2EYoGUzZ1qVmFCSQ4hS70zao+3WD/IOIBRnfiieUeoorKh PPgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dCwJEKOeshLtdpXMCJCbKaY22zxydQd9xaL4ctaCW2c=; b=gKoj6K/+J0p9QyeQ8WmVq4vuCbVF3zwq2zieXST2JWVPS5oVOEcHQxjcCkxNZi19OQ AM/2o9foo9r83BL77GWltAc88leUCy/e/yBmkj/kMOrOrmxdf+wsWeZXKw09wL1pIY7q 9c3XooZj7kzBvHgcZ4r+pXo1UiWgm0Dqr/euoLnMOD1oMXnZDNJNBJWUDfC49dtZbqRy qeusNtdBNcvRuPkTost6fE5N3TSc+5TA1WLUSih7iuU90AcVgsaOwqEXSlkOBEZBde2F /0CkcDecRVW/k+GJHT5jJ/31jFqpwYdJ2HUBc9T0SHVyRe3/6XZmh6Y7rDSIy5p/7zOk /xpA== X-Gm-Message-State: AOAM533PyS1pyuFSBu3nLqt5bVQS/PaK7ZrRrU46bW226JQfCkVfLUCI L2bSLkQ8rJeWk1nMWPzs99FQzmKFXZ0= X-Google-Smtp-Source: ABdhPJzqSuU+a8MI+u+dhiTTU4BAGSqBdUe0bD0WW0bDmQGk2X1mcM/iBvDCB612DJzFOaZ+HWOOyA== X-Received: by 2002:adf:80eb:: with SMTP id 98mr12352232wrl.370.1631536925976; Mon, 13 Sep 2021 05:42:05 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:05 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 01/20] trace-cmd library: Add support for compression algorithms Date: Mon, 13 Sep 2021 15:41:44 +0300 Message-Id: <20210913124203.3677760-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 907 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 962 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 045fd5a1..04b3cb41 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -36,6 +36,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -154,6 +155,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -487,6 +489,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b29236dc..bd95a1fa 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -32,6 +32,9 @@ struct data_file_write { unsigned long long doffset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..b198b8dc --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,907 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handler + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handler + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handler + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + handle->buffer = malloc(s_uncompressed); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handler + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handler + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handler, used to encode the data + * @msg_handle: message handler, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handler. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handler + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handler + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index d28b6ec2..97db6ed9 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -625,6 +625,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Mon Sep 13 12:41:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2080C4332F for ; Mon, 13 Sep 2021 12:42:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 88B6D60FDA for ; Mon, 13 Sep 2021 12:42:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239048AbhIMMnY (ORCPT ); Mon, 13 Sep 2021 08:43:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238985AbhIMMnX (ORCPT ); Mon, 13 Sep 2021 08:43:23 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A3EFC061760 for ; Mon, 13 Sep 2021 05:42:08 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id d21so6779628wra.12 for ; Mon, 13 Sep 2021 05:42:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sK2aoDnvq7JgG8byEWgz4DnvgCvWptaFw6S3jxHl3kA=; b=ks33LWNN5YVbcHW9rmeINNHrj2gheGuqifXhtCnG/XClGmuH19WBS/5mNfOCQAs6bI /1vfSDqwbu0brqprg4CbcANzBTFF/1oNQAoTAxpo0727T535CgzkBb/4VKgwMnq63x28 5g/6W9i+Q7U3IVMKUIBhOKB3GZLBUBPbKY9WsJkcLHSXU3jcul6zpIEs1CAyZlvpo+xn 83dEVgXN4/PczTh74r8dbyILEupun3rB6qo2D3g7u1qzxjnQmQoeFU8NePwAnILrMOYv eviKm3ttJcUBNSpZnGMDOtWiNxvvL27TGXemiY9oTSstoUSlXXE+qKz89uQ7l0991L+O 1guA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sK2aoDnvq7JgG8byEWgz4DnvgCvWptaFw6S3jxHl3kA=; b=XCskLsnqd1AK+iRzuyyJ/dVOnHL91tEcIITggsJXzRTckBwclKbRoFkLRHt4q+Ala4 706yP6mHqWwN8JzA+E1WUBH2ihbpicQYP7kjfn3+zWWpOLLK+dPjmFCs5WFIxgFFbp+d wIclA4/jENDAPqN4aIKL/nnloL0lYSyhMXzBaBFVbYaypAjZ/Lc2mK1VHGCJRg+TYLsM 1nuOtr5D+/aLV4RyvxYaYFOYtzftEZ84s+w1VIEdFBJhI52H9QlwlZbqF+TNN36Y0U/y Veqq4rJmLhJtSqIxYO7b3RcoiGUauaXQQoKkUlpB0bvMGS8LKvnDI99KdAvivZSSTNC8 Ejow== X-Gm-Message-State: AOAM5324nEcB+5tUvPDRX3BfTk37WSUNO1Hpuvcp9G0yCdsaXI3ilbLz +MgF+7AJTOsmZE/BSj7NIq97jBCCxV8= X-Google-Smtp-Source: ABdhPJyhHU7NxGDiiYeWH6+GNJSmwznz75BOhxJJh2xXh4lgcztLh3iU05CzFY69/RXb/wvJc8wVAQ== X-Received: by 2002:adf:8144:: with SMTP id 62mr12465063wrm.144.1631536926750; Mon, 13 Sep 2021 05:42:06 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:06 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 02/20] trace-cmd library: Internal helpers for compressing data Date: Mon, 13 Sep 2021 15:41:45 +0300 Message-Id: <20210913124203.3677760-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index bd95a1fa..304bec49 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -38,6 +38,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 42151799..2a694f7f 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -65,6 +65,8 @@ struct tracecmd_output { unsigned long file_version; tsize_t options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -89,18 +91,27 @@ struct list_event_system { static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -128,6 +139,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + /** * tracecmd_set_quiet - Set if to print output to the screen * @quiet: If non zero, print no output to the screen @@ -1552,7 +1600,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Mon Sep 13 12:41:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BE2FC433F5 for ; Mon, 13 Sep 2021 12:42:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 46DE060FC0 for ; Mon, 13 Sep 2021 12:42:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239101AbhIMMnZ (ORCPT ); Mon, 13 Sep 2021 08:43:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239085AbhIMMnY (ORCPT ); Mon, 13 Sep 2021 08:43:24 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E71FC061574 for ; Mon, 13 Sep 2021 05:42:09 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id g16so14506738wrb.3 for ; Mon, 13 Sep 2021 05:42:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Jdv8rK6vXgqUw+vNfuhyVo0YJpHG5XQUVh2ccXuM4d8=; b=g5O8mAJJWkQzoqm+nu7proO3SwZWN1t9ag0NB7opb7FfqUOUPO4adk23A+i6I+r0Dl uLnc7SS020+S7qcKi5ZYBKHDyXCcDevmnZssCLj4bQJ73uWNYj7PgcmJrlr19wRAfrfc //0AbMLNYTDmEoJjtl6BStezhn8dQcdSRBQgWOWMK56OD/gIsVVaIt6CGQo8CdUFjqlf yVZSwSxSiY96DZ3o7yPmu2l1pNu2tzvS0mPGqsbJF48lGdMbkzC+fpKw/47GGbhNkYYX DtAnXRB7Tu/bg0LpLY1FSkvVBkT8ronOBjxtoJpSxfN6kQtLpX1pWL/Zfq/WWPVZO12s QH+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Jdv8rK6vXgqUw+vNfuhyVo0YJpHG5XQUVh2ccXuM4d8=; b=siugycrze+Knk4ddXw1Qhk1JrybIpUkTk8NTg/iV4PlOXtdF38vWJw8AOzl+jRtemO 60n2TRUPMiWDO0530zKsdGvftqpqs/s8BIQbVJByk+VcwwmC30gwWuAN6Jjw+01CETv8 ULOUZvoQxpIH+9purR93BT3bJp+6HKX3pmlILK/ktRa1Hhj4An/1wRiXYkklHHWcxTq9 crJWNltDigVBFq757tfidbLgu1SYhs5jK/yopm18UY2Mg6EFHU9O1m+kvfdRbRr4naFQ gt82lK5O4QmcGrbg4zoDeFHyEkHBd540gOCOLVVoE6rWwOd21EU/egQfCYSxKnkeemyg 0hpg== X-Gm-Message-State: AOAM533b2NZYj8BDYdviAfbdHRxtRP4HtP6jumdDgmEuzJEeMl+seTIT ziONC2f0JAHmjZxY548wW95in5cv+Cw= X-Google-Smtp-Source: ABdhPJyM+KmdB5SMalHbe+LMNkUHYzt/Tn0aYmskF3Gvz4RePypC5fxk+6HWiKuYCf+vKFV6YeMP4A== X-Received: by 2002:adf:82cf:: with SMTP id 73mr12756696wrc.366.1631536927711; Mon, 13 Sep 2021 05:42:07 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:07 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Mon, 13 Sep 2021 15:41:46 +0300 Message-Id: <20210913124203.3677760-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 304bec49..14223a08 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -45,6 +45,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index b7d78a3a..25d376f7 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -155,6 +155,9 @@ struct tracecmd_input { long long ts_offset; struct tsc2nsec tsc_calc; + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; unsigned int cpustats_size; @@ -262,13 +265,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -280,6 +283,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -304,9 +323,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -332,7 +349,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -396,6 +413,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Mon Sep 13 12:41:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A987C433F5 for ; Mon, 13 Sep 2021 12:42:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E976960FC0 for ; Mon, 13 Sep 2021 12:42:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236325AbhIMMn0 (ORCPT ); Mon, 13 Sep 2021 08:43:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238985AbhIMMnZ (ORCPT ); Mon, 13 Sep 2021 08:43:25 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A411C061574 for ; Mon, 13 Sep 2021 05:42:10 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id s24so6543933wmh.4 for ; Mon, 13 Sep 2021 05:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fCEcFUPk4LuocBOj4/4arEZQy38dO9Jl5rIjrAm+ExY=; b=qeZXDI8YoMYyMK7E8lveTweKaC7rvLVwlsLuE9X5DFSu6A0RMfXztbxqzxB5/J5m8X h7oVGxRhlksh/2C3IAzksjvj1wJvXdFqL0SL6FUqtqOTSrovwBUOq+GSAkhdMPuSRvgP 3vmaBxfnthVhbKRT2SyQ91+oepN+cziPOPyqBRKylOzHXkvHHwy0lZ5ffkGvu0n23dXd y6MVMiVI6e+WAbogZdCHV5gXl6ULeZydQBDq60v1eh++DH5LteieEvf0CZ947eJti6/j zZnjGIojBavQCx4YqckUyPManrKx5KzM+SeoMIuuiXHFQlu9qia2OQpAhP1Gou7koMKI 8O8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fCEcFUPk4LuocBOj4/4arEZQy38dO9Jl5rIjrAm+ExY=; b=IS0HpSDhC9N1WgbdP++MsIO/4np+JC/DkLq6wqbxHblnwuBes5YnZJrKMvNl/Wik6D V3+NsWaLfqe220XK7JzRClDhoQXjYnx9v9UCQmvXskEYKXcDxfFCC3XJm/IyQLGQ31BI w5GSyJ6bbGizM5CkvdcHCKOKOCEz6ag5mIqkj16Q71vGqWJIoa6f7xUJSpyZjjWo3qdv yhP10QxpovQKpii+L+4hDlwKzjYLhj6AhXuWDWE/XvzpQnChA9WloV7QNGc1ufA5yjOE EvJWcSqmMFaSNTXtpoHzXPXQFJdLHZWpSOwoE/cZLeePI9Nw2PPtaPcsTHsfVQQq3YHp NTRg== X-Gm-Message-State: AOAM5303+zE4L//6+Feg3mmde53ItmQvrtyoFaf14Ljz4nRR6RScCoT7 MwOB1ffv104UaAH2DZkbiWV1/vdaawM= X-Google-Smtp-Source: ABdhPJwbIyWVVw+YuErUHYrO+ZGVbRrnvK25hqJ2Ehx1/ABajVNPxm3L70zXZ0JmUvpqF2fUnhBBGA== X-Received: by 2002:a1c:1d56:: with SMTP id d83mr5564061wmd.74.1631536928605; Mon, 13 Sep 2021 05:42:08 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:08 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Mon, 13 Sep 2021 15:41:47 +0300 Message-Id: <20210913124203.3677760-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 ++++++++++++++++ lib/trace-cmd/trace-output.c | 19 +++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 04b3cb41..30315c00 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -255,6 +255,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 25d376f7..2ea8f5a5 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4606,6 +4606,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 2a694f7f..a655d35f 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1213,6 +1213,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *ka */ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handler || !ihandle || handler->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1227,6 +1230,14 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct trace else handler->big_endian = false; + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handler->compress = tracecmd_compress_alloc(cname, cver, handler->fd, + handler->pevent, handler->msg_handle); + if (!handler->compress) + return -1; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->file_version = FILE_VERSION_COMPRESSION; + } return 0; } @@ -2166,6 +2177,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2205,6 +2218,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Mon Sep 13 12:41:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 520B5C433EF for ; Mon, 13 Sep 2021 12:42:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3B21560FF2 for ; Mon, 13 Sep 2021 12:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239109AbhIMMna (ORCPT ); Mon, 13 Sep 2021 08:43:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239085AbhIMMn3 (ORCPT ); Mon, 13 Sep 2021 08:43:29 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBDD5C061574 for ; Mon, 13 Sep 2021 05:42:10 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id m9so14500750wrb.1 for ; Mon, 13 Sep 2021 05:42:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GysL5SYJIU/rSFQSYU0TJCpWOlQDAsa29SfJkSuSj2A=; b=JttTGDd76898bSrfzCpobCHcIKPLHliYaDUvAudyclF/u4A8cc6qmp171zTpMMCPZC Uf9aYuOmLvJgz1Tp1hB3nRcKGM1pIeWdkybuhlqF1y3dSJEIN/sm9tybPfAw+bfgInkR cuEySVMl6lJ7Ng5ihoUFUy5EOjrO7S8XjnjnYjPmlbnkQbE6Uq4WR6SvCrXkPZm3okdc 8MwZKWMNWZs2GSDffovOM8wuHzK5KOfY99A1pk4FlrF5RDstPO4Pm2i4gJMcmi9VKsmz IAo8uUGKvKcbvdhKJfDX2735H4zUK6AQ5ShYFinAkCnXlNU8QwpDOfq5c3DqDbW7vjGO dOXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GysL5SYJIU/rSFQSYU0TJCpWOlQDAsa29SfJkSuSj2A=; b=BPOCEj63v/8P5NjKIg2EZ+PEu8sWTrnk0cVPgv++xdGzHwWD5bF7yrTNda2x950AjJ bxA8YzAN1S0uQod/bIwRKV2zbEzgonkHP3ON027t/q+52gv5PQ5y8B1ErhLXjlSQBMMc Fmeh3i3d1/TYPL5wa1C5hbO0bYmFpvPWZy1JSECrZs2oZYPKFPgkVyvqf1Seupc8lvjs fpSlSay6K88OFtzp4VIRZ7aP3vZu9Z1IrMN5Et9vG/stF8UskqgG9qLugMYg3wkkPUr/ d9lkR9ztjBSLma3EmBQj/s5aynT/5Gp9RWsKBnaab4WdZ063C3d43P81U95Gl10uO1v8 oXvA== X-Gm-Message-State: AOAM532AUbGTnah2RxhLRgGgrulgyDfM2HwjtCqsYlOHql4fxu9vNdB8 jelS1LQuNujdPZt+4lmD1jgG8Y0uP4c= X-Google-Smtp-Source: ABdhPJwiAH36qFtIXmeDtnrXMhlzDxGgCSMA5xu1YIaa34cyq92UHR9mzSZc6A1e9fLhbg80/CnonA== X-Received: by 2002:adf:f904:: with SMTP id b4mr12417570wrr.403.1631536929421; Mon, 13 Sep 2021 05:42:09 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:09 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 05/20] trace-cmd library: New API to configure compression on an output handler Date: Mon, 13 Sep 2021 15:41:48 +0300 Message-Id: <20210913124203.3677760-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handler to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 57 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 30315c00..8f5408ff 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -297,12 +297,13 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *t int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression); int tracecmd_output_write_init(struct tracecmd_output *handler); int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_output *tracecmd_create_init_fd(int fd); struct tracecmd_output *tracecmd_create_init_file(const char *output_file); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index a655d35f..802d35ed 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -234,6 +234,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) free(option); } free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1258,9 +1259,57 @@ int tracecmd_output_set_version(struct tracecmd_output *handler, int file_versio if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handler->file_version = file_version; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->compress = NULL; return 0; } +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handler + * @handle: output handler to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_init(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handler is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression) +{ + if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handler->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handler->compress = tracecmd_compress_alloc(NULL, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handler->compress = tracecmd_compress_alloc(compression, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handler->compress && handler->file_version < FILE_VERSION_COMPRESSION) { + handler->file_version = FILE_VERSION_COMPRESSION; + if (handler->msg_handle) + tracecmd_msg_handle_cache(handler->msg_handle); + } + + return 0; +} /** * tracecmd_output_write_init - Write the initial magics in the trace file @@ -1846,7 +1895,7 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1863,6 +1912,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in goto out_free; if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_init(handle)) goto out_free; if (tracecmd_output_write_headers(handle, NULL)) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 421bb7d4..fb6de81b 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4516,7 +4516,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Mon Sep 13 12:41:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1151EC433F5 for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EBE9160FDA for ; Mon, 13 Sep 2021 12:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239125AbhIMMna (ORCPT ); Mon, 13 Sep 2021 08:43:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239116AbhIMMn3 (ORCPT ); Mon, 13 Sep 2021 08:43:29 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B26E4C061760 for ; Mon, 13 Sep 2021 05:42:11 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id m9so14500829wrb.1 for ; Mon, 13 Sep 2021 05:42:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tC1J6R2ziZNlKPTE9xNNUQxKtbzGe6ytt8wDdKjpt/o=; b=cvuPkFzdSdUXd+zXIxbLgrKmzkNp9W2i/TXzYTnc4FXo7hfmfpD2j9npSu9wWFuQfB pWcPLSZq9lTfxAeiWo9rLZQIQdK8xSj4U0h6v2BUXR9aBGOPl0rLRlg0FSvNXXZJqDY+ tFLO48XsUe7RGqoNGSvrIolopr+1ep/RgyTWCYWnPkixTCUhMCBRUO880IYBBB+Gv/6z iTbYjPWYJUZkJ4d+ETz2hSbcyf0AsOwDCE3ZygYaq7oKGTnyHb0ftW5VQkKT9lxRaqrs xST+AbX49frp4zzYeCZwwtgTh3D4JLRrfC/w9vCh8l5+E38Vpap9ePsugpw00C/5o91W 45nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tC1J6R2ziZNlKPTE9xNNUQxKtbzGe6ytt8wDdKjpt/o=; b=H4M/M1+yWpKlvEclzGIWO5u9GDXu7HMsGHFUgzPMehe/+FR742tn3FWgDbVQ3CogDf XX7RdBxMACuxJVtlFUkq1WH+WBaSn6pbFyX6YHAyrGvBPJX0ZA6F+fVPCfpXXjY4Pebh 4doTfppJNp1immIhxVKgJQdNk6gfJk+J1cNLWSsqK53NT/Ghc6enRQndcBWLWijZzYZZ 038fbpx23Bao55+YQeIeqvG8iXLUF1TLPlzy4gfUwvoqOJYSJBBSRX/Nq/hBQnQU1kLu a2CDINq98Yu/TX61/XqYfM54HHiGS57qApgiIzs/Gm5c4P+3GyF46twZ16EyLz0jHUPx wVEA== X-Gm-Message-State: AOAM533NgkwE4iyIdMudlSF507MkpO1ZHgQ53O9nhsQfY+737jFWvMvU yQmWUG74yVk1DogSEiiVPmEWfNFUIkY= X-Google-Smtp-Source: ABdhPJxQswzCJcO2FXluIwEmQa3EsB9zYDcawKUzLBDZ9Xn+h6Zb7H3qD2o/LVKl9w8EljmhZQ26Bw== X-Received: by 2002:adf:b319:: with SMTP id j25mr7863614wrd.256.1631536930296; Mon, 13 Sep 2021 05:42:10 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:09 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 06/20] trace-cmd library: Write compression header in the trace file Date: Mon, 13 Sep 2021 15:41:49 +0300 Message-Id: <20210913124203.3677760-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 802d35ed..c979a6ed 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1076,6 +1076,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_allocate - allocate new output handler to a trace file * @handle: file descriptor to an empty file, it can be -1 if the handler @@ -1362,6 +1380,10 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) endian4 = convert_endian_4(handler, handler->page_size); if (do_write_check(handler, &endian4, 4)) return -1; + if (handler->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handler)) + return -1; + } if (HAS_SECTIONS(handler)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Mon Sep 13 12:41:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2C57C433FE for ; Mon, 13 Sep 2021 12:42:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C6E6C60FBF for ; Mon, 13 Sep 2021 12:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238985AbhIMMna (ORCPT ); Mon, 13 Sep 2021 08:43:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239137AbhIMMn3 (ORCPT ); Mon, 13 Sep 2021 08:43:29 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3B00C061762 for ; Mon, 13 Sep 2021 05:42:12 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id t18so14524815wrb.0 for ; Mon, 13 Sep 2021 05:42:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HmiwOJgr3gAoYyCOjpOlJRmmLd9p3R2Povp8m2DfuOQ=; b=YqRaYPnyIsrZROvtLAM5PC7+AO9/q9ETA/oqmnd3JiXHZgR750z8vlsINtAN5qPPyG RTU2rWOguZIVTudB8jIiJ+xOTVrskTe5MR1xfGS1IswMfW8gWn3aytYgloYNCpWesamI K2fkoBtBr0BNDSDax86jNJ2qVAd25NJiEUtVNkCseZwvjuM4vc/i32KX7U6dC4GvJn8X /FT6DH3fAE3YT3zA7dQGJOStL0hZtbsUZKiyRNH2n/2xDFkPsat6+w3Gqvoc/pjcK+d/ gFGOX+1LI4Hp/fNwKvbE//Xh3LvRDWL5eU+EemtAOJsad+m4YccUx6daVyz97QhkW9hC sMqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HmiwOJgr3gAoYyCOjpOlJRmmLd9p3R2Povp8m2DfuOQ=; b=j8YA0owmJQYPSENUKkv17JuKs5bxXz9pStuE54sxX4ruyqJK8LHQM0MMg8NvQCh9ye Xy23lC8VCA26HfDS0I8dfTFigus1IweDDj0MmBMGumzCWUNjErHau+Jl8yTVyMXhocCQ rjgJVcFfoBSt7idgBxBH4Wr790Z63RwSOpSyhkEgCDZlVxbEWaQABERulmZ6bovdf2G0 rLGhiKHHP2xG0vByOVeFrELlLekYfZawHbuF5MiND5trwrxi0ESQwlVQb4pyOOtFMcM5 q8n04oOtjyR4IUKCI7zAr2Ww7ZYSpcJNjTcJMvdZQHDXI8EvxfifvougnRE/hfnNOVzo iXRw== X-Gm-Message-State: AOAM531tbKhsQ1zIu4+LnkNUAHgpdyufhUFn+gILzPDDo6c5h5GOoueX QpLi2muxIb9BTFgB8nX7d2aLBV+fRVw= X-Google-Smtp-Source: ABdhPJzflAkNNkr9P4a4Kbmf7o3PCSDvUkJFXifOmjLI+N1eVSn1Si9RBr/pYOLcmMNDMcXFvZfFwA== X-Received: by 2002:a5d:6d42:: with SMTP id k2mr12014786wri.116.1631536931282; Mon, 13 Sep 2021 05:42:11 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:10 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 07/20] trace-cmd library: Compress part of the trace file Date: Mon, 13 Sep 2021 15:41:50 +0300 Message-Id: <20210913124203.3677760-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 81 +++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 14 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index c979a6ed..1c6e9728 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -394,6 +394,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -449,7 +451,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, unsigned return 0; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -469,11 +471,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -487,6 +492,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -539,6 +546,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -546,6 +555,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -774,7 +784,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -788,15 +798,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -822,7 +837,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -840,6 +855,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -860,7 +877,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -875,6 +892,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -882,6 +902,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -928,7 +950,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -946,11 +968,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -976,14 +1001,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1002,10 +1032,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1028,12 +1061,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1414,21 +1450,25 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list) { + bool compress = false; + if (!handler || handler->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handler->file_state < TRACECMD_FILE_INIT && tracecmd_output_write_init(handler)) return -1; - if (read_header_files(handler)) + if (handler->compress) + compress = true; + if (read_header_files(handler, compress)) return -1; - if (read_ftrace_files(handler)) + if (read_ftrace_files(handler, compress)) return -1; - if (read_event_files(handler, list)) + if (read_event_files(handler, list, compress)) return -1; - if (read_proc_kallsyms(handler)) + if (read_proc_kallsyms(handler, compress)) return -1; - if (read_ftrace_printk(handler)) + if (read_ftrace_printk(handler, compress)) return -1; return 0; } @@ -1799,6 +1839,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1808,14 +1849,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Mon Sep 13 12:41:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55F09C4332F for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 41EDE60FDA for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239137AbhIMMnb (ORCPT ); Mon, 13 Sep 2021 08:43:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239946AbhIMMn3 (ORCPT ); Mon, 13 Sep 2021 08:43:29 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E38A8C061764 for ; Mon, 13 Sep 2021 05:42:13 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id g128so2115241wma.5 for ; Mon, 13 Sep 2021 05:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t7LybXcZs8hWo1S+cBmstKmaqsKoBt8ivugmVPGLzTg=; b=ZMa4NcSlH07WCAPrBhsIaENzhdLhJikXM/ZdV5+70pLm63JFOzSOkpr2Ig793lixd6 24zbXko11bPN2KpaCLqr99SHNv1v/wEe7Nx2C1EYzzD27iUer0klKX6JppSuXbUkmib/ Y3a2htlNsvI2fkSLAjHqgVKG+UnhW0s4ztfCWqSOdxVIcL7W7kVTZBThtA0UsxDVE/n+ cRcgYnf9FEpFw6J1/cmImWwbMgliSSxHQWTOeV1hgEZ0lwPbOj3XH/ykt1funP2GeuA6 frOfKi55dE6f73r3PqAoD6fMxCgoDg8F/9pLGfUX7BbVLaY42+Nf++6Qc5fzY1Cc0XMa BQJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t7LybXcZs8hWo1S+cBmstKmaqsKoBt8ivugmVPGLzTg=; b=2qenT27tw6iJcWxUihnQG5AI8SqvxLauidi6tAbaC9Z5fW9OvPZ//ZL4AIpEDCl5fU aW1FMCMIQqE5SwDYC1kzrPoekakSLCE9UIooPJfH1vBB85u1WgRxw+m55suTewvINAF2 x6hpaSjtEcFYzrzuZjhAUfAoyveyZ/xHwjButlTdjkHreb6p/DvmMl8vW+hhwo5r+mUL ApBVn3PC5bel7HVx+AJ5G935xlA6XwvHXgjCOmsFPbGBh0v/0vpNTcC0U1Rdpso5NzMU 1yWElpt0+ZobADldssX3VPqs6rpo6zgC1TB1lG1WR56dZGcjRn6cVYG2tAX2E5ZBpqdw b90g== X-Gm-Message-State: AOAM5302jHI+4/8rXaSPjRpRRsTLM1upysQliLDo+dB3PQCj7tNPMrjq ki/Q+bEI+tcy5ickF5ly9xoTM1GA5xU= X-Google-Smtp-Source: ABdhPJwfYyommN3Whsu5bYwM4GsvKhjkyYD1ZwabQcJ8FQSUNPowrSgk28niqZAEimN/V889X59kmQ== X-Received: by 2002:a1c:7e85:: with SMTP id z127mr11050909wmc.141.1631536932059; Mon, 13 Sep 2021 05:42:12 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:11 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 08/20] trace-cmd library: Add local helper function for data compression Date: Mon, 13 Sep 2021 15:41:51 +0300 Message-Id: <20210913124203.3677760-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 1c6e9728..c69d6d4f 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -283,18 +283,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -312,12 +321,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -513,7 +572,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -539,7 +598,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2177,7 +2236,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; if (data[i].size) { - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Mon Sep 13 12:41:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3DDFC43217 for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8EAFD60FC0 for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239946AbhIMMnb (ORCPT ); Mon, 13 Sep 2021 08:43:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239085AbhIMMna (ORCPT ); Mon, 13 Sep 2021 08:43:30 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62092C061574 for ; Mon, 13 Sep 2021 05:42:14 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id w29so13708748wra.8 for ; Mon, 13 Sep 2021 05:42:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+cl/O5tgZWtqAOva1SZj6PlCWSuAGY002QXYdogqg0g=; b=beRZdmOXaKLnNxjeM9QBnyxytQFU1Ju6+d1SQvTSFiEVY6ZslYfkN0az1sZ/RozB/6 8RilO/+lMO3f8e8pgJgM+fGo/XKPPUUT/4Ch8xq4I0TgUTWQFgZxC8SFr5Dd+jSZSnjw tOx+9kY8zGesyHWoDNlT28F6EamxcwqxSXIn90wHRBd00pABaAYBquZQlgM+jmcNUecm L/O9h178t+uoEEi6mk5xxlhbP3LWWbRXv4YSocsoqWW/t1nEdrP4h/SgCLVAqKvNZS5V PoUWw6dAeEuTA5FlR5Uziw5TyG7nln4UahMLj07T/WM66xwoT0bDvuT+MguiliD43S6t jh4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+cl/O5tgZWtqAOva1SZj6PlCWSuAGY002QXYdogqg0g=; b=a0Dgb4Q7oFjWfgSaoSZqulFObg+LlvJGi2BzzSlUHiuKzZbBY7x0CRrRfcIpj7kZcd nSb1z3kwMjVdhr+GxbVBL4hAqvYdGBmgIluNyZ1E3DHg9svxP2blwHZnJP/dzroZXNGg ON28n1aA2p34wUxnB+qL1ImSLE3+jepUOZs6jq6LNCMb40bY+G8o7egwk0D6bvGWhp6j BXPxlFp5mu4pwzTZbTloZ3Xz49hBJGjWHVf95WskjqYCqvgZSai0h8HpGcNWvPGiYIpB ifCKQUZHzdo8Y/mj+ygJh/WxIggiCfWqo0miviOCCWN8fKKks08vC56Is8JwcGUQNXDP w0jA== X-Gm-Message-State: AOAM532XiIsBxV3GRl7od+BZcBFYps1DZQCpJMka+S7Zyn6STUc96nX8 OCBe4n6pQ8uWRoGIdM0CgAU= X-Google-Smtp-Source: ABdhPJx25mQtluW4SWcIlSui7QYc5NJY8p4eCgaGZEdnDtU+uKXJd7bUnuVkL/8Vs/1kVloZxDDkXQ== X-Received: by 2002:adf:e408:: with SMTP id g8mr12383698wrm.138.1631536933018; Mon, 13 Sep 2021 05:42:13 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:12 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 09/20] trace-cmd library: Compress the trace data Date: Mon, 13 Sep 2021 15:41:52 +0300 Message-Id: <20210913124203.3677760-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index c69d6d4f..89b502b4 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2086,11 +2086,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2191,6 +2193,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2236,14 +2240,14 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; if (data[i].size) { - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Mon Sep 13 12:41:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EED8FC433EF for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DCF7660FC0 for ; Mon, 13 Sep 2021 12:42:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239085AbhIMMnb (ORCPT ); Mon, 13 Sep 2021 08:43:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239116AbhIMMnb (ORCPT ); Mon, 13 Sep 2021 08:43:31 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 544DEC061574 for ; Mon, 13 Sep 2021 05:42:15 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id j17-20020a05600c1c1100b002e754875260so6503099wms.4 for ; Mon, 13 Sep 2021 05:42:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=e697NvpF9tuRuBTrXAkHfX8ZI1uKeZM8RJGq29d0nbI=; b=YPdqO5ffR7RKOM2P9+5Zr/w5iwszy9+84EmYZwdQqfemD51VjpCRHBHEdGF1gy3E/V vhgwZ3I30PBtwgpUXz3OBikPgs5uIrcvRNwwzunrJEMDwZel31C6TF1yUWgN3fjw+EKg Yk/jWl+NdrhKURuklUfp8cjqvF1FgceRShHaMFEIrJgbKEu9IBREZZ/to+sqTD3hEku6 Z12C5e0B+alqBMpcxzouGWIiyOLpVjL6mg6iJKvR7sdg553Z9Qh4TwwO4DZBsDJgpYJ1 Qrr+qYXFg/tXkA/sqCMKYCgW0I1dSHVT4b+Lu7qMwtLg1eYHh8NwoT0/IQ3l9bFcgken O+EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e697NvpF9tuRuBTrXAkHfX8ZI1uKeZM8RJGq29d0nbI=; b=wM64OploBEyI1elYFoKz+P+5mkZUrY0oQTvZwjRB1GKTc/pNAofhstL3Z8tUDtzbQd Ah6XHEDDBdWIj8SQgAksT2Uy4V10SycZTPyMx5K9ubErMlJoS4Yx+VzetiiHq8sBZiMV E2Vb+rDPE/TI1+1hdn2vtjk5FLwspQKWzk/HW/SAPJjYI5f9fHWASu6FsoVsLpAEC4sZ aabrAPw/SVoYd5UxrrOkqpsNORqPPeMs5OPa5oMsncByw68NTPY1KDImVSHqt7Va/jIN IJnMFdnRrYF47e65JzO51yFb0JoGdWJXi9TimUw5O6+23P/rUnGrcP2GnNoYnUXNBguC k3xg== X-Gm-Message-State: AOAM533nPOuHgCvb7etF5+B4FgfU0ACf+8JdSQ7+Tm90lAjgSLQqlKWx s4JCtRH6G4IztSdKEJgvesVB1XZqfX8= X-Google-Smtp-Source: ABdhPJwxSI/zgwBC9Qpj5s9HSUtDpXzINDfCyb8PTZX66iBIJXKouoTvxo4Jit2wQg/29PTRiE+1tQ== X-Received: by 2002:a05:600c:4e86:: with SMTP id f6mr3993938wmq.52.1631536933973; Mon, 13 Sep 2021 05:42:13 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:13 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Mon, 13 Sep 2021 15:41:53 +0300 Message-Id: <20210913124203.3677760-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 41 ++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 2ea8f5a5..320561d0 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3036,6 +3036,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3047,23 +3048,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3113,7 +3123,7 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: @@ -3122,7 +3132,7 @@ static int handle_options(struct tracecmd_input *handle) handle->cpustats_size + size + 1); if (!cpustats) { ret = -ENOMEM; - return ret; + goto out; } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; @@ -3133,7 +3143,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3191,6 +3201,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3203,7 +3215,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Mon Sep 13 12:41:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAB4BC433FE for ; Mon, 13 Sep 2021 12:42:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA71F61004 for ; Mon, 13 Sep 2021 12:42:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239998AbhIMMnd (ORCPT ); Mon, 13 Sep 2021 08:43:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239986AbhIMMnd (ORCPT ); Mon, 13 Sep 2021 08:43:33 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45487C061574 for ; Mon, 13 Sep 2021 05:42:16 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id d207-20020a1c1dd8000000b00307e2d1ec1aso725662wmd.5 for ; Mon, 13 Sep 2021 05:42:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BqdXmJM70z8KjHEjztcSICoax63heaUEMM003tJ1TUg=; b=DEbS0LHe1j4Q2LbFeKQcQcOn8WHjWuJQmhkoXG+E7/+Xfj6CQ5SbIuMH4147Fxg/64 UPH0pIH5tfNQ/tveRqjsowkb1+O2MpTEUlYsVGViRqrcSJQ4ukktatzrUWJBCtnkI1/r u01ZSYo0YSXWY44yZf6GsWnU4hqVgnoG1+cJRw/ute+CGZDN7D4uH/ARH5CgFoV0B3v3 I1wjKgpsrhqIBOUh1WrUc/vRINiaU/A7yDHB6SV0YOs3lLQX2UkYGM65P9IAefzicFIb vAiMOef+ie3bzP57h1CvK+YmLV5+zdgBIWF/sSK6cR/dcod0kzOJTqGo4uqNFbgBLELn 5OuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BqdXmJM70z8KjHEjztcSICoax63heaUEMM003tJ1TUg=; b=wn6p80m2BKf+gmxlDP3sR8qQIY6o30F6lKPI3DzTE9Q2pk1IJA6L8SbD8XLxyeCwwG bCAHqZvKNLOffQTUTEUKSR8wb7HZy9olD5LqXJc6iaFF/mFz8SnH6faTDgpBZ6tY94Zd vvHIf5Kgg3Hlyntd+CAEkPOMjamp7gxZqvX05ybYLoNkyX0McnYgg6D7LNG9rd7fouT5 wreFOP7Up0NwOjawZiwv0d7dm7zAmoB+/ZW7OEUodMDJSfX1O2//NpdO59y7YYxoI+iN 3nAT2Glt7JnNLDP99UGv5xlkpfI8L5rh++tGLkuU1aJaFErV2oWt1hRm2MrJF+G+tUnN 6AsA== X-Gm-Message-State: AOAM531oCLsFSp13o1bHGxiy9Ysnw1Zzf142KWbkAUVBq1rMOMHl+Ohx kO1uSK5LY3iDB30AFwdQv27YKXj1j2k= X-Google-Smtp-Source: ABdhPJy0+Mmwpi6Ds4d1EclGVcxxYmqVzhKaPy4YpxCbIC/fjJgcK9aAfeDl+fARe1k/4jz4rUot+Q== X-Received: by 2002:a05:600c:1c11:: with SMTP id j17mr11419120wms.138.1631536934846; Mon, 13 Sep 2021 05:42:14 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:14 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 11/20] trace-cmd library: Read compression header Date: Mon, 13 Sep 2021 15:41:54 +0300 Message-Id: <20210913124203.3677760-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 320561d0..19a86269 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -189,6 +189,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3795,7 +3796,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3833,9 +3836,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3865,6 +3871,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3877,6 +3903,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4075,7 +4104,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Mon Sep 13 12:41:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C090FC433F5 for ; Mon, 13 Sep 2021 12:42:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AC61960FF2 for ; Mon, 13 Sep 2021 12:42:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239116AbhIMMnd (ORCPT ); Mon, 13 Sep 2021 08:43:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239963AbhIMMnd (ORCPT ); Mon, 13 Sep 2021 08:43:33 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C580C061762 for ; Mon, 13 Sep 2021 05:42:17 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id d207-20020a1c1dd8000000b00307e2d1ec1aso725692wmd.5 for ; Mon, 13 Sep 2021 05:42:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hFneIwwflyvYS50+7Kf5cz199vOtKGcFApls84TqVf4=; b=cyJXTo2w/paUDVsrmRRUG36e80vtRxv/pxKE8C/zJ1wOuX2w3OWMt9lUKpObTkU2+r wkTmclkKPi4ouGAt/dlam18lMeB/lxe5ZanbChziiOUHb58fJvrKVfPYg4n+pDpCMZhT vKPrwsciYz76+ylhalcKZPUZrbAE+3LCl3Wji1Pa5o2dJZos7flPbcwHIZJ3aH6lKnYV bYODKyU8sLQskaK7EkYbL9KMl3XU0k9fXtosXpERiYzn+kTh6PfE0tzutAoC2e6EcmHy WHkOiT9ldQLIYaDmxdmQkVzXASzHN5Dq1Fgn5jAcuMdiwRAfRNc09RzV+3tCvugv3Dtk qLAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hFneIwwflyvYS50+7Kf5cz199vOtKGcFApls84TqVf4=; b=A8ZM70WNhXl9EN8PnTKy+ilwh3hSRvOqUN4D+qXJH8jQYif0q+97nSdpvB0WWyzK+6 xQBztCOHGnfqMxh53bBkPMuf9ZpmDi8V8JNYt/yQ+p+w09qX4Zn3ptc1+NYPz52C3F4L nzl/NyNDHbu7CD67UmBbTgzGt0kbJOlotf2EpScjpfou1hc6McMTz/rgUiifikceIlR9 /dTj2DIo/F8ouL6E2huH9oZdT/PuQhosnkhx7DfYlIgs5UROjHNspq7ZA6zyh8xOQcC5 5B/5Wpk4oUgXnmQSQInxhxDvHgf5jYM04qsCAwQ415yQUYvlAr/uqzKYq/lu+yJC0oaF e2vg== X-Gm-Message-State: AOAM530kPlxW6ab4thF2lU+AFoDY6+mukwmfjor9hd9t9lskPD/9Pa0Q 7OPiW+UNhgTdC22XKejb4aFOaJeTXWA= X-Google-Smtp-Source: ABdhPJx3coxDHRXP6FpIkfEjjnON+sn5A3dCnYayZKVrKloiy4Ll0vIxwkhy86FizVTkUb1PGNlsRQ== X-Received: by 2002:a05:600c:3641:: with SMTP id y1mr10733425wmq.43.1631536935804; Mon, 13 Sep 2021 05:42:15 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:15 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Mon, 13 Sep 2021 15:41:55 +0300 Message-Id: <20210913124203.3677760-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 19a86269..ce74252e 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -150,6 +169,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; struct cpu_data *cpu_data; long long ts_offset; @@ -3300,6 +3321,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4041,6 +4063,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4060,17 +4083,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Mon Sep 13 12:41:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9A5AC43217 for ; Mon, 13 Sep 2021 12:42:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D1DEF60FC0 for ; Mon, 13 Sep 2021 12:42:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239986AbhIMMne (ORCPT ); Mon, 13 Sep 2021 08:43:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239995AbhIMMnd (ORCPT ); Mon, 13 Sep 2021 08:43:33 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A2E7C061574 for ; Mon, 13 Sep 2021 05:42:18 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id n7-20020a05600c3b8700b002f8ca941d89so6504993wms.2 for ; Mon, 13 Sep 2021 05:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8lM7r9FSX9P4bmH1zte4dndLPKHCRYsUoKmK8Uty7CQ=; b=cU8U/xdL+CiqrO2MJT66/l+YkYfj+OKegCl3DvUPykl51L0CJ7PXYdAxESKNQZaDvk SzUKAfP7aVSccUPuNO4yzHrA+xXgGyviuTwp8nieK+ZZ6pLPKgzXMbRK9T1DjchrrCx7 YXzi2+PXRsZgvjcqr0Ia+Itk08mYwuz+j6xw7+E+7unQ2zAXc0XwTxiWIDL+92nL66ls 4Ey0F19U55D/txJ0Ceb6Wccsf4lq2KG87X0SueeuOIhvTc7To+MR3U+CGWRXiylVgYDC WnS9kIzbtVhXL2wrnnM67k7Urx6iF1goG0tSONHxIwntSPIBpPHYy2QRP+YWnD3DnPXZ aHhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8lM7r9FSX9P4bmH1zte4dndLPKHCRYsUoKmK8Uty7CQ=; b=BZkhURKyu7rByK8nqDShgTPCWP++unBd2MEtWcCXAJQKV5fHZS3hOKnA6iNn39NxUF 3VrZWdbyiCJS0bCsEH2P8be3FRU70yT2msweQoaPFJQHC0DIBbs0a2OWMOSJR7NmNnjl i1i2mU92B4035Q0cO25d8IBteDK5cb27a3uXqirM3H5x40P6tq6R3IDltzLo6ys6BNst Sz24B4kWKv8TxpxFghVHGYoG0ews8w37InHNm1MGDK/+hktcq4PRmGiqvAeHVZSbOsmj Y8zIgJz0zd2EG35cOanrsvlfIZgAfZCD5KZBrPwdT/plJ1iZB0Cj6QUNG3x/e7NUkZGi dqEw== X-Gm-Message-State: AOAM533e+Pd7f7cLQa6w2n7HybL1MSinS1ZzBe75tCaF0aANS3zfTZlH 5nFsvFvQOugyuX48fEBPHX+JzLq4qzI= X-Google-Smtp-Source: ABdhPJypuYLEw3/pSW6XbuVQ0KBDvFe/it0uYsWz0BvAxjA7tmXqkFOn9DYBJITr6d//tBpVNXTq/w== X-Received: by 2002:a1c:2c3:: with SMTP id 186mr11008186wmc.43.1631536936702; Mon, 13 Sep 2021 05:42:16 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:16 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Mon, 13 Sep 2021 15:41:56 +0300 Message-Id: <20210913124203.3677760-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index ce74252e..8d1e1c28 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1266,6 +1266,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1305,12 +1306,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2502,16 +2506,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3378,6 +3442,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Mon Sep 13 12:41:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4806FC433EF for ; Mon, 13 Sep 2021 12:42:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 347A160FF2 for ; Mon, 13 Sep 2021 12:42:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240000AbhIMMng (ORCPT ); Mon, 13 Sep 2021 08:43:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240006AbhIMMnf (ORCPT ); Mon, 13 Sep 2021 08:43:35 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0327C061766 for ; Mon, 13 Sep 2021 05:42:18 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 196-20020a1c04cd000000b002fa489ffe1fso6851418wme.4 for ; Mon, 13 Sep 2021 05:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GwTARa2VtynWMrU0jrXWmNwlc0ut56tBqfgsJMIxJ6Q=; b=K9EtPfnFXCEoMvsbihJRVCLCyogIFuhOGeWtnYp8qhWuL9xVpiV7mLZk15+N4Eb3Hd Hxn97i4RCrF43Ruw4QRbUiKr9PW2eUox52KNARbrvFTc3kjL7//4pS/VilHtPrWddW7e XQL4GU93B0r6DB5bFDWpFk4KNz0o3SuObEJDLp7aIagrqiuAx0zxAndKz20AK1fSnSFr oH4flmEZFhn5YBPQad7Y3WOw2C1dT/yBMx387/HwmUFDgV63xj1FKRyThs0x2w+l8aSN +vSibp/TZUa36/2gkRShHhrGF67udK9hxcJbc9WSEdpWk9s5ItHfN8ef9swtAOobcsgG xOEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GwTARa2VtynWMrU0jrXWmNwlc0ut56tBqfgsJMIxJ6Q=; b=0Tu7+jUpKBli3tdM6erBb3pG8N0hRWk+khDv5toZdUDq088TwnjhK+J6cKeLdilAnr IVvZDfwNbcUdx7LFHZ4YUcT/NAiMrbRNQKF/fTy8UDMA6L3fEK4Ofq7ixyZQ1aBb2vi7 s/1FsByUKONcrICv0T5CgyxlCVHasHqJBsZcfa0WOw6O4E/hj91GLsqbjwtiSZnG1i7f yRG6yoevXeDZblF7Jf11zB8exABYTcHZl9Kul0OV8XxkKebO/RSranBZVab3ukV6pi8u dGA0TBILy0S02NhIte6nLiumvEFlkyY1BzIiKXWpr8HdhYtCmXs/2jKBnZ6Zo5XvMoFs NfPw== X-Gm-Message-State: AOAM533YiLpGdKvRUoTLzdjhbo3eH+WYSs1jYtRT+bd8+yKqXihwvmiq kx+On90eU623tpm7zqXICq4PsSGj+0Y= X-Google-Smtp-Source: ABdhPJxmgVoKD1FhDOv3oEMDqegwkDvNNrhnxzdZi/dFkfd6f7/Al24AspNs5z9FANTrj3FnW11sDw== X-Received: by 2002:a1c:f00a:: with SMTP id a10mr10685882wmb.112.1631536937514; Mon, 13 Sep 2021 05:42:17 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:17 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 14/20] trace-cmd library: Add logic for in-memory decompression Date: Mon, 13 Sep 2021 15:41:57 +0300 Message-Id: <20210913124203.3677760-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 8d1e1c28..fab8b9ba 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1257,6 +1260,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1268,6 +1370,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1410,6 +1515,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3900,6 +4007,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Mon Sep 13 12:41:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A059EC433FE for ; Mon, 13 Sep 2021 12:42:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 894D960FF2 for ; Mon, 13 Sep 2021 12:42:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240001AbhIMMng (ORCPT ); Mon, 13 Sep 2021 08:43:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239995AbhIMMnf (ORCPT ); Mon, 13 Sep 2021 08:43:35 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D406BC061574 for ; Mon, 13 Sep 2021 05:42:19 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 196-20020a1c04cd000000b002fa489ffe1fso6851449wme.4 for ; Mon, 13 Sep 2021 05:42:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bp2hT6LArxwN/6m8wavIayM35MrPWkhPiC4Vw46dQks=; b=AjhHg/snZ7iDqq2oGx8O5vcK2M+Lv+21SMy48A2h1cGgaAMzgUx2ix1lkY5JJ8k0eh v03XaapQIR1uZZqZGnjlTCd9PfjhjRb9L//rxab8dLWfPcXFfuLztISUlthqaMQvTszA GO5XFAAFGwL/Xb+f+zaT4TFE9d8irOha9HXQ7PmZkjE4CVE/nscoDUvEVA4scBcHZEYr haFwY8FFVOuvE/GvW9dPfQ7C+3HgjSAPQhu3IDlaMzxvdRIQHOm4EI18B4HCRnNgkiTr crQ/i8Bp6nRGaRebnD37hJNPx550he0F34OvhTaGkF30U2qmcubMWy3FaGycOnc8PZPi /OAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bp2hT6LArxwN/6m8wavIayM35MrPWkhPiC4Vw46dQks=; b=NNfhCAbytrfpgqTDVJrRUYc5jLw5jyOiFxRfJBqsZ6zVLex4UMQzqm3khLbTLKrXFx iex9TH4+T6zGVvHGD4gSrvjLV4X76m0ZQOY/sJnoGSy3KcZVK5eVBOCRyebG1PjhypRJ aihcqqzpdcvIvswu/wqTeyPQSOGp3WUJLBFCy9l/wkvSOGlAT59lJXvPmfcksEdEsXWD 7zbBTbi3wVVU6MqpTu6R02Yp03Z0OR5YuSuKmaGg3Ff8l0F0pNW6rKmY1mqDuxCbVq+c X5vkjPlfG0hP3el1scqe0gMHEt3vdZx0P30R3msZettUyVM+Jcs1aIY1+SFpVGYKuzHD nZ6g== X-Gm-Message-State: AOAM530ylyZdeWBErvGoCsxMqBY4/+p1VK7SAU+zJ2F1StzBOLO+WGJd OLQ32Ki7fyJ+l6ZEN8p7JXIjZ0Hcluo= X-Google-Smtp-Source: ABdhPJx3YUtx6UhhLzSKGmKqA9sZ+gMcfNRAWy2767ahCxYd/SkDk4j/xnxEiLbtk5EslvC+zBhhJg== X-Received: by 2002:a7b:cd93:: with SMTP id y19mr1602381wmj.110.1631536938467; Mon, 13 Sep 2021 05:42:18 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:18 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 15/20] trace-cmd library: Read compressed latency data Date: Mon, 13 Sep 2021 15:41:58 +0300 Message-Id: <20210913124203.3677760-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index fab8b9ba..c901be0e 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -175,6 +175,7 @@ struct tracecmd_input { bool read_zpage; /* uncompress pages in memory, do not use tmp files */ bool cpu_compressed; int file_version; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3450,20 +3451,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3531,7 +3564,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -4004,6 +4057,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4293,7 +4347,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->trace_clock); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Mon Sep 13 12:41:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EC38C4332F for ; Mon, 13 Sep 2021 12:42:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 10F6160FC0 for ; Mon, 13 Sep 2021 12:42:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239995AbhIMMng (ORCPT ); Mon, 13 Sep 2021 08:43:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239992AbhIMMng (ORCPT ); Mon, 13 Sep 2021 08:43:36 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90CB6C061760 for ; Mon, 13 Sep 2021 05:42:20 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id d21so6780729wra.12 for ; Mon, 13 Sep 2021 05:42:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WSh20c6YzUYMR96Re8GlZB2NA1Vr5MUK5hyf06tEWoQ=; b=ZkaKXAX3eySQPletyY8SKHyfahtk9c05TWU9CQlPlMSi9fLwjJErTK5pGy++48vLOl mhPaVEYkgywPxwXJFTIASUvosQkNcCmRGDbJtBQ7gcII8HxKtinoa7qBrt72Ud5QsJeD jsHKacWBzYLVeX4tT3lNwfG/Kk7OVZGDztEUIhMpgLW2NWTbWAvnTq9h8CPEZwNYbvq/ RCBl1yHs6hFtdNpS4wGt2KVenTKjxIOstaaiBE8XkSx6WJVcBuQ0dUGv7iXey6KBRsEG 10r1s+VW0/TdHlPUaFkaQ6UK+uBrhjqB6YeCBorsOX+9A72Ab9/4Nj9CAlNGtc7rvihA 9+Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WSh20c6YzUYMR96Re8GlZB2NA1Vr5MUK5hyf06tEWoQ=; b=0GLvfG7z7itj0kHxPC1EqHUljq2pxL49Ps1e/G9j6MPUsPIA9h2dynjjT32TqA4VZF Ki+MmwgmgKp4q/2XPaX5ArlgrLWFzRJlBhdIfq4LXKrEjTu8W743b9QNS168QQXEKKeM ZPKVYkOYg3KMyLPke1xqhyBzrPLAlmKUr18kcK/0Ke6jDUlr21XFxXycnq1znPrWYk5u UJuQfJyKi3IXkMxsNv7MlMW81H7N8ldh7Bw6eJP4hCixS9v+PUyJW3avviyK9hKq1V6r OGMMMILWV3yF+rNT7XfKlW6Kzql4bpNZMsDdxpwtcc2BhGGGPLVTdWPvSSL/Yquv+1HC 2rYQ== X-Gm-Message-State: AOAM532rUAS4/P9dsqTXc5pTW7z12zBuj+xr7n8KpcJcmBUgv6eYA9QQ mmHdzMtqnTedHA9tN2nbSe/XhW+5pe8= X-Google-Smtp-Source: ABdhPJyw1wb7H1UxyL4LjxkvKWdJtGo8zBvBAGFLJuP32/MIOTCUpajfrHX1Ev21yWmHeG0dS4q0dg== X-Received: by 2002:adf:d209:: with SMTP id j9mr11490090wrh.97.1631536939238; Mon, 13 Sep 2021 05:42:19 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:18 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 16/20] trace-cmd library: Decompress file sections on reading Date: Mon, 13 Sep 2021 15:41:59 +0300 Message-Id: <20210913124203.3677760-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index c901be0e..6ed5edf9 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -480,12 +480,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1116,6 +1119,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1141,6 +1146,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } From patchwork Mon Sep 13 12:42:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1FEDC433F5 for ; Mon, 13 Sep 2021 12:42:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9A11B60FF2 for ; Mon, 13 Sep 2021 12:42:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240006AbhIMMnh (ORCPT ); Mon, 13 Sep 2021 08:43:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239992AbhIMMnh (ORCPT ); Mon, 13 Sep 2021 08:43:37 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 765C0C061574 for ; Mon, 13 Sep 2021 05:42:21 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id b6so14454144wrh.10 for ; Mon, 13 Sep 2021 05:42:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CTrf7INUA20Bk1rTfPxVSMlhRbenOJ5lSnPqezUKuCY=; b=hJCi9K74QlOa/MgsuxslNO5Dxt2gpN99o/cPPGfYprS2N3a/syaQHAi96NO9hetPNo K0FxLbf4nqxcpiau/e2JA30RMkh+4epwWHwW52buNRZf/z3rpO01AM3VGLSzJgbiscy+ NHN+HRErpNpMMb5hPg99yB4Aa+v1gEJXIW7gXm8/Ju8n9r7VOF8FUW/Bx1B/1ihqhQlO 6ayouIxmXgA+sx2gfxzsajd1YsT8t2DPWw7kivXum1S6glH/iMpUDv9FmtJh7mdTRk4C KpOBtWlT2sUDJtZQTqpwSfWZsgun6IotuGTjTxPDrCz/TSBfCdJjHsMrFYBgJ+v2P7TU Y+mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CTrf7INUA20Bk1rTfPxVSMlhRbenOJ5lSnPqezUKuCY=; b=jRlnb0rXQQ4AQP0uKdZ2V76IlAR+hszyZkY3mDf76//7Ljfe7ZX0YyhWpchMtA3sya iDUlYyxJofR5WidP5MGVqw/Rz7MpWFErgMNIILuv+fb9ONwiUiysEqvbhEqR+BJ6ciVa PKYtZv6hJZODLzPizv/pS7bA1VRR51dIMBzd5Ox8yZ5k8SULmRUQEa3F7elpd5INhRmp +7x7s2vFn8gIqtvwGOtzs7I7l2drebp3pyAM/3XvY2PIXaIw58FnVUCKnFMIzsu6goFi HwVSSG1YOH9OojrajjbWLbnvvWY/wnUANyv3KpeMVn1K581H1YZNhObS5zxWTUR0jkgt AQUg== X-Gm-Message-State: AOAM531sClfr3jqbgtYDAfVs75aPePgjl1KvzBNgG7rc3/LoqCgglTTN giK6sJ5ImNdJtgUZVFbGRt4XAJ+l2hg= X-Google-Smtp-Source: ABdhPJxapoRg2kcIxrs1AZ/grb5s7e1JceIJpKxeGV7ZpuoPa2XVY5DFG5Bwe8cf1E8XbWTx7QEZ9g== X-Received: by 2002:a5d:560e:: with SMTP id l14mr11978316wrv.205.1631536940097; Mon, 13 Sep 2021 05:42:20 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:19 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 17/20] trace-cmd library: Add zlib compression algorithm Date: Mon, 13 Sep 2021 15:42:00 +0300 Message-Id: <20210913124203.3677760-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index b198b8dc..c1f37cdf 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -356,6 +356,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Mon Sep 13 12:42:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FE2CC433EF for ; Mon, 13 Sep 2021 12:42:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1886560FC0 for ; Mon, 13 Sep 2021 12:42:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240009AbhIMMnj (ORCPT ); Mon, 13 Sep 2021 08:43:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239992AbhIMMni (ORCPT ); Mon, 13 Sep 2021 08:43:38 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D824C061574 for ; Mon, 13 Sep 2021 05:42:22 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id m9so14501516wrb.1 for ; Mon, 13 Sep 2021 05:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=nGq71YQ2Gpgv88+qbEVM6uj7kxpDaV3N7NTiuied0/zwe7N4nljtFC6MSoT/XpQe1M zIUo2HtVsGklEW313HJ5nyCxryPj3AtdAXsp7yIn+a+s3JIDPXN9b8W1rBvnOGMAegqa JsBGKll6mawjAM34A9cNCYytltmfpJ348yLkdWo3FaL5b98zVEUPG0XoGdb3V+eGbEbB ulORWPi48bujDYAGbDIxCTpXZDmw1aBlA+P2lDTPfOzCUkWmf5iEuFCyrzCzLFeqNZlk 30WLMkMVdBUixssjHxtq+BzZv8Bsp4nBt685zDMfa52cJo+q8NTDyrWWfpT/CEibbCwN ACRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=XUo3WaI+ZXYuxbGCEsDnnsDYgvieKZ5mW4x2l8HKBAy5Mma8VloJtNL6HQtQd6WAnc 94W4RD9CvdNnFsaLK4eIwpQ6VgOywv9JvUSFOvKATbY04Ps0VodYuOrXUIuVUu8oLd7j XZano71u8OAWP0wI2OubmaZXhsQHCJWrxXwZz1hFqt5uGb7OhUV6DDFhaDRPZIoVEbzZ DV/R1gV4VAH1bDyFeXCr1OhOAkYcGWI9dpQZUNnnAdNAdcrx4V02xghv11KHilp6gZ+f 3bWKo4lvgjJHfKrxGmy9TYdGDe6i/l5dYVh6PSK0eCBVqgPg16+n55ti1EhY5SpCIQNF aoMA== X-Gm-Message-State: AOAM532xthhaQLhCSlCaMxnQxr7RA5LacpMKv2PaGl1To/6nuS8D9T4z iaiYs8OswBO3GTPMAkYVDccnae0axNA= X-Google-Smtp-Source: ABdhPJyT/sgR4If5u0Ebhx59TyhgIGIXxTkVuIE9Gv4D21mYa4tCGLk3Ds0lFJu49ZsqRQyjsPZbrg== X-Received: by 2002:adf:80eb:: with SMTP id 98mr12353567wrl.370.1631536940996; Mon, 13 Sep 2021 05:42:20 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:20 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 18/20] trace-cmd list: Show supported compression algorithms Date: Mon, 13 Sep 2021 15:42:01 +0300 Message-Id: <20210913124203.3677760-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Mon Sep 13 12:42:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB29C433F5 for ; Mon, 13 Sep 2021 12:42:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7BF660FF2 for ; Mon, 13 Sep 2021 12:42:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240007AbhIMMnj (ORCPT ); Mon, 13 Sep 2021 08:43:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240010AbhIMMnj (ORCPT ); Mon, 13 Sep 2021 08:43:39 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4565DC061574 for ; Mon, 13 Sep 2021 05:42:23 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id d6so14455700wrc.11 for ; Mon, 13 Sep 2021 05:42:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yao3HSHynO3fw2qebc1cDpu949X61cymkG+rYeq5V9A=; b=o1bm22BQ8qraueUTGC8UdhCt2XxbzQyxTdyN4WLzw223HcRVJOMlNcFor4oeD0oGrh dnZmrVRkBFanLlbQuaNRcFCBwihx/lAE4enrqxdLMUKOsNAYLEkq4aq8jXi4XnpS/tNQ UysYXw4j02JBNVg2c7EfG3sFkEbgunzK3XzAn3gUPGShwWHmM31XthB2afK1OHCutawt zhsiiezWoFLjqUdN1iz18XYl+Y2K9slDXC+0RjMrbHtgZSa55SJAPhVIDJfH35aYivm4 PQyofYxGmATWaUOUrfsdh43IXRxcyt+Cx44Hi6S4DDVxQz01TXCMLhoyeOdN06P5jSR9 TLfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yao3HSHynO3fw2qebc1cDpu949X61cymkG+rYeq5V9A=; b=FMCBHyfLwARY0e0jsTWiMHgppWNHHaRDFGzMFBwCeWnRqxQb/50o7hWqYYxi1s4SqL qidVcESNzSlP1crvtB8JDsbkvBqNwIHh9Si3kAaRTUgDSVg//Bvb8LqL4zJva716rVLd FlJCGarLT/GKwT6fVGsNa1pJPL99pmark7DiQIGVdURUGeiOl0RzX69bMsdqlK1rDbJv pY9J+hQpBlZcx+JiPPyigC0waI/3qYNtYd+La6vxjp9MDM1wLTz3RgKJXgavYNrRSQmk zLdw40HrcbMJ/YSn92yf01NlqTTR4mnubLREewfMe3mBJaU6rYV2OYQBmRc/zQU1CyFC J6jw== X-Gm-Message-State: AOAM53157mKOmNFkusMa5OV3hSSaT1NWfrp6c/2sE51aub/1+V/0TzeJ jbnwT/wDj613LCfYycyXY/kQYdVk5Tg= X-Google-Smtp-Source: ABdhPJyCREIsigpQ04CMTb+BWu09qjCc6gR0YFflL1ZD48M3ntRigh7DHG5jYM91VKaDYuv+F/1y1g== X-Received: by 2002:adf:82cf:: with SMTP id 73mr12757962wrc.366.1631536941873; Mon, 13 Sep 2021 05:42:21 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:21 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 19/20] trace-cmd record: Add compression to the trace context Date: Mon, 13 Sep 2021 15:42:02 +0300 Message-Id: <20210913124203.3677760-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index fb6de81b..72592c60 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4481,6 +4494,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; return out; @@ -4516,7 +4535,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Mon Sep 13 12:42:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12488847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 829FAC433FE for ; Mon, 13 Sep 2021 12:42:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F1EF60FF2 for ; Mon, 13 Sep 2021 12:42:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240008AbhIMMnk (ORCPT ); Mon, 13 Sep 2021 08:43:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239992AbhIMMnk (ORCPT ); Mon, 13 Sep 2021 08:43:40 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D46DC061574 for ; Mon, 13 Sep 2021 05:42:24 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id q11so14461576wrr.9 for ; Mon, 13 Sep 2021 05:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r6lgmGCkcNCWvC0yzxh7nAD0rs8LGbIWc6U6pt0m1uQ=; b=ZCGLHpsvrtpI2VCNABwY386iFJOmbObTlzVWnfH3kudJal7jVq9z4FkztEcINYrviF ythDJAZC4gSSImNefSc4MvQtmta4DgM+JOCIzh/lKf2hJJ95p3f8AGrAIBi6o9n2Tvsq s9kEByoEQdcVC1IzIejq2BpiiNVCa0zbqEDK0G4usTzcBeb6ZDpa6vXebW4Vj63AyCsU YDKCdBJcKbJFyVvRZY54IaleLGtItPkf4e2kSWEu02gNf6Jua4D/Wre0DXTxYPyRKvxx vzgmgGiyGkXgGWhAhhNfW6TPVdqAAk1q/V+Aa4apEdY7kqPOi/sN48HJhg3+VeKpbKyl s7gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r6lgmGCkcNCWvC0yzxh7nAD0rs8LGbIWc6U6pt0m1uQ=; b=26kOqvvNDz4wo2o8PXIb8HTRlayiVnrPQVgilOCjxFCcadg6zv9nshl00HitVg4Rzu DqogkvzoxQaPyWA3ZKG6ijh6qlJM01frFH/vtFo3VqFbpWVZ5sPepMvvzlo7d6m81H85 TUu8ygtPwJuk37b3u3adQcs5mMMdPfuftYzUx/r5ro/m3KIiOVYc59Np6DOQVzzsLZTl UhhlUUSZ5ZVAUPFaYBPKQYXzZ/hnBT23uTjOLPPoPsOe+K/v+TUOBKWUxNJvtvhvpFKp P+ClrZB96br8Lw+XQ3QsZc1BNbgg9k1H8R/niBkPZcWK8w9PlT8yiF3quVuuMdb4gebh uS3g== X-Gm-Message-State: AOAM530OK6HG1oDTWc7QMKGxeBj5WALJpkSdzEZfa1XB/oDF1H93EUT2 hi191zzxnXNMDIVVeOYD4FfyGBYlncM= X-Google-Smtp-Source: ABdhPJxoGKb0sMHRVD0dqC/Ci009jdAT4i30PA9o2trtv+VjfFrfLhT7IAXn5A5ub5mUOVanyv4GUw== X-Received: by 2002:a5d:43c8:: with SMTP id v8mr12574336wrr.68.1631536942810; Mon, 13 Sep 2021 05:42:22 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id u13sm7244290wrt.41.2021.09.13.05.42.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 05:42:22 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH 20/20] trace-cmd report: Add new parameter for trace file compression Date: Mon, 13 Sep 2021 15:42:03 +0300 Message-Id: <20210913124203.3677760-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210913124203.3677760-1-tz.stoyanov@gmail.com> References: <20210913124203.3677760-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 72592c60..dceed783 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5810,6 +5810,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6250,6 +6251,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6676,6 +6678,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version"); cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version"); diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",