From patchwork Fri Dec 10 10:59:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB61EC433EF for ; Fri, 10 Dec 2021 10:59:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237990AbhLJLDS (ORCPT ); Fri, 10 Dec 2021 06:03:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35550 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232315AbhLJLDR (ORCPT ); Fri, 10 Dec 2021 06:03:17 -0500 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 900A8C061746 for ; Fri, 10 Dec 2021 02:59:42 -0800 (PST) Received: by mail-ed1-x52a.google.com with SMTP id g14so27966240edb.8 for ; Fri, 10 Dec 2021 02:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NK3Cjgqb9YIQ7ImxFkIK4Lz1EGgU5pPXsobVAm8qm9M=; b=hseLwruQ3iyJHv4wRFYMtV4L3rC4Dq7i7DW3Mb8ioDnpsE/QBggwE6qXgjnDERyg3w rAyvMzdTx9kldPEEE0YFI6vU+Zph8XhtSoFGbGTAvcu2xf15YN7gX95M7t53iBJORUDd 2ySaODZPv7nKzU8n1UULg0genX6ZXAbu4qmnm/pS6v/j01uCTAxhzWfwckhGYeMeb6Hx PqWMHKMXNb5TrH2oV5z7wGBScm6XAP79Exuqnnpw+GmyHxqTUNz2UuxfOruzp7KAIPOQ SvP25WUcEaB4m8jOCxBnpl8I3oYTOfe3l1v0ws6HHjxQhP2IFy1Tq0+4dR91ItDSy1I6 mQmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NK3Cjgqb9YIQ7ImxFkIK4Lz1EGgU5pPXsobVAm8qm9M=; b=SqoYrHMOg1HQZUc+iqDmAf2lEEplhYWEyHFArHvmUZC70RNVwoplU+mh5PZ9m01F8L 3sIYWm+SWXtPwccg/SYK49MZbTiFGAO5al7AL+d3Vrjp2iHi4yX9ww9He+rqjYUzVGLq rkDAZ+uV4W+d5+yegZLb5aTlAOdpeiZAtgFXalQXE9DPy78ciwiOxDYgUeuBoGvL7BG5 98akvU/IE3v3bgK67BoO+gAbwywRu/XdDt8oc3fmfme9CQkhAa6Clk6o+xl7CU5cCIO6 K67NWZd5UhFCZbHrtv413O4WfwBchCd/W8catsWEo7oNcSEUiQZ//mVfXRL92tnjQKek ENHw== X-Gm-Message-State: AOAM5338BxBQGr5NUybpOd9bY3fLsp4vAXo3n7n2GRjpUtNL4ZUAIL2D g77yT67VL8P2+mOH3HuVfDYY3ciroO8= X-Google-Smtp-Source: ABdhPJzCEnd6u6yFyhPjLO/ZTyV5JB8Lec374bBGkEoY8m6sVISak8xYOMBPiTXCseFGeYjLN+N+JA== X-Received: by 2002:a05:6402:1a58:: with SMTP id bf24mr37620151edb.16.1639133980649; Fri, 10 Dec 2021 02:59:40 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:40 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 01/20] trace-cmd library: Add support for compression algorithms Date: Fri, 10 Dec 2021 12:59:19 +0200 Message-Id: <20211210105938.98250-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 910 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 965 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 6fa3ca3a..139c0a78 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -35,6 +35,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -155,6 +156,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -487,6 +489,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b4f3d8c8..d4047429 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -36,6 +36,9 @@ struct data_file_write { unsigned long long file_data_offset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..883cf669 --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,910 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handle + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handle + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handle + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handle + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int size; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + size = s_uncompressed > s_compressed ? s_uncompressed : s_compressed; + handle->buffer = malloc(size); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + s_uncompressed = size; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handle + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handle + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handle, used to encode the data + * @msg_handle: message handle, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handle. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handle + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handle + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handle + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index 21f1b065..06071f6c 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -635,6 +635,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Fri Dec 10 10:59:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E8F3C433FE for ; Fri, 10 Dec 2021 10:59:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232315AbhLJLDS (ORCPT ); Fri, 10 Dec 2021 06:03:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDR (ORCPT ); Fri, 10 Dec 2021 06:03:17 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9888C0617A1 for ; Fri, 10 Dec 2021 02:59:42 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id v1so29002881edx.2 for ; Fri, 10 Dec 2021 02:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nf8qIehI0YZ6GHzWhu+FaDOt0HIZ/zp1dMbXGKZ9l1U=; b=ppWpOveK8xZBtorGax1YeNIYbP7Mq5oFO/6hzi4NWX3FvV+8EwpGi4kNequG2cshxi WJwgvyOB2pBVwP0G6DfATCRxPO5PNvLTCu4DKd7cQUnCz7SdTaTwTP/LtO7+rboLSbBb jyEcvCQ6fph05LdXLRLXxHC5JxcgXg0V6UOibNXuXH1JmyGSgN0l/vzyvSL7ZBGqiJWL xyJZz7Xu0acKboTbteGDbx2c9bCT9l/D5FcWFHfoI8lO4N+/BAHjeSBeglv/a+1u8cnw nYdomP3DR9d00nk1/IVNFVgvZW2MUR1aMQ514dqRdNCHvKG/3zJn8MD9RHP071S3Gm3B Ns0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nf8qIehI0YZ6GHzWhu+FaDOt0HIZ/zp1dMbXGKZ9l1U=; b=uiYNzpuXX04XQIcnITr0ESqnzppUMoc49XhjvUNnit4BOOIpdDkyWNwWevqD0LhIsT C70CkYlr2o95+ji+MB6pv19XYWzBk5D1wAxEIesCX5CTRWw5uu4acRuvpFO8Y8sxzumG ODqJ9xi/ZEXggPGOBEUzoSlUWiePTEZQwv7yXW5FjNfSwL1026dm2vI98j/AdgW6VHYM g5Z4sJE0Ia1AwJrs8TcgvQ/4mXBxAj6RZsCP/4nvf+Jzd0cUEMFS0X4pTZ/2v0doOSSm ILkYsGftuY1zpibU+14epO1B+Dka0ltC+jN9h3mIGu4H2kT4PP6dqBBBSAXhOk6zgrP4 oYbg== X-Gm-Message-State: AOAM530zY/hv03wwGfHoeBeM/L/pu84zCJgGTm+J6CiaHeBx61/ooZlI BIPxTgXTSJIMwxzl3cdTnj1ErBlelqs= X-Google-Smtp-Source: ABdhPJw+UGJZfq98FJdx5Xeg48e0HlOlw9leuYCnsqzgpYGtUpVJ+Nn+WPrpfcI/ouuD1rZGOla/5w== X-Received: by 2002:a05:6402:280b:: with SMTP id h11mr38184182ede.341.1639133981522; Fri, 10 Dec 2021 02:59:41 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:41 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 02/20] trace-cmd library: Internal helpers for compressing data Date: Fri, 10 Dec 2021 12:59:20 +0200 Message-Id: <20211210105938.98250-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index d4047429..b848514e 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -42,6 +42,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 38a0699b..6dda61f1 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -66,6 +66,8 @@ struct tracecmd_output { unsigned long strings_offs; tsize_t options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -95,18 +97,27 @@ static int save_string_section(struct tracecmd_output *handle); static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -134,6 +145,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + static long add_string(struct tracecmd_output *handle, const char *string) { int size = strlen(string) + 1; @@ -1638,7 +1686,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Fri Dec 10 10:59:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 005E2C433F5 for ; Fri, 10 Dec 2021 10:59:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238259AbhLJLDT (ORCPT ); Fri, 10 Dec 2021 06:03:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDS (ORCPT ); Fri, 10 Dec 2021 06:03:18 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1F2FC061746 for ; Fri, 10 Dec 2021 02:59:43 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id x15so29282835edv.1 for ; Fri, 10 Dec 2021 02:59:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8zafahj+BIQ+EoEjSAS5NYP2Tv9+YwHaM4X6PQNieU8=; b=D4Xj6Kg+OtEh7RG9PUtCtOj9bOLifVZeUgFD2KTyKDsPpN3msLd243S31ME8q59olF smpRt3fFDenY204IkdXid1ZjRrcnU5zo4oQSfl6g36lK+j0o3igxZv9gjtgOD1S0U81i C1+8ENE4nvrIjugZ1CMmCGbQDzdmCS9e/X5oXyOWx1BWv6hXV4yO8qBg2AfZykj4qo8H 7syOpi7vQFSVrfNz0meDLU9rD1XAYo1m16kWR6X2wMXE5REQ/AOzZ8EZD3AnwWxAEaMH Y6y46AdZJ0SfSFFPwnVEJh67qFRU/AgGVXgZQMry2JZKm8BGroGqDBy5ZdHg0a99+vuH cSFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8zafahj+BIQ+EoEjSAS5NYP2Tv9+YwHaM4X6PQNieU8=; b=asKXxLcSJkG/EQe1dKQDqb0VRuFTA27z79nSJZIMAfplXdRFuoY38eFaUf6KBE1Wh4 GAIovzbP710/7kU3+x9JN55qPBSZsaLHo34PU61X0kZboR2B30yJUvzQ51WIlmBUcGtl oj0aw84RfGJF8zRuNnxLXn8dz8eAW7mRI73it9D8MmSkyOVRo4tR4qiO7MagDLxd8PGq Y/pZJem91VVPxm94k4yJb33j7FYeu0vPoe5mlLyKiiae/igfwM7/u6zsTR+pYdfKAX7a qG+29HbVPzOKF3jsmk9UB+LLt7ip0xEuZ0vwejFhgNyyB3JG+aesRRO2nZDgD6AZSVQ2 MK2g== X-Gm-Message-State: AOAM533WmlvrT60hpgQ2cploAa79uI5GOdJtmaQomyUUCCNObEESACfD pklApACGE6VMnLIcVSaoWopPLOi27NY= X-Google-Smtp-Source: ABdhPJyHWeakeGEP/zgYdwNS/k7eAC6FwGCfE0zs/QLe+yx7/gxPMcRb/yK2p29au5Fbk8EnmqNAfg== X-Received: by 2002:a05:6402:2792:: with SMTP id b18mr37329675ede.329.1639133982500; Fri, 10 Dec 2021 02:59:42 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:42 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Fri, 10 Dec 2021 12:59:21 +0200 Message-Id: <20211210105938.98250-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b848514e..ca682ed9 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -49,6 +49,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, int flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index bc4e2885..eb1359b8 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -160,6 +160,9 @@ struct tracecmd_input { unsigned int strings_size; /* size of the metadata strings */ char *strings; /* metadata strings */ + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; char * cpustats; @@ -266,13 +269,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -284,6 +287,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -308,9 +327,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -336,7 +353,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -400,6 +417,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Fri Dec 10 10:59:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1FC6C4332F for ; Fri, 10 Dec 2021 10:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240090AbhLJLDU (ORCPT ); Fri, 10 Dec 2021 06:03:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDT (ORCPT ); Fri, 10 Dec 2021 06:03:19 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0DF5C061746 for ; Fri, 10 Dec 2021 02:59:44 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id y13so28281260edd.13 for ; Fri, 10 Dec 2021 02:59:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PCHlHvkzHaJ4SiSlcY0lkejT6x1X8SjCTCr8Wg+3s5Q=; b=YKv6zmPbYDUYNlr/Hk2ww966NKF3NdEenhf9Ri8xCYZa8dgXME4cMbsM8J8yeXI/HZ YKo0vIxwQdoKfkLliiU5ztZDHJV0AKveTUYODFev2okAaWDgc63akZOeDolZEbASegQh 3xvZTnIb9JYuXaVMQcqHezL/JdsGyvT/JUSL/11EVN62tgZoFgXk1TbZgn7mxNgzzFys 2nl8y8g59zGiruDYl1MW2tm+s89C/yGOe/ttdBiW+pXfFvIgDgGeDIQJu7YLx0GNEo+g OPNtXI7AVfB/0j5OEVr+zsLSlWXvYx9WJ8xMZPDnEsLB5GqQFCeZ023ASHu9zE5QcbTv COag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PCHlHvkzHaJ4SiSlcY0lkejT6x1X8SjCTCr8Wg+3s5Q=; b=i9OXNtZBkg79nPecY3Jxq7e5jLVO+yuQC0bYtSSE96JkUBSUY32X6AWMY6rHxo+Spj V7nvlI3a9SAcbOundnZoF2EB5M1Uw43NpaQ/T9crAFbAqf33hmsiwAJb8tqXdpAocY3Q m7gmXfT2mn9EuoCetpFP49dJ76epKUBgerKoLdB11GgNaYcMRiw53n/ZyOB4/TkzrWI0 AkIMzgdg9hUbOeZXchJ2OBCruU1Oz2vefXTKh63SBwt/nZEusSwNHnm2w5SsDEF5k9L5 I9YO1NHl55/ftPSqnP6DtzMtRZvuty6wWfXMCgqOEdxn4wAtJF+UNNJ4NfF1irUW7g9I 5KDA== X-Gm-Message-State: AOAM533dF2vopCSfvJCnK/DfChpwVTh8Vm+oEgUu1+cFNlZeUaIpfmB8 wpTX9Dvu+OGwJERqGbzWirEcWW855Xw= X-Google-Smtp-Source: ABdhPJy9BE4iqkJTdY9fVbiZHzdP9Zqa1MevMfvpzgmvyDOwkKTpa/wegziPJoCg4Adp8sIliUw9xw== X-Received: by 2002:a05:6402:514d:: with SMTP id n13mr37779268edd.380.1639133983318; Fri, 10 Dec 2021 02:59:43 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:42 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Fri, 10 Dec 2021 12:59:22 +0200 Message-Id: <20211210105938.98250-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 +++++++++++++++ lib/trace-cmd/trace-output.c | 20 +++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 139c0a78..924efeeb 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -256,6 +256,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index eb1359b8..e35e3c74 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4649,6 +4649,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 6dda61f1..a4d29d8e 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1294,6 +1294,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kal */ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handle || !ihandle || handle->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1305,6 +1308,15 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracec handle->file_version = tracecmd_get_in_file_version(ihandle); handle->big_endian = tep_is_file_bigendian(handle->pevent); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + return -1; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->file_version = FILE_VERSION_COMPRESSION; + } + return 0; } @@ -2254,6 +2266,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2294,6 +2308,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Fri Dec 10 10:59:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8527EC43217 for ; Fri, 10 Dec 2021 10:59:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240099AbhLJLDU (ORCPT ); Fri, 10 Dec 2021 06:03:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDU (ORCPT ); Fri, 10 Dec 2021 06:03:20 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96D69C061746 for ; Fri, 10 Dec 2021 02:59:45 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id z5so29196398edd.3 for ; Fri, 10 Dec 2021 02:59:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tEv+d2Yp4nwWrdZaUchV59IXo+3StH/VDh4Teg9zL+Y=; b=Ck26dCnAw6M40BU1oHnnghBtHh2wMyTUtfeZdPBR1iGkVCzlmFVoL0eoIR5AujTQKj wl+IEjaOpDX7ombPzSj3y2THRlNj/RnWEkF4SOaltNO7b1ULJ5E4R1lUdBvmFVQAxg+B pX2cwcfG1BEtEN/p0Ax/CwT5mzNKkPByCSinoR75RLerqkzQ6e8FSJ6D66isEU6Pasuw TBDmLG6/CXygKCGr/SmEORHzc01YBw2gl4zAUugzS2oRPpsaBdF+1RZo10SP8h5GvP9S M7z+tUTSY9nXsDIVwcz1GTUgxRU/Yz8xGnxmvNZm/JGzCwTzUQaZTGFGvKCJcvoQ+nIO /HJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tEv+d2Yp4nwWrdZaUchV59IXo+3StH/VDh4Teg9zL+Y=; b=ykqYL/w7GjRZ/Wzx8sKE4c9L2ke+gNyqGizXzL3/ZsdowVrtPFJzIpjdgi5LNXK9hE A/DqjCIMMplRwROzMwuKZ6V1AwBHQhYMeBFQJD5FZhjT5qYoUBtxNMC88ug5e1APejo3 41jYwDTMGNCpk/eip3LsJdvfbQF1ZqLtOVaNYOXwcQoQHLY8t/UBTrGYprRLqX1koDJt 7AuSTbvnsMFlrkphlk+BbQjAIftCw+kpsjWlbTcw5/WRDaN7cEF2T6dV5far5x4nVd4v 6lJAqXLOKc68xGesyj7YyPUueJjFS6EawqNmack51C1PbheKDr2zW8NANSFIhtJ0roO0 20hQ== X-Gm-Message-State: AOAM530LY8r5u/Idx7cYx7/Kmve+SzHi4e5pmUswlwDJAgFFr0AkWZBO Klc8QdQL0RbF7asMPr7YnzyyTzaqSm0= X-Google-Smtp-Source: ABdhPJwVYnLm1S2iV3yhTSPAXPU6Ui5bdBdBi8RUOEdN+CtE2rI3C0De76ltg727sq96INxyKhIV7g== X-Received: by 2002:a05:6402:3496:: with SMTP id v22mr36562591edc.177.1639133984138; Fri, 10 Dec 2021 02:59:44 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:43 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 05/20] trace-cmd library: New API to configure compression on an output handler Date: Fri, 10 Dec 2021 12:59:23 +0200 Message-Id: <20211210105938.98250-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handle to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 58 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 60 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 924efeeb..c385c6db 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -297,13 +297,14 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handle, const char *tr int tracecmd_output_set_kallsyms(struct tracecmd_output *handle, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handle, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression); int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_output_create(const char *output_file); struct tracecmd_output *tracecmd_output_create_fd(int fd); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_option *tracecmd_add_option(struct tracecmd_output *handle, unsigned short id, int size, diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index a4d29d8e..9c996cd0 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -258,6 +258,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) free(handle->strings); free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1336,6 +1337,55 @@ int tracecmd_output_set_version(struct tracecmd_output *handle, int file_version if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handle->file_version = file_version; + if (handle->file_version < FILE_VERSION_COMPRESSION) + handle->compress = NULL; + return 0; +} + +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handle + * @handle: output handle to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_headers(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handle is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handle, const char *compression) +{ + if (!handle || handle->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handle->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handle->compress = tracecmd_compress_alloc(NULL, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handle->compress = tracecmd_compress_alloc(compression, NULL, handle->fd, + handle->pevent, + handle->msg_handle); + if (!handle->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handle->compress && handle->file_version < FILE_VERSION_COMPRESSION) { + handle->file_version = FILE_VERSION_COMPRESSION; + if (handle->msg_handle) + tracecmd_msg_handle_cache(handle->msg_handle); + } + return 0; } @@ -1933,7 +1983,7 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1946,6 +1996,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_headers(handle, NULL)) goto out_free; /* diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 1b806e78..21937562 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4511,7 +4511,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Fri Dec 10 10:59:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9D15C433EF for ; Fri, 10 Dec 2021 10:59:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240100AbhLJLDV (ORCPT ); Fri, 10 Dec 2021 06:03:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDV (ORCPT ); Fri, 10 Dec 2021 06:03:21 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53B65C061746 for ; Fri, 10 Dec 2021 02:59:46 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id l25so28878957eda.11 for ; Fri, 10 Dec 2021 02:59:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dHwGyEPp8asw5TFDr8bVu7XDrtbEXAAbBY9t8ZhHKvI=; b=e/EVgiiElZemDBcUXdMOypJfJH3iQXMHYJqJD2UNu9BENNAqoyx2u7um8VvYW105ed lYv3P62hME9Lebet/gkOL/AjGruL7CHnPmj7t4Ym2KDotxP51C+4PQnQ2Ohc4qS3wm5S 0frGBgDVpb687g3rJi7qgrltUjf7AoWIMF8LfFZ8zjz7o3e8zCOyEgm6qipokkbB5nKt oWKJ5s8f/YImfIWuQ5ErG3pdECGzbmKSwUGjqWH5T5OdDSQag+AwffBuWND1IfoNesP8 VrkYemkMSPE1ZVUyrR9qH0kfRoj9OHePrpJ+NkcSe08liSJXhqcJknj8zShpeG6PC4lD 5+jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dHwGyEPp8asw5TFDr8bVu7XDrtbEXAAbBY9t8ZhHKvI=; b=xk9WKhXlL3PXi0uWnhmcz1p/Yl85gMq8veojzWGec1AtZ22ohEF37VCcoazEwQ6pvp 9mjX6Iw4Mu0Y3Fiabwds1eNryRnSvCA49htHpsU9UUPT0y8LvNEwyOB85+8AS2fscRdp 5aMsvBHug/uS3oGfIKUM9TkwMnyS5zahzbTPTR706i0NUVkHYVeafgJRdy+I3cr+tr/n vVZFllUIfIRoOxQmkwsZ+e1I4uPUafPzE7sF+jF/6xBDAnUO6Nw68tAGgW2fJSBtO1Uq ZxgKcuMC7RukWZ8M85p+J7Wgt4PfQt+lvbyRbPZVhCQgZ77wGfpyn3JXjKrnZ1h3B8Mz bZrg== X-Gm-Message-State: AOAM531q6UrYit0K3Wow0lbANiWBf9zzVS3EVrEokTaMHPSoU3nnad9H zqCfa3gh/yjqynoLxtKKKHmaLSIMxtQ= X-Google-Smtp-Source: ABdhPJxKLs2CHk5wS1wUljxqCf0VZpTav4jeAMrMqJZbtF98yo0ZmEFNgbeY/yo6PCRnA7YDjRg2lg== X-Received: by 2002:a17:907:7ba8:: with SMTP id ne40mr23087114ejc.391.1639133984911; Fri, 10 Dec 2021 02:59:44 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:44 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 06/20] trace-cmd library: Write compression header in the trace file Date: Fri, 10 Dec 2021 12:59:24 +0200 Message-Id: <20211210105938.98250-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 9c996cd0..43667c8b 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1151,6 +1151,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_create_fd - allocate new output handle to a trace file * @fd: File descriptor for the handle to write to. @@ -1443,6 +1461,10 @@ static int output_write_init(struct tracecmd_output *handle) endian4 = convert_endian_4(handle, handle->page_size); if (do_write_check(handle, &endian4, 4)) return -1; + if (handle->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handle)) + return -1; + } if (HAS_SECTIONS(handle)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Fri Dec 10 10:59:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63649C4332F for ; Fri, 10 Dec 2021 10:59:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240104AbhLJLDW (ORCPT ); Fri, 10 Dec 2021 06:03:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDW (ORCPT ); Fri, 10 Dec 2021 06:03:22 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E1D7C061746 for ; Fri, 10 Dec 2021 02:59:47 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id g14so27967060edb.8 for ; Fri, 10 Dec 2021 02:59:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GSWtlPxKuX6lEMTp9uGEQk3/jlh12JO3c+vR8eul//k=; b=STk0ndlXcEKYM5QnOK8I0AE6wc+x609JSKYtYfrTfZzjXIAMUq5G3Wl1iti4s8wgor IT6NGYhKeX4cAd2x8u4Pu7ShIWYK15E8dML+7S+fy7cDjEtIzydtAaGRtymYYCG3bKPL ctc5xJ7MwCFpGuiBZgvT+83zbhfJsKaaPaZvCRjy9a2/MUZB3pzSuBO9hpIEQm4rGvzZ 9fjo7nrjPc2mtMdXsnXGe1BKWA44sV1pErEIBDGDvXJu661D5dlyy1u0o1ksE96qpiaz hLEWyE2OqUafryDsZdkjDsoAQNdE3vcaHBeAcS6+piAxCAx/7aGfuVaWwc9HCDOjume+ PJhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GSWtlPxKuX6lEMTp9uGEQk3/jlh12JO3c+vR8eul//k=; b=FUAlPs/qtuuKA82ZZu7CmHnIuntxPhmJ5H5yJJEDy04z1yavG2JnsFsjIC1gwItv2r nUeJa2Uk5kCoudMjI8JOOttoQByHUr4mfYOR7gIqDTlRO8NJMG2TBURy/YMZRVmim/Mj CUOnv8EitW+2uPyV4Pdj90bhwfnNXumh9loVIawZYBx+YgWTkTACTKnEg6nMSYry8mlX RRQHnSwF9KcPv/U0eX2PWNvX9ItRZiF3veT9ivIGEeGqglG6bF6W9C85y5eNcKHuGoxz MEf8gYPm1q4RlNqEqhN+ukcA0Od+uBdqJZhZntBtakBE+juMIMN425AWzODp+SrpxkFq VSkA== X-Gm-Message-State: AOAM532aNuLp4u3bu9OOiyey3jV+q6Y+UGuklNUJ2eUt7phoPlgYkQKb 01F8tqsklze9xVPmLA8lRBwLIVU7vPc= X-Google-Smtp-Source: ABdhPJwjDrKHpIdr4guEBRxjiziWqErZZOUGLPYW1U/KCpoCnb/+mkmeKUCWQZdW6BIenC1I+XA9Og== X-Received: by 2002:a17:906:90da:: with SMTP id v26mr23465978ejw.442.1639133985857; Fri, 10 Dec 2021 02:59:45 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:45 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 07/20] trace-cmd library: Compress part of the trace file Date: Fri, 10 Dec 2021 12:59:25 +0200 Message-Id: <20211210105938.98250-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name - strings Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 97 +++++++++++++++++++++++++++++------- 1 file changed, 79 insertions(+), 18 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 43667c8b..a5878636 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -92,7 +92,7 @@ struct list_event_system { #define HAS_SECTIONS(H) ((H)->file_version >= FILE_VERSION_SECTIONS) static int write_options_v7(struct tracecmd_output *handle); -static int save_string_section(struct tracecmd_output *handle); +static int save_string_section(struct tracecmd_output *handle, bool compress); static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) @@ -272,7 +272,7 @@ void tracecmd_output_close(struct tracecmd_output *handle) write_options_v7(handle); /* write strings section */ - save_string_section(handle); + save_string_section(handle, true); } if (handle->fd >= 0) { @@ -428,6 +428,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -489,7 +491,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, tsize_t o return 0; } -static int save_string_section(struct tracecmd_output *handle) +static int save_string_section(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t offset; @@ -503,13 +505,20 @@ static int save_string_section(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_STRINGS, "strings", flags, false); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + if (do_write_check(handle, handle->strings, handle->strings_p)) goto error; + if (out_compression_end(handle, compress)) + goto error; + if (out_update_section_header(handle, offset)) return -1; @@ -521,10 +530,11 @@ static int save_string_section(struct tracecmd_output *handle) return 0; error: + out_compression_reset(handle, compress); return -1; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -544,11 +554,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -562,6 +575,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -614,6 +629,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -621,6 +638,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -849,7 +867,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -863,15 +881,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -897,7 +920,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -915,6 +938,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -935,7 +960,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -950,6 +975,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -957,6 +985,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -1003,7 +1033,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1021,11 +1051,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1051,14 +1084,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1077,10 +1115,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1103,12 +1144,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1495,21 +1539,25 @@ static int output_write_init(struct tracecmd_output *handle) int tracecmd_output_write_headers(struct tracecmd_output *handle, struct tracecmd_event_list *list) { + bool compress = false; + if (!handle || handle->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handle->file_state < TRACECMD_FILE_INIT && output_write_init(handle)) return -1; - if (read_header_files(handle)) + if (handle->compress) + compress = true; + if (read_header_files(handle, compress)) return -1; - if (read_ftrace_files(handle)) + if (read_ftrace_files(handle, compress)) return -1; - if (read_event_files(handle, list)) + if (read_event_files(handle, list, compress)) return -1; - if (read_proc_kallsyms(handle)) + if (read_proc_kallsyms(handle, compress)) return -1; - if (read_ftrace_printk(handle)) + if (read_ftrace_printk(handle, compress)) return -1; return 0; } @@ -1738,7 +1786,7 @@ int tracecmd_write_meta_strings(struct tracecmd_output *handle) if (!HAS_SECTIONS(handle)) return 0; - return save_string_section(handle); + return save_string_section(handle, true); } int tracecmd_write_options(struct tracecmd_output *handle) @@ -1887,6 +1935,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1896,14 +1945,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Fri Dec 10 10:59:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D5F5C433FE for ; Fri, 10 Dec 2021 10:59:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240105AbhLJLDX (ORCPT ); Fri, 10 Dec 2021 06:03:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDX (ORCPT ); Fri, 10 Dec 2021 06:03:23 -0500 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 145E2C061746 for ; Fri, 10 Dec 2021 02:59:48 -0800 (PST) Received: by mail-ed1-x52f.google.com with SMTP id t5so28322011edd.0 for ; Fri, 10 Dec 2021 02:59:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J0pHtu//6As10l0ieQVxU0zhtHdqLlD61BrnvWG8TG8=; b=IKBBW1LLerE84F9AbGJmvjDcupU9JzYNf6RodzuC/roqgq5c1CeTCUZ2rp2QMRwblj 3RCxoN9ZD2Lgxwco+1T8ARTkCleYZPVPR0stp5Qco13RQlB5/A/5Vp6T8I5WvWIbe7xE efzo8RiVqBOrgzQi5aon+zKhomPEnJYg2CKBTyif+Txw9wAE54dJVMjTlqPIgyWSsDQd uCWp4NaIS3K4UtWwoWqGFgWol6Aqu3M47nOSH1m4yjfDNC23yC68/7Ua6+ylHrnLVUSl 9HWc2ainr6VDsqtk2wW9uxE52Vzl9rlBMtwyCnlqCMB+aDNLsFYnDfGqoWWQzOdmnTrz P/1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J0pHtu//6As10l0ieQVxU0zhtHdqLlD61BrnvWG8TG8=; b=Jk8CgX4m0LKDdLC6vA39yLy/C558ViaeBE52+qhdDf7u9T8pXqtZG4eO1yfRW/IQtk xg71TOjpfDzF+jIrzLZ537911cjmsyCK5S4bB5z5y4RnoXPU6y7fupmHyNBp/TRGi0ou 1KpePlckqhWnWVFtJj1e0d+siQ6+yy3e24RCV/bLILwBjSr9oAIs87Sxkxwnt2BcnpLD UbnfFE9JD1AttUNNVcGHvb2k2Fw5VocDjUpwFMJZ6M1t5xAOro/zTnGVW9P4CqV8GdvZ QuqYL3otRfqK6MvgrJE8D484gMSt8XPrxq6HhBoyN3LDroe5zNxcH8mDOwo7LZy0yJvD Xnuw== X-Gm-Message-State: AOAM530nrx6+JbFIAulSF0EkiLsM4GT9+2JjolIxuBGnJeZ7wByVHnCn ovVX9LjWnM1ewjOVxjJmfCt4B+jGohg= X-Google-Smtp-Source: ABdhPJxLFcNwGu+gcsXdKNWRmrZSSeM0lPcZ67PrtPOxTZHEWpx5TzRVCAgYVYHK8fmFEE30MtmuyQ== X-Received: by 2002:a17:907:608b:: with SMTP id ht11mr23789434ejc.80.1639133986670; Fri, 10 Dec 2021 02:59:46 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:46 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 08/20] trace-cmd library: Add local helper function for data compression Date: Fri, 10 Dec 2021 12:59:26 +0200 Message-Id: <20211210105938.98250-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index a5878636..3bd67312 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -315,18 +315,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -344,12 +353,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -596,7 +655,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -622,7 +681,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2276,7 +2335,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Fri Dec 10 10:59:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A8B9C433F5 for ; Fri, 10 Dec 2021 10:59:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240106AbhLJLDY (ORCPT ); Fri, 10 Dec 2021 06:03:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDY (ORCPT ); Fri, 10 Dec 2021 06:03:24 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E941C061746 for ; Fri, 10 Dec 2021 02:59:49 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id x15so29283739edv.1 for ; Fri, 10 Dec 2021 02:59:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yjPuy8hn8HL+lC9Ml5tr34DiybXLoAWp5sZItqSMxoM=; b=WhrvK7/5+TdnCDeib5102nE/R9zSWTmXYYx8wodk2noGwFjy5ILm7TMsXvPSxOVPfY USd9iVq+H/mn4SDW2Dpy+8fQpqp517+GKjsw+vRP37OlJJfCT1duBNIdT+LrxRPcyl3K NRTGhZ/TZk44AxT1vozqQssLe9l5VF0fGOnrLKdfSac/+DEOE21FPyDjvZTzCjz1LR44 QoD27PxjO+Y7ABB7gNPJyIKehWhwsaCNvRuAXdodoemHVlHmKqiAo2pVe/R4w26gZ3Hu 4OzbKfN/uExZ6gOXWOdNOo7KOkj42GvSE4YzDQmISbPtkbUYA58GCcYHQ9rpA2yEZOUI u4Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yjPuy8hn8HL+lC9Ml5tr34DiybXLoAWp5sZItqSMxoM=; b=NBpjX5rXBo0wrMZIIy5T1Eg/1vNdvxcEdWyLrJ3NX4lCmN9qaneq5kghsw2f8MKaav qHXqIrS3V+WqoJMZ+iPGK49Yd7MpKcFUH4WeXzf8HjI6pNeQJYBOMb/7UoZSIV30w9jy 7J3fan8eDSwKSu4bPw5g04+q1/r2EaBTYW0ePKzQYjvSUdgjWspTEFRpJqAaPAAuwVaX ccZJHK2PWi7be+y/PlvwwA/4go2mE8bOWjUZqGBfuP+L+pccwjN0HcdmVMApgM+m8LWy 6zgCnCaP8SDwAnC/SYyNxKWIcGirDqdLFNj11CjJYrEGfRaeZGhmgg5WTmvEATCpGcRM YIrw== X-Gm-Message-State: AOAM531Z8yBD8weF3fnJHJ3N6HHkabnwYG9Ba4PGlM+8Krm2Z+tfHj3O zpPLox+tQQMjDIKaBGn54y3gyTgNs5M= X-Google-Smtp-Source: ABdhPJyvKBszgA2/HRzeJqf+HXEZLcEEzwjW+fP46W2jPB7e38miEl8qiJzFbBTgtqmi3HABwZ7GOQ== X-Received: by 2002:a17:906:dc8d:: with SMTP id cs13mr23563496ejc.323.1639133987667; Fri, 10 Dec 2021 02:59:47 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:47 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 09/20] trace-cmd library: Compress the trace data Date: Fri, 10 Dec 2021 12:59:27 +0200 Message-Id: <20211210105938.98250-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 3bd67312..4ba971c4 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2176,11 +2176,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2283,6 +2285,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2335,14 +2339,15 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); + if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Fri Dec 10 10:59:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DDE4C433FE for ; Fri, 10 Dec 2021 10:59:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240110AbhLJLDZ (ORCPT ); Fri, 10 Dec 2021 06:03:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDY (ORCPT ); Fri, 10 Dec 2021 06:03:24 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1F07C061746 for ; Fri, 10 Dec 2021 02:59:49 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id v1so29004174edx.2 for ; Fri, 10 Dec 2021 02:59:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T30AEnGqNs/0l3fiuhjZCDhcwPjBp3J5XxtplF3N0Xk=; b=dIBgBQPPmdsj8xJaCGBUibSksfp1WB8jA9QrIYUnE7g5uRHqtHMYCmGzZaok9j1IRp o0EGRsAmG0jfaRosvDjBqrpadNYBP5aBya9P8jZ/8SYDGOoGH4zlSLwTcfBKOFm2tPFk DLM47FqjD2h2D/k9G2iuCux/CpJfFVJc2WMcp4HSTNOIUUuQQj8gIyOrdLUsBsiWG+t7 +IybqXdJxoVCCk/ChfvW3OhQSanFBs0MFPxtPnwGTHtPEUcjdmjJBlm/PrHq20/K+6im pjFGMaSFGFG1mw3wp67NlCkcs6m3HD5M75Mc6eyEaz4fSeprG7cERwsNHzwFEwe8/9zr fGbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T30AEnGqNs/0l3fiuhjZCDhcwPjBp3J5XxtplF3N0Xk=; b=mc5zaXXOVzR/c/4c5kY6Xr89cVvaMuKtfmPovsNo9IEkNChDNrHkMmZQVg1/b+C7el 8Joc+Hhp8Y1XDpSNsgPxT9KKAMFOXULUyCuw+CI5mte2rSgSataOVuS4JjXNIzHfuXaP V/91igKjB/ColsjMeY+ydzTRLQiGSyjaurUs57uG9d6qpyObsMd/YPRo8PKEhGJ0A3Ls +VqfoOHkNV0tEot94CGYCu0TkZA0JBgYQkm2WHbQ6XcQl2j5noekHCI8sNkY1nuqXeVL yfHAz+jI2EWUoBuR9t8mlXH+o8AZOt+dyNOBa82+uJvNpLum/5jVfI/CgrtGvYm/ATaw LSJA== X-Gm-Message-State: AOAM530TdeLpkSdr9AH8yOoolwuMw5OJaIvweNfOgrkz51FjWZ0cS0LQ BDvA8Pe4G5SCpQm4YE8Aiw8OWWIH7/0= X-Google-Smtp-Source: ABdhPJzpXhjegbrWZyikAy55/nAgt21loBvmAXzwYfZDATDFATnb+0B6xmY2z8relVEi1xLSyAwwMg== X-Received: by 2002:a17:907:2da5:: with SMTP id gt37mr23186863ejc.316.1639133988584; Fri, 10 Dec 2021 02:59:48 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:48 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Fri, 10 Dec 2021 12:59:28 +0200 Message-Id: <20211210105938.98250-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 45 ++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index e35e3c74..f35b61ce 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3035,6 +3035,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3046,23 +3047,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3116,15 +3126,17 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: buf[size-1] = '\n'; cpustats = realloc(handle->cpustats, handle->cpustats_size + size + 1); - if (!cpustats) - return -ENOMEM; + if (!cpustats) { + ret = -ENOMEM; + goto out; + } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; cpustats[handle->cpustats_size] = 0; @@ -3134,7 +3146,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3193,6 +3205,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3205,7 +3219,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Fri Dec 10 10:59:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B4C0C4332F for ; Fri, 10 Dec 2021 10:59:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240115AbhLJLD1 (ORCPT ); Fri, 10 Dec 2021 06:03:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240113AbhLJLD0 (ORCPT ); Fri, 10 Dec 2021 06:03:26 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA29EC0617A1 for ; Fri, 10 Dec 2021 02:59:51 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id w1so28500707edc.6 for ; Fri, 10 Dec 2021 02:59:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rGybjosqsQwLOzfXnuAsdnckYzIGrHvfQXkGG/d0SNw=; b=cmdBf8O9uCMAXiuy6zPyantu0rOXQwbbSzA96rQLtwd8WeJnAODC+7culUssoumv// uHE0Z7zZp5b1fdiEIbv1a3jKRmbxJItqdCMzqSX1LIJlVshajsfkLXZmWrqrhZJ9pNWM IeodZ89VN6QrAcewbo+tIeSWeg0uOcHjEfAGaFZ9wNaRh/SYm8xemxLdLvBGY9DtHdqe 9ur5XjFXaYRwsRAizD44A+6HcXPkHRm8/aXksqUY+NnawbVLRwsbnD7MPXaYtahs+1yv TJV1CzcI6kYaBY8zEGr9+u/y4r/WeykAzUl+6xJQ3ujq+7VR7BvMe3d0B6m+iVrohc8b z0Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rGybjosqsQwLOzfXnuAsdnckYzIGrHvfQXkGG/d0SNw=; b=BlLTyyRoMkTUEUIT5fyBS/wn5R3fPD/7NW7AdQrrfHh5scVzTQAUWAGr+n1dV1npTw 8Bpo87mBB8GBJiIlyACasChk0yvddMMwYiiMxKkW/VVKc2HHHszIbADCnP5Gko8gVVg3 Zl66cr7u6fEZRdAjW5axtUls67tLcOgGYKxNw4MY67B5ClWp4YYPo23CaczOyYhAMWWk 4QC4yD65EwWGrY5EwOf7nYRf0ZHFEfiumhJL2f80XXbf4FcyPBwkZxsNopsShnlocivD kf2jpqbuXmmodbqZUcEmeTvIK4H1EZSnaBciRPH/uqPQE7LjLqA5HIBEryoZeUlI1ugG OK8A== X-Gm-Message-State: AOAM531qR5RWTq3nqZmedB6CNlnVqlmTMXRm2ihofIYXWJrVRcBS5zF/ Vw8BU7N85CkhGLLrpWsBhdzXBxUt1SA= X-Google-Smtp-Source: ABdhPJw8ac+0k6EY1xum239HnAiblBfpyCeB8ddxcO3KV/VYpnxf3NEvpvYmvPhcd3klyyOw1J+xvQ== X-Received: by 2002:a17:906:f0d4:: with SMTP id dk20mr22947690ejb.257.1639133989495; Fri, 10 Dec 2021 02:59:49 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:49 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 11/20] trace-cmd library: Read compression header Date: Fri, 10 Dec 2021 12:59:29 +0200 Message-Id: <20211210105938.98250-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index f35b61ce..78271fdf 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -193,6 +193,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3838,7 +3839,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3876,9 +3879,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3908,6 +3914,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3921,6 +3947,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4120,7 +4149,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Fri Dec 10 10:59:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE3E6C433EF for ; Fri, 10 Dec 2021 10:59:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240114AbhLJLD1 (ORCPT ); Fri, 10 Dec 2021 06:03:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLD0 (ORCPT ); Fri, 10 Dec 2021 06:03:26 -0500 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B731DC061746 for ; Fri, 10 Dec 2021 02:59:51 -0800 (PST) Received: by mail-ed1-x536.google.com with SMTP id e3so29073647edu.4 for ; Fri, 10 Dec 2021 02:59:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0JLOGryTX6Dx+lCa8BZt5/FliAFVkF7DjTVQAb57Za8=; b=TTNwR+NLXHuMZewLbYSq7/2rpynzJ1q0j2Kady/WrGcHxBQI6UYh+9ffa1llE++Z96 Q3cIQTe+/K/D6sn2NTz9p+K+itaMmrCBUltSzH3O0CQa+mguDPEQ0kX/HbQ40I7Mqt4/ Jy6MOlHA+g6bLJxcbjRYtdsik2JHlhBMZIDJLvsUfYUvAA6wvaWENDD5PO6w9CmFvPmQ XwoH89FB5ciLcnwpqqxvhFJXyFWiPe4izC4y4QPZq98WqVK+7uCrp3QRpQRhUdABlyOk gp1V3Nss66H7AiHCpZsxZrx9ey3tT5TYEQeTSDsJEu5IPxD9j1qxGk+7XRWbrwrh9cZ0 M8tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0JLOGryTX6Dx+lCa8BZt5/FliAFVkF7DjTVQAb57Za8=; b=17bSOjVAWGrAmjTwHAtyiEDhJrlkfcZzQHNzKBSm6PFMCvXGS4sJ7fp7uCDr3n96Qp qLUvX/BVL7QOmIRd13eb4ZYshevxVXapZENCNVOPdhzOpj4Gl4E1BSUru9oxcLT7LATh qwOjmkBu/k8tOqxk9ap+q5WYZXQLtOXaKNme33dD833V0SLc2rK0JfUyhoomGVI+sMvY 7VQsYDC73VMvJILTI9NbK817NEd1okMiCNcQMtmh9wx7ck1v4TStTUOSBl8LCuid7Ycc 2qLsit79Nr0tygVc9t/73BOS9zwOkUiDZ1Qv8AORUvaLhEegqZbQX0s+vysqZySeoeJt aiwg== X-Gm-Message-State: AOAM531huPMpcHnLkQMPFceVD1nhfnGG/5SrKDJbRvfZoaNyIE/cXeTO aL39jN1Xxy+gONv8RbwYqP05ceRGRu0= X-Google-Smtp-Source: ABdhPJyHL1L9PCmQgc25Y87GHdOUYA8BcY7hMaqqLv4ehs02GHS/TicQIVmPh8GK0Nfrk81m8nxkZA== X-Received: by 2002:a17:907:629b:: with SMTP id nd27mr22941678ejc.24.1639133990304; Fri, 10 Dec 2021 02:59:50 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:49 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Fri, 10 Dec 2021 12:59:30 +0200 Message-Id: <20211210105938.98250-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 78271fdf..dbfd881b 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -151,6 +170,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; unsigned int cpustats_size; struct cpu_data *cpu_data; @@ -3304,6 +3325,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4085,6 +4107,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4104,17 +4127,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Fri Dec 10 10:59:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D44CDC433F5 for ; Fri, 10 Dec 2021 10:59:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240113AbhLJLD2 (ORCPT ); Fri, 10 Dec 2021 06:03:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLD1 (ORCPT ); Fri, 10 Dec 2021 06:03:27 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 869B2C0617A1 for ; Fri, 10 Dec 2021 02:59:52 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id y13so28282529edd.13 for ; Fri, 10 Dec 2021 02:59:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qAQrCfQ7XX2f+uDKVEsDU5jZpBO8BjHGndI47OL11NA=; b=i1M+NZ8aE50+sm+ZXBD9RRNauH4LA/Mi8Ji2eU3w3kTbBVgJ/DUwMDJM0W5LuNBflD a0GRsVIkXwVkivpHPtFPPzrgS7Yq6ZEPRcQQr+9uh6NU19g1uCL/s4UPPxqVTUG+lC2P er667/s8+YeZYQsLSQFmyhJseAC6L1QQ6vN0xlv1ShjbUVJxTRftWAzjQ37eqOUoC698 DBn4F/wSBzVdxuj/2kBmGxLPKcTAcctNOSAeAcWAaq/+UO4KTwjO/7XlT2soLAaNwCPJ I5f4y/Q2mloJ8UPNKmzGb4dELf8BmNmTfR7oLUGCfB9E060+7ZdadI5OjaLdEmO2Do9g gRFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qAQrCfQ7XX2f+uDKVEsDU5jZpBO8BjHGndI47OL11NA=; b=xfth95Y5pmnftCzhL/1k9mMGCWclpMIbFay7ce21THZa6SH8iWbgKC3rp4Tljj+fFE NjMVC2EL9eb1RCfEMwbi/jSPMK3v4vmY3i7Dy95VPGfwrXQZFfS021jzJtZlvvphrtJh VtK1QT/51YuAvtFnjR1l5EKX0ooPWXeXRSna2qV2djc/iAO4tUFluVV95+5LxxP5KUOh NSAQ1CAZMgkn3pwEWb12zj9Gpn362Di/RBLwbLBOXdJWOnvIGChl9xGsykJDBeY+O+8L bHezzC1IvwQO8oTtdS3AzOJLZa/TI9BUHIXfMaK2uDd3XcjDq0/C8aCkkoGLao4l+QTq k0ZQ== X-Gm-Message-State: AOAM530us8HTmAurui18ua0KeaM2S0iKIXLU7Fo6jqQetJy4UvvQo9vU j+6rW0h+twsdwf055H2ZS0D8R40Qz7I= X-Google-Smtp-Source: ABdhPJwaSptISys4ze9oFpcFSppejJNft24jm/0ssb9nXylNCXOGkDTrBdhfdD/DxE8JfPRI+IfUQg== X-Received: by 2002:a17:907:16ac:: with SMTP id hc44mr22182544ejc.363.1639133991091; Fri, 10 Dec 2021 02:59:51 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:50 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Fri, 10 Dec 2021 12:59:31 +0200 Message-Id: <20211210105938.98250-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index dbfd881b..7ebf8b20 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1264,6 +1264,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1303,12 +1304,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2496,16 +2500,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3382,6 +3446,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Fri Dec 10 10:59:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8331BC433FE for ; Fri, 10 Dec 2021 10:59:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240117AbhLJLD2 (ORCPT ); Fri, 10 Dec 2021 06:03:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLD2 (ORCPT ); Fri, 10 Dec 2021 06:03:28 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70C25C061746 for ; Fri, 10 Dec 2021 02:59:53 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id y12so28104475eda.12 for ; Fri, 10 Dec 2021 02:59:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jmb32HJXyhXWwu8C99kX/Txjg2qdJIps23LRbBGkBAQ=; b=Wpumoi5Y+Q7G3/I7p4KDqXL5FqjimlquiaE6XH/yHatt+3X0iubApvueht2p9tTzgc ByngjKKZ+XyeALTDaN0RuoEVJFdK2/nDDHPB9aNBzhGpogPt1TmeVnCSCj85SCQJ/lK1 cTRq0Mf0ncs3X28rbw0NXAyhLv9iNCKho409itafY807gZCM+vYKckEKtb+G2DmXZLsx edCVE8QnzTNcrGMlQgyTQmUeu6f4LBRPgvLCT9+pqzL34cKLbuljskL++gnf522X2jc7 uCCTpNjvS0W+jx4p57wL7BqbumIC7dIbwFrIo0ZOAwa3K6J64GbRfE3+hcr3oN5KWQmU F+vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jmb32HJXyhXWwu8C99kX/Txjg2qdJIps23LRbBGkBAQ=; b=Opgi4c306X1jsxpjKIyapdbScZwWhh4wYrxXQZfsHtPIpo6Ekhmyszv+IlBBR2f8vR tw/F34AkU01CVLGNHU/jRZjxteSX/cuMK4RIlEKJeKGnUyl96JAkLbbfuFC+qU7EV1Ca WhicpKOLKfFXYvwhRyE67VAbqfX1WKVatjiOmX8ro+Jv4vrEBHaqWrcKbhFu1eYtEUNV rUiwC83AR9qRVq6+BEcI7lwnJrpaG0t4rkGuOoMmTlZ9nG7PDRDFhnSnkHgQhR5SCzT/ LDmCODvddpgxe76ZJ356NAiBQ6zlwfWavWc6Dwbs6daKr/0I8rZBQqasU0BMR5nXbFOJ yaMA== X-Gm-Message-State: AOAM531ysEaTDYV9apiZL8xbj7V96i5qovNYaDpntOtFTjJp7mHWLIbe 10oraw/ArNy+ZwetZFEJ+QPzem6LsCU= X-Google-Smtp-Source: ABdhPJyL4NnY7C7St3Vs3ctiSAaExf13oleJ461MrzJYHpwWL+HFnhUADEsqCL4s9FSgvEhaQ5t8Hg== X-Received: by 2002:a17:906:ae83:: with SMTP id md3mr23187591ejb.31.1639133991918; Fri, 10 Dec 2021 02:59:51 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:51 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 14/20] trace-cmd library: Add logic for in-memory decompression Date: Fri, 10 Dec 2021 12:59:32 +0200 Message-Id: <20211210105938.98250-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 7ebf8b20..f151954a 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1255,6 +1258,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1266,6 +1368,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1408,6 +1513,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3943,6 +4050,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Fri Dec 10 10:59:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25DA8C4332F for ; Fri, 10 Dec 2021 10:59:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240119AbhLJLD3 (ORCPT ); Fri, 10 Dec 2021 06:03:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLD3 (ORCPT ); Fri, 10 Dec 2021 06:03:29 -0500 Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26E68C061746 for ; Fri, 10 Dec 2021 02:59:54 -0800 (PST) Received: by mail-ed1-x530.google.com with SMTP id l25so28880361eda.11 for ; Fri, 10 Dec 2021 02:59:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yArGJOm0EyvU1cDsxj2ccxRvQwNkhsYWa8UP6TkXvps=; b=oErn7xqgKz4ZDiINC/qqNh4F2ajlWwOV3oBxUrn7UwZR+WnikKUWIJFALrlNp9JYPd CZNWY1NLwS2JCPiGvkb2lWOpyuPyp/yRgrYNiPbFD/Rro33+ODY0vqZ/DPfArGWK+zLh GCvi2wbqeTnufCTIgNiuaxxLtdgDaNAoLdjPTUOV1WzikpGDraPFJyqYzA/of2rlhhY+ lOKC6yh4+RSU75RaJ+SxPd6YQMyQ8BnlyZF0mLR8blS1uBymSCMWn2c4Np3pyUa8yhmO 1ws70pTB+UR7y/PNMLycw0KPqaFGkRYcePNRng23r0oUXtYCxonEyyerkNnqvhQQI1D3 uwfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yArGJOm0EyvU1cDsxj2ccxRvQwNkhsYWa8UP6TkXvps=; b=FB0wUiA+H/60TQr2iLpHim715s6bDnLfz1U87RPkWFEzzGBFlzFmMC1sKvdWgBDL02 wqHlwzLEaF+wYN4ReQCOiG+Z7QOnmMduPdawD4tm8gtCCn/ziUX+s9eUI/nsY3ku1WDi Db+qJdmJSPY8MTV1HQ/xlnIPHU200koRCzUlvzCwE2jkOs6DBvMtc8y6ZEEjoq0Q2ck5 s43BJBGtDEEegPuBvevy+ZrRR89tCNDUa5TT+FRxiKlhIHhnUQipBkjodsASY5EDPbgR Q3I9KGY5LGRpHhjjcCm5VeIanU22QXnc3ULLWEILKwV8R2/8AydmaqYUramqqAMq+fYD Vkcw== X-Gm-Message-State: AOAM530Q3agRWfNuLiiYfgBlnDsveZv2x4Luly9tvBAc1kQvcOSGbQtR XoLG3jPOrSsynv9A5vP/FPhvEweblNA= X-Google-Smtp-Source: ABdhPJx2puOw0eomGIiSh+nvpG+fYd+ueQLpDePoQDzBWoPS0F64/tG4+fojU6eyZ3UAML7FBbrb2Q== X-Received: by 2002:a17:907:7ba8:: with SMTP id ne40mr23087838ejc.391.1639133992742; Fri, 10 Dec 2021 02:59:52 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:52 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 15/20] trace-cmd library: Read compressed latency data Date: Fri, 10 Dec 2021 12:59:33 +0200 Message-Id: <20211210105938.98250-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index f151954a..a631692a 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -177,6 +177,7 @@ struct tracecmd_input { bool cpu_compressed; int file_version; unsigned int cpustats_size; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3454,20 +3455,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3535,7 +3568,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -4047,6 +4100,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4338,7 +4392,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->strings); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Fri Dec 10 10:59:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F91BC433EF for ; Fri, 10 Dec 2021 10:59:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240121AbhLJLDa (ORCPT ); Fri, 10 Dec 2021 06:03:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDa (ORCPT ); Fri, 10 Dec 2021 06:03:30 -0500 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C695C061746 for ; Fri, 10 Dec 2021 02:59:55 -0800 (PST) Received: by mail-ed1-x532.google.com with SMTP id w1so28501279edc.6 for ; Fri, 10 Dec 2021 02:59:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TSnm0a26M58DHTqXW7gtU3LVcAdETyef38QT6KMLRlA=; b=WUUeqDexX9NB/QfNFuTOPDg5daooBO0Bz+1bKRYV/qPab0npQ9WcYD2sSoKfhO6uEB 2l/oY+6QcuNXom98xwzwwRuIXXJfwPw2gD1wq+/RXmZRVXyoy1jeobg7SyLZnpQewaa5 DFsYK0ME2rc31p9D/tyRT70LOCO7Y2N/E4iU7etLHCJdhZ4tD4o66io6XsGd+gmlaCWF p+flbUUahf4o3J3U1Sgo1OCzbAlpqgunfKGgxPPF9diP4AiyCNe8K6NQ3U7/tGWwoNnG BkvIlxQXQBKfElgecKcUZaD6fNyRuMs55lokPTnA331/WEpLCXPqUF+/5QkdTuAeCJxB W6sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TSnm0a26M58DHTqXW7gtU3LVcAdETyef38QT6KMLRlA=; b=M+HZaGRQoUdbAmX8qhkAKmkgC87QrrnKhbKKN42edK+Gy20Hohv/QERODjVSBs7g8U ethcknBnt5hLrbjRQIOj2UGEVkwArvT0GiQuaHLCZgmZetKn/sH1wVDOV9mU/pKAYN4e G39wqoQJtiR4z3OYR5nHMeoPiR9k6KidVyKBfytZo7+1qoTKECE2vbtxEbxrhI/7F+bG lglKN3NXoZlhkhVygjvuBnH2Y9W7LDmxprhAckbXltYMNoTcCzAVzi5VTV2thkhokiJE klrWDVN638FWWSosN54rgLUPK2Z6ra1QIsRrqAy/PCXCEB0Ea23BdRlbjRq61e4Hoc/q jxXw== X-Gm-Message-State: AOAM532yzcytnwRedmfNel8Lk3d9L39+qFJhz2Zd4pElKkQwKGAWF9vs Yi6c5PjM3A8/9YaDfjNwgrnS+OU1YjU= X-Google-Smtp-Source: ABdhPJyy3BF5LXFbNNln4uL8N0pbSWyhbEwUca2WHo34n7zfsGNyZ8PRe3vINGBMf8QW8ROIFxvNww== X-Received: by 2002:a17:906:4fc5:: with SMTP id i5mr23501859ejw.475.1639133993579; Fri, 10 Dec 2021 02:59:53 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:53 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 16/20] trace-cmd library: Decompress file sections on reading Date: Fri, 10 Dec 2021 12:59:34 +0200 Message-Id: <20211210105938.98250-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index a631692a..97ad47cc 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -484,12 +484,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1114,6 +1117,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1139,6 +1144,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } @@ -4044,6 +4052,7 @@ static int read_metadata_strings(struct tracecmd_input *handle) unsigned short flags; int found = 0; unsigned short id; + unsigned int csize, rsize; unsigned long long size; off64_t offset; @@ -4053,7 +4062,18 @@ static int read_metadata_strings(struct tracecmd_input *handle) break; if (id == TRACECMD_OPTION_STRINGS) { found++; - init_metadata_strings(handle, size); + if ((flags & TRACECMD_SEC_FL_COMPRESS)) { + read4(handle, &csize); + read4(handle, &rsize); + do_lseek(handle, -8, SEEK_CUR); + if (in_uncompress_block(handle)) + break; + } else { + rsize = size; + } + init_metadata_strings(handle, rsize); + if (flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } else { if (lseek64(handle->fd, size, SEEK_CUR) == (off_t)-1) break; From patchwork Fri Dec 10 10:59:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D965AC433EF for ; Fri, 10 Dec 2021 10:59:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240133AbhLJLDb (ORCPT ); Fri, 10 Dec 2021 06:03:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDa (ORCPT ); Fri, 10 Dec 2021 06:03:30 -0500 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1B9DC061746 for ; Fri, 10 Dec 2021 02:59:55 -0800 (PST) Received: by mail-ed1-x535.google.com with SMTP id w1so28501455edc.6 for ; Fri, 10 Dec 2021 02:59:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=W7WACWTRxYHvxphLFaDkLLl6Ip7NTeMCnyKHUYDHUMg=; b=kZ7j8jqvv0savlSF1X/sOqaJPDHzd85twWwd+xI95TerZ4zOjC5rD2dAwo5CDzrUP5 d8XEbFbLRFwbZSaBHlhbgd4RV9EIXgyi220Tpa7dvmla5ZVlOJMYXQK2LtqadQ/2hff+ tXCKw4NzSHcq3JWKruHThQQHXxohTYDEINRWrkL/o9h5hRTtrXnknea9bpQYXAiiCyyC Cv1G/c7j6fhez/qJwpNe7TldJ2dL9JEQ5sQguy4y0Px/KZLz9koEYmnUMqv+rsJB5IsV 9Mh1Hz98UszeAtw3zhmRjuZ8md0soEcU1ifGHS31WXLLXIuqYvep21InX8my21YbvVFn av+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W7WACWTRxYHvxphLFaDkLLl6Ip7NTeMCnyKHUYDHUMg=; b=0Vq0ODnnAgtr/YBZgYR4eyh6G6S7hXTfvMG4Vs0Rx0L5ap7bIO9PkEiE2eA4MdBNJ3 CiCP5i7xWb/4aeci7Ei+p2bt4DAtows5a3fLNRC7Sbk4EXg5W0shwOf6U2mMHKiiDOic hXhla7eZmR99I6RV7sOnnSp9wf5sG2sGPVsGoZa0wBYuJacG/f2rQHPDZDzDpG20VJFQ sE59zkCOV6WUoYrVggaVsqwckKXhjh+HiZaHe7+/JF8OVVWYJOI0qpovmXXpyEhBHgUK 8+G5o0j280kiRsDI/OFCAJf4w1oHToHFC88naENRVK7KylwwBR4q0HdLZKUfopdbvnhP vtlQ== X-Gm-Message-State: AOAM531D3iSNsI/V06w+6rxBczv+MxM55g431MVSM/d19UrPoD6r5YjA /cewUBqaYHLFxpHLZuR/Ti4gVMB83/U= X-Google-Smtp-Source: ABdhPJzhmehVQZrPru3d/BOPDgTO/s461ty7ovIZz/+UVG+T7f1WQv3p0336ktF3TBLEVns945sOxg== X-Received: by 2002:a17:907:3f19:: with SMTP id hq25mr21973859ejc.225.1639133994371; Fri, 10 Dec 2021 02:59:54 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:54 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 17/20] trace-cmd library: Add zlib compression algorithm Date: Fri, 10 Dec 2021 12:59:35 +0200 Message-Id: <20211210105938.98250-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index 883cf669..3d8f31bf 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -359,6 +359,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Fri Dec 10 10:59:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CE03C433EF for ; Fri, 10 Dec 2021 11:00:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240153AbhLJLDe (ORCPT ); Fri, 10 Dec 2021 06:03:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhLJLDb (ORCPT ); Fri, 10 Dec 2021 06:03:31 -0500 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A924FC061746 for ; Fri, 10 Dec 2021 02:59:56 -0800 (PST) Received: by mail-ed1-x534.google.com with SMTP id g14so27968725edb.8 for ; Fri, 10 Dec 2021 02:59:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0vF4/kyI9N2FIR8o+/109ZpzkBALfqAJzj8PQ0/ofkw=; b=fyxqnbEGrNQrt+x4t99u5eleWrZpyMmtWorYBK92/UcinfbLp70pdtrNGwoSHyd3l+ LdCVabI496q57DVFxGy72QoadyV20nODL6rlalkqoxE3pvuI/TqovRUk0DqUOQw9qksz bBLb2KO++MNGJKgq4ZQfdWrwueis7eG4PsTZTD4WQSBcFkkTzvg7N1brgsXnZL5YX0qF Jl8KRxArJov7p7mBNVlaitv68kMEEy/jDczSWL/2kOPAHq6WOacdT6oGKMOxbTu9LSXH FVyIDjbFkgGfYfdEaXyCgmVFzfE4g4BdyDS+ucoIrYOiQjlDJbGr5DSYgiVlDv7R2O0g dDkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0vF4/kyI9N2FIR8o+/109ZpzkBALfqAJzj8PQ0/ofkw=; b=oLQxwheAuacryk8zkXhOgIIqZeYd/rQxD4130fjXWp7v5QVL3k/cy9dIohWrchm+IA h/o3nBGGtrp2yKNhB2niEtQC8w+qppBcVTYDf3sjUGpZmSpWvBmZy98nsFxwkjNtIqSN nnJgkb2YnzXQBP/fleVMR5IvuUbgzVCnuXUhUPaMmzFcYvN6t7uuPgXxwOQZeONSZ7Wn zf5zsdGBCRBMAPaXqgTwOk2s37JPWQM1siYYOnNGibknUDkgoARvHgjbVHcU4KUQjdXz uPdEOoE/3kcixllBtJarH0TW6tXjxj/AvDU3sIcRa4pZqO5q0s0zjYbsRPGg4qXBT2WK pL/w== X-Gm-Message-State: AOAM53333uaIpp1KW2zzcQIuQZB7WLN6OqOtfUor5j5k8iGe1OvlO7Dj Bw4o4coJJlGp72Hvu8pCSzhXnms0tRQ= X-Google-Smtp-Source: ABdhPJxsEouga/CX+FRcq4EscL7aHGvvlOD7TmvSvbvoSqXwMvEvOmDNYI31RyvFtFZjcTaHP9sQbg== X-Received: by 2002:a05:6402:2551:: with SMTP id l17mr38067339edb.142.1639133995187; Fri, 10 Dec 2021 02:59:55 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:54 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 18/20] trace-cmd list: Show supported compression algorithms Date: Fri, 10 Dec 2021 12:59:36 +0200 Message-Id: <20211210105938.98250-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Fri Dec 10 10:59:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1442C433FE for ; Fri, 10 Dec 2021 11:00:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237285AbhLJLDe (ORCPT ); Fri, 10 Dec 2021 06:03:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240134AbhLJLDc (ORCPT ); Fri, 10 Dec 2021 06:03:32 -0500 Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E4E7C0617A1 for ; Fri, 10 Dec 2021 02:59:57 -0800 (PST) Received: by mail-ed1-x530.google.com with SMTP id e3so29074705edu.4 for ; Fri, 10 Dec 2021 02:59:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nhok2iNIEn15a1HvxTL91Q46+hpNPYfftkRan6XLC8o=; b=Np5GnbzF859fkf1ceQicTO7DSP/AnfsXLrEt6R9eSX3HSqNhUwRp8IcuuJCO4ZAvLP t4tzHJHdHSD7AVv3D67ejMUQVJZY7msouqYUsho40Ok18tOWxhH9gbfW+bJ6Gxwlul2E Jx5q8a6MqGacQvWRX1cvGraPY4IeTX+1Y8ja677Bvu0gvXvkf2lKN72AbIK4A8KPQe6d EoaxwPJJ5D84Br7JqnxTmR9L13pROr5Rn/rDCmcKtGliHdskpsayqCmc/gbcFhrqnabo HMiBlTNCfnpOQ8inrVyQzCsMIDUwd2CpAN8bRnW99cgkMu1og5RyYVYBpfK0XTTCx9J7 dpTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nhok2iNIEn15a1HvxTL91Q46+hpNPYfftkRan6XLC8o=; b=fxTWFlC/x/TRsUoMwtDX6CyKWc/e4jl1HOZ7b6ewbi4G6zH1MSw/EnOdhJz1cvXG5w hxralMK+xDFaM5qrcHAVV6teZDQ5jNtJDGlUsQoxgGNnW6RqZKM+gZAt9vZ0OT82jmkm owapxogi/qz91Uc5KIulx4HOmqSSoGedAp9fa9M85pBABlDFIszpZIc/76czM+ba3+wg Zxi4/sqlAHCrauL3TWxGvwHds8epUVYc8a8V8Tu9PajI8vYqTut7UA5V38KGVO8+B6bu uFhcShF3I3fSYqxP4acK98AXW9X9JU6rwLOAhUL2rT15pMLAra9P+/J/mhGNYivY8xdL KGXw== X-Gm-Message-State: AOAM533DgvAgFTLsykEJvVaVxEaQK1pXOVwELkW2iFYp+PtoGMcxQnkT fSQBLHCy99gwhZd3RYqy4Bl2OcCYg7M= X-Google-Smtp-Source: ABdhPJwwL5CcTyWL2wgcpS3OjzgvEQqdyRDDhApJEq3rhurAg/OeAct8e05Dxe9Pk/yaQClm+kdwOQ== X-Received: by 2002:a05:6402:8:: with SMTP id d8mr37900392edu.61.1639133996043; Fri, 10 Dec 2021 02:59:56 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:55 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 19/20] trace-cmd record: Add compression to the trace context Date: Fri, 10 Dec 2021 12:59:37 +0200 Message-Id: <20211210105938.98250-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 21937562..20a36a62 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4477,6 +4490,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -4511,7 +4530,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Fri Dec 10 10:59:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12669341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F11B4C433F5 for ; Fri, 10 Dec 2021 11:00:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240134AbhLJLDf (ORCPT ); Fri, 10 Dec 2021 06:03:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240141AbhLJLDd (ORCPT ); Fri, 10 Dec 2021 06:03:33 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3938DC0617A2 for ; Fri, 10 Dec 2021 02:59:58 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id r25so28192152edq.7 for ; Fri, 10 Dec 2021 02:59:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+wDebInUk6oMh6gDZ8LDaSc9uhgNRDbCGsd6OpAfeNE=; b=BQ4V445wBiXmmHeVAv60HD73ymkNUF6AJEU6p0bbE51yyAP+wDlzl4kFwBFJRCbLrQ /eMZcMV+FEuLxubaGcZ0pgJBKsL1oohXHc0ZWxCGMoDME5oHxq4j5mpJMcuvLjFoElFD C5rx3kLZa8V2E/TrGuWuCW5vISAHD1+WAH33rTU4qvogCWEnur+kc0TkKbIHuphPsSyU kJAXrLnDo/kToejeqS+J60q1ZJHHA0e4flBqxT2JoE8nIZz5ZTaveT3ll6MdKNPwkBdq 96t6309EF6VTG1yryUbsCU4caxHfnpq6dQK0SC7RsEK3IA3BgS38eKAW8sJsE7zOhLUV qZxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+wDebInUk6oMh6gDZ8LDaSc9uhgNRDbCGsd6OpAfeNE=; b=THMe7W5dyKYNhrs5y0uPT7Tn7mO4lYZkOMmddu/6b8MKRDUGLqHP+W6P1Ufo2m9PKE eJoumYZZ7tynGMqAQ+fGGJxnY5fB/oZRmsmboVqi97WyPmgRUBrr8njcnFuDrINHij/s 3XtsI3lYU9gpVexZuxHfFaQ/AVMwqeHhu1Td7vv5EO2SHkPAj62Kk0ZLdN9EHGz7/IaL 3kFukoUmP9xIR7b6Ghkjx4+6B733d8nXF4F7ll3oWkAMwS18RD8DAHRnAy9eFSo+ZrOr sruoVFnHET4bRzFwzapiSbIg6oMIAoHB9Q5JYApSYxwod/Nb23WJlVlooNrxk2Rni+K1 jtfA== X-Gm-Message-State: AOAM533mWxs2vrTd0/TPxj95XgY/erqAsETzmBNsA+gSdwRTTX1SqQyq VaKRSMq3w2F4DLD0zonlh3dYgcrk8F4= X-Google-Smtp-Source: ABdhPJz7/ucCxcKzfFbPmqqp2Lo1OGVCSTR4zaAmFVOJyBu8zB32vrgesrn7sDKVe8SvMkDfY8KYkg== X-Received: by 2002:a05:6402:b26:: with SMTP id bo6mr39100197edb.207.1639133996831; Fri, 10 Dec 2021 02:59:56 -0800 (PST) Received: from oberon.zico.biz.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id j17sm1320379edj.0.2021.12.10.02.59.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Dec 2021 02:59:56 -0800 (PST) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 20/20] trace-cmd report: Add new parameter for trace file compression Date: Fri, 10 Dec 2021 12:59:38 +0200 Message-Id: <20211210105938.98250-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211210105938.98250-1-tz.stoyanov@gmail.com> References: <20211210105938.98250-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 20a36a62..7ab8fec1 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5804,6 +5804,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6244,6 +6245,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6670,6 +6672,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version"); cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version"); diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",