From patchwork Tue Sep 14 13:16:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AF42C433EF for ; Tue, 14 Sep 2021 13:16:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4030A60C51 for ; Tue, 14 Sep 2021 13:16:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233077AbhINNSH (ORCPT ); Tue, 14 Sep 2021 09:18:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233025AbhINNSH (ORCPT ); Tue, 14 Sep 2021 09:18:07 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C01E3C061574 for ; Tue, 14 Sep 2021 06:16:49 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id t8so15131685wrq.4 for ; Tue, 14 Sep 2021 06:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dCwJEKOeshLtdpXMCJCbKaY22zxydQd9xaL4ctaCW2c=; b=Eo9DBLZS+wi6AHdwM3el5ACe9K7CPgI+NfxnacRwyPsJHQCbzC6AVayLbC2NVOHU7f Wj3Zfe8aW1E3lD+GOi26t9LX/Ar10mA0UvG9JJap82H813kPDNR2oZRaXr3Oh2o6GQrv m809jYoZADt0ZrqltLgo8RPjDbV0xFAKzJV7w6XNQgGkRVNGrqExGz14ecJu3sph/0/H NFVYtX+2Yeida76bXX7pE71Km/eGWAsaTllgXvPHwYmumx4ZGKQsYqsADRcZbL3PVzGH SLazez6t6iNuq2SDUw0BJh3kZ8ta5lOeHzDBxFlkhe2ELQVBwhbd9rle7m8wLeNM4cAe HeVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dCwJEKOeshLtdpXMCJCbKaY22zxydQd9xaL4ctaCW2c=; b=FJwUkaGRr5RZUzeG/AuLn7erGnSuQlVGNlKIsdJK57OfUVJyeQa1//tz2ofkQdTSjA fxgux/SCKcYxGFwtO/qV9ztd/LbMTsgS86lnQ3O+YIidBC6JHimO7J3MddtbyISHkPry CXQO9zTC84dOTZBXIHamrlUFhujj8LC3QtSQsuBM2SEwF4HzDshh3Pfd7pQC21/cTKBX uRZ3lo5vG3jyHTce6BK1vrsOqBtUG+tbg0QF7hNeKa9mDkAJMfE9hIHWt2w7dmtcSU71 d5vn0f6N2qiOBn+kudmOQRuDdF8S2GXCZEwS3jGxmG1u0mrQG/KIpcT9ffhwMI55FXgT zl3g== X-Gm-Message-State: AOAM5301QCCZVG9z2GndwPu6/zpAiTk+HueOp8bK5zzgS/SMMgvd+Q8r j4esOoke/wSTJhYS1XSYifkUAyl107Y= X-Google-Smtp-Source: ABdhPJzI6trTMUZ6faRGp9T0DsWX8GSyW1oXLZogYjJXA5L5cSs9vPObJuuTGDTI1bQQ9wBYBr+8xg== X-Received: by 2002:a5d:554e:: with SMTP id g14mr19060192wrw.48.1631625408171; Tue, 14 Sep 2021 06:16:48 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:47 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 01/20] trace-cmd library: Add support for compression algorithms Date: Tue, 14 Sep 2021 16:16:26 +0300 Message-Id: <20210914131645.3966308-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added infrastructure to trace-cmd library for compression. Introduced various new APIs to work with this new functionality: struct tracecmd_compression tracecmd_compress_init() tracecmd_compress_free() tracecmd_compress_alloc() tracecmd_compress_destroy() tracecmd_compress_block() tracecmd_uncompress_block() tracecmd_compress_reset() tracecmd_compress_read() tracecmd_compress_pread() tracecmd_compress_write() tracecmd_compress_lseek() tracecmd_compress_proto_get_name() tracecmd_compress_is_supported() tracecmd_compress_protos_get() tracecmd_compress_proto_register() tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() tracecmd_uncompress_chunk() tracecmd_load_chunks_info() The compression algorithms are not part of this patch. Added trace-cmd library constructor and destructor routines, used to initialize and free compression context. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/Makefile | 1 + .../include/private/trace-cmd-private.h | 41 + lib/trace-cmd/include/trace-cmd-local.h | 3 + lib/trace-cmd/trace-compress.c | 907 ++++++++++++++++++ lib/trace-cmd/trace-util.c | 10 + 5 files changed, 962 insertions(+) create mode 100644 lib/trace-cmd/trace-compress.c diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index 17600318..bab4322d 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -25,6 +25,7 @@ ifeq ($(VSOCK_DEFINED), 1) OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif +OBJS += trace-compress.o # Additional util objects OBJS += trace-blk-hack.o diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 045fd5a1..04b3cb41 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -36,6 +36,7 @@ int *tracecmd_add_id(int *list, int id, int len); #define FILE_VERSION_MAX 7 #define FILE_VERSION_SECTIONS 7 +#define FILE_VERSION_COMPRESSION 7 enum { RINGBUF_TYPE_PADDING = 29, @@ -154,6 +155,7 @@ enum { TRACECMD_FL_IN_USECS = (1 << 2), TRACECMD_FL_RAW_TS = (1 << 3), TRACECMD_FL_SECTIONED = (1 << 4), + TRACECMD_FL_COMPRESSION = (1 << 5), }; struct tracecmd_ftrace { @@ -487,6 +489,45 @@ void tracecmd_tsync_free(struct tracecmd_time_sync *tsync); int tracecmd_write_guest_time_shift(struct tracecmd_output *handle, struct tracecmd_time_sync *tsync); +/* --- Compression --- */ +struct tracecmd_compress_chunk { + unsigned int size; + unsigned int zsize; + off64_t zoffset; + off64_t offset; +}; +struct tracecmd_compression; +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle); +void tracecmd_compress_destroy(struct tracecmd_compression *handle); +int tracecmd_compress_block(struct tracecmd_compression *handle); +int tracecmd_uncompress_block(struct tracecmd_compression *handle); +void tracecmd_compress_reset(struct tracecmd_compression *handle); +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len); +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset); +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size); +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence); +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version); +bool tracecmd_compress_is_supported(const char *name, const char *version); +int tracecmd_compress_protos_get(char ***names, char ***versions); +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*comress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)); +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data); +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index b29236dc..bd95a1fa 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -32,6 +32,9 @@ struct data_file_write { unsigned long long doffset; }; +void tracecmd_compress_init(void); +void tracecmd_compress_free(void); + bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c new file mode 100644 index 00000000..b198b8dc --- /dev/null +++ b/lib/trace-cmd/trace-compress.c @@ -0,0 +1,907 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Copyright (C) 2021, VMware, Tzvetomir Stoyanov tz.stoyanov@gmail.com> + * + */ +#include +#include +#include +#include +#include + +#include "trace-cmd-private.h" +#include "trace-cmd-local.h" + +struct compress_proto { + struct compress_proto *next; + char *proto_name; + char *proto_version; + int weight; + + int (*compress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + int (*uncompress_block)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes); + unsigned int (*compress_size)(unsigned int bytes); + bool (*is_supported)(const char *name, const char *version); +}; + +static struct compress_proto *proto_list; + +struct tracecmd_compression { + int fd; + unsigned int capacity; + unsigned long pointer; + char *buffer; + struct compress_proto *proto; + struct tep_handle *tep; + struct tracecmd_msg_handle *msg_handle; +}; + +static int read_fd(int fd, char *dst, int len) +{ + size_t size = 0; + int r; + + do { + r = read(fd, dst+size, len); + if (r > 0) { + size += r; + len -= r; + } else + break; + } while (r > 0); + + if (len) + return -1; + return size; +} + +static long long write_fd(int fd, const void *data, size_t size) +{ + long long tot = 0; + long long w; + + do { + w = write(fd, data + tot, size - tot); + tot += w; + + if (!w) + break; + if (w < 0) + return w; + } while (tot != size); + + return tot; +} + +static long long do_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + int ret; + + if (handle->msg_handle) { + ret = tracecmd_msg_data_send(handle->msg_handle, data, size); + if (ret) + return -1; + return size; + } + return write_fd(handle->fd, data, size); +} + +static inline int buffer_extend(struct tracecmd_compression *handle, unsigned int size) +{ + int extend; + char *buf; + + if (size <= handle->capacity) + return 0; + + extend = ((size - handle->capacity) / BUFSIZ + 1) * BUFSIZ; + buf = realloc(handle->buffer, handle->capacity + extend); + if (!buf) + return -1; + handle->buffer = buf; + handle->capacity += extend; + + return 0; +} + +/** + * tracecmd_compress_lseek - Move the read/write pointer into the compression buffer + * @handle: compression handler + * @offset: number of bytes to move the pointer, can be negative or positive + * @whence: the starting position of the pointer movement, + * + * Returns the new file pointer on success, or -1 in case of an error. + */ +off64_t tracecmd_compress_lseek(struct tracecmd_compression *handle, off64_t offset, int whence) +{ + unsigned long p; + + if (!handle || !handle->buffer) + return (off64_t)-1; + + switch (whence) { + case SEEK_CUR: + p = handle->pointer + offset; + break; + case SEEK_END: + p = handle->capacity + offset; + break; + case SEEK_SET: + p = offset; + break; + default: + return (off64_t)-1; + } + + if (buffer_extend(handle, p)) + return (off64_t)-1; + + handle->pointer = p; + + return p; +} + +static int compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int s; + + if (handle->pointer + len > handle->capacity) + s = handle->capacity - handle->pointer; + else + s = len; + memcpy(dst, handle->buffer + handle->pointer, s); + + return s; +} + +/** + * tracecmd_compress_pread - pread() on compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * @offset: offset in the buffer of data to be read + * + * Read a @len of data from the compression buffer at given @offset, + * without updating the buffer pointer. + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_pread(struct tracecmd_compression *handle, char *dst, int len, off_t offset) +{ + int ret; + + if (!handle || !handle->buffer || offset > handle->capacity) + return -1; + + ret = tracecmd_compress_lseek(handle, offset, SEEK_SET); + if (ret < 0) + return ret; + return compress_read(handle, dst, len); +} + +/** + * tracecmd_compress_read - read() from compression buffer + * @handle: compression handler + * @dst: return, store the read data + * @len: length of data to be read + * + * Read a @len of data from the compression buffer + * + * On success returns the number of bytes read, or -1 on failure. + */ +int tracecmd_compress_read(struct tracecmd_compression *handle, char *dst, int len) +{ + int ret; + + if (!handle || !handle->buffer) + return -1; + + ret = compress_read(handle, dst, len); + if (ret > 0) + handle->pointer += ret; + + return ret; +} + +/** + * tracecmd_compress_reset - Reset the compression buffer + * @handle: compression handler + * + * Reset the compression buffer, any data currently in the buffer will be destroyed. + * + */ +void tracecmd_compress_reset(struct tracecmd_compression *handle) +{ + if (!handle) + return; + + free(handle->buffer); + handle->buffer = NULL; + handle->pointer = 0; + handle->capacity = 0; +} + +/** + * tracecmd_uncompress_block - uncompress a memory block + * @handle: compression handler + * + * Read compressed memory block from the file and uncompress it into internal buffer. + * The tracecmd_compress_read() can be used to read the uncompressed data from the buffer + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_uncompress_block(struct tracecmd_compression *handle) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + char *bytes = NULL; + char buf[4]; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + tracecmd_compress_reset(handle); + + if (read(handle->fd, buf, 4) != 4) + return -1; + s_compressed = tep_read_number(handle->tep, buf, 4); + if (read(handle->fd, buf, 4) != 4) + return -1; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + + handle->buffer = malloc(s_uncompressed); + if (!handle->buffer) + return -1; + bytes = malloc(s_compressed); + if (!bytes) + goto error; + + if (read_fd(handle->fd, bytes, s_compressed) < 0) + goto error; + ret = handle->proto->uncompress_block(bytes, s_compressed, + handle->buffer, &s_uncompressed); + if (ret) + goto error; + free(bytes); + handle->pointer = 0; + handle->capacity = s_uncompressed; + return 0; +error: + tracecmd_compress_reset(handle); + free(bytes); + return -1; +} + +/** + * tracecmd_compress_block - compress a memory block + * @handle: compression handler + * + * Compress the content of the internal memory buffer and write the compressed data in the file + * The tracecmd_compress_write() can be used to write data into the internal memory buffer, before + * calling this API. + * + * Returns 0 on success, or -1 in case of an error. + */ +int tracecmd_compress_block(struct tracecmd_compression *handle) +{ + unsigned int size; + char *buf; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_size || !handle->proto->compress_block) + return -1; + + size = handle->proto->compress_size(handle->pointer); + buf = malloc(size); + if (!buf) + return -1; + ret = handle->proto->compress_block(handle->buffer, handle->pointer, buf, &size); + if (ret < 0) + goto out; + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &handle->pointer, 4); + ret = do_write(handle, &endian4, 4); + if (ret != 4) + goto out; + /* Write compressed data */ + ret = do_write(handle, buf, size); + ret = ((ret == size) ? 0 : -1); +out: + tracecmd_compress_reset(handle); + free(buf); + return ret; +} + +/** + * tracecmd_compress_write - write() to compression buffer + * @handle: compression handler + * @data: data to be written + * @size: size of @data + * + * Write @data of @size in the compression buffer + * + * Returns 0 on success, or -1 on failure. + */ +int tracecmd_compress_write(struct tracecmd_compression *handle, + const void *data, unsigned long long size) +{ + if (!handle) + return -1; + + if (buffer_extend(handle, (handle->pointer + size))) + return -1; + + memcpy(&handle->buffer[handle->pointer], data, size); + handle->pointer += size; + return 0; +} + +/** + * tracecmd_compress_init - initialize the library with available compression algorithms + */ +void tracecmd_compress_init(void) +{ + struct timeval time; + + gettimeofday(&time, NULL); + srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); + +} + +static struct compress_proto *compress_proto_select(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *selected = NULL; + + while (proto) { + if (!selected || selected->weight > proto->weight) + selected = proto; + proto = proto->next; + } + + return selected; +} + +/** + * tracecmd_compress_alloc - Allocate a new compression context + * @name: name of the compression algorithm, if NULL - auto select the best available algorithm + * @version: version of the compression algorithm, can be NULL + * @fd: file descriptor for reading / writing data + * @tep: tep handler, used to encode the data + * @msg_handle: message handler, use it for reading / writing data instead of @fd + * + * Returns NULL on failure or pointer to allocated compression context. + * The returned context must be freed by tracecmd_compress_destroy() + */ +struct tracecmd_compression *tracecmd_compress_alloc(const char *name, const char *version, + int fd, struct tep_handle *tep, + struct tracecmd_msg_handle *msg_handle) +{ + struct tracecmd_compression *new; + struct compress_proto *proto; + + if (name) { + proto = proto_list; + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + break; + proto = proto->next; + } + } else { + proto = compress_proto_select(); + } + if (!proto) + return NULL; + + new = calloc(1, sizeof(*new)); + if (!new) + return NULL; + new->fd = fd; + new->tep = tep; + new->msg_handle = msg_handle; + new->proto = proto; + return new; +} + +/** + * tracecmd_compress_destroy - Free a compression context + * @handle: handle to the compression context that will be freed + */ +void tracecmd_compress_destroy(struct tracecmd_compression *handle) +{ + tracecmd_compress_reset(handle); + free(handle); +} + +/** + * tracecmd_compress_is_supported - check if compression algorithm with given name and + * version is supported + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * + * Returns true if the algorithm with given name and version is supported or false if it is not. + */ +bool tracecmd_compress_is_supported(const char *name, const char *version) +{ + struct compress_proto *proto = proto_list; + + if (!name) + return NULL; + + while (proto) { + if (proto->is_supported && proto->is_supported(name, version)) + return true; + proto = proto->next; + } + return false; +} + +/** + * tracecmd_compress_proto_get_name - get name and version of compression algorithm + * @compress: compression handler. + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_compress_proto_get_name(struct tracecmd_compression *compress, + const char **name, const char **version) +{ + if (!compress || !compress->proto) + return -1; + if (name) + *name = compress->proto->proto_name; + if (version) + *version = compress->proto->proto_version; + return 0; +} + +/** + * tracecmd_compress_proto_register - register a new compression algorithm + * @name: name of the compression algorithm. + * @version: version of the compression algorithm. + * @weight: weight of the compression algorithm, lower is better. + * @compress: compression hook, called to compress a memory block. + * @uncompress: uncompression hook, called to uncompress a memory block. + * @compress_size: hook, called to get the required minimum size of the buffer for compression + * given number of bytes. + * @is_supported: check hook, called to check if compression with given name and version is + * supported by this plugin. + * + * Returns 0 on success, or -1 in case of an error. If algorithm with given name and version is + * already registered, -1 is returned. + */ +int tracecmd_compress_proto_register(const char *name, const char *version, int weight, + int (*compress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + int (*uncompress)(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes), + unsigned int (*compress_size)(unsigned int bytes), + bool (*is_supported)(const char *name, const char *version)) +{ + struct compress_proto *new; + + if (!name || !compress || !uncompress) + return -1; + if (tracecmd_compress_is_supported(name, version)) + return -1; + + new = calloc(1, sizeof(*new)); + if (!new) + return -1; + + new->proto_name = strdup(name); + if (!new->proto_name) + goto error; + new->proto_version = strdup(version); + if (!new->proto_version) + goto error; + new->compress_block = compress; + new->uncompress_block = uncompress; + new->compress_size = compress_size; + new->is_supported = is_supported; + new->weight = weight; + new->next = proto_list; + proto_list = new; + return 0; + +error: + free(new->proto_name); + free(new->proto_version); + free(new); + return -1; +} + +/** + * tracecmd_compress_free - free the library resources, related to available compression algorithms + * + */ +void tracecmd_compress_free(void) +{ + struct compress_proto *proto = proto_list; + struct compress_proto *del; + + while (proto) { + del = proto; + proto = proto->next; + free(del->proto_name); + free(del->proto_version); + free(del); + } + proto_list = NULL; +} + +/** + * tracecmd_compress_protos_get - get a list of all supported compression algorithms and versions + * @names: return, array with names of all supported compression algorithms + * @versions: return, array with versions of all supported compression algorithms + * + * On success, the size of @names and @versions arrays is returned. Those arrays are allocated by + * the API and must be freed with free() by the caller. Both arrays are with same size, each name + * from @names corresponds to a version from @versions. + * On error -1 is returned and @names and @versions arrays are not allocated. + */ +int tracecmd_compress_protos_get(char ***names, char ***versions) +{ + struct compress_proto *proto = proto_list; + char **n = NULL; + char **v = NULL; + int c, i; + + for (c = 0; proto; proto = proto->next) + c++; + + if (c < 1) + return c; + + n = calloc(c, sizeof(char *)); + if (!n) + goto error; + v = calloc(c, sizeof(char *)); + if (!v) + goto error; + + proto = proto_list; + for (i = 0; i < c && proto; i++) { + n[i] = proto->proto_name; + v[i] = proto->proto_version; + proto = proto->next; + } + + *names = n; + *versions = v; + return c; + +error: + free(n); + free(v); + return -1; +} + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int all; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + all = 0; + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + do { + r = read(fd, buf_from + all, rchunk - all); + all += r; + + if (r <= 0) + break; + } while (all != rchunk); + + + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += all; + size = csize; + if (all > 0) { + ret = handle->proto->compress_block(buf_from, all, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &all, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (all > 0); + free(buf_from); + free(buf_to); + if (all) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_load_chunks_info - Read compression chunks information from the file + * @handle: compression handler + * @chunks_info: return, array with compression chunks information + * + * This function reads information of all compression chunks in the current compression block from + * the file and fills that information in a newly allocated array @chunks_info which is returned. + * + * On success count of compression chunks is returned. Array of that count is allocated and + * returned in @chunks_info. Each entry describes one compression chunk. On error -1 is returned. + * In case of success, @chunks_info must be freed by free(). + */ +int tracecmd_load_chunks_info(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk **chunks_info) +{ + struct tracecmd_compress_chunk *chunks = NULL; + unsigned long long size = 0; + unsigned int count = 0; + off64_t offset; + int ret = -1; + char buf[4]; + int i; + + if (!handle) + return -1; + + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (offset == (off64_t)-1) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + count = tep_read_number(handle->tep, buf, 4); + if (!count) { + ret = 0; + goto out; + } + chunks = calloc(count, sizeof(struct tracecmd_compress_chunk)); + if (!chunks) + goto out; + for (i = 0; i < count; i++) { + chunks[i].zoffset = lseek64(handle->fd, 0, SEEK_CUR); + if (chunks[i].zoffset == (off_t)-1) + goto out; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].zsize = tep_read_number(handle->tep, buf, 4); + chunks[i].offset = size; + if (read(handle->fd, buf, 4) != 4) + goto out; + chunks[i].size = tep_read_number(handle->tep, buf, 4); + size += chunks[i].size; + if (lseek64(handle->fd, chunks[i].zsize, SEEK_CUR) == (off64_t)-1) + goto out; + } + + ret = count; +out: + if (lseek64(handle->fd, offset, SEEK_SET) == (off64_t)-1) + ret = -1; + + if (ret > 0 && chunks_info) + *chunks_info = chunks; + else + free(chunks); + + return ret; +} + +/** + * tracecmd_uncompress_chunk - Uncompress given compression chunk. + * @handle: compression handler + * @chunk: chunk, that will be uncompressed in @data + * @data: Preallocated memory for uncompressed data. Must have enough space to hold + * the uncompressed data + * + * This function uncompresses the chunk described by @chunk and stores the uncompressed data in + * the preallocated memory @data. + * + * On success 0 is returned and the uncompressed data is stored in @data. On error -1 is returned. + */ +int tracecmd_uncompress_chunk(struct tracecmd_compression *handle, + struct tracecmd_compress_chunk *chunk, char *data) +{ + char *bytes_in = NULL; + unsigned int size; + int ret = -1; + + if (!handle || !handle->proto || !handle->proto->uncompress_block || !chunk || !data) + return -1; + + if (lseek64(handle->fd, chunk->zoffset + 8, SEEK_SET) == (off_t)-1) + return -1; + bytes_in = malloc(chunk->zsize); + if (!bytes_in) + return -1; + if (read_fd(handle->fd, bytes_in, chunk->zsize) < 0) + goto out; + size = chunk->size; + if (handle->proto->uncompress_block(bytes_in, chunk->zsize, data, &size)) + goto out; + ret = 0; +out: + free(bytes_in); + return ret; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out = 0; + int size_in = 0; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} diff --git a/lib/trace-cmd/trace-util.c b/lib/trace-cmd/trace-util.c index d28b6ec2..97db6ed9 100644 --- a/lib/trace-cmd/trace-util.c +++ b/lib/trace-cmd/trace-util.c @@ -625,6 +625,16 @@ bool tracecmd_is_version_supported(unsigned int version) return false; } +static void __attribute__ ((constructor)) tracecmd_lib_init(void) +{ + tracecmd_compress_init(); +} + +static void __attribute__((destructor)) tracecmd_lib_free(void) +{ + tracecmd_compress_free(); +} + __hidden bool check_file_state(unsigned long file_version, int current_state, int new_state) { switch (new_state) { From patchwork Tue Sep 14 13:16:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0CADC433F5 for ; Tue, 14 Sep 2021 13:16:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8AD5160724 for ; Tue, 14 Sep 2021 13:16:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233093AbhINNSI (ORCPT ); Tue, 14 Sep 2021 09:18:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233025AbhINNSH (ORCPT ); Tue, 14 Sep 2021 09:18:07 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84CBCC061760 for ; Tue, 14 Sep 2021 06:16:50 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id k5-20020a05600c1c8500b002f76c42214bso2057981wms.3 for ; Tue, 14 Sep 2021 06:16:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/WbC7tNCfgykiPCeK3OcLPHTUQbxsfeQD6Twyvant50=; b=D+tb9qml4avBfO7U3YEiEkvMiDM71qMmK0LB5q8N19jZtC/p1KPkwTMeW1fnQqi5Ey wKLc4Qc5z3Spn2UwVleli3FZuqZbZGKdMQ6OkqCrWhcQoUnxQvfkeiUDk3luFtfE4I3R WN9H0MpfLhxjCo06V0I0iiQFYugm3bsptXmA++sAn0TrNBV893xLaPNkF9jnszg3xVWJ 0bNTrDk1Hq1p9VWVpgA399SNIsFS1AYwbwwLqozj1t4Wqdql8IBrc2W8RSbMWzTVjzLi 7lQ7H3oplxjrQ7wYfZ0y0jHm0nH6o7XkwO9Oi/A6Mdi87hDGDMQM6zmvrhSkF6NI+rSE LKQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/WbC7tNCfgykiPCeK3OcLPHTUQbxsfeQD6Twyvant50=; b=2PMSP8l9R5Tg1EEXkTe7qrEaauzL8EJbvzPRtg5n0d45yttyA2ovSj1Hmbj7o1RCk2 rjWtFFsRirC68irguc32uydMLCX3zvrfsLA5N0IJTbkSc68vIiw06MoQfARI3pF2dULY VqsIJSbIr9vGjh64guw3z9IFlWInA+m9XTkkNyYC3Abmdm0qbJOYbKczSQYGYFH9J1S3 y7W3MTSVTxL8/SXipCS2yM5w13hm134J+J/FtgB7GzbzorYhqYni4EyhceuCRfbdaGvM FW2GsbtFJU0wE2xkfon79/ijBxUL4dWet8USyd41+jCvght/dDFyi5uiRlxd9EicXVSY gkHw== X-Gm-Message-State: AOAM533WxC9T++b4yA8I6CCyvCPvS+KzOkOkcqn7qukzVtZzyltBCZAl sDdWL/A+luudpIDY1jWQGCQCMDidx80= X-Google-Smtp-Source: ABdhPJw4aUtFXYxuNJUQDhB+BbmXaIDZV8e7H4/L3VVREV+S+aeCZDxnohSC9mwyr+6luMgDk6lJCw== X-Received: by 2002:a1c:2b04:: with SMTP id r4mr2149862wmr.168.1631625409119; Tue, 14 Sep 2021 06:16:49 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:48 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 02/20] trace-cmd library: Internal helpers for compressing data Date: Tue, 14 Sep 2021 16:16:27 +0300 Message-Id: <20210914131645.3966308-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the output trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 7 ++++ lib/trace-cmd/trace-output.c | 56 +++++++++++++++++++++++-- 2 files changed, 59 insertions(+), 4 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index bd95a1fa..304bec49 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -38,6 +38,13 @@ void tracecmd_compress_free(void); bool check_file_state(unsigned long file_version, int current_state, int new_state); bool check_out_state(struct tracecmd_output *handle, int new_state); +int out_uncompress_block(struct tracecmd_output *handle); +int out_compression_start(struct tracecmd_output *handle, bool compress); +int out_compression_end(struct tracecmd_output *handle, bool compress); +void out_compression_reset(struct tracecmd_output *handle, bool compress); +unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size); unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 645a8ab7..d9502023 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -65,6 +65,8 @@ struct tracecmd_output { unsigned long file_version; tsize_t options_start; bool big_endian; + bool do_compress; + struct tracecmd_compression *compress; struct list_head options; struct list_head buffers; @@ -89,18 +91,27 @@ struct list_event_system { static stsize_t do_write_check(struct tracecmd_output *handle, const void *data, tsize_t size) { + if (handle->do_compress) + return tracecmd_compress_write(handle->compress, data, size); if (handle->msg_handle) return tracecmd_msg_data_send(handle->msg_handle, data, size); - return __do_write_check(handle->fd, data, size); } static inline off64_t do_lseek(struct tracecmd_output *handle, off_t offset, int whence) { + if (handle->do_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); if (handle->msg_handle) return msg_lseek(handle->msg_handle, offset, whence); - else - return lseek64(handle->fd, offset, whence); + return lseek64(handle->fd, offset, whence); +} + +static inline int do_preed(struct tracecmd_output *handle, void *dst, int len, off_t offset) +{ + if (handle->do_compress) + return tracecmd_compress_pread(handle->compress, dst, len, offset); + return pread(handle->fd, dst, len, offset); } static short convert_endian_2(struct tracecmd_output *handle, short val) @@ -128,6 +139,43 @@ static unsigned long long convert_endian_8(struct tracecmd_output *handle, return tep_read_number(handle->pevent, &val, 8); } +__hidden void out_compression_reset(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return; + tracecmd_compress_reset(handle->compress); + handle->do_compress = false; +} + +__hidden int out_uncompress_block(struct tracecmd_output *handle) +{ + int ret = 0; + + if (!handle->compress) + return 0; + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->do_compress = true; + return ret; +} + +__hidden int out_compression_start(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + tracecmd_compress_reset(handle->compress); + handle->do_compress = true; + return 0; +} + +__hidden int out_compression_end(struct tracecmd_output *handle, bool compress) +{ + if (!compress || !handle->compress) + return 0; + handle->do_compress = false; + return tracecmd_compress_block(handle->compress); +} + /** * tracecmd_set_quiet - Set if to print output to the screen * @quiet: If non zero, print no output to the screen @@ -1552,7 +1600,7 @@ static int append_options_v6(struct tracecmd_output *handle) if (offset == (off_t)-1) return -1; - r = pread(handle->fd, &option, 2, offset); + r = do_preed(handle, &option, 2, offset); if (r != 2 || option != TRACECMD_OPTION_DONE) return -1; From patchwork Tue Sep 14 13:16:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C754C433FE for ; Tue, 14 Sep 2021 13:16:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 75E7860F36 for ; Tue, 14 Sep 2021 13:16:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233129AbhINNSJ (ORCPT ); Tue, 14 Sep 2021 09:18:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233025AbhINNSI (ORCPT ); Tue, 14 Sep 2021 09:18:08 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8256CC061574 for ; Tue, 14 Sep 2021 06:16:51 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id s24so9227817wmh.4 for ; Tue, 14 Sep 2021 06:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/OE6tdFd+yfVYNsUI1RW9md2K5qwYRUbMEMY+ELH0Ts=; b=kVQdZILE85pf4AsxLBqpQhNznUgPvIb3vUNnkqfdrekdU2eiNGBKoOUDS5ms5b51nJ Ci6l4UcyKeoXJXvII6FCcFNwkcvkVzo2pdl1UxMhohpfUaDQr1CjNLeJVmxF2K14EgKC Hsb1L0sfU9h3AqpA1DdOWUMAy++ZQGI0pZF56XpKgu8GrgnmD13PWi5wAtsFQEa1/DP0 HHVTrqmszU+tcSoq9aU8/iZVbSQ1JQXz48cAebc/s+jQSD0MLMo0ChzQLFWwhyeFz6gs me1uL5cGCo9KMJrRa+DvqWM8PgfiPS8S7TogTAefFWb1a+vQowimUDfNJy/YGHPJqfOV DltA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/OE6tdFd+yfVYNsUI1RW9md2K5qwYRUbMEMY+ELH0Ts=; b=V9Li9V07dEJnPCZFC43b1MZvOZClg4WqvdwcHsvDFBgv2K+YWNih7LhQcpCs2O7J9R GI1WP1UkmasfbsbdfQlmNE7AXX2y7Rk1deC1y7PrWQd/QeoqL52Lpo/FUoqJUhHaV5U8 6DqtLdnxjYW5GnrgIvcrugw3TbxiQf14821CUnAJT/vz8aHjrsvbe/igTRDmrqVSTWKA CNrqtKXe6slLR+LO5UNE+VgUpxqrZny59BxxzFJzcE6cpckkSOTg7fT5FJMP/yO8QnXP Y3jUIemzUCCd/DJeN0ugm7Uf8sFm1Bjx1g02GLTMqtYranr2AcKJjNbJx0MZCtdz03wK pbwQ== X-Gm-Message-State: AOAM533nZygUX5MK0865NPVh+Y+R3cWql1sz/LYO0PwKuJf3zR2+C1GM zuXL0AsKZSUmqMeQT7AWuICnDwbC6GI= X-Google-Smtp-Source: ABdhPJxmzKYU8tqcjhB/aqRJjjOlKjy4+y5pyze8n4OMD5TrbxQRP+zrXk9KOaGdS/3bqxVixBDO1A== X-Received: by 2002:a1c:7dc8:: with SMTP id y191mr2151285wmc.6.1631625410162; Tue, 14 Sep 2021 06:16:50 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:49 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 03/20] trace-cmd library: Internal helpers for uncompressing data Date: Tue, 14 Sep 2021 16:16:28 +0300 Message-Id: <20210914131645.3966308-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New library internal helper functions are introduced, to add compression functionality to the input trace handler. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/include/trace-cmd-local.h | 3 ++ lib/trace-cmd/trace-input.c | 49 ++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/include/trace-cmd-local.h b/lib/trace-cmd/include/trace-cmd-local.h index 304bec49..14223a08 100644 --- a/lib/trace-cmd/include/trace-cmd-local.h +++ b/lib/trace-cmd/include/trace-cmd-local.h @@ -45,6 +45,9 @@ void out_compression_reset(struct tracecmd_output *handle, bool compress); unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, int fd, unsigned long long max, unsigned long long *write_size); +void in_uncompress_reset(struct tracecmd_input *handle); +int in_uncompress_block(struct tracecmd_input *handle); + unsigned long long out_write_section_header(struct tracecmd_output *handle, unsigned short header_id, char *description, enum tracecmd_section_flags flags, bool option); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 222b9c89..3eef539f 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -155,6 +155,9 @@ struct tracecmd_input { long long ts_offset; struct tsc2nsec tsc_calc; + bool read_compress; + struct tracecmd_compression *compress; + struct host_trace_info host; double ts2secs; unsigned int cpustats_size; @@ -262,13 +265,13 @@ static const char *show_records(struct page **pages, int nr_pages) static int init_cpu(struct tracecmd_input *handle, int cpu); -static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +static ssize_t do_read_fd(int fd, void *data, size_t size) { ssize_t tot = 0; ssize_t r; do { - r = read(handle->fd, data + tot, size - tot); + r = read(fd, data + tot, size - tot); tot += r; if (!r) @@ -280,6 +283,22 @@ static ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) return tot; } +static inline int do_lseek(struct tracecmd_input *handle, int offset, int whence) +{ + if (handle->read_compress) + return tracecmd_compress_lseek(handle->compress, offset, whence); + else + return lseek(handle->fd, offset, whence); +} + +static inline ssize_t do_read(struct tracecmd_input *handle, void *data, size_t size) +{ + if (handle->read_compress) + return tracecmd_compress_read(handle->compress, data, size); + else + return do_read_fd(handle->fd, data, size); +} + static ssize_t do_read_check(struct tracecmd_input *handle, void *data, size_t size) { @@ -304,9 +323,7 @@ static char *read_string(struct tracecmd_input *handle) for (;;) { r = do_read(handle, buf, BUFSIZ); - if (r < 0) - goto fail; - if (!r) + if (r <= 0) goto fail; for (i = 0; i < r; i++) { @@ -332,7 +349,7 @@ static char *read_string(struct tracecmd_input *handle) } /* move the file descriptor to the end of the string */ - r = lseek(handle->fd, -(r - (i+1)), SEEK_CUR); + r = do_lseek(handle, -(r - (i+1)), SEEK_CUR); if (r < 0) goto fail; @@ -396,6 +413,26 @@ static int read8(struct tracecmd_input *handle, unsigned long long *size) return 0; } +__hidden void in_uncompress_reset(struct tracecmd_input *handle) +{ + if (handle->compress) { + handle->read_compress = false; + tracecmd_compress_reset(handle->compress); + } +} + +__hidden int in_uncompress_block(struct tracecmd_input *handle) +{ + int ret = 0; + + if (handle->compress) { + ret = tracecmd_uncompress_block(handle->compress); + if (!ret) + handle->read_compress = true; + } + return ret; +} + static struct file_section *section_get(struct tracecmd_input *handle, int id) { struct file_section *sec; From patchwork Tue Sep 14 13:16:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC103C433F5 for ; Tue, 14 Sep 2021 13:16:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A695860F36 for ; Tue, 14 Sep 2021 13:16:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233145AbhINNSK (ORCPT ); Tue, 14 Sep 2021 09:18:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233025AbhINNSK (ORCPT ); Tue, 14 Sep 2021 09:18:10 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F343C061574 for ; Tue, 14 Sep 2021 06:16:52 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id 196-20020a1c04cd000000b002fa489ffe1fso2053636wme.4 for ; Tue, 14 Sep 2021 06:16:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x257PmiUOYlA+KiZnBHtOFBwk3HICB+mn/P/kKYhzPw=; b=pBrm0+TSteTZ/NfQbfW5+dx/dfQ/o5Gkj3zgfSdDbBRUEy6ckpNQL3q6C0M3SoV7gf jbA/rBCIjd49VoIw//rYTY9Fit/eONmrCO7goJDZXbjtzBlfNCOk4dcfmuBGvw+zTNG5 KzptDpuqDhFlDG3lXJs7A3+QK36F5VYxDRLsDDzGbex7t21iAIroKiIBgXmXBXXcJTuH RdKL4mtjdx/b1V4ixG/Qd3oDIZda8gtUriJ1qe/j2dITN9HL7OA5Qp/y0G9mhFHT90SA +9Af4Kvc8CM+SU508WqghVM7v/xN6tzH62iRfSh8o2sqbB8YemBJIaHPYYEDNwNlNgqp jTzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x257PmiUOYlA+KiZnBHtOFBwk3HICB+mn/P/kKYhzPw=; b=IM52Jo96nYw56nnP6P4BEB5Tma7E0mRh54Bh5QfXQ+9TWHGk4kHphdBqz21djtU+dr /dVy4RCnDdCgF4mb50MIvrlVmzDPrpUOPthOD+QOR2ERuoE+FTUQ+XnldrKPg5GWozG4 UV4J+bi/Ef6EZ8lmZBaoAFI3UOuhflg6IiN2b6+hqZpIoKSEK7zF18AKEZ3913xYWm8V atrbicGy8APnVWMq6LXixzi9xrciLUzL5lRhhS0L0p8QEcyIv8CpXTWdxKemwjk+ckVA 1bd/HudGoU9CSwxWC21XLTc0F/0ar/Uj5l8X6dPB4fzPdfSQEuIKMaWOZaXbssoA8MgZ LCaQ== X-Gm-Message-State: AOAM533pkL8Mxep1kpKjAzADwEu9728CajBiFISRy/ITV2BzCOhLkE/q r19oPq4O7zw/FP0xu/VxlhkKZymOMxs= X-Google-Smtp-Source: ABdhPJzRybLOG8dkrBRcBfNRE3kuBgAP5YQW5zAOrmtHhY6tQBoW0T8SQJOwGPTodVFN04FXdqGDtQ== X-Received: by 2002:a1c:7f89:: with SMTP id a131mr2119106wmd.31.1631625411259; Tue, 14 Sep 2021 06:16:51 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:50 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 04/20] trace-cmd library: Inherit compression algorithm from input file Date: Tue, 14 Sep 2021 16:16:29 +0300 Message-Id: <20210914131645.3966308-5-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When a new trace file output handler is allocated, based on given trace file input handler - use the same compression algorithm. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 2 ++ lib/trace-cmd/trace-input.c | 16 ++++++++++++++++ lib/trace-cmd/trace-output.c | 19 +++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 04b3cb41..30315c00 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -255,6 +255,8 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu); unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle); size_t tracecmd_get_options_offset(struct tracecmd_input *handle); +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version); int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo); bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle); diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 3eef539f..11ee6470 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -4608,6 +4608,22 @@ unsigned long tracecmd_get_in_file_version(struct tracecmd_input *handle) return handle->file_version; } +/** + * tracecmd_get_file_compress_proto - get name and version of compression algorithm, + * used to compress the trace file + * @handle: input handle for the trace.dat file + * @name: return, name of the compression algorithm. + * @version: return, version of the compression algorithm. + * + * Returns 0 on success, or -1 in case of an error. If 0 is returned, the name and version of the + * algorithm are stored in @name and @version. The returned strings must *not* be freed. + */ +int tracecmd_get_file_compress_proto(struct tracecmd_input *handle, + const char **name, const char **version) +{ + return tracecmd_compress_proto_get_name(handle->compress, name, version); +} + /** * tracecmd_get_use_trace_clock - return use_trace_clock * @handle: input handle for the trace.dat file diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index d9502023..edf05cfe 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1213,6 +1213,9 @@ int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *ka */ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle) { + const char *cname = NULL; + const char *cver = NULL; + if (!handler || !ihandle || handler->file_state != TRACECMD_FILE_ALLOCATED) return -1; @@ -1227,6 +1230,14 @@ int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct trace else handler->big_endian = false; + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handler->compress = tracecmd_compress_alloc(cname, cver, handler->fd, + handler->pevent, handler->msg_handle); + if (!handler->compress) + return -1; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->file_version = FILE_VERSION_COMPRESSION; + } return 0; } @@ -2166,6 +2177,8 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) { struct tracecmd_output *handle = NULL; struct tracecmd_input *ihandle; + const char *cname = NULL; + const char *cver = NULL; int fd2; /* Move the file descriptor to the beginning */ @@ -2205,6 +2218,12 @@ struct tracecmd_output *tracecmd_get_output_handle_fd(int fd) list_head_init(&handle->options); list_head_init(&handle->buffers); + if (!tracecmd_get_file_compress_proto(ihandle, &cname, &cver)) { + handle->compress = tracecmd_compress_alloc(cname, cver, handle->fd, + handle->pevent, handle->msg_handle); + if (!handle->compress) + goto out_free; + } tracecmd_close(ihandle); return handle; From patchwork Tue Sep 14 13:16:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB41AC43217 for ; Tue, 14 Sep 2021 13:16:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C349F60724 for ; Tue, 14 Sep 2021 13:16:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233025AbhINNSL (ORCPT ); Tue, 14 Sep 2021 09:18:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233172AbhINNSK (ORCPT ); Tue, 14 Sep 2021 09:18:10 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DBF2C061574 for ; Tue, 14 Sep 2021 06:16:53 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id d21so12468775wra.12 for ; Tue, 14 Sep 2021 06:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RfqiSNfAxwO7yHzNNGqqUA/eKrI3uCiNUxgmJyywPk8=; b=expFe/il4E14UNai57r+PXzGTRNzjPdgwhvxUUYM32KQyNSkFf27jKDUNX1V5XgBED l6aXmwFP8ro3zI+dJhdlZSqrQQ868CJdgsjUK6vCBfnfdW1KfORKTruH2nBn8e8f9O1f mJ6/j1E4GKvnGlmOmZHTyDWZ+20EoCX3pYGa/qJGdDFCP19A6Y1R+gjlfp5iVTGsQ0tw 7j2te6VjtN7sZWpC5IcFdYDXywUL039Yuq9UMJuiU6dqBzLxXtBCVM5cOmcP0AFO218v UiX6Yz7MQehpxakwkNvDmz5yx1m+a7K2LPzitc3S9FC33BlJehZJHT7291f8srO7o9iX KiXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RfqiSNfAxwO7yHzNNGqqUA/eKrI3uCiNUxgmJyywPk8=; b=CalnZG9zhhdjnn4X0IGc9PeeqAazrUCKd7jXGE9AefR7umkK2oQNC6eMRCXiR3DS+o Yg7u6YfVyX1I7eKYZl/4c2ZDhqwwYc+VG481auAgDS5ypyrjaMedyikKmmQLqWcPZYuV vBfa5vEFVGGwogMeJth0e7aom17B2nXUUFIQJ4AKRNarVfz+tpN9BzX7wx2YXO8O28Gk YgzCdZcAllGh/Pp68O0OfVNt3o+IhM02YKHLsOiRKyMlrw6WKCVCxtzOcHt3dnrkZhnu 3DS17mYHoZZEOQMCD1iIBJM04UBRtNNG4Nc77Fw3CeJBTGkfUi3u47VwNjA2m3ivfKr3 OPeg== X-Gm-Message-State: AOAM531mi9S09dESaWNdeWeg+HQiTidFhplOvFx1VRALlPeHfN6hTd+m tDUmQn2+F09sONuKrL02EnLyqoZiDGQ= X-Google-Smtp-Source: ABdhPJytkGjtJ4n060uoZGAOT15Ugf3/7ZqI1gd4TE3lAEqSIfBZLNFYiYpRO+iqGmOvL1Qwhvn1hg== X-Received: by 2002:a05:6000:1623:: with SMTP id v3mr19124788wrb.288.1631625412206; Tue, 14 Sep 2021 06:16:52 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:51 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 05/20] trace-cmd library: New API to configure compression on an output handler Date: Tue, 14 Sep 2021 16:16:30 +0300 Message-Id: <20210914131645.3966308-6-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The new API can be used to configure compression algorithm on an output handler to a trace file. tracecmd_output_set_compression() The API for creation of latency trace file is extended with compression parameter. Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 3 +- lib/trace-cmd/trace-output.c | 57 ++++++++++++++++++- tracecmd/trace-record.c | 2 +- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 30315c00..8f5408ff 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -297,12 +297,13 @@ int tracecmd_output_set_trace_dir(struct tracecmd_output *handler, const char *t int tracecmd_output_set_kallsyms(struct tracecmd_output *handler, const char *kallsyms); int tracecmd_output_set_from_input(struct tracecmd_output *handler, struct tracecmd_input *ihandle); int tracecmd_output_set_version(struct tracecmd_output *handler, int file_version); +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression); int tracecmd_output_write_init(struct tracecmd_output *handler); int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list); struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version); + int file_version, const char *compression); struct tracecmd_output *tracecmd_create_init_fd(int fd); struct tracecmd_output *tracecmd_create_init_file(const char *output_file); diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index edf05cfe..c15c6979 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -234,6 +234,7 @@ void tracecmd_output_free(struct tracecmd_output *handle) free(option); } free(handle->trace_clock); + tracecmd_compress_destroy(handle->compress); free(handle); } @@ -1258,9 +1259,57 @@ int tracecmd_output_set_version(struct tracecmd_output *handler, int file_versio if (file_version < FILE_VERSION_MIN || file_version > FILE_VERSION_MAX) return -1; handler->file_version = file_version; + if (handler->file_version < FILE_VERSION_COMPRESSION) + handler->compress = NULL; return 0; } +/** + * tracecmd_output_set_compression - Set file compression algorithm of the output handler + * @handle: output handler to a trace file. + * @compression: name of the desired compression algorithm. Can be one of: + * - "none" - do not use compression + * - "all" - use the best available compression algorithm + * - or specific name of the desired compression algorithm + * + * This API must be called before tracecmd_output_write_init(). + * + * Returns 0 on success, or -1 in case of an error: + * - the output file handler is not allocated or not in expected state. + * - the specified compression algorithm is not available + */ +int tracecmd_output_set_compression(struct tracecmd_output *handler, const char *compression) +{ + if (!handler || handler->file_state != TRACECMD_FILE_ALLOCATED) + return -1; + + handler->compress = NULL; + if (compression && strcmp(compression, "none")) { + if (!strcmp(compression, "any")) { + handler->compress = tracecmd_compress_alloc(NULL, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) + tracecmd_warning("No compression algorithms are supported"); + } else { + handler->compress = tracecmd_compress_alloc(compression, NULL, handler->fd, + handler->pevent, + handler->msg_handle); + if (!handler->compress) { + tracecmd_warning("Compression algorithm %s is not supported", + compression); + return -1; + } + } + } + if (handler->compress && handler->file_version < FILE_VERSION_COMPRESSION) { + handler->file_version = FILE_VERSION_COMPRESSION; + if (handler->msg_handle) + tracecmd_msg_handle_cache(handler->msg_handle); + } + + return 0; +} /** * tracecmd_output_write_init - Write the initial magics in the trace file @@ -1846,7 +1895,7 @@ out_add_buffer_option_v7(struct tracecmd_output *handle, const char *name, } struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, int cpus, - int file_version) + int file_version, const char *compression) { enum tracecmd_section_flags flags = 0; struct tracecmd_output *handle; @@ -1863,6 +1912,12 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in goto out_free; if (file_version && tracecmd_output_set_version(handle, file_version)) goto out_free; + if (compression) { + if (tracecmd_output_set_compression(handle, compression)) + goto out_free; + } else if (file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(handle, "any"); + } if (tracecmd_output_write_init(handle)) goto out_free; if (tracecmd_output_write_headers(handle, NULL)) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 421bb7d4..fb6de81b 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -4516,7 +4516,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version); + ctx->file_version, NULL); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Tue Sep 14 13:16:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C422AC433F5 for ; Tue, 14 Sep 2021 13:16:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AFBB960724 for ; Tue, 14 Sep 2021 13:16:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233182AbhINNSM (ORCPT ); Tue, 14 Sep 2021 09:18:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233172AbhINNSL (ORCPT ); Tue, 14 Sep 2021 09:18:11 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 950FFC061574 for ; Tue, 14 Sep 2021 06:16:54 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id t18so20211982wrb.0 for ; Tue, 14 Sep 2021 06:16:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fxMT8pTuiX7OpXgchCrCJXn/ox+XMUxVn/EbfGJgxww=; b=ZcY52P61wrJvCt7eSljJ7p8LL0s08VSbaxMgVFONW5flO2l8bgIOJHyCK6bePfnjk9 S/N1QVRRYqxnQ6ZUsCQGgaT14njPhT+DilzKom3xBdoJTMGpVsVKjuDBdy1X3yK01/7a MkY1fDbhRuRbpqZEjYoxJjw1IUTLYobXmjOzVEYCarVULPdkyA6aDzvxC10JFYN/B/M3 MgLbMBNEkuWB/TvFkl8S5eUS/TP3x5E5aLSKSuBtwiEZtz1I864aDXAh50nwSJLaot11 ED0uvja9IJTzwmbg9N9BNHibh3+ZmYVaveFLs5f3lpGs/b50tA1J0M+s0bNo69szlCQK QboA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fxMT8pTuiX7OpXgchCrCJXn/ox+XMUxVn/EbfGJgxww=; b=s2g5MdV8VqPI+XwTo6PgmvXn2AsKZfSLALZpf8GbIR6sP+Afkfp/hvR/1pv5w1OIEY BG6DZtoblw6ncnnmSdYqsn3d8ljHyTwLMrd44gInQu1+Fh5ESFQs+bUB2LQ0H4UWHlpn waPiaaKT6vwqaFjqS+7A8kevPRimuz4M3z+g2bRhwsxqcJ7CkT5T8kNwszZW8flMPiHT KXjexED/vhU+72JNVFQQYkti+DMgF/MT7Hk87amAMrs2qTgFXU2WPZZXqUDo8uUhXbfV dEXhy0f20IDcIQ9IYekV6HI2AvIeBFqA35MwPcrJJjGbexFD/2sdR3O/Mh2C7hobEYmj qqOg== X-Gm-Message-State: AOAM532e60JxiyomOKYN1Cz7Ifq6okiGIU4fyslx/IiV4UkBKfqp6e4W PXfjQih9loiw4Z65WumWQJIyRY6thls= X-Google-Smtp-Source: ABdhPJwylErFH7x7Kbb2JMiU/yzx47l9Km3aT0bBFbTZOLddJ5uyzQrf+MAsEUfxtYnpWZY4fU4TOQ== X-Received: by 2002:a5d:61c1:: with SMTP id q1mr17099246wrv.154.1631625413257; Tue, 14 Sep 2021 06:16:53 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:52 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 06/20] trace-cmd library: Write compression header in the trace file Date: Tue, 14 Sep 2021 16:16:31 +0300 Message-Id: <20210914131645.3966308-7-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If there is a compression configured on the output file handler and if the file version is at least 7, write compression header in the file. The compression header is two null terminated strings - name and version of the compression algorithm, used to compress some parts of the file. The header is located after the page size in the file. The new header is mandatory for trace files version 7. If no compression is used, the string "none" is saved as name of the compression algorithm and empty string as compression algorithm version. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index c15c6979..d087b1ba 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -1076,6 +1076,24 @@ out_free: return ret; } +static int write_compression_header(struct tracecmd_output *handle) +{ + const char *name = NULL; + const char *ver = NULL; + int ret; + + ret = tracecmd_compress_proto_get_name(handle->compress, &name, &ver); + if (ret < 0 || !name || !ver) { + name = "none"; + ver = ""; + } + if (do_write_check(handle, name, strlen(name) + 1)) + return -1; + if (do_write_check(handle, ver, strlen(ver) + 1)) + return -1; + return 0; +} + /** * tracecmd_output_allocate - allocate new output handler to a trace file * @handle: file descriptor to an empty file, it can be -1 if the handler @@ -1362,6 +1380,10 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) endian4 = convert_endian_4(handler, handler->page_size); if (do_write_check(handler, &endian4, 4)) return -1; + if (handler->file_version >= FILE_VERSION_COMPRESSION) { + if (write_compression_header(handler)) + return -1; + } if (HAS_SECTIONS(handler)) { /* Write 0 as options offset and save its location */ offset = 0; From patchwork Tue Sep 14 13:16:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E405EC433EF for ; Tue, 14 Sep 2021 13:16:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA57C604DC for ; Tue, 14 Sep 2021 13:16:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233172AbhINNSN (ORCPT ); Tue, 14 Sep 2021 09:18:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233143AbhINNSN (ORCPT ); Tue, 14 Sep 2021 09:18:13 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB7D4C061574 for ; Tue, 14 Sep 2021 06:16:55 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 196-20020a1c04cd000000b002fa489ffe1fso2053788wme.4 for ; Tue, 14 Sep 2021 06:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7mj5sKFfRAlnMRQNcOoFxrVGCprv71RGgPKPLFewszg=; b=AmrDTS5RTyUov7wWQrkBG/uJWCoXlTjQOKfta10qShIGrhUYr2ktJGlMCEKwSHhm3Z Bt+WpX/S47wcIj9/rbDxS2G2bp7MNhQL8tHD5kcON05m+dBJRfclrwxrzNkcexH2cZVh HR2gkvGvn15Q5HFptiHnKZZSB/3p3su6oOfCXKYSspdaRMQHiafD+HTS1HeJYLtKAxoa UCUKJpCqBsnitiXILR5tLxyETvAwXH7qESkYEv6mVePqYR8W2Xc6FRmc/+xrhagwLAMe njrNiLKGsONASLdBx/Ngp1c25GrgUcImXp2le/eWTs4g6HzimlLmyh3Aq5YbPyYC9xEK rqhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7mj5sKFfRAlnMRQNcOoFxrVGCprv71RGgPKPLFewszg=; b=ejY+aTzqIN0qXIxvoGt7z48/06OdnRhACitDLoEdvkeY3pVme+lefP9GcZON1GOx5f sq3CrGkLukO8PHTPI3ZVytQgDdvKW3ktjzPWdYJypgVTsXqWHr+1JdnqD4cXF238NaZm TrDBjdGIZ4yjs9Ec0lzbuzId2k6l4mKBVh5f7a0R+W639s9Q5HMsJAUKW55bIN8F/Ear Moy/qjB6+z47cy5PrWS8QzgBvCTYnZlC0RgtAWxqFl/021olSPl+12ZEUXg1TlNENdeq wFmtWOQ1cO1mlLqrnmHJDbkVlS7Tg94XzfgQgheLhuKWc94CTdZcIXoT8SSxvnRI+ZIL u4bA== X-Gm-Message-State: AOAM5312cnkmP5MZBaI4G8gYhCoiN5gtO8SpzASlCgCTE1gRvzjh/p/k Gz74pn0Ody7Gt0YYZ0UBBNjco7dC1EA= X-Google-Smtp-Source: ABdhPJzjKc30nfutwP5jRY5FiUhD9Aa8aUjiScMhhQ6KghrnBBUV2D0xwfwfnl0n9abWy//H7qel0w== X-Received: by 2002:a05:600c:a0b:: with SMTP id z11mr2174175wmp.147.1631625414298; Tue, 14 Sep 2021 06:16:54 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:53 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 07/20] trace-cmd library: Compress part of the trace file Date: Tue, 14 Sep 2021 16:16:32 +0300 Message-Id: <20210914131645.3966308-8-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Compress part of the trace.dat file metadata. If there is compression support, compress these parts of the file: - ftrace events format - format of recorded events - information of the mapping of function addresses to the function names - trace_printk() format strings - information of the mapping a PID to a process name Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 81 +++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 14 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index d087b1ba..9c80042f 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -394,6 +394,8 @@ out_write_section_header(struct tracecmd_output *handle, unsigned short header_i return -1; if (!HAS_SECTIONS(handle)) return 0; + if (!handle->compress) + flags &= ~TRACECMD_SEC_FL_COMPRESS; offset = do_lseek(handle, 0, SEEK_CUR); if (option) { endian8 = convert_endian_8(handle, offset); @@ -449,7 +451,7 @@ __hidden int out_update_section_header(struct tracecmd_output *handle, unsigned return 0; } -static int read_header_files(struct tracecmd_output *handle) +static int read_header_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; tsize_t size, check_size, endian8; @@ -469,11 +471,14 @@ static int read_header_files(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_HEADER_INFO, "headers", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* old style did not show this info, just add zero */ @@ -487,6 +492,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; if (do_write_check(handle, &size, 8)) goto out_close; + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; return 0; @@ -539,6 +546,8 @@ static int read_header_files(struct tracecmd_output *handle) goto out_close; } put_tracing_file(path); + if (out_compression_end(handle, compress)) + goto out_close; if (out_update_section_header(handle, offset)) goto out_close; handle->file_state = TRACECMD_FILE_HEADERS; @@ -546,6 +555,7 @@ static int read_header_files(struct tracecmd_output *handle) return 0; out_close: + out_compression_reset(handle, compress); if (fd >= 0) close(fd); return -1; @@ -774,7 +784,7 @@ create_event_list_item(struct tracecmd_output *handle, tracecmd_warning("Insufficient memory"); } -static int read_ftrace_files(struct tracecmd_output *handle) +static int read_ftrace_files(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems = NULL; @@ -788,15 +798,20 @@ static int read_ftrace_files(struct tracecmd_output *handle) return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_FTRACE_EVENTS, "ftrace events", flags, true); if (offset == (off64_t)-1) return -1; create_event_list_item(handle, &systems, &list); - + out_compression_start(handle, compress); ret = copy_event_system(handle, systems); - + if (!ret) + ret = out_compression_end(handle, compress); + else + out_compression_reset(handle, compress); free_list_events(systems); if (ret) return ret; @@ -822,7 +837,7 @@ create_event_list(struct tracecmd_output *handle, } static int read_event_files(struct tracecmd_output *handle, - struct tracecmd_event_list *event_list) + struct tracecmd_event_list *event_list, bool compress) { enum tracecmd_section_flags flags = 0; struct list_event_system *systems; @@ -840,6 +855,8 @@ static int read_event_files(struct tracecmd_output *handle, return -1; } + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_EVENT_FORMATS, "events format", flags, true); if (offset == (off64_t)-1) @@ -860,7 +877,7 @@ static int read_event_files(struct tracecmd_output *handle, for (slist = systems; slist; slist = slist->next) count++; - + out_compression_start(handle, compress); ret = -1; endian4 = convert_endian_4(handle, count); if (do_write_check(handle, &endian4, 4)) @@ -875,6 +892,9 @@ static int read_event_files(struct tracecmd_output *handle, } ret = copy_event_system(handle, slist); } + if (ret) + goto out_free; + ret = out_compression_end(handle, compress); if (ret) goto out_free; ret = out_update_section_header(handle, offset); @@ -882,6 +902,8 @@ static int read_event_files(struct tracecmd_output *handle, out_free: if (!ret) handle->file_state = TRACECMD_FILE_ALL_EVENTS; + else + out_compression_reset(handle, compress); free_list_events(systems); @@ -928,7 +950,7 @@ err: tracecmd_warning("can't set kptr_restrict"); } -static int read_proc_kallsyms(struct tracecmd_output *handle) +static int read_proc_kallsyms(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -946,11 +968,14 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) if (handle->kallsyms) path = handle->kallsyms; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_KALLSYMS, "kallsyms", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -976,14 +1001,19 @@ static int read_proc_kallsyms(struct tracecmd_output *handle) } set_proc_kptr_restrict(1); + ret = out_compression_end(handle, compress); + if (ret) + goto out; ret = out_update_section_header(handle, offset); out: if (!ret) handle->file_state = TRACECMD_FILE_KALLSYMS; + else + out_compression_reset(handle, compress); return ret; } -static int read_ftrace_printk(struct tracecmd_output *handle) +static int read_ftrace_printk(struct tracecmd_output *handle, bool compress) { enum tracecmd_section_flags flags = 0; unsigned int size, check_size, endian4; @@ -1002,10 +1032,13 @@ static int read_ftrace_printk(struct tracecmd_output *handle) if (!path) return -1; + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_PRINTK, "printk", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); ret = stat(path, &st); if (ret < 0) { /* not found */ @@ -1028,12 +1061,15 @@ static int read_ftrace_printk(struct tracecmd_output *handle) out: put_tracing_file(path); + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; handle->file_state = TRACECMD_FILE_PRINTK; return 0; fail: put_tracing_file(path); + out_compression_reset(handle, compress); return -1; } @@ -1414,21 +1450,25 @@ int tracecmd_output_write_init(struct tracecmd_output *handler) int tracecmd_output_write_headers(struct tracecmd_output *handler, struct tracecmd_event_list *list) { + bool compress = false; + if (!handler || handler->file_state < TRACECMD_FILE_ALLOCATED) return -1; /* Write init data, if not written yet */ if (handler->file_state < TRACECMD_FILE_INIT && tracecmd_output_write_init(handler)) return -1; - if (read_header_files(handler)) + if (handler->compress) + compress = true; + if (read_header_files(handler, compress)) return -1; - if (read_ftrace_files(handler)) + if (read_ftrace_files(handler, compress)) return -1; - if (read_event_files(handler, list)) + if (read_event_files(handler, list, compress)) return -1; - if (read_proc_kallsyms(handler)) + if (read_proc_kallsyms(handler, compress)) return -1; - if (read_ftrace_printk(handler)) + if (read_ftrace_printk(handler, compress)) return -1; return 0; } @@ -1799,6 +1839,7 @@ static tsize_t get_buffer_file_offset(struct tracecmd_output *handle, const char int tracecmd_write_cmdlines(struct tracecmd_output *handle) { enum tracecmd_section_flags flags = 0; + bool compress = false; tsize_t offset; int ret; @@ -1808,14 +1849,26 @@ int tracecmd_write_cmdlines(struct tracecmd_output *handle) return -1; } + if (handle->compress) + compress = true; + + if (compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_CMDLINES, "command lines", flags, true); if (offset == (off64_t)-1) return -1; + out_compression_start(handle, compress); + ret = save_tracing_file_data(handle, "saved_cmdlines"); - if (ret < 0) + if (ret < 0) { + out_compression_reset(handle, compress); return ret; + } + + if (out_compression_end(handle, compress)) + return -1; if (out_update_section_header(handle, offset)) return -1; From patchwork Tue Sep 14 13:16:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9761C433FE for ; Tue, 14 Sep 2021 13:16:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF66160C51 for ; Tue, 14 Sep 2021 13:16:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233105AbhINNSO (ORCPT ); Tue, 14 Sep 2021 09:18:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSO (ORCPT ); Tue, 14 Sep 2021 09:18:14 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3973C061574 for ; Tue, 14 Sep 2021 06:16:56 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id d21so12469014wra.12 for ; Tue, 14 Sep 2021 06:16:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ga6slhVdV+CGvPHwW+wkYug19e5c110ATjll1l/h8/I=; b=S0/zmU2gawfVRUcRRsk1y5NURhR5vwXKpEa8vf9g7PeX93fjFlepcL08lORvxe5jW+ 5N7QdD7W76omtuoUKEVlDV5e1NxmmeXIBJeAupow5x8qPAUqe+5ZmjxurfuwpoWIe0p1 MgTPdXsBa8sjCmWdMTgd3waU9+NRcC+m+rO8x46a6baI3iJUPVtVb+XsEHgrw9XhbMi6 hrkEeJAXlh3JDg8naXA00w0hAYy0cM7Wt9iURhTooZuWobRDk6saINsAu099QqfWibOx 1l1JCMeWtAi7HvZdR56TQHk0qIYEms+vipibA7R7F22G2rqDogeUYOSuB6mAOyglkJ9+ PP9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ga6slhVdV+CGvPHwW+wkYug19e5c110ATjll1l/h8/I=; b=PdNwTmSEKHfbD6xJHSsXDbA11RdB0wb4BI/AVbhT1gFhoitO6lewG+IPAmhtQw0lmO G3ahvaIomcoYpGL3QGq0ErCR3bjBDTKTV4g4wBeZMahNBaao41jDGxFqNpOW+oq3zwv7 t7bRcqEgVG2N9Xjk9xsqG8UvxbZojRwRiwidwQrssqJsM+EQ5aUgJA5AI11gmx0tRF9p Z8G2OOKH5Gr4u9KDQWxmhoZFig8vXiA1ka6F9Z/ShYlXSNjutIGlAeZAlXL3Zn3Kqe6a vt/gLzyEidhIhkK9kNwJw0/SppeX70VaqDc7u9SmbFnh793Hg/NgvpMozRYVpWluiQSX K1Ew== X-Gm-Message-State: AOAM531EE6FhEgtH+41ATvcP5+UofTdR1hAEsGOl7Rvkw+cDZxQnLRId qAkjNpKkAjnET+oyastAWFaq7agAUNs= X-Google-Smtp-Source: ABdhPJxexdSJWNgcbbSiN4qE7JmbSgNo6Wol3STYw4+tuVODF045MuaVENN2YiY4FsBojr0mrw+TLA== X-Received: by 2002:a5d:66d1:: with SMTP id k17mr14330432wrw.200.1631625415312; Tue, 14 Sep 2021 06:16:55 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:54 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 08/20] trace-cmd library: Add local helper function for data compression Date: Tue, 14 Sep 2021 16:16:33 +0300 Message-Id: <20210914131645.3966308-9-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The newly added helper functions read data from a file and compress it, before writing into the trace file. The trace data is compressed in chunks, which are page aligned. A new local define is introduced: PAGES_IN_CHUNK which can be used to tune how big a compression chunk is. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 71 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 9c80042f..3fafdb1f 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -283,18 +283,27 @@ static unsigned long get_size(const char *file) return size; } -static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd) +static tsize_t copy_file_fd(struct tracecmd_output *handle, int fd, unsigned long long max) { + tsize_t rsize = BUFSIZ; tsize_t size = 0; char buf[BUFSIZ]; stsize_t r; do { - r = read(fd, buf, BUFSIZ); + if (max && rsize > max) + rsize = max; + + r = read(fd, buf, rsize); if (r > 0) { size += r; if (do_write_check(handle, buf, r)) return 0; + if (max) { + max -= r; + if (!max) + break; + } } } while (r > 0); @@ -312,12 +321,62 @@ static tsize_t copy_file(struct tracecmd_output *handle, tracecmd_warning("Can't read '%s'", file); return 0; } - size = copy_file_fd(handle, fd); + size = copy_file_fd(handle, fd, 0); close(fd); return size; } +#define PAGES_IN_CHUNK 10 +__hidden unsigned long long out_copy_fd_compress(struct tracecmd_output *handle, + int fd, unsigned long long max, + unsigned long long *write_size) +{ + unsigned long long rsize = 0; + unsigned long long wsize = 0; + unsigned long long size; + int ret; + + if (handle->compress) { + rsize = max; + ret = tracecmd_compress_copy_from(handle->compress, fd, + PAGES_IN_CHUNK * handle->page_size, + &rsize, &wsize); + if (ret < 0) + return 0; + + size = rsize; + if (write_size) + *write_size = wsize; + } else { + size = copy_file_fd(handle, fd, max); + if (write_size) + *write_size = size; + } + + return size; +} + +static tsize_t copy_file_compress(struct tracecmd_output *handle, + const char *file, unsigned long long *write_size) +{ + int ret; + int fd; + + fd = open(file, O_RDONLY); + if (fd < 0) { + tracecmd_warning("Can't read '%s'", file); + return 0; + } + + ret = out_copy_fd_compress(handle, fd, 0, write_size); + if (!ret) + tracecmd_warning("Can't compress '%s'", file); + + close(fd); + return ret; +} + /* * Finds the path to the debugfs/tracing * Allocates the string and stores it. @@ -513,7 +572,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s' size=%lld read=%lld", path, size, check_size); @@ -539,7 +598,7 @@ static int read_header_files(struct tracecmd_output *handle, bool compress) endian8 = convert_endian_8(handle, size); if (do_write_check(handle, &endian8, 8)) goto out_close; - check_size = copy_file_fd(handle, fd); + check_size = copy_file_fd(handle, fd, 0); close(fd); if (size != check_size) { tracecmd_warning("wrong size for '%s'", path); @@ -2177,7 +2236,7 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd); + read_size = copy_file_fd(handle, data[i].fd, data[i].size); if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", From patchwork Tue Sep 14 13:16:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAF18C433EF for ; Tue, 14 Sep 2021 13:16:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C9A8460F38 for ; Tue, 14 Sep 2021 13:16:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233143AbhINNSP (ORCPT ); Tue, 14 Sep 2021 09:18:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSP (ORCPT ); Tue, 14 Sep 2021 09:18:15 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC196C061574 for ; Tue, 14 Sep 2021 06:16:57 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id e26so9255409wmk.2 for ; Tue, 14 Sep 2021 06:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1xeFo4S6vZ7qlDDHpiyWHcYeuT+Npnk8w3qOqgBkQtg=; b=kRgQll3y1oZaHKyaDF81iQAFhBZI2m8vZ+CGLNHLS37/ip3uuX0oBH+7RRB/GhckrQ r+47ZvLjv80qrGOrwVL4n5ranUev28vVhe1+jH57Ii7TyhrRJ7ctZIfY/vx2PyS9QnTJ Z+5rL62lj/+LaqUp/YgWcvfz9eLcDR9w0/vm8yqDtq9HEvxqRrBbzmkve2nn0kfBKZRU iBHL3+NEEWUlt4CBI+0KPvctq4GTHVr2+7ghKAr6yIsWg8m+bU3odtGPKm3Tm/gYSJwP iFdXWXeKx3yG2++eSBaRau/1slAoeHgkaNI2+yJYlZxAiqOmCsL5TeaOPXwCaE+Z3u87 4bCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1xeFo4S6vZ7qlDDHpiyWHcYeuT+Npnk8w3qOqgBkQtg=; b=2Czm5Kw7OMYln96SvO83XRaHkdCFtu08gG8IdNTCA0ZLW1w+3LiTqHlXG4Gd4rg7aJ vXDmewXErJNuIA7Tzn8aw4NvdHJ9EW/12idTvbx/qqJWSuMFMkzngx3zKbCo/TSPt2w+ 5JVnk1T3/ku+6DELy571gJHBelcrA3Dq1PNmfmwe4t9t0Pnb5dPUzfKWzByNwQRXSvt+ Ssn2WnnKN3o7cFOi3kUVpP8KFGmgqkXsBAu/Qh+BhJIdFAFZzaUE+bhaFUDjx/0/djZe KTzzEk6CiT2t4dJCrbo4e+vN+G1xfgCJlTMzfpP5+nd1HXV9ecErSqGyJTpjHlPct4G0 fDKw== X-Gm-Message-State: AOAM5320wTiLAADPmLDTANS9CQBS1BgzaS0xGnSU4ZSY6AHsob2c6RVk 6xxzWzla008KSSEJ6Ik0W8CccJMkZpc= X-Google-Smtp-Source: ABdhPJztcXsVXWCY3Psnp7mle22us6ulhPfkj+983pTj3h3N6oscejfharTVFHHji70zZjqZ9cTDZw== X-Received: by 2002:a1c:9d4e:: with SMTP id g75mr2203595wme.20.1631625416319; Tue, 14 Sep 2021 06:16:56 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:55 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 09/20] trace-cmd library: Compress the trace data Date: Tue, 14 Sep 2021 16:16:34 +0300 Message-Id: <20210914131645.3966308-10-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org If the output file handler supports compression, use it to compress the flyrecord and latency trace data. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-output.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-output.c b/lib/trace-cmd/trace-output.c index 3fafdb1f..33fb812b 100644 --- a/lib/trace-cmd/trace-output.c +++ b/lib/trace-cmd/trace-output.c @@ -2086,11 +2086,13 @@ struct tracecmd_output *tracecmd_create_file_latency(const char *output_file, in if (HAS_SECTIONS(handle) && !out_add_buffer_option_v7(handle, "", TRACECMD_OPTION_BUFFER_TEXT, offset, 0, NULL)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER_TEXT, "buffer latency", flags, false); - copy_file(handle, path); + copy_file_compress(handle, path, NULL); if (out_update_section_header(handle, offset)) goto out_free; @@ -2191,6 +2193,8 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (!HAS_SECTIONS(handle) && do_write_check(handle, "flyrecord", 10)) goto out_free; + if (handle->compress) + flags |= TRACECMD_SEC_FL_COMPRESS; if (asprintf(&str, "buffer flyrecord %s", buff_name) < 1) goto out_free; offset = out_write_section_header(handle, TRACECMD_OPTION_BUFFER, str, flags, false); @@ -2236,14 +2240,15 @@ __hidden int out_write_cpu_data(struct tracecmd_output *handle, if (data[i].size) { if (lseek64(data[i].fd, data[i].offset, SEEK_SET) == (off64_t)-1) goto out_free; - read_size = copy_file_fd(handle, data[i].fd, data[i].size); + read_size = out_copy_fd_compress(handle, data[i].fd, + data[i].size, &data_files[i].write_size); + if (read_size != data_files[i].file_size) { errno = EINVAL; tracecmd_warning("did not match size of %lld to %lld", read_size, data_files[i].file_size); goto out_free; } - data_files[i].write_size = read_size; } else { data_files[i].write_size = 0; } From patchwork Tue Sep 14 13:16:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2670CC433FE for ; Tue, 14 Sep 2021 13:17:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0F5D660C51 for ; Tue, 14 Sep 2021 13:17:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233195AbhINNSQ (ORCPT ); Tue, 14 Sep 2021 09:18:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSQ (ORCPT ); Tue, 14 Sep 2021 09:18:16 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1758C061574 for ; Tue, 14 Sep 2021 06:16:58 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id u16so20167075wrn.5 for ; Tue, 14 Sep 2021 06:16:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HVAcQFUvhJRbVIAGcoLA1dmhOPwqFtnMRxVkdR4wv1w=; b=SxUT6xWreeUl3H8CarfgzNkRw3IV/yY9ZMsM/2PFXBOFWY9aGFSAmj6tbo0YnRC1FJ KctuJpFgcoYU0krm2hjyPexaAq+tTuN0665VkUpyNKyFTRag4F0GSjLCm8Yq/Zjq0uFp 4BDH09lSb5bPnsj46j3TjTYe4vvGi1oUqdLw7n0dCV5DksP3f40uqQjjcnJ1zpdNWx8q mSRIB3jtSX1FWxMrFAOFoGzPEPPv2IC8vD17x5tT52RCz97KFiwp5+8RKTZsZG9GAITQ /rd/o59fJlSSk1oHhicetmSee6T1E7MUzmt8WbgPRI1IsUi8rQnulzK1o2Nr4UgcHezA kO4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HVAcQFUvhJRbVIAGcoLA1dmhOPwqFtnMRxVkdR4wv1w=; b=X0U49bu+Ivrxj57b7IK/g5790mu0/JQ2GUKyO2dGdksgeJ4nBqH2Mdy09ep/krp3ls j6zUUKyZzMt82te47VIl6U9zQHdcUGRUSWQudm9rwgUR2H79CX45NEfUEMhR0yslaoCk 2ypf3AP9Buqib32YSBM/oVMDuaHrf/iqVNltzlO7PEOfaH331B8q4DK7qBNeXrvLQ6dX i0pHk0ywnFR3VrtmuJoU753IwfId//fDqSGMEpb1p54VR1l9uNgnQ1axapZV3h4ArEoj 1eWtnHdATveOeuQm6T7Qo1ZnEwEe0NLchl/fJ1b9zEOaYlOqOt6ahyBhVqPS/jsJSK+C XGhQ== X-Gm-Message-State: AOAM533qrj7x6hEQxRTPe3sHL4utLf9vJtvWCV1mC7kxTMYBiQpBIdor 8KpVC1QYkYvA8RCagYvnMrqScgy09tY= X-Google-Smtp-Source: ABdhPJy+VhXvMjqd6Et+rzVxHLBk6bNHi780QD6bcuU5xPFdB8/+MRiAB3EHyxbxRsX/XOXY2DkPOA== X-Received: by 2002:a5d:49c6:: with SMTP id t6mr18064659wrs.201.1631625417312; Tue, 14 Sep 2021 06:16:57 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:56 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 10/20] trace-cmd library: Decompress the options section, if it is compressed Date: Tue, 14 Sep 2021 16:16:35 +0300 Message-Id: <20210914131645.3966308-11-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In trace file version 7, options section can be compressed. Extended the options handling decompression if needed . Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 41 ++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 11ee6470..41b60649 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -3038,6 +3038,7 @@ static int handle_options(struct tracecmd_input *handle) unsigned short id, flags; char *cpustats = NULL; struct hook_list *hook; + bool compress = false; char *buf; int cpus; int ret; @@ -3049,23 +3050,32 @@ static int handle_options(struct tracecmd_input *handle) return -1; if (id != TRACECMD_OPTION_DONE) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + compress = true; } + if (compress && in_uncompress_block(handle)) + return -1; for (;;) { - if (read2(handle, &option)) - return -1; + ret = read2(handle, &option); + if (ret) + goto out; if (!HAS_SECTIONS(handle) && option == TRACECMD_OPTION_DONE) break; /* next 4 bytes is the size of the option */ - if (read4(handle, &size)) - return -1; + ret = read4(handle, &size); + if (ret) + goto out; buf = malloc(size); - if (!buf) - return -ENOMEM; - if (do_read_check(handle, buf, size)) - return -1; + if (!buf) { + ret = -ENOMEM; + goto out; + } + ret = do_read_check(handle, buf, size); + if (ret) + goto out; switch (option) { case TRACECMD_OPTION_DATE: @@ -3115,7 +3125,7 @@ static int handle_options(struct tracecmd_input *handle) buf + 8, 4); ret = tsync_cpu_offsets_load(handle, buf + 12, size - 12); if (ret < 0) - return ret; + goto out; tracecmd_enable_tsync(handle, true); break; case TRACECMD_OPTION_CPUSTAT: @@ -3124,7 +3134,7 @@ static int handle_options(struct tracecmd_input *handle) handle->cpustats_size + size + 1); if (!cpustats) { ret = -ENOMEM; - return ret; + goto out; } memcpy(cpustats + handle->cpustats_size, buf, size); handle->cpustats_size += size; @@ -3135,7 +3145,7 @@ static int handle_options(struct tracecmd_input *handle) case TRACECMD_OPTION_BUFFER_TEXT: ret = handle_buffer_option(handle, option, buf, size); if (ret < 0) - return ret; + goto out; break; case TRACECMD_OPTION_TRACECLOCK: if (!handle->ts2secs) @@ -3193,6 +3203,8 @@ static int handle_options(struct tracecmd_input *handle) tep_read_number(handle->pevent, buf, 8), 0); break; case TRACECMD_OPTION_DONE: + if (compress) + in_uncompress_reset(handle); ret = handle_option_done(handle, buf, size); free(buf); return ret; @@ -3205,7 +3217,12 @@ static int handle_options(struct tracecmd_input *handle) } - return 0; + ret = 0; + +out: + if (compress) + in_uncompress_reset(handle); + return ret; } static int read_options_type(struct tracecmd_input *handle) From patchwork Tue Sep 14 13:16:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 106EBC433F5 for ; Tue, 14 Sep 2021 13:17:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F0F7260724 for ; Tue, 14 Sep 2021 13:17:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233135AbhINNSR (ORCPT ); Tue, 14 Sep 2021 09:18:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSR (ORCPT ); Tue, 14 Sep 2021 09:18:17 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB872C061574 for ; Tue, 14 Sep 2021 06:16:59 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id g19-20020a1c9d13000000b003075062d4daso2552701wme.0 for ; Tue, 14 Sep 2021 06:16:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mlxclYoJfY56ZTfpJxTpE6EKYQGSGZp1shRRwl5bhFw=; b=po4gU9MFQ/uit/e+CC9Rtp9nUeQYT98+cIz8myQFzYv5VtAM1AbbswGmdth3ut+yoH vWyQviGZXM9XgyUrtMmPGSd9YWOZgVH3AhrMfwMAmosI/K1q9pXuEhTQcm1TX94usArH rzd7DEKa+jTeL2f75NJv69v7Ygan+WQU7l/7RfD8AklfvtDoS/6kFwACXHhu2oCHjyk8 o69u4ZqwdX/lWHiEVqRZa7fqadBpAx0T4shVkf+LD0/IZPQnsTplAYH4LjRJg+mTUjw6 jsIv9RTeGXs6Nkk2aiCGnWIHyjdNKDzRbjZPidkSz/wiv5QnI6cU5g5oC+O3vTTj2ClS CWBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mlxclYoJfY56ZTfpJxTpE6EKYQGSGZp1shRRwl5bhFw=; b=j0vH5JohKGQEBw8oLsyQNzmFa474PUFufcqGVp5tB1KVKBmpxP2nj9jMmrLokdrRKD /JGTx/D3vCUG19d5THtvYPqo9c0XIS4jQ8VoVzPEXOyvnSi6yyYipOdjH7tzG/jiCGMD TtsEOKtpVbSYplxDbxQ5Xsb0+KA/aueJjqdur7G8VZfR3mcDcaEc+jswU/eNmL7HzKLQ w4I3XK3Vgj5JTOJoTvU5kzFzO5gArYYIqddcZ80ShUN7VsCucWVKvQ4iOMp4YRSB6Wk7 o6XoFZ6Iq3ZcyZLgTi9+sOx0QUhz747hNju1YkxPtm/FM/IcMbJ734K+NpoFT2zfTBt5 Av2A== X-Gm-Message-State: AOAM5305qpIRupLJvGM/sSX2rE4jGpSHuukVyLeVvqtMMZevqCc9JDqR OC7nAMJPb3+4SQ57ceYG9OdZ4VT94Pk= X-Google-Smtp-Source: ABdhPJyilL0oq/xCG68nL8v6U0Lh4OXhYKU4fM06aOELYScaSS3NJ28zN0QM/F/geGdmYrBXf5JdQg== X-Received: by 2002:a1c:7e8a:: with SMTP id z132mr2119612wmc.75.1631625418335; Tue, 14 Sep 2021 06:16:58 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:57 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 11/20] trace-cmd library: Read compression header Date: Tue, 14 Sep 2021 16:16:36 +0300 Message-Id: <20210914131645.3966308-12-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace file version 7 introduced new mandatory compression header, storing information about the compression algorithm used to compress the trace file. Added code to read that header and to initialize compression context according to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 41b60649..d4728c86 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -189,6 +189,7 @@ __thread struct tracecmd_input *tracecmd_curr_thread_handle; #define CHECK_READ_STATE(H, S) ((H)->file_version < FILE_VERSION_SECTIONS && (H)->file_state >= (S)) #define HAS_SECTIONS(H) ((H)->flags & TRACECMD_FL_SECTIONED) +#define HAS_COMPRESSION(H) ((H)->flags & TRACECMD_FL_COMPRESSION) static int read_options_type(struct tracecmd_input *handle); @@ -3797,7 +3798,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) char test[] = TRACECMD_MAGIC; unsigned int page_size; size_t offset; - char *version; + char *version = NULL; + char *zver = NULL; + char *zname = NULL; char buf[BUFSIZ]; unsigned long ver; @@ -3835,9 +3838,12 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) } handle->file_version = ver; free(version); + version = NULL; if (handle->file_version >= FILE_VERSION_SECTIONS) handle->flags |= TRACECMD_FL_SECTIONED; + if (handle->file_version >= FILE_VERSION_COMPRESSION) + handle->flags |= TRACECMD_FL_COMPRESSION; if (do_read_check(handle, buf, 1)) goto failed_read; @@ -3867,6 +3873,26 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->total_file_size = lseek64(handle->fd, 0, SEEK_END); lseek64(handle->fd, offset, SEEK_SET); + if (HAS_COMPRESSION(handle)) { + zname = read_string(handle); + if (!zname) + goto failed_read; + zver = read_string(handle); + if (!zver) + goto failed_read; + if (strcmp(zname, "none")) { + handle->compress = tracecmd_compress_alloc(zname, zver, + handle->fd, + handle->pevent, NULL); + if (!handle->compress) { + tracecmd_warning("Unsupported file compression %s %s", zname, zver); + goto failed_read; + } + } + free(zname); + free(zver); + } + if (HAS_SECTIONS(handle)) { if (read8(handle, &(handle->options_start))) { tracecmd_warning("Filed to read the offset of the first option section"); @@ -3879,6 +3905,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) return handle; failed_read: + free(version); + free(zname); + free(zver); free(handle); return NULL; @@ -4077,7 +4106,8 @@ void tracecmd_close(struct tracecmd_input *handle) if (handle->flags & TRACECMD_FL_BUFFER_INSTANCE) tracecmd_close(handle->parent); else { - /* Only main handle frees plugins and pevent */ + /* Only main handle frees plugins, pevent and compression context */ + tracecmd_compress_destroy(handle->compress); tep_unload_plugins(handle->plugin_list, handle->pevent); tep_free(handle->pevent); } From patchwork Tue Sep 14 13:16:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81B41C433FE for ; Tue, 14 Sep 2021 13:17:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 68B28604DC for ; Tue, 14 Sep 2021 13:17:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233206AbhINNSS (ORCPT ); Tue, 14 Sep 2021 09:18:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSS (ORCPT ); Tue, 14 Sep 2021 09:18:18 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B47F5C061574 for ; Tue, 14 Sep 2021 06:17:00 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id t18so20212550wrb.0 for ; Tue, 14 Sep 2021 06:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IBSnXtLUWdnoAGzwCpkqt/zTiW5sZ5SNi9LgmB9E914=; b=GIBzkx5QO9D+zcdPWRmRCrdvQTZfba7TISnXOWVXeK0PFg1MkmU1Na5eExoPQ4sPD8 ziyy3LEG9iWmjRjzveYOJaytH/1Mo6BZBsFyEX0t/2qPW5Xvt3y1daGaIb9IPIx8aueC RJuEPBkWDqZswxWYe3cWAK8lgMg1D+jImLNoKNgVr/pSVq4bTEa24ht6w3fqynBW52+2 G9hceCeStkb9uiVgdCsRDWC88xb99Vij7yWxStcJZtTcHCghEnuewavMoauOm6qwQsCL 2JJplvvxvKlZ1jK1aQ4GQs9nAtGG2m3lh4JO0cJfzKnq9RTAYmOcIGtL9fJiHxgCvRAl /ZeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IBSnXtLUWdnoAGzwCpkqt/zTiW5sZ5SNi9LgmB9E914=; b=uUB6vRICnMRIJY1SHAiLQveez+Fzh+8HGd8yPt5fLKyGvQqHwQT/hY/vlzNaXK6Dgb B59NgNw7aogmojeoIDp5ERG4Maf7En7C0bOUfI16rl8/tUwatewgKBSZzjgcE4P/6XoO 5mXcfZcIB3fYuA6IiuuHcbTZFN6zfq9xd1FvSw3gE1GUZDEneKqL6tqVBej9KZJWVx7m rq/UZhlIJZtVsYYgUxtXtGmKrveHqDNb0vk6AZ7yINY542JwahIZywzXMhVcS/pZ3M2i BXvMdhpG0nwyLlWzt6OFmdcyuXuKsuJeq6Gx+zuvrjtCss/vcM3Kgy2uDtCfkEhtC4wd VqXA== X-Gm-Message-State: AOAM531+mLpk2Us5Y7yqtl/eEiMDBJJ5ZtLAvmxrdAc7qKpi4bIF3TTf n1uuCSXPd3I4cSHPBlJxJXbhoquaM9U= X-Google-Smtp-Source: ABdhPJyahCONK/pilp86o5df+FtrVLdeJ0XFs0QL/JCFna3BEO6AT/q3tDc6nLGEF+d2ERmHSn7woA== X-Received: by 2002:a5d:554e:: with SMTP id g14mr19061293wrw.48.1631625419341; Tue, 14 Sep 2021 06:16:59 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:16:58 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 12/20] trace-cmd library: Extend the input handler with trace data decompression context Date: Tue, 14 Sep 2021 16:16:37 +0300 Message-Id: <20210914131645.3966308-13-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The CPU trace data is compressed in chunks, as chunk's size is multiple trace pages. The input handler is extended with the necessary structures, to control the data decompression. There are two approaches for data decompression, both are supported and can be used in different use cases: - in-memory decompression, page by page. - using a temporary file Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 59 ++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index d4728c86..658855a5 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -54,6 +54,24 @@ struct page { #endif }; +struct zchunk_cache { + struct list_head list; + struct tracecmd_compress_chunk *chunk; + void *map; + int ref; +}; + +struct cpu_zdata { + /* uncompressed cpu data */ + int fd; + char file[26]; /* strlen(COMPR_TEMP_FILE) */ + unsigned int count; + unsigned int last_chunk; + struct list_head cache; + struct tracecmd_compress_chunk *chunks; +}; + +#define COMPR_TEMP_FILE "/tmp/trace_cpu_dataXXXXXX" struct cpu_data { /* the first two never change */ unsigned long long file_offset; @@ -72,6 +90,7 @@ struct cpu_data { int page_cnt; int cpu; int pipe_fd; + struct cpu_zdata compress; }; struct cpu_file_data { @@ -150,6 +169,8 @@ struct tracecmd_input { bool use_trace_clock; bool read_page; bool use_pipe; + bool read_zpage; /* uncompress pages in memory, do not use tmp files */ + bool cpu_compressed; int file_version; struct cpu_data *cpu_data; long long ts_offset; @@ -3302,6 +3323,7 @@ static int init_cpu_data(struct tracecmd_input *handle) endian = KBUFFER_ENDIAN_LITTLE; for (cpu = 0; cpu < handle->cpus; cpu++) { + handle->cpu_data[cpu].compress.fd = -1; handle->cpu_data[cpu].kbuf = kbuffer_alloc(long_size, endian); if (!handle->cpu_data[cpu].kbuf) goto out_free; @@ -4043,6 +4065,7 @@ static inline void free_buffer(struct input_buffer_instance *buf) */ void tracecmd_close(struct tracecmd_input *handle) { + struct zchunk_cache *cache; struct file_section *del_sec; int cpu; int i; @@ -4062,17 +4085,31 @@ void tracecmd_close(struct tracecmd_input *handle) /* The tracecmd_peek_data may have cached a record */ free_next(handle, cpu); free_page(handle, cpu); - if (handle->cpu_data && handle->cpu_data[cpu].kbuf) { - kbuffer_free(handle->cpu_data[cpu].kbuf); - if (handle->cpu_data[cpu].page_map) - free_page_map(handle->cpu_data[cpu].page_map); - - if (handle->cpu_data[cpu].page_cnt) - tracecmd_warning("%d pages still allocated on cpu %d%s", - handle->cpu_data[cpu].page_cnt, cpu, - show_records(handle->cpu_data[cpu].pages, - handle->cpu_data[cpu].nr_pages)); - free(handle->cpu_data[cpu].pages); + if (handle->cpu_data) { + if (handle->cpu_data[cpu].kbuf) { + kbuffer_free(handle->cpu_data[cpu].kbuf); + if (handle->cpu_data[cpu].page_map) + free_page_map(handle->cpu_data[cpu].page_map); + + if (handle->cpu_data[cpu].page_cnt) + tracecmd_warning("%d pages still allocated on cpu %d%s", + handle->cpu_data[cpu].page_cnt, cpu, + show_records(handle->cpu_data[cpu].pages, + handle->cpu_data[cpu].nr_pages)); + free(handle->cpu_data[cpu].pages); + } + if (handle->cpu_data[cpu].compress.fd >= 0) { + close(handle->cpu_data[cpu].compress.fd); + unlink(handle->cpu_data[cpu].compress.file); + } + while (!list_empty(&handle->cpu_data[cpu].compress.cache)) { + cache = container_of(handle->cpu_data[cpu].compress.cache.next, + struct zchunk_cache, list); + list_del(&cache->list); + free(cache->map); + free(cache); + } + free(handle->cpu_data[cpu].compress.chunks); } } From patchwork Tue Sep 14 13:16:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77B28C433F5 for ; Tue, 14 Sep 2021 13:17:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5E072610D1 for ; Tue, 14 Sep 2021 13:17:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233196AbhINNST (ORCPT ); Tue, 14 Sep 2021 09:18:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNST (ORCPT ); Tue, 14 Sep 2021 09:18:19 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C76BC061574 for ; Tue, 14 Sep 2021 06:17:02 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id s24so9228270wmh.4 for ; Tue, 14 Sep 2021 06:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+kcfbV8tKqZo1OCdmsj/KRe6lM3F32WAwHEZnU5jWKA=; b=F75rZj8JBNkKEQgbx3O7Ae9w2cT249yayKNd+ZGCT6AZAFPDGp5n5whlM3Qr/IPmsg LkV2yIG9/Ns3rbaC/syrZ/yQah8ngWlt2jJML23TgYGG/Pq2I5gd7EluxVpeai+J/qGt bAY2B2GsIqYjfMe4Gy65N4ZF9dmagKl+u+pEjdNG3E4OTHsK5uWbQEWvDJKmOAPL6/j1 wYheprLgmiZAwtYYas2ptRjf8sAFjz7Tx2gbpdpC3gMX0RLupOnoaCIXksaVFU5I2MJC UksbYZxwg1OUzFQ5LQIUZUEHEYWgYfQj3ITiX9fHKb2tzxWtoqsRo4jQwI8rqhvKTEbp 0PXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+kcfbV8tKqZo1OCdmsj/KRe6lM3F32WAwHEZnU5jWKA=; b=CPOf26m3bzTJgov0GTLy9Yy4jNDQkgj6c/wM3RP6w/Jh3aJLKqUSJ38TZqmSapWZ/1 cNCqKo3fjwsKyu4MsBIYsFaV66hBWMUr3ik24IzuaigjZka1x+AGgl378hhH+XCGIQG8 abZ++Iwj/6+28iQZfZejus23K658M2vt1h3x4CH/er2QcQ0ybqTl/StTYP7xOPsFBa2v wTmo0u9X0rSmdtHk4p7mbWLiq8xT+wCFDYdxnGAOB/Erm+miEVfinqblQq9drkDHM/dP qr6ZCcsGkWkvD8dfhibaSVJPWqd0LImuqxp1duIMe29PzlkDwSNX8GAHb3hsOIYaWOiC NqHg== X-Gm-Message-State: AOAM533mnahPMgiNQfRsRvdIbWJ6Lzr0HWNOy9W3485k+J53doRQO+LI sM3W6slf2aaDEJB7Es51JhffVi8x43w= X-Google-Smtp-Source: ABdhPJzdgTEonL2Jv1xo/+GsGyXQrBgBi9ALTrqnHVn2X4rCshobwoF0eUvghO43dKJfpWFNjAYLyg== X-Received: by 2002:a7b:c8c3:: with SMTP id f3mr2212031wml.30.1631625420609; Tue, 14 Sep 2021 06:17:00 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.16.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:00 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 13/20] trace-cmd library: Initialize CPU data decompression logic Date: Tue, 14 Sep 2021 16:16:38 +0300 Message-Id: <20210914131645.3966308-14-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org On CPU data initialization stage, initialize decompression context for both in-memory and temporary file decompression logics. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 72 +++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 658855a5..0fd1a451 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -1266,6 +1266,7 @@ static void *allocate_page_map(struct tracecmd_input *handle, off64_t map_offset; void *map; int ret; + int fd; if (handle->read_page) { map = malloc(handle->page_size); @@ -1305,12 +1306,15 @@ static void *allocate_page_map(struct tracecmd_input *handle, map_size -= map_offset + map_size - (cpu_data->file_offset + cpu_data->file_size); + if (cpu_data->compress.fd >= 0) + fd = cpu_data->compress.fd; + else + fd = handle->fd; again: page_map->size = map_size; page_map->offset = map_offset; - page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, - handle->fd, map_offset); + page_map->map = mmap(NULL, map_size, PROT_READ, MAP_PRIVATE, fd, map_offset); if (page_map->map == MAP_FAILED) { /* Try a smaller map */ @@ -2502,16 +2506,76 @@ tracecmd_read_prev(struct tracecmd_input *handle, struct tep_record *record) /* Not reached */ } -static int init_cpu(struct tracecmd_input *handle, int cpu) +static int init_cpu_zfile(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data; + unsigned long long size; + off64_t offset; + + cpu_data = &handle->cpu_data[cpu]; + offset = lseek64(handle->fd, 0, SEEK_CUR); + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + strcpy(cpu_data->compress.file, COMPR_TEMP_FILE); + cpu_data->compress.fd = mkstemp(cpu_data->compress.file); + if (cpu_data->compress.fd < 0) + return -1; + if (tracecmd_uncompress_copy_to(handle->compress, cpu_data->compress.fd, NULL, &size)) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + cpu_data->offset = 0; + cpu_data->file_offset = 0; + cpu_data->file_size = size; + cpu_data->size = size; + return 0; +} + +static int init_cpu_zpage(struct tracecmd_input *handle, int cpu) { struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int count; int i; + if (lseek64(handle->fd, cpu_data->file_offset, SEEK_SET) == (off_t)-1) + return -1; + + count = tracecmd_load_chunks_info(handle->compress, &cpu_data->compress.chunks); + if (count < 0) + return -1; + cpu_data->compress.count = count; + cpu_data->compress.last_chunk = 0; + + cpu_data->file_offset = 0; + cpu_data->file_size = 0; + for (i = 0; i < count; i++) + cpu_data->file_size += cpu_data->compress.chunks[i].size; cpu_data->offset = cpu_data->file_offset; cpu_data->size = cpu_data->file_size; + return 0; +} + +static int init_cpu(struct tracecmd_input *handle, int cpu) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + int ret; + int i; + + if (handle->cpu_compressed && cpu_data->file_size > 0) { + if (handle->read_zpage) + ret = init_cpu_zpage(handle, cpu); + else + ret = init_cpu_zfile(handle, cpu); + if (ret) + return ret; + } else { + cpu_data->offset = cpu_data->file_offset; + cpu_data->size = cpu_data->file_size; + } cpu_data->timestamp = 0; list_head_init(&cpu_data->page_maps); + list_head_init(&cpu_data->compress.cache); if (!cpu_data->size) { printf("CPU %d is empty\n", cpu); @@ -3380,6 +3444,8 @@ static int init_buffer_cpu_data(struct tracecmd_input *handle, struct input_buff return -1; if (read_section_header(handle, &id, &flags, NULL, NULL)) return -1; + if (flags & TRACECMD_SEC_FL_COMPRESS) + handle->cpu_compressed = true; if (buffer->latency) { handle->file_state = TRACECMD_FILE_CPU_LATENCY; return init_latency_data(handle) == 0 ? 1 : -1; From patchwork Tue Sep 14 13:16:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C08BC433FE for ; Tue, 14 Sep 2021 13:17:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 75A9E610CE for ; Tue, 14 Sep 2021 13:17:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233181AbhINNSV (ORCPT ); Tue, 14 Sep 2021 09:18:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSU (ORCPT ); Tue, 14 Sep 2021 09:18:20 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FF61C061574 for ; Tue, 14 Sep 2021 06:17:03 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id d21so12469641wra.12 for ; Tue, 14 Sep 2021 06:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y3/WTAvwdoRnVjxeqxmfI6+6BiqDEX3drwjk1TFdTIo=; b=F/aS1eu5uz4gHQ5tRR1iGBgVVrwjHhvxHC2ZFgjmStNrbKJ4wdYt84k5HyFuPChxGo YFPCRLmkufxkSTmSX50TkSB2VITlrr7SFu4NczPtVT9g8JIfDYrPhEjFI30dEEiGxeHc iWS8tfGw4e9Fpkc5Ic5DsFEPTNN2q5eCEumwStKvHhWeEHMxkPamJ9zrprCMvnocfv34 zdLqPl8W3WXh/0zJlSYAW6Deg63/HcBPoYmQjPETHjuJTUSEp7SpKHYz64hGZ3bWwRRz COkpWtCwXUpvO5tVC8wUh+lveBoJ3KLwf710VhBLt6TMJZwewbCbb3UzrtFSZCN4mX1W tnJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y3/WTAvwdoRnVjxeqxmfI6+6BiqDEX3drwjk1TFdTIo=; b=DAPhaDnqzr1AcR8xjrFNv34D0/hE5sgAwj12a6CxiOWH9knIez4HouLHEe9D4nJ5zp OPXC6IHVr1BwsEYzltpSvlqaQkUfn5MM2zq2o7x8ffrPjuwR2PWub33TOHVwD/pF4GwK Td/hLnuWneQRWnWDGkJJ6GhkW1UhPplMUpy9Fpgrj8egTOOxsvwusm8m13ETbcubEq66 yTszrroNxKwYDHN4AMUiks0QWd3F3m8YXrE4GDNL+4OMF8z7a4REDYr+1lkR/m4BLx9J VK0CVgyOm9r+e7Sf4ZmzpzZjUcwq4lO6nqCqJJB3tSNS7YrpSFtMGcYJ9WCUpx+7rky3 AvZw== X-Gm-Message-State: AOAM530iVVgxyeIzDDPqONOp+9rhrr2sV9CWZvjDQhBHtHBBAG+NAmmG bwxxW6WTzafm1wGL8aCgBdOSYgHslSk= X-Google-Smtp-Source: ABdhPJw5R6BY0t2XIaXGiuR/dtyACAs3qZjLLXkkTWdB+dNAYj+3u2WJK07XMmYQYpp4MN2ThxjGwQ== X-Received: by 2002:a5d:6750:: with SMTP id l16mr18697268wrw.174.1631625421752; Tue, 14 Sep 2021 06:17:01 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:01 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 14/20] trace-cmd library: Add logic for in-memory decompression Date: Tue, 14 Sep 2021 16:16:39 +0300 Message-Id: <20210914131645.3966308-15-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org There are two approaches to read compressed trace data: - use a temporary file to decompress entire trace data before reading - use in-memory decompression of requested trace data chunk only In-memory decompression seems to be more efficient, but selecting which approach to use depends in the use case. A compression chunk consists of multiple trace pages, that's why a small cache with uncompressed chunks is implemented. The chunk stays in the cache until there are pages which have reference to it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 110 ++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 0fd1a451..040fb391 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -29,6 +29,9 @@ #define COMMIT_MASK ((1 << 27) - 1) +/* force uncompressing in memory */ +#define INMEMORY_DECOMPRESS + /* for debugging read instead of mmap */ static int force_read = 0; @@ -1257,6 +1260,105 @@ static void free_page_map(struct page_map *page_map) free(page_map); } +#define CHUNK_CHECK_OFFSET(C, O) ((O) >= (C)->offset && (O) < ((C)->offset + (C)->size)) +static struct tracecmd_compress_chunk *get_zchunk(struct cpu_data *cpu, off64_t offset) +{ + struct cpu_zdata *cpuz = &cpu->compress; + int min, mid, max; + + if (!cpuz->chunks) + return NULL; + if (offset > (cpuz->chunks[cpuz->count - 1].offset + cpuz->chunks[cpuz->count - 1].size)) + return NULL; + + /* check if the requested offset is in the last requested chunk or in the next chunk */ + if (CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + cpuz->last_chunk++; + if (cpuz->last_chunk < cpuz->count && + CHUNK_CHECK_OFFSET(cpuz->chunks + cpuz->last_chunk, offset)) + return cpuz->chunks + cpuz->last_chunk; + + /* do a binary search to find the chunk holding the given offset */ + min = 0; + max = cpuz->count - 1; + mid = (min + max)/2; + while (min <= max) { + if (offset < cpuz->chunks[mid].offset) + max = mid - 1; + else if (offset > (cpuz->chunks[mid].offset + cpuz->chunks[mid].size)) + min = mid + 1; + else + break; + mid = (min + max)/2; + } + cpuz->last_chunk = mid; + return cpuz->chunks + mid; +} + +static void free_zpage(struct cpu_data *cpu_data, void *map) +{ + struct zchunk_cache *cache; + + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (map <= cache->map && map > (cache->map + cache->chunk->size)) + goto found; + } + return; + +found: + cache->ref--; + if (cache->ref) + return; + list_del(&cache->list); + free(cache->map); + free(cache); +} + +static void *read_zpage(struct tracecmd_input *handle, int cpu, off64_t offset) +{ + struct cpu_data *cpu_data = &handle->cpu_data[cpu]; + struct tracecmd_compress_chunk *chunk; + struct zchunk_cache *cache; + void *map = NULL; + int pindex; + int size; + + /* Look in the cache of already loaded chunks */ + list_for_each_entry(cache, &cpu_data->compress.cache, list) { + if (CHUNK_CHECK_OFFSET(cache->chunk, offset)) { + cache->ref++; + goto out; + } + } + + chunk = get_zchunk(cpu_data, offset); + if (!chunk) + return NULL; + size = handle->page_size > chunk->size ? handle->page_size : chunk->size; + map = malloc(size); + if (!map) + return NULL; + if (tracecmd_uncompress_chunk(handle->compress, chunk, map) < 0) + goto error; + + cache = calloc(1, sizeof(struct zchunk_cache)); + if (!cache) + goto error; + cache->ref = 1; + cache->chunk = chunk; + cache->map = map; + list_add(&cache->list, &cpu_data->compress.cache); + + /* a chunk can hold multiple pages, get the requested one */ +out: + pindex = (offset - cache->chunk->offset) / handle->page_size; + return cache->map + (pindex * handle->page_size); +error: + free(map); + return NULL; +} + static void *allocate_page_map(struct tracecmd_input *handle, struct page *page, int cpu, off64_t offset) { @@ -1268,6 +1370,9 @@ static void *allocate_page_map(struct tracecmd_input *handle, int ret; int fd; + if (handle->cpu_compressed && handle->read_zpage) + return read_zpage(handle, cpu, offset); + if (handle->read_page) { map = malloc(handle->page_size); if (!map) @@ -1410,6 +1515,8 @@ static void __free_page(struct tracecmd_input *handle, struct page *page) if (handle->read_page) free(page->map); + else if (handle->read_zpage) + free_zpage(cpu_data, page->map); else free_page_map(page->page_map); @@ -3902,6 +4009,9 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; +#ifdef INMEMORY_DECOMPRESS + handle->read_zpage = 1; +#endif if (do_read_check(handle, buf, 3)) goto failed_read; From patchwork Tue Sep 14 13:16:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C357EC433F5 for ; Tue, 14 Sep 2021 13:17:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A1C1F610D1 for ; Tue, 14 Sep 2021 13:17:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233122AbhINNSW (ORCPT ); Tue, 14 Sep 2021 09:18:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSV (ORCPT ); Tue, 14 Sep 2021 09:18:21 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D79C061574 for ; Tue, 14 Sep 2021 06:17:04 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id v20-20020a1cf714000000b002e71f4d2026so1948792wmh.1 for ; Tue, 14 Sep 2021 06:17:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dzEv3aV9AetR6VmyaD18uFySaeAq9X3N7n9UMXGUAdA=; b=WiIjpF91SAshBBAkL8W1qweuW9hGs3AOpl2wblR87spEknRHjMymeg9LafyFfKx/il mtjSVr64YmjcsCy446cUx7u1a+BFblBKdloU8GPh3oTTTNzzJgoEWA9m+zbry4JIfdm0 h+Hl1kVEO7ZpZ+NPo8xqoM+XsDkYlAKUJlpBkb2rfBX9it7AGstdD0HF3GMDp2oWamdt k2XdxEEwKJdpmXuoYq7QXck+4okKKpl25Jqs/dcCFf9/I+TVuAnfxyNbpI/XoTyvLWMs 1gP5sSjeFKkoPRe0KJVMvXoC1IcbwidervfN2Nt1eckw18tR0uvcMbeBEX1BrTFEy+Mw 8MiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dzEv3aV9AetR6VmyaD18uFySaeAq9X3N7n9UMXGUAdA=; b=nt+pNUh6ezNJj+jlOtqp1Tgi0m4AaejJ9QixVcW3YrLGVWYGthjxrd+2KNHyLgjlaS M+GYivCdXWiVlCWJ7//FhcyFTXO0a5M7JX+xRXiADXIzAvj3AmQ7dqkFMcc8YYY6FXZv VQsYeUYEDPLas3KM64iocxQlSpILFBPrsgeWCI/PPfH+jnp/18X9hvw/+u+gYBFKZLMX TNY/I0Xiljvk/0RijmhYfWkPyfzNmZOa8yKogEJltIPI6KI7GPyXGZxtWe9tOoi3yY+O 6UJaWnPw0jT7QJWvTdOVMSz238HDlC514kB/zIqL296ijoACkZ5ezhllGGaBwsk4CjLV Hz/Q== X-Gm-Message-State: AOAM531qfV3FxWe8zcaFrzNiIGdwzAl1FNXOjkaxIaywvogKK9Q49nBr iG6e9nKBDVZNVjQ0e+HAuMFUfBhP9H8= X-Google-Smtp-Source: ABdhPJyHoYgiGFfmFp/U9y/fW7sWaNfwerflx8ITb1J4OevD/Pm+VPH2l0e8pMg46Kx6HgdhW8ir6Q== X-Received: by 2002:a1c:2b04:: with SMTP id r4mr2151070wmr.168.1631625422715; Tue, 14 Sep 2021 06:17:02 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:02 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 15/20] trace-cmd library: Read compressed latency data Date: Tue, 14 Sep 2021 16:16:40 +0300 Message-Id: <20210914131645.3966308-16-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Extended the latency read logic for reading comperssed latency trace data. Both decompressing approaches are supported: in-memory and using temporary file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 74 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 8 deletions(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 040fb391..80230ba5 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -175,6 +175,7 @@ struct tracecmd_input { bool read_zpage; /* uncompress pages in memory, do not use tmp files */ bool cpu_compressed; int file_version; + struct cpu_zdata latz; struct cpu_data *cpu_data; long long ts_offset; struct tsc2nsec tsc_calc; @@ -3452,20 +3453,52 @@ static int read_options_type(struct tracecmd_input *handle) int tracecmd_latency_data_read(struct tracecmd_input *handle, char **buf, size_t *size) { + struct cpu_zdata *zdata = &handle->latz; + void *data; + int rsize; + int fd = -1; + int id; + if (!handle || !buf || !size) return -1; if (handle->file_state != TRACECMD_FILE_CPU_LATENCY) return -1; - /* Read data from a file */ - if (!(*buf)) { - *size = BUFSIZ; - *buf = malloc(*size); - if (!(*buf)) + if (!handle->cpu_compressed) { + fd = handle->fd; + } else if (!handle->read_zpage) { + if (zdata->fd < 0) return -1; + fd = zdata->fd; } - return do_read(handle, *buf, *size); + /* Read data from a file */ + if (fd >= 0) { + if (!(*buf)) { + *size = BUFSIZ; + *buf = malloc(*size); + if (!(*buf)) + return -1; + } + return do_read_fd(fd, *buf, *size); + } + + /* Uncompress data in memory */ + if (zdata->last_chunk >= zdata->count) + return 0; + id = zdata->last_chunk; + if (!*buf || *size < zdata->chunks[id].size) { + data = realloc(*buf, zdata->chunks[id].size); + if (!data) + return -1; + *buf = data; + *size = zdata->chunks[id].size; + } + if (tracecmd_uncompress_chunk(handle->compress, &zdata->chunks[id], *buf)) + return -1; + rsize = zdata->chunks[id].size; + zdata->last_chunk++; + return rsize; } static int init_cpu_data(struct tracecmd_input *handle) @@ -3533,7 +3566,27 @@ static int init_cpu_data(struct tracecmd_input *handle) int init_latency_data(struct tracecmd_input *handle) { - /* To do */ + unsigned long long wsize; + int ret; + + if (!handle->cpu_compressed) + return 0; + + if (handle->read_zpage) { + handle->latz.count = tracecmd_load_chunks_info(handle->compress, &handle->latz.chunks); + if (handle->latz.count < 0) + return -1; + } else { + strcpy(handle->latz.file, COMPR_TEMP_FILE); + handle->latz.fd = mkstemp(handle->latz.file); + if (handle->latz.fd < 0) + return -1; + ret = tracecmd_uncompress_copy_to(handle->compress, handle->latz.fd, NULL, &wsize); + if (ret) + return -1; + lseek64(handle->latz.fd, 0, SEEK_SET); + } + return 0; } @@ -4006,6 +4059,7 @@ struct tracecmd_input *tracecmd_alloc_fd(int fd, int flags) handle->fd = fd; handle->ref = 1; + handle->latz.fd = -1; /* By default, use usecs, unless told otherwise */ handle->flags |= TRACECMD_FL_IN_USECS; @@ -4295,7 +4349,11 @@ void tracecmd_close(struct tracecmd_input *handle) free(handle->trace_clock); free(handle->version); close(handle->fd); - + free(handle->latz.chunks); + if (handle->latz.fd >= 0) { + close(handle->latz.fd); + unlink(handle->latz.file); + } while (handle->sections) { del_sec = handle->sections; handle->sections = handle->sections->next; From patchwork Tue Sep 14 13:16:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CC32C433EF for ; Tue, 14 Sep 2021 13:17:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7642860C51 for ; Tue, 14 Sep 2021 13:17:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233217AbhINNS1 (ORCPT ); Tue, 14 Sep 2021 09:18:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbhINNSW (ORCPT ); Tue, 14 Sep 2021 09:18:22 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94A52C061574 for ; Tue, 14 Sep 2021 06:17:05 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id s24so9228416wmh.4 for ; Tue, 14 Sep 2021 06:17:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KUu8wEBDFHkS++DKeU8RnSaGh748X1gceNhCKSNAVhw=; b=GFyTubmtZgIMrsjOEeG2amciVajMQcyVmPUUFtYmFW0CCGE+TIHQ0f3KXnSFtLTcsh yVTORaopzcAuUZvKy5EbJJlw63F0UNrACiTAihby7ywJiw3UaGrAQc5TDH8ihxJ+9A1t tNRkArCFUkA+Akf0tF7+P36vw2kT12CWuUoehPkjUgTLcOLB4XhmkJvf3GhvXML68prw oEMGvTRv/GET4X1cfeG1YJWq7jzHoqnJ4fhbcYYwugNfJsFMtKmn58LSvDui3OgS8Dgf Po6vMxQcomZbwJbaXZAn0t817hA5jKmmk+xGUdR0YDWng53ofhPiBGUByOj1DPUjAfpi 48fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KUu8wEBDFHkS++DKeU8RnSaGh748X1gceNhCKSNAVhw=; b=bSsXdpqt6NyT6jlvPlxlHxV2U1eUmG/YByo2XlsuquWggirLvVftQyTSg/6tBUSW1E CHkfpI7NXeI3di0O21/bcm8GKSvcpdMt12gZlLFBTpYAkIanOeXYFstNHACRAJmONjzN qnA0VcfOu2qc0RP+oWLkEBwup6NIfWFanKXMUtQ3r1FMicSBiuLUcj+DA0fkq4Sp3GYW iIX6PAI4EdxvhWctUu5IA91j1RccOwdPiksN0r2EzbP7kN32ChPZtnzJjVATNkOyEVE6 tED6v0rta/cCol6eM+NnVY3NXXbt++wcPdVB/HPRdgXLpla4IhbctYy57mtyxOkwAN7R yQ8A== X-Gm-Message-State: AOAM5300LvenvQrnTQcFazvTi1/roP4LxOPvz/M5kx3/RCyi+H1oOcmW xRQ7X2y5TPujmwr055lkuT5wVRXGSSk= X-Google-Smtp-Source: ABdhPJyMo05Qsjnd4q1F0CJzO18MdeFUD0gNZk/EIfpqJBg18/mRg5aUXKs3hA3T3E9sxHUec2XXWA== X-Received: by 2002:a1c:20c5:: with SMTP id g188mr2128756wmg.182.1631625423825; Tue, 14 Sep 2021 06:17:03 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:03 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 16/20] trace-cmd library: Decompress file sections on reading Date: Tue, 14 Sep 2021 16:16:41 +0300 Message-Id: <20210914131645.3966308-17-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When reading sections from trace file v7 - check section flags to find out if the section is compressed and decompress it. Signed-off-by: Tzvetomir Stoyanov (VMware) --- lib/trace-cmd/trace-input.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/trace-cmd/trace-input.c b/lib/trace-cmd/trace-input.c index 80230ba5..290bcf5e 100644 --- a/lib/trace-cmd/trace-input.c +++ b/lib/trace-cmd/trace-input.c @@ -480,12 +480,15 @@ static struct file_section *section_open(struct tracecmd_input *handle, int id) if (lseek64(handle->fd, sec->data_offset, SEEK_SET) == (off64_t)-1) return NULL; + if ((sec->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return NULL; return sec; } static void section_close(struct tracecmd_input *handle, struct file_section *sec) { - /* To Do */ + if (sec->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); } static int section_add_or_update(struct tracecmd_input *handle, int id, int flags, @@ -1116,6 +1119,8 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se return -1; section->data_offset = lseek64(handle->fd, 0, SEEK_CUR); + if ((section->flags & TRACECMD_SEC_FL_COMPRESS) && in_uncompress_block(handle)) + return -1; switch (section->id) { case TRACECMD_OPTION_HEADER_INFO: @@ -1141,6 +1146,9 @@ static int handle_section(struct tracecmd_input *handle, struct file_section *se break; } + if (section->flags & TRACECMD_SEC_FL_COMPRESS) + in_uncompress_reset(handle); + return ret; } From patchwork Tue Sep 14 13:16:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC450C433F5 for ; Tue, 14 Sep 2021 13:17:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A3B2F610E6 for ; Tue, 14 Sep 2021 13:17:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232989AbhINNS2 (ORCPT ); Tue, 14 Sep 2021 09:18:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233087AbhINNSX (ORCPT ); Tue, 14 Sep 2021 09:18:23 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6888EC061760 for ; Tue, 14 Sep 2021 06:17:06 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id z184-20020a1c7ec1000000b003065f0bc631so2097990wmc.0 for ; Tue, 14 Sep 2021 06:17:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CTrf7INUA20Bk1rTfPxVSMlhRbenOJ5lSnPqezUKuCY=; b=pnpBhR+fCp2gh0tEnUkmvN2t/ch01tSlphaISctRzOcgDY56ODrb0yrJovrKTUOwiG bQsbkd39LFGIt24pwPighKFN+/TZu7NVFj0LJxxF1R8jWXkf+vB6vIvt3MTDHsVMPu03 0fRYWnBoM7PakVareDnVPff93w77V7eKHKMrOcW2i/u/aMjSeAdX2i2GwwddQYyUlapL sDDYKKfU0dm3XZAg20uCFpHguNwbiEKzpRStYjuU126l+o6xHhMvyxNngXxHPylYH4TE qZXXvgcGRMg2GDJREmFHaxGh1JU+jvv/bEPhLHBhP28KGGQmf6ZnHfbmsWNdzMhfjoFZ lV0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CTrf7INUA20Bk1rTfPxVSMlhRbenOJ5lSnPqezUKuCY=; b=r0hfqEL1Nj2+KFHk0308irRjXWoF6kqx1yWgr89EAVZVLOmXgHz8T+CgJLpn7RupwR IRdAJratRS+RabaBmz76p9aaZ4e0ivcBWW1ydLI2vkmiSHUWMKjvv5GFnkqSM8xdtNwH RaV66UrqarBQS1JdCagKNYKkIOhF8rA2anOZNxYk83tF7J9GSL205HRF+MOzHciH0kge eW3qKDBZN3Ry2QkC/nlrlZPdxzyCbP0it9DCmBAm0qxswbxWLS1aX7uh1y8ilQD9lci6 ym3xAOgQGZa01+Bmzpigt8GaEXrdXQD9r3/1YO6J+1QnVLZkI3HCVOKgq56dJC6tbXe0 P9GA== X-Gm-Message-State: AOAM530Nc12aRcBCpsmcdtktlUx4b0+80o73uNpq0KCwUmr+AOG0/bII iMu/3ruIhGlxHHax8dqzA5rvN5N+G1Y= X-Google-Smtp-Source: ABdhPJzBAham+DnXVUKkWpoYRxZoVUFnXa9klH44DXxANcy2DpAmL+lbOo1dEq2NUB0BLCoHZc8QPA== X-Received: by 2002:a7b:cb86:: with SMTP id m6mr2187850wmi.4.1631625425003; Tue, 14 Sep 2021 06:17:05 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:04 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 17/20] trace-cmd library: Add zlib compression algorithm Date: Tue, 14 Sep 2021 16:16:42 +0300 Message-Id: <20210914131645.3966308-18-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Added implementation for zlib compression algorithm, using libz library, if available. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Makefile | 7 ++ lib/trace-cmd/Makefile | 7 ++ lib/trace-cmd/include/trace-cmd-local.h | 4 + lib/trace-cmd/trace-compress-zlib.c | 109 ++++++++++++++++++++++++ lib/trace-cmd/trace-compress.c | 3 + tracecmd/Makefile | 4 + 6 files changed, 134 insertions(+) create mode 100644 lib/trace-cmd/trace-compress-zlib.c diff --git a/Makefile b/Makefile index 91f62859..c9faf8db 100644 --- a/Makefile +++ b/Makefile @@ -300,6 +300,13 @@ ifeq ($(PERF_DEFINED), 1) CFLAGS += -DPERF endif +ZLIB_INSTALLED := $(shell if (printf "$(pound)include \n void main(){deflateInit(NULL, Z_BEST_COMPRESSION);}" | $(CC) -o /dev/null -x c - -lz >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) +ifeq ($(ZLIB_INSTALLED), 1) +export ZLIB_INSTALLED +CFLAGS += -DHAVE_ZLIB +$(info Have zlib compression support) +endif + CUNIT_INSTALLED := $(shell if (printf "$(pound)include \n void main(){CU_initialize_registry();}" | $(CC) -o /dev/null -x c - -lcunit >/dev/null 2>&1) ; then echo 1; else echo 0 ; fi) export CUNIT_INSTALLED diff --git a/lib/trace-cmd/Makefile b/lib/trace-cmd/Makefile index bab4322d..1ee6ac4c 100644 --- a/lib/trace-cmd/Makefile +++ b/lib/trace-cmd/Makefile @@ -26,6 +26,9 @@ OBJS += trace-timesync-ptp.o OBJS += trace-timesync-kvm.o endif OBJS += trace-compress.o +ifeq ($(ZLIB_INSTALLED), 1) +OBJS += trace-compress-zlib.o +endif # Additional util objects OBJS += trace-blk-hack.o @@ -47,6 +50,10 @@ $(LIBTRACECMD_STATIC): $(OBJS) LIBS = $(LIBTRACEEVENT_LDLAGS) $(LIBTRACEFS_LDLAGS) -lpthread +ifeq ($(ZLIB_INSTALLED), 1) +LIBS += -lz +endif + $(LIBTRACECMD_SHARED_VERSION): $(LIBTRACECMD_SHARED) @ln -sf $( + * + */ +#include +#include +#include +#include + +#include "trace-cmd-private.h" + +#define __ZLIB_NAME "zlib" +#define __ZLIB_WEIGTH 10 + +static int zlib_compress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = compress2((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes, Z_BEST_COMPRESSION); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_STREAM_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static int zlib_decompress(const char *in, unsigned int in_bytes, + char *out, unsigned int *out_bytes) +{ + unsigned long out_size = *out_bytes; + int ret; + + ret = uncompress((unsigned char *)out, &out_size, + (unsigned char *)in, (unsigned long)in_bytes); + *out_bytes = out_size; + errno = 0; + switch (ret) { + case Z_OK: + return 0; + case Z_BUF_ERROR: + errno = -ENOBUFS; + break; + case Z_MEM_ERROR: + errno = -ENOMEM; + break; + case Z_DATA_ERROR: + errno = -EINVAL; + break; + default: + errno = -EFAULT; + break; + } + + return -1; +} + +static unsigned int zlib_compress_bound(unsigned int in_bytes) +{ + return compressBound(in_bytes); +} + +static bool zlib_is_supported(const char *name, const char *version) +{ + const char *zver; + + if (!name) + return false; + if (strlen(name) != strlen(__ZLIB_NAME) || strcmp(name, __ZLIB_NAME)) + return false; + + if (!version) + return true; + + zver = zlibVersion(); + if (!zver) + return false; + + /* Compare the major version number */ + if (atoi(version) <= atoi(zver)) + return true; + + return false; +} + +int tracecmd_zlib_init(void) +{ + return tracecmd_compress_proto_register(__ZLIB_NAME, zlibVersion(), __ZLIB_WEIGTH, + zlib_compress, zlib_decompress, + zlib_compress_bound, zlib_is_supported); +} diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index b198b8dc..c1f37cdf 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -356,6 +356,9 @@ void tracecmd_compress_init(void) gettimeofday(&time, NULL); srand((time.tv_sec * 1000) + (time.tv_usec / 1000)); +#ifdef HAVE_ZLIB + tracecmd_zlib_init(); +#endif } static struct compress_proto *compress_proto_select(void) diff --git a/tracecmd/Makefile b/tracecmd/Makefile index 80c69bbb..b7a23dc4 100644 --- a/tracecmd/Makefile +++ b/tracecmd/Makefile @@ -51,6 +51,10 @@ CONFIG_INCLUDES = CONFIG_LIBS = -lrt -lpthread $(TRACE_LIBS) CONFIG_FLAGS = +ifeq ($(ZLIB_INSTALLED), 1) +CONFIG_LIBS += -lz +endif + all: $(TARGETS) $(bdir): From patchwork Tue Sep 14 13:16:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22560C433FE for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0DCC7610E6 for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233087AbhINNS2 (ORCPT ); Tue, 14 Sep 2021 09:18:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233113AbhINNSZ (ORCPT ); Tue, 14 Sep 2021 09:18:25 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 943B6C061762 for ; Tue, 14 Sep 2021 06:17:07 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id s24so9228496wmh.4 for ; Tue, 14 Sep 2021 06:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=oXtpLjTiWXbn35JVk50LIfxlFqFgTcshU4lyACBZp7Lhh7w86qK6N3qu1jzsQPqNMh Y2q6GSwOk75d28cEuGf79kIqS0spUqOt82KARuBOmSzomQFATEdXNKqBUvacXnw/l/Av 8V78ikP1WtlLgX3fbPR2SNheWOvqI81nbb1OWRsMviDsLxqB6ih1ha5jNlOYY+3AGxT+ H+HUc9Q7DT2iqFSNzANqNXc1KSKgrRn7uzUAhNJms8zyYHtW0PwDiF2XPxG7CRSgGIfj SL4sqfhvsmqDRy8R+j3ISe+xazKmljxti17KRUqQ5xugOm2qtnfpZtshBVZPxATNKoZ1 IfWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cGVmGD9Qf7kvIF9D5l30TnsDDlX1yRgwYMIHptNl3Bo=; b=Bfqh9jPWdWkdavpxzjAqEGVc+senpW5Ga3+CKb2ICskoMYiu9QHOOGNitVCGcYtYb5 +FaVhGLHYEiWchRGUYDOfYMiG8x4w8Q3lLfgS2xREZULaWDAhcD5C2Y4dPVpA3xtn85n eDtHjDJWUNHEiWPlS60EL0sSxwA4vZh+uMm7H8UXQikZ57X84V+kGlhLUuxHslRvFnhx gB1gsQfdaqAB5GEII1FY/+QGh7HpcTqwm454CPzthel9LJE1NVjWUUBf/0k1vb4tKawy QZL12gEzcbZa45Em0Gw9naSjy2lbh7wNVA9T0bbcHm7aKkH6Nwnjp9JXFExyXWUSNX+b y2hA== X-Gm-Message-State: AOAM533J8AFHViR32lCIrsgSUfnQDg2bSEDNjecI77dlVpuuGLCpVARS /Vt8Vo/SMR292HfriswJjjCUbkCQp8s= X-Google-Smtp-Source: ABdhPJzcUBjIKRQH+nvcbCh3gwjWLaPJcZggiqYzUXam6W9ezHM9VUw7vEpuySoJQITabXy9h1cNeA== X-Received: by 2002:a7b:c18c:: with SMTP id y12mr2166698wmi.3.1631625426209; Tue, 14 Sep 2021 06:17:06 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:05 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 18/20] trace-cmd list: Show supported compression algorithms Date: Tue, 14 Sep 2021 16:16:43 +0300 Message-Id: <20210914131645.3966308-19-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Add new parameter "trace-cmd list -c" to show supported compression algorithms. Signed-off-by: Tzvetomir Stoyanov (VMware) --- Documentation/trace-cmd/trace-cmd-list.1.txt | 3 +++ tracecmd/trace-list.c | 26 ++++++++++++++++++++ tracecmd/trace-usage.c | 1 + 3 files changed, 30 insertions(+) diff --git a/Documentation/trace-cmd/trace-cmd-list.1.txt b/Documentation/trace-cmd/trace-cmd-list.1.txt index a5c6b16c..b77e3460 100644 --- a/Documentation/trace-cmd/trace-cmd-list.1.txt +++ b/Documentation/trace-cmd/trace-cmd-list.1.txt @@ -71,6 +71,9 @@ OPTIONS List defined clocks that can be used with trace-cmd record -C. The one in brackets ([]) is the active clock. +*-c*:: + List the available trace file compression algorithms. + SEE ALSO -------- trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), diff --git a/tracecmd/trace-list.c b/tracecmd/trace-list.c index d060c810..900da73b 100644 --- a/tracecmd/trace-list.c +++ b/tracecmd/trace-list.c @@ -549,6 +549,24 @@ static void show_plugins(void) tep_free(pevent); } +static void show_compression(void) +{ + char **versions, **names; + int c, i; + + c = tracecmd_compress_protos_get(&names, &versions); + if (c <= 0) { + printf("No compression algorithms are supported\n"); + return; + } + printf("Supported compression algorithms:\n"); + for (i = 0; i < c; i++) + printf("\t%s, %s\n", names[i], versions[i]); + + free(names); + free(versions); +} + void trace_list(int argc, char **argv) { int events = 0; @@ -562,6 +580,7 @@ void trace_list(int argc, char **argv) int flags = 0; int systems = 0; int show_all = 1; + int compression = 0; int i; const char *arg; const char *funcre = NULL; @@ -626,6 +645,10 @@ void trace_list(int argc, char **argv) systems = 1; show_all = 0; break; + case 'c': + compression = 1; + show_all = 0; + break; case '-': if (strcmp(argv[i], "--debug") == 0) { tracecmd_set_debug(true); @@ -670,6 +693,8 @@ void trace_list(int argc, char **argv) show_clocks(); if (systems) show_systems(); + if (compression) + show_compression(); if (show_all) { printf("event systems:\n"); show_systems(); @@ -679,6 +704,7 @@ void trace_list(int argc, char **argv) show_tracers(); printf("\noptions:\n"); show_options(); + show_compression(); } return; diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index ac12b066..34c6cc35 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -348,6 +348,7 @@ static struct usage_help usage_help[] = { " -O list plugin options\n" " -B list defined buffer instances\n" " -C list the defined clocks (and active one)\n" + " -c list the supported trace file compression algorithms\n" }, { "restore", From patchwork Tue Sep 14 13:16:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5B0EC43219 for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 905FA610CE for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233214AbhINNS3 (ORCPT ); Tue, 14 Sep 2021 09:18:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233199AbhINNS0 (ORCPT ); Tue, 14 Sep 2021 09:18:26 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B63A9C061764 for ; Tue, 14 Sep 2021 06:17:08 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u16so20167981wrn.5 for ; Tue, 14 Sep 2021 06:17:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yao3HSHynO3fw2qebc1cDpu949X61cymkG+rYeq5V9A=; b=HgrT/2TETgdp8SrUiZYK5L45+IfAGrMFWL9Zshf6mk9qeOATdvqMJWATrvYq/se27J pcrmkXgBwcOd8EGBSAzpvcLV4kC24ZLgy/gbzWntWaiNMJvbwKA8CNe4k4tlGd7ak9sX 5dvxmv0+91suRNq1dYWm5T8xCvhjrjJyx4Jc1U6TIhcJXl+oQEzNM63hoso4RZCt7VAT 2RbtiY2nob1/Deu6FUqT8NBZ9j1y0LRdf8mgbC7R+pXnHhb5sou4gjmBZab7wg/guh5G pWB2A6Hl8Kr9LHoONx9zHTtXQszGPTIfyL0Wc6+B9wGYS9tlSEtS4Aow1V2j9x9dYvTU SIow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yao3HSHynO3fw2qebc1cDpu949X61cymkG+rYeq5V9A=; b=WKYuEjeFdBjiLLyRtUO2+VXnP7retxVkmmF6WVhJahEmfFpPrbEvztB58mazT8M4Ci Fm4Sm44R0P1mlR6a2FPSSlwVo4KgSecxnJ3ZEiPZsagwrYwH2xkIWbkS//b+wuHRiM6Q IDrOaWnCc37cW7Nim0wq4jZ2fFT672qYK2rVSQlSVNO0+S4iQKICOzl+4OIEiuVqFNZa z+2BEZEsLDNb2d9FBThuCRhH/LqxckPsLlTqmfmz9rY1otKAOxWvG9jcBCUFUx8qZhop +38XUVH5oJLIfNqthptjLeIiEBAcagC78T1xpsibVjgq484ZK/U0o33xexvyjt5scCiK Ulxw== X-Gm-Message-State: AOAM5339xR0+O2P9L5c+luNIVDbymyxEAH0X4pqVUn8aPWBgFG7JP8Wl LXruPzwnGeS0KOvf9i1ubO1xynak2Dg= X-Google-Smtp-Source: ABdhPJwEw+gZw5eJKx6Q2TGLbpJAtV5TeNqAIte7YwhaEhMhbr6BoPbZ5cjs2WMFee8mbuoxwonT+g== X-Received: by 2002:a5d:61c1:: with SMTP id q1mr17100691wrv.154.1631625427285; Tue, 14 Sep 2021 06:17:07 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:06 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 19/20] trace-cmd record: Add compression to the trace context Date: Tue, 14 Sep 2021 16:16:44 +0300 Message-Id: <20210914131645.3966308-20-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org As the trace-cmd library supports trace file compression, trace-cmd record command should have a way to configure this functionality. Trace context is extended to hold the compression algorithm, used to compress the file. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index fb6de81b..72592c60 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -199,6 +199,7 @@ struct common_record_context { char *date2ts; char *user; const char *clock; + const char *compression; struct tsc_nsec tsc2nsec; int data_flags; int tsync_loop_interval; @@ -3702,6 +3703,12 @@ static struct tracecmd_output *create_net_output(struct common_record_context *c goto error; if (tracecmd_output_set_msg(out, msg_handle)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; @@ -3748,6 +3755,12 @@ setup_connection(struct buffer_instance *instance, struct common_record_context goto error; if (tracecmd_output_set_version(network_handle, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(network_handle, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(network_handle, "any"); + } if (tracecmd_output_write_headers(network_handle, listed_events)) goto error; tracecmd_set_quiet(network_handle, quiet); @@ -4481,6 +4494,12 @@ static struct tracecmd_output *create_output(struct common_record_context *ctx) goto error; if (ctx->file_version && tracecmd_output_set_version(out, ctx->file_version)) goto error; + if (ctx->compression) { + if (tracecmd_output_set_compression(out, ctx->compression)) + goto error; + } else if (ctx->file_version >= FILE_VERSION_COMPRESSION) { + tracecmd_output_set_compression(out, "any"); + } if (tracecmd_output_write_headers(out, listed_events)) goto error; return out; @@ -4516,7 +4535,7 @@ static void record_data(struct common_record_context *ctx) if (latency) { handle = tracecmd_create_file_latency(ctx->output, local_cpu_count, - ctx->file_version, NULL); + ctx->file_version, ctx->compression); tracecmd_set_quiet(handle, quiet); } else { if (!local_cpu_count) From patchwork Tue Sep 14 13:16:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12493329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530C2C4332F for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 406E2610A6 for ; Tue, 14 Sep 2021 13:17:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233113AbhINNS2 (ORCPT ); Tue, 14 Sep 2021 09:18:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233214AbhINNS1 (ORCPT ); Tue, 14 Sep 2021 09:18:27 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEC0EC061574 for ; Tue, 14 Sep 2021 06:17:09 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id b6so20150619wrh.10 for ; Tue, 14 Sep 2021 06:17:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r6lgmGCkcNCWvC0yzxh7nAD0rs8LGbIWc6U6pt0m1uQ=; b=bJaVe0Ec51fgupu86UCtSZ84j1ckVCCcm30LqtA489LteI/DHXrnbSY8xeYVT3FxBV SyC9wcf698h8+kactdn9023VcJQriCSjFgnHSqhR59NR3M8+IALnSPwRpU6Lp6SbYA5M PSkKdxrT1OKk6VPhP79exTm0zzCj+SCrvPpxwCdzS3s54qxQcZTiF/19uk/ac6RX3NMF pu3JXuFP0XYKeahXmuZtfs+u7IvXJBJHVLSZJlvF2CSbBgHvoQrmya5nz/Q6DMghaDrU gR+x1frk0IMry0AKpgdErt7pWjWWyrxyYFvGIE4YBPxtE3DksQW0j7G8AgCWTmOdxnYC iTQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r6lgmGCkcNCWvC0yzxh7nAD0rs8LGbIWc6U6pt0m1uQ=; b=BTIYdFNG/Wqck2adH3WLDJg1Sx7kbJ12jUTMDRzn5t8tX6uut34jplZhn77byunzaj SAfuoWwhe+fwcR53wko8Yp0h78UOPHdCw2n3dTzkfWWuFK4Z1gimukR+T6gqjy9N0vlX G2mGHue4I3f46eSi7CH8LsQcNS6yae5mnJtxUGEG7YB7LQ0gsgR2Wf9AqnfApq+2GIcy QwfUPamdAVg9AmgzG8tyMFEnh3pRcJ9JLyAk7R90m8ifCDix+Ch41FlXnzxUHvPPe/7A fZV41nd4ywrSZLIqGiJjGb7v6Fr+weAw6e/kB8pR0oEUBPktqbj2++hE1Kp7tqfq9Dev 23bg== X-Gm-Message-State: AOAM532XwKG1nLIdnqUw49AAamrq3VCX0Zofxl5jMoF1U/cuS12W9mtf lICbserIvuuAHOmDI5zQL8b4qu7zEJ4= X-Google-Smtp-Source: ABdhPJwf3lqLxazfYrg4PYFlzNdGyXX0NZ/3IgF/YpnkdNRTbhqgZAhEmjaDvTI1eemTZSQ57U5r/A== X-Received: by 2002:a5d:66d1:: with SMTP id k17mr14331838wrw.200.1631625428412; Tue, 14 Sep 2021 06:17:08 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id q10sm1119624wmq.12.2021.09.14.06.17.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 06:17:07 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v2 20/20] trace-cmd report: Add new parameter for trace file compression Date: Tue, 14 Sep 2021 16:16:45 +0300 Message-Id: <20210914131645.3966308-21-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210914131645.3966308-1-tz.stoyanov@gmail.com> References: <20210914131645.3966308-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org A new parameter is added, which can be used to set desired compression the output trace file. "trace-cmd report --compression " Where the string can be compression algorithm name, "any" for the best available algorithm or "none" for no compression. Signed-off-by: Tzvetomir Stoyanov (VMware) --- tracecmd/trace-record.c | 13 +++++++++++++ tracecmd/trace-usage.c | 5 +++++ 2 files changed, 18 insertions(+) diff --git a/tracecmd/trace-record.c b/tracecmd/trace-record.c index 72592c60..dceed783 100644 --- a/tracecmd/trace-record.c +++ b/tracecmd/trace-record.c @@ -5810,6 +5810,7 @@ void init_top_instance(void) } enum { + OPT_compression = 237, OPT_file_ver = 238, OPT_verbose = 239, OPT_tsc2nsec = 240, @@ -6250,6 +6251,7 @@ static void parse_record_options(int argc, {"tsc2nsec", no_argument, NULL, OPT_tsc2nsec}, {"poll", no_argument, NULL, OPT_poll}, {"verbose", optional_argument, NULL, OPT_verbose}, + {"compression", required_argument, NULL, OPT_compression}, {"file-version", required_argument, NULL, OPT_file_ver}, {NULL, 0, NULL, 0} }; @@ -6676,6 +6678,17 @@ static void parse_record_options(int argc, cmd_check_die(ctx, CMD_set, *(argv+1), "--poll"); recorder_flags |= TRACECMD_RECORD_POLL; break; + case OPT_compression: + cmd_check_die(ctx, CMD_start, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_set, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_extract, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_stream, *(argv+1), "--compression"); + cmd_check_die(ctx, CMD_profile, *(argv+1), "--compression"); + if (strcmp(optarg, "any") && strcmp(optarg, "none") && + !tracecmd_compress_is_supported(optarg, NULL)) + die("Compression algorithm %s is not supported", optarg); + ctx->compression = strdup(optarg); + break; case OPT_file_ver: cmd_check_die(ctx, CMD_start, *(argv+1), "--file_version"); cmd_check_die(ctx, CMD_set, *(argv+1), "--file_version"); diff --git a/tracecmd/trace-usage.c b/tracecmd/trace-usage.c index 34c6cc35..77898c1c 100644 --- a/tracecmd/trace-usage.c +++ b/tracecmd/trace-usage.c @@ -70,6 +70,11 @@ static struct usage_help usage_help[] = { " at the beginnig and at the end of the trace\n" " --poll don't block while reading from the trace buffer\n" " --file-version set the desired trace file version\n" + " --compression compress the trace output file, one of these strings can be passed:\n" + " any - auto select the best available compression algorithm\n" + " none - do not compress the trace file\n" + " name - the name of the desired compression algorithms\n" + " available algorithms can be listed with trace-cmd list -c\n" }, { "set",