From patchwork Wed Mar 27 16:03:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873727 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C3A5922 for ; Wed, 27 Mar 2019 16:03:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88B6428759 for ; Wed, 27 Mar 2019 16:03:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D42C287C2; Wed, 27 Mar 2019 16:03:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A31828759 for ; Wed, 27 Mar 2019 16:03:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727389AbfC0QDq (ORCPT ); Wed, 27 Mar 2019 12:03:46 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:32927 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726990AbfC0QDp (ORCPT ); Wed, 27 Mar 2019 12:03:45 -0400 Received: by mail-pg1-f195.google.com with SMTP id b12so10178436pgk.0 for ; Wed, 27 Mar 2019 09:03:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lPWoOzH5YTvnrwVFuW5joRTOY72FphCBaUp1V74GVWQ=; b=Shmr3vtjoYx9pZ+yiXWyZiuOm4C//KDokzYhN2q1sGDLKKD6v2xh0zoRh5uObnX1rB CUZBWb1ecx0nzthBHNX34X6/uLM9iZNI8Wi5Y8F0Y9oBUls15FSp6xikFWfZZHPe/YVA 0cn6835gsNL8zBJtScRUlUwdOQd68ovlvbyEdLQTRPAlp8l7N+Bw8n0Sp5qZfnAKDFuT 5fywHOCxPwhabEKuiEXQIjDD6UTWFeGbc56naAQ7C4V5VwjFKq/uzzYBIGWSvMabnWh6 qyoti4jJGynGwO3XjptO5axqC8Z/c/zvbkzavywvI5OrD6UDobJ8o2NZGsfLU6nWCsDE XVAw== X-Gm-Message-State: APjAAAVbMABo+eKS3ZP0/HtKErP8NjmKS/604AcvZH9+lSw2dPZC+doZ 8QXJUpmulrEMiTWg6LYunRtR+kBhIhc= X-Google-Smtp-Source: APXvYqxEAyyZp16eDeuW2p6a2KY/QHrgT6hlkH+z991RanyoJezhciFCtibPFtOnddw8wibOLmUQSg== X-Received: by 2002:aa7:8c42:: with SMTP id e2mr6796652pfd.24.1553702625312; Wed, 27 Mar 2019 09:03:45 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:44 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 1/7] kernel-shark: kshark_string_config_alloc() must take no arguments Date: Wed, 27 Mar 2019 18:03:17 +0200 Message-Id: <20190327160323.31654-2-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The function is not meant to have arguments, so it must explicitly state this. A functions taking unspecified list of parameters will be a problem for the Pythion interface. Signed-off-by: Yordan Karadzhov --- kernel-shark/src/libkshark-configio.c | 2 +- kernel-shark/src/libkshark.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel-shark/src/libkshark-configio.c b/kernel-shark/src/libkshark-configio.c index 2bd5600..9106522 100644 --- a/kernel-shark/src/libkshark-configio.c +++ b/kernel-shark/src/libkshark-configio.c @@ -369,7 +369,7 @@ kshark_filter_config_new(enum kshark_config_formats format) * @returns kshark_config_doc instance on success, otherwise NULL. Use free() * to free the object. */ -struct kshark_config_doc *kshark_string_config_alloc() +struct kshark_config_doc *kshark_string_config_alloc(void) { return kshark_config_alloc(KS_CONFIG_STRING); } diff --git a/kernel-shark/src/libkshark.h b/kernel-shark/src/libkshark.h index a1b1f91..c218b61 100644 --- a/kernel-shark/src/libkshark.h +++ b/kernel-shark/src/libkshark.h @@ -541,7 +541,7 @@ kshark_record_config_new(enum kshark_config_formats); struct kshark_config_doc * kshark_filter_config_new(enum kshark_config_formats); -struct kshark_config_doc *kshark_string_config_alloc(); +struct kshark_config_doc *kshark_string_config_alloc(void); bool kshark_type_check(struct kshark_config_doc *conf, const char *type); From patchwork Wed Mar 27 16:03:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873729 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EB2917E0 for ; Wed, 27 Mar 2019 16:03:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A3E428759 for ; Wed, 27 Mar 2019 16:03:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E3CD287C2; Wed, 27 Mar 2019 16:03:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DDDA28759 for ; Wed, 27 Mar 2019 16:03:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727405AbfC0QDs (ORCPT ); Wed, 27 Mar 2019 12:03:48 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:34607 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726990AbfC0QDs (ORCPT ); Wed, 27 Mar 2019 12:03:48 -0400 Received: by mail-pf1-f193.google.com with SMTP id b3so8550169pfd.1 for ; Wed, 27 Mar 2019 09:03:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S4vA5wGkuKQVJQjjTTVBOBVpJVgygzrAd4O1sEikO9o=; b=mCpAuuFZ4wRXkHwgsmCuPhBYHOTCq+0dBiqFj2rGKTSYZzlTDRWGCEy/CU9Mo6KaMe S/Ij4gNvXAnzrQJtI8PTzN8V1ZT46VPDKevDZheL/uyxJdi5KqwBIeFs0Tsj9RvGO+yP bkrRQFnU7XW5sQIwKrWoGFJuaDaQTJSC4egHpMM9clQOL5CMzAYoHUH62RdNdNPujO41 O0xkoIYMMJT/9dv6OpErq0j8F8w2QyHH9HU4Bw+5SObIdYeFsRUHP9/UCwWywo4mfzxc 57Nglb6bHHxA0XL0g6MqnLLH+nvZcG6e+qLpo4CG4IdpEaXgxUpFEOvXoW56EcYnr3dG jQ7Q== X-Gm-Message-State: APjAAAWDXh7F7JQIk3NPLjUVGJS93RMGjtqvpV3z3PwLXckKAgRcY7YA wjkgPotfLB72eHHeEMN1taA= X-Google-Smtp-Source: APXvYqx3rvZC5ySmfbKGti36DwCN23zJOIpqOr4fILagvVzvP21mxEqi17hpO//+dSG1WPb915SOIg== X-Received: by 2002:a62:174c:: with SMTP id 73mr36156740pfx.33.1553702627345; Wed, 27 Mar 2019 09:03:47 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:46 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 2/7] kernel-shark: Add new dataloading method to be used by the NumPu interface Date: Wed, 27 Mar 2019 18:03:18 +0200 Message-Id: <20190327160323.31654-3-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The new function loads the content of the trace data file into a table / matrix, made of columns / arrays of data having various integer types. Later those arrays will be wrapped as NumPy arrays. Signed-off-by: Yordan Karadzhov --- kernel-shark/src/libkshark.c | 128 +++++++++++++++++++++++++++++++++++ kernel-shark/src/libkshark.h | 7 ++ 2 files changed, 135 insertions(+) diff --git a/kernel-shark/src/libkshark.c b/kernel-shark/src/libkshark.c index a886f80..20b1594 100644 --- a/kernel-shark/src/libkshark.c +++ b/kernel-shark/src/libkshark.c @@ -959,6 +959,134 @@ ssize_t kshark_load_data_records(struct kshark_context *kshark_ctx, return -ENOMEM; } +static bool data_matrix_alloc(size_t n_rows, uint64_t **offset_array, + uint8_t **cpu_array, + uint64_t **ts_array, + uint16_t **pid_array, + int **event_array) +{ + if (offset_array) { + *offset_array = calloc(n_rows, sizeof(**offset_array)); + if (!offset_array) + goto free_all; + } + + if (cpu_array) { + *cpu_array = calloc(n_rows, sizeof(**cpu_array)); + if (!cpu_array) + goto free_all; + } + + if (ts_array) { + *ts_array = calloc(n_rows, sizeof(**ts_array)); + if (!ts_array) + goto free_all; + } + + if (pid_array) { + *pid_array = calloc(n_rows, sizeof(**pid_array)); + if (!pid_array) + goto free_all; + } + + if (event_array) { + *event_array = calloc(n_rows, sizeof(**event_array)); + if (!event_array) + goto free_all; + } + + return true; + + free_all: + fprintf(stderr, "Failed to allocate memory during data loading.\n"); + + if (offset_array) + free(offset_array); + + if (cpu_array) + free(cpu_array); + + if (ts_array) + free(ts_array); + + if (pid_array) + free(pid_array); + + if (event_array) + free(event_array); + + return false; +} + +/** + * @brief Load the content of the trace data file into a table / matrix made + * of columns / arrays of data. The user is responsible for freeing the + * elements of the outputted array + * + * @param kshark_ctx: Input location for the session context pointer. + * @param offset_array: Output location for the array of record offsets. + * @param cpu_array: Output location for the array of CPU Ids. + * @param ts_array: Output location for the array of timestamps. + * @param pid_array: Output location for the array of Process Ids. + * @param event_array: Output location for the array of Event Ids. + * + * @returns The size of the outputted arrays in the case of success, or a + * negative error code on failure. + */ +size_t kshark_load_data_matrix(struct kshark_context *kshark_ctx, + uint64_t **offset_array, + uint8_t **cpu_array, + uint64_t **ts_array, + uint16_t **pid_array, + int **event_array) +{ + enum rec_type type = REC_RECORD; + struct rec_list **rec_list; + struct tep_record *rec; + size_t count, total = 0; + bool status; + int n_cpus; + + total = get_records(kshark_ctx, &rec_list, REC_RECORD); + if (total < 0) + goto fail; + + status = data_matrix_alloc(total, offset_array, + cpu_array, + ts_array, + pid_array, + event_array); + if (!status) + goto fail; + + n_cpus = tracecmd_cpus(kshark_ctx->handle); + + for (count = 0; count < total; count++) { + int next_cpu; + + next_cpu = pick_next_cpu(rec_list, n_cpus, type); + if (next_cpu >= 0) { + rec = rec_list[next_cpu]->rec; + + (*offset_array)[count] = rec->offset; + (*cpu_array)[count] = rec->cpu; + (*ts_array)[count] = rec->ts; + (*pid_array)[count] = tep_data_pid(kshark_ctx->pevent, rec); + (*event_array)[count] = tep_data_type(kshark_ctx->pevent, rec); + + rec_list[next_cpu] = rec_list[next_cpu]->next; + } + } + + free_rec_list(rec_list, n_cpus, type); + + return total; + + fail: + fprintf(stderr, "Failed to allocate memory during data loading.\n"); + return -ENOMEM; +} + static const char *kshark_get_latency(struct tep_handle *pe, struct tep_record *record) { diff --git a/kernel-shark/src/libkshark.h b/kernel-shark/src/libkshark.h index c218b61..92ade41 100644 --- a/kernel-shark/src/libkshark.h +++ b/kernel-shark/src/libkshark.h @@ -149,6 +149,13 @@ ssize_t kshark_load_data_entries(struct kshark_context *kshark_ctx, ssize_t kshark_load_data_records(struct kshark_context *kshark_ctx, struct tep_record ***data_rows); +size_t kshark_load_data_matrix(struct kshark_context *kshark_ctx, + uint64_t **offset_array, + uint8_t **cpu_array, + uint64_t **ts_array, + uint16_t **pid_array, + int **event_array); + ssize_t kshark_get_task_pids(struct kshark_context *kshark_ctx, int **pids); void kshark_close(struct kshark_context *kshark_ctx); From patchwork Wed Mar 27 16:03:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873731 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF11F14DE for ; Wed, 27 Mar 2019 16:03:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAC9F28759 for ; Wed, 27 Mar 2019 16:03:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF700287C8; Wed, 27 Mar 2019 16:03:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4962928759 for ; Wed, 27 Mar 2019 16:03:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727446AbfC0QDu (ORCPT ); Wed, 27 Mar 2019 12:03:50 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:35636 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726990AbfC0QDu (ORCPT ); Wed, 27 Mar 2019 12:03:50 -0400 Received: by mail-pl1-f196.google.com with SMTP id p19so3527551plo.2 for ; Wed, 27 Mar 2019 09:03:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8zVIyzNbGKGvrkTMc44Mf2RnGpRFEVAMRC3Fuut3EY4=; b=MAwxM9rXmTIMTEr+1B+KNng7xELfalSTh9B1SNeaCjholkhEjatGYEx8SdZbOHiFtM 06q/QyqA0R4OIerrVJXtdwZOoWiXWx4s1J+pxum19dkKJkn2Oi+BmfytfNkF2Wz8Z/pA EAR6CvRXix7wjFm5sA2ppSTCsHDYXmWxMaLZJBl4dFFwYt3UxPS4nKPYBHV1/XE0vsKq ba5Hj/k2JYFHKwtJYYNuKRSLntXeG6Er15OsPFymgYd67y/zX0f1a/n/wTfi+yGKsD3F aD5BEy/yrPWmvG+bhxL/YjrldFKq/lh8tn+u7fM8VlExHWtOiGsSjcsh64NBoIdrlGRa KGTQ== X-Gm-Message-State: APjAAAUq20NLFPsbOp6a0doFDXiBY0e8w1BLznsfPhxfqix+578wa+3R GV6Y2gGehkf6GO+gfktEMfg= X-Google-Smtp-Source: APXvYqy2Yes96WdAD9NBd90azT188FCf6od3fEiMGW4R9kNYMng2pHvtGdP3+xqclolikrS3EZrjvQ== X-Received: by 2002:a17:902:e684:: with SMTP id cn4mr14690498plb.71.1553702629385; Wed, 27 Mar 2019 09:03:49 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:48 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 3/7] kernel-shark: Prepare for building the NumPy interface Date: Wed, 27 Mar 2019 18:03:19 +0200 Message-Id: <20190327160323.31654-4-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch prepares the Cmake build infrastructure for the introduction of a the NumPy interface. We add building of a static version of the C API library to be used by the interface. The NumPy interface itself will be added in the following patches. Signed-off-by: Yordan Karadzhov --- kernel-shark/CMakeLists.txt | 3 +++ kernel-shark/README | 12 ++++++++++-- kernel-shark/src/CMakeLists.txt | 19 +++++++++++++++++++ 3 files changed, 32 insertions(+), 2 deletions(-) diff --git a/kernel-shark/CMakeLists.txt b/kernel-shark/CMakeLists.txt index 20ced14..0c5d7a7 100644 --- a/kernel-shark/CMakeLists.txt +++ b/kernel-shark/CMakeLists.txt @@ -34,6 +34,9 @@ if (Qt5Widgets_FOUND) endif (Qt5Widgets_FOUND) +find_package(PythonLibs) +include(${KS_DIR}/build/FindNumPy.cmake) + set(LIBRARY_OUTPUT_PATH "${KS_DIR}/lib") set(EXECUTABLE_OUTPUT_PATH "${KS_DIR}/bin") diff --git a/kernel-shark/README b/kernel-shark/README index efc6748..5a723a4 100644 --- a/kernel-shark/README +++ b/kernel-shark/README @@ -12,7 +12,11 @@ KernelShark has the following external dependencies: sudo apt-get install freeglut3-dev libxmu-dev libxi-dev -y sudo apt-get install qtbase5-dev -y -1.1 I you want to be able to generate Doxygen documentation: +1.1 I you want to be able to use the NumPu Interface of KernelShark: + sudo apt-get install libpython-dev cython -y + sudo apt-get install python-numpy python-matplotlib -y + +1.2 I you want to be able to generate Doxygen documentation: sudo apt-get install graphviz doxygen-gui -y @@ -21,7 +25,11 @@ KernelShark has the following external dependencies: dnf install freeglut-devel redhat-rpm-config -y dnf install qt5-qtbase-devel -y -2.1 I you want to be able to generate Doxygen documentation: +2.1 I you want to be able to use the NumPu Interface of KernelShark: + dnf install python2-devel python-Cython -y + dnf install python-numpy python3-matplotlib -y + +2.2 I you want to be able to generate Doxygen documentation: dnf install graphviz doxygen -y diff --git a/kernel-shark/src/CMakeLists.txt b/kernel-shark/src/CMakeLists.txt index 1e0a794..a3c6743 100644 --- a/kernel-shark/src/CMakeLists.txt +++ b/kernel-shark/src/CMakeLists.txt @@ -28,6 +28,25 @@ if (OPENGL_FOUND AND GLUT_FOUND) endif (OPENGL_FOUND AND GLUT_FOUND) +if (PYTHONLIBS_FOUND AND CYTHON_FOUND AND NUMPY_FOUND) + + message(STATUS "kshark_wrapper") + + add_library(pykshark STATIC libkshark.c + libkshark-model.c + libkshark-plugin.c + libkshark-configio.c + libkshark-collection.c) + + target_compile_options(pykshark PUBLIC "-fPIC") + + target_link_libraries(pykshark ${TRACEEVENT_LIBRARY} + ${TRACECMD_LIBRARY} + ${CMAKE_DL_LIBS} + ${JSONC_LIBRARIES}) + +endif (PYTHONLIBS_FOUND AND CYTHON_FOUND AND NUMPY_FOUND) + if (Qt5Widgets_FOUND AND Qt5Network_FOUND) message(STATUS "libkshark-gui") From patchwork Wed Mar 27 16:03:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873733 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B9F6922 for ; Wed, 27 Mar 2019 16:03:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47E8C28759 for ; Wed, 27 Mar 2019 16:03:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C91C287C8; Wed, 27 Mar 2019 16:03:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7B66287C2 for ; Wed, 27 Mar 2019 16:03:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726990AbfC0QDw (ORCPT ); Wed, 27 Mar 2019 12:03:52 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:34614 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727469AbfC0QDw (ORCPT ); Wed, 27 Mar 2019 12:03:52 -0400 Received: by mail-pf1-f193.google.com with SMTP id b3so8550253pfd.1 for ; Wed, 27 Mar 2019 09:03:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Hs6abDd+R5XRjHVJGRT3Zm2JeiFasU70eNA248oAoTA=; b=iyMCb2G1JnMnYeIYzdzFYf5RUUeKRVBpkeM95fqQMapkeAZ9oBlP9H+TEdWObwk3Cq 3D/KPyj5IkKXU1OH53dak1A5vNmcNjRCfjZRkiAUVTS9OooiFEmy6/TixOXHyiJahL6h 4Mca5Lur8UYZ3nZcoZ06I4dbc+pYhZXyeyCeVDdzchcKCheD6QOiPsc2f5NBPAHmsS/Z +/kuR76VcEdGM9TCQ3Cew0Uz9FgI6O+YFWMV3vN2ZnxQF7yue8niz4yEj34sT8z1fsfT ue+b12cauE4pP/OoA6VRfNdD4BNr+ORpfIT1olcaV1zzHjiRA/ekHo5sTEWECKt8soz6 fFXQ== X-Gm-Message-State: APjAAAUe55rElLvTRE7VIprWzYzgnpw3QW2aajf4O3zU9IsyPCLAWQb7 IRR7cOl+kqULTn8ZeYtUUtc= X-Google-Smtp-Source: APXvYqx/gAu2Z8EOUCl++y5PIP8GWyKuZj8PL71vJ1+lDGQp2jgXPxyQ6v84fVcG2rex49xCYGYPBw== X-Received: by 2002:aa7:8453:: with SMTP id r19mr8736500pfn.44.1553702631252; Wed, 27 Mar 2019 09:03:51 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:50 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 4/7] kernel-shark: Add the core components of the NumPy API Date: Wed, 27 Mar 2019 18:03:20 +0200 Message-Id: <20190327160323.31654-5-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The NumPy API is meant to operate on top of the C API-s of trace-cmd and KernelShark and to provide only the minimum of basic functionalities needed in order to processor the tracing data. The NumPy API itself is made of two layers. A bottom-one written in C and a top-one which implements the interface, written in Cython (C-Python). This patch introduces the C layer. Signed-off-by: Yordan Karadzhov --- kernel-shark/src/libkshark-py.c | 176 ++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 kernel-shark/src/libkshark-py.c diff --git a/kernel-shark/src/libkshark-py.c b/kernel-shark/src/libkshark-py.c new file mode 100644 index 0000000..4c589f3 --- /dev/null +++ b/kernel-shark/src/libkshark-py.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: LGPL-2.1 + +/* + * Copyright (C) 2019 VMware Inc, Yordan Karadzhov + */ + + /** + * @file libpykshark-py.c + * @brief Python API for processing of FTRACE (trace-cmd) data. + */ + +// KernelShark +#include "libkshark.h" +#include "libkshark-model.h" + +bool kspy_open(const char *fname) +{ + struct kshark_context *kshark_ctx = NULL; + + if (!kshark_instance(&kshark_ctx)) + return false; + + return kshark_open(kshark_ctx, fname); +} + +void kspy_close(void) +{ + struct kshark_context *kshark_ctx = NULL; + + if (!kshark_instance(&kshark_ctx)) + return; + + kshark_close(kshark_ctx); + kshark_free(kshark_ctx); +} + +static int compare(const void * a, const void * b) +{ + return ( *(int*)a - *(int*)b ); +} + +size_t kspy_get_tasks(int **pids, char ***names) +{ + struct kshark_context *kshark_ctx = NULL; + const char *comm; + ssize_t i, n; + int ret; + + if (!kshark_instance(&kshark_ctx)) + return 0; + + n = kshark_get_task_pids(kshark_ctx, pids); + if (n == 0) + return 0; + + qsort(*pids, n, sizeof(**pids), compare); + + *names = calloc(n, sizeof(char*)); + if (!(*names)) + goto fail; + + for (i = 0; i < n; ++i) { + comm = tep_data_comm_from_pid(kshark_ctx->pevent, (*pids)[i]); + ret = asprintf(&(*names)[i], "%s", comm); + if (ret < 1) + goto fail; + } + + return n; + + fail: + free(*pids); + free(*names); + return 0; +} + +size_t kspy_trace2matrix(uint64_t **offset_array, + uint8_t **cpu_array, + uint64_t **ts_array, + uint16_t **pid_array, + int **event_array) +{ + struct kshark_context *kshark_ctx = NULL; + size_t total = 0; + + if (!kshark_instance(&kshark_ctx)) + return false; + + total = kshark_load_data_matrix(kshark_ctx, offset_array, + cpu_array, + ts_array, + pid_array, + event_array); + + return total; +} + +int kspy_get_event_id(const char *sys, const char *evt) +{ + struct kshark_context *kshark_ctx = NULL; + struct tep_event *event; + + if (!kshark_instance(&kshark_ctx)) + return -1; + + event = tep_find_event_by_name(kshark_ctx->pevent, sys, evt); + + return event->id; +} + +uint64_t kspy_read_event_field(uint64_t offset, const char *sys, + const char *evt, + const char *field) +{ + struct kshark_context *kshark_ctx = NULL; + struct tep_format_field *evt_field; + struct tep_record *record; + struct tep_event *event; + unsigned long long val; + int ret; + + if (!kshark_instance(&kshark_ctx)) + return 0; + + event = tep_find_event_by_name(kshark_ctx->pevent, sys, evt); + if (!event) + return 0; + + evt_field = tep_find_any_field(event, field); + if (!evt_field) + return 0; + + record = tracecmd_read_at(kshark_ctx->handle, offset, NULL); + if (!record) + return 0; + + ret = tep_read_number_field(evt_field, record->data, &val); + free_record(record); + + if (ret != 0) + return 0; + + return val; +} + +void kspy_new_session_file(const char *data_file, const char *session_file) +{ + struct kshark_context *kshark_ctx = NULL; + struct kshark_trace_histo histo; + struct kshark_config_doc *session; + struct kshark_config_doc *filters; + struct kshark_config_doc *markers; + struct kshark_config_doc *model; + struct kshark_config_doc *file; + + if (!kshark_instance(&kshark_ctx)) + return; + + session = kshark_config_new("kshark.config.session", + KS_CONFIG_JSON); + + file = kshark_export_trace_file(data_file, KS_CONFIG_JSON); + kshark_config_doc_add(session, "Data", file); + + filters = kshark_export_all_filters(kshark_ctx, KS_CONFIG_JSON); + kshark_config_doc_add(session, "Filters", filters); + + ksmodel_init(&histo); + model = kshark_export_model(&histo, KS_CONFIG_JSON); + kshark_config_doc_add(session, "Model", model); + + markers = kshark_config_new("kshark.config.markers", KS_CONFIG_JSON); + kshark_config_doc_add(session, "Markers", markers); + + kshark_save_config_file(session_file, session); +} From patchwork Wed Mar 27 16:03:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873735 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAE0214DE for ; Wed, 27 Mar 2019 16:03:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9301F28759 for ; Wed, 27 Mar 2019 16:03:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 87492287C8; Wed, 27 Mar 2019 16:03:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7D6E28759 for ; Wed, 27 Mar 2019 16:03:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727537AbfC0QDy (ORCPT ); Wed, 27 Mar 2019 12:03:54 -0400 Received: from mail-pl1-f169.google.com ([209.85.214.169]:43441 "EHLO mail-pl1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727469AbfC0QDy (ORCPT ); Wed, 27 Mar 2019 12:03:54 -0400 Received: by mail-pl1-f169.google.com with SMTP id m10so3527415plt.10 for ; Wed, 27 Mar 2019 09:03:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kDkNZESp4lQbZ512JZjn7zRF1O4+2aKvr3Mkvj2Pd9k=; b=gcDwlx+5RjWtAt0ThECcRYIxH/+swOg1aTjFEEfpAMIu/eq3ubGUW1DRzOb1Ssae+A OUeryWwhXyk5Jc8w1QAraUj1qfXO1YK3WBvz2t2YUb6tQTJhQ8WiMuWqhKAGod7/yv9G 3cPZazFX04a1GPGeKv1wp6qODfVUUn1snO3am9MczkZjH615INd7yA/L7S26o7/dDxkM p9otZAsWvLwEhQQgqCh6AadGurrdPzO5hOYKlZ6WjtGNts8mJuFmwgv3Y0xr3mMPh1fP 61I7v34RvCPP+DM/uifejOVGkte32v18aTewBpICl8j30C/HFqZ6cfv3pM7ByK7eFFUF ANVQ== X-Gm-Message-State: APjAAAWFaiC2vfC+LjCTJr/KUyCHOy7QNcYpmZfXU+1rlxEh95gXL0KP Vdnb7WOindGhqbFMzlU1khTO7NYgut8= X-Google-Smtp-Source: APXvYqx10DgnsgcsQs5TTzB7zmz1PT4ii1i/7InoQnTpxCR1GkTetRoMWr0pcF7DuDoqmjtjSt6g0Q== X-Received: by 2002:a17:902:380c:: with SMTP id l12mr37993439plc.238.1553702633355; Wed, 27 Mar 2019 09:03:53 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:52 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 5/7] kernel-shark: Add Numpy Interface for processing of tracing data Date: Wed, 27 Mar 2019 18:03:21 +0200 Message-Id: <20190327160323.31654-6-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch contains the Cython implementation of the Interface together with a Python script used to build the corresponding library (libkshark_wrapper.so) Signed-off-by: Yordan Karadzhov --- kernel-shark/build/py/libkshark_wrapper.pyx | 264 ++++++++++++++++++++ kernel-shark/build/py/np_setup.py | 87 +++++++ 2 files changed, 351 insertions(+) create mode 100644 kernel-shark/build/py/libkshark_wrapper.pyx create mode 100644 kernel-shark/build/py/np_setup.py diff --git a/kernel-shark/build/py/libkshark_wrapper.pyx b/kernel-shark/build/py/libkshark_wrapper.pyx new file mode 100644 index 0000000..825850f --- /dev/null +++ b/kernel-shark/build/py/libkshark_wrapper.pyx @@ -0,0 +1,264 @@ +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright (C) 2017 VMware Inc, Yordan Karadzhov +""" + +# Import the Python-level symbols of numpy +import numpy as np + +# Import the C-level symbols of numpy +cimport numpy as np + +import ctypes +import json + +cdef extern from "stdint.h": + ctypedef unsigned short uint8_t + ctypedef unsigned short uint16_t + ctypedef unsigned long long uint64_t + +from libcpp cimport bool + +cdef extern from "numpy/ndarraytypes.h": + int NPY_ARRAY_CARRAY + +# Declare the prototype of the C function we are interested in calling +cdef extern from "../../src/libkshark-py.c": + bool kspy_open(const char *fname); + +cdef extern from "../../src/libkshark-py.c": + bool kspy_close() + +cdef extern from "../../src/libkshark-py.c": + size_t kspy_trace2matrix(uint64_t **offset_array, + uint8_t **cpu_array, + uint64_t **ts_array, + uint16_t **pid_array, + int **event_array); + +cdef extern from "../../src/libkshark-py.c": + int kspy_get_event_id(const char *sys, const char *evt) + +cdef extern from "../../src/libkshark-py.c": + uint64_t kspy_read_event_field(uint64_t offset, const char *sys, + const char *evt, + const char *field) + +cdef extern from "../../src/libkshark-py.c": + ssize_t kspy_get_tasks(int **pids, char ***names) + +cdef extern from "../../src/libkshark.h": + int KS_EVENT_OVERFLOW + +cdef extern from "../../src/libkshark-py.c": + void kspy_new_session_file(const char *data_file, const char *session_file) + +EVENT_OVERFLOW = KS_EVENT_OVERFLOW + +from libc.stdlib cimport free + +#from libcpp.string cimport string + +from cpython cimport PyObject, Py_INCREF + +# Numpy must be initialized!!! +np.import_array() + +cdef class KsDataWrapper: + cdef int item_size + cdef int data_size + cdef int data_type + cdef void* data_ptr + + cdef init(self, int data_type, + int data_size, + int item_size, + void* data_ptr): + """ This initialization cannot be done in the constructor because we use + C-level arguments. + """ + self.item_size = item_size + self.data_size = data_size + self.data_type = data_type + self.data_ptr = data_ptr + + def __array__(self): + """ Here we use the __array__ method, that is called when numpy + tries to get an array from the object. + """ + cdef np.npy_intp shape[1] + shape[0] = self.data_size + + ndarray = np.PyArray_New(np.ndarray, + 1, shape, + self.data_type, + NULL, + self.data_ptr, + self.item_size, + NPY_ARRAY_CARRAY, + NULL) + + return ndarray + + def __dealloc__(self): + """ Frees the data. This is called by Python when all the + references to the object are gone. + """ + free(self.data_ptr) + +def open_file(fname): + """ Open a tracing data file. + """ + return kspy_open(fname) + +def close(): + """ Open the session file. + """ + kspy_close() + +def read_event_field(offset, sys, event, field): + """ Read the value of a specific field of the trace event. + """ + cdef uint64_t v + + v = kspy_read_event_field(offset, sys, event, field) + return v + +def event_id(system, event): + """ Get the unique Id of the event + """ + return kspy_get_event_id(system, event) + +def c_str_cast(char *c_str): + """ String convertion C -> Python + """ + return ctypes.c_char_p(c_str).value + +def get_tasks(): + """ get a dictionary of all task's PIDs + """ + cdef int *pids + cdef char **names + cdef int size = kspy_get_tasks(&pids, &names) + + task_dict = {} + + for i in range(0, size): + task_dict.update({c_str_cast(names[i]) : pids[i]}) + + return task_dict + +def load_data(): + """ Python binding of the 'kshark_load_data_matrix' function that does not + copy the data. + """ + cdef uint64_t *ofst + cdef uint8_t *cpu + cdef uint64_t *ts + cdef uint16_t *pid + cdef int *evt + + cdef np.ndarray ndarray_ofst + cdef np.ndarray ndarray_cpu + cdef np.ndarray ndarray_ts + cdef np.ndarray ndarray_pid + cdef np.ndarray ndarray_evt + + # Call the C function + cdef int size = kspy_trace2matrix(&ofst, &cpu, &ts, &pid, &evt) + + array_wrapper_ofst = KsDataWrapper() + array_wrapper_cpu = KsDataWrapper() + array_wrapper_ts = KsDataWrapper() + array_wrapper_pid = KsDataWrapper() + array_wrapper_evt = KsDataWrapper() + + array_wrapper_ofst.init(data_type=np.NPY_UINT64, + item_size=0, + data_size=size, + data_ptr= ofst) + + array_wrapper_cpu.init(data_type=np.NPY_UINT8, + data_size=size, + item_size=0, + data_ptr= cpu) + + array_wrapper_ts.init(data_type=np.NPY_UINT64, + data_size=size, + item_size=0, + data_ptr= ts) + + array_wrapper_pid.init(data_type=np.NPY_UINT16, + data_size=size, + item_size=0, + data_ptr= pid) + + array_wrapper_evt.init(data_type=np.NPY_INT, + data_size=size, + item_size=0, + data_ptr= evt) + + ndarray_ofst = np.array(array_wrapper_ofst, copy=False) + ndarray_cpu = np.array(array_wrapper_cpu, copy=False) + ndarray_ts = np.array(array_wrapper_ts, copy=False) + ndarray_pid = np.array(array_wrapper_pid, copy=False) + ndarray_evt = np.array(array_wrapper_evt, copy=False) + + # Assign our object to the 'base' of the ndarray object + ndarray_ofst.base = array_wrapper_ofst + ndarray_cpu.base = array_wrapper_cpu + ndarray_ts.base = array_wrapper_ts + ndarray_pid.base = array_wrapper_pid + ndarray_evt.base = array_wrapper_evt + + # Increment the reference count, as the above assignement was done in + # C, and Python does not know that there is this additional reference + Py_INCREF(array_wrapper_ofst) + Py_INCREF(array_wrapper_cpu) + Py_INCREF(array_wrapper_ts) + Py_INCREF(array_wrapper_pid) + Py_INCREF(array_wrapper_evt) + + return ndarray_ofst, ndarray_cpu, ndarray_evt, ndarray_pid, ndarray_ts + +def save_session(session, s): + """ Save a KernelShark session description of a JSON file. + """ + s.seek(0) + json.dump(session, s, indent=4) + s.truncate() + +def new_session(fname, sname): + """ Generate and save a default KernelShark session description file (JSON). + """ + kspy_new_session_file(fname, sname) + + with open(sname, "r+") as s: + session = json.load(s) + + session['Filters']['filter mask'] = 7 + + session['CPUPlots'] = [] + + session['TaskPlots'] = [] + + session['Splitter'] = [1, 1] + + session['MainWindow'] = [1200, 800] + + session['ViewTop'] = 0 + + session['ColorScheme'] = 0.75 + + session['Model']['bins'] = 1000 + + session['Markers']['markA'] = {} + session['Markers']['markA']['isSet'] = False + + session['Markers']['markB'] = {} + session['Markers']['markB']['isSet'] = False + + session['Markers']['Active'] = 'A' + + save_session(session, s) diff --git a/kernel-shark/build/py/np_setup.py b/kernel-shark/build/py/np_setup.py new file mode 100644 index 0000000..602a99c --- /dev/null +++ b/kernel-shark/build/py/np_setup.py @@ -0,0 +1,87 @@ +#!/usr/bin/python + +import sys, getopt +import numpy +from Cython.Distutils import build_ext + +def libs(argv): + pykslibdir = '' + evlibdir = '' + evincdir = '' + trlibdir = '' + trincdir = '' + jsnincdir = '' + + try: + opts, args = getopt.getopt(argv,"l:t:i:e:n:j:", \ + ["pykslibdir=", \ + "trlibdir=", \ + "trincdir=", \ + "evlibdir=", \ + "evincdir=", \ + "jsnincdir="]) + except getopt.GetoptError: + sys.exit(2) + for opt, arg in opts: + if opt in ("-l", "--pykslibdir"): + pykslibdir = arg + elif opt in ("-t", "--trlibdir"): + trlibdir = arg + elif opt in ("-i", "--trincdir"): + trincdir = arg + elif opt in ("-e", "--evlibdir"): + evlibdir = arg + elif opt in ("-n", "--evincdir"): + evincdir = arg + elif opt in ("-j", "--jsnincdir"): + jsnincdir = arg + + cmd1 = 1 + for i in xrange(len(sys.argv)): + if sys.argv[i] == "build_ext": + cmd1 = i + + sys.argv = sys.argv[:1] + sys.argv[cmd1:] + print "jsnincdir:", jsnincdir + + return pykslibdir, evlibdir, evincdir, trlibdir, trincdir, jsnincdir + +def configuration(parent_package='', top_path=None, \ + libs=["pykshark", "tracecmd", "traceevent", "json-c"], \ + libdirs=['.'], \ + incdirs=['.']): + """ Function used to build our configuration. + """ + from numpy.distutils.misc_util import Configuration + + # The configuration object that hold information on all the files + # to be built. + config = Configuration('', parent_package, top_path) + config.add_extension("libkshark_wrapper", + sources=["libkshark_wrapper.pyx"], + libraries=libs, + library_dirs=libdirs, + depends=["../../src/libkshark.c"], + include_dirs=incdirs) + + return config + +def main(argv): + # Retrieve third-party libraries + pykslibdir, evlibdir, evincdir, trlibdir, trincdir, jsnincdir = libs(sys.argv[1:]) + + # Retrieve the parameters of our local configuration + params = configuration(top_path='', \ + libdirs=[pykslibdir, trlibdir, evlibdir], \ + incdirs=[trincdir, evincdir, jsnincdir]).todict() + + #Override the C-extension building so that it knows about '.pyx' + #Cython files + params['cmdclass'] = dict(build_ext=build_ext) + + # Call the actual building/packaging function (see distutils docs) + from numpy.distutils.core import setup + setup(**params) + +if __name__ == "__main__": + main(sys.argv[1:]) From patchwork Wed Mar 27 16:03:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873737 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01E87922 for ; Wed, 27 Mar 2019 16:03:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E160D287A7 for ; Wed, 27 Mar 2019 16:03:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D5C602882F; Wed, 27 Mar 2019 16:03:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5CDEB287C8 for ; Wed, 27 Mar 2019 16:03:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726803AbfC0QD4 (ORCPT ); Wed, 27 Mar 2019 12:03:56 -0400 Received: from mail-pl1-f179.google.com ([209.85.214.179]:41303 "EHLO mail-pl1-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727391AbfC0QDz (ORCPT ); Wed, 27 Mar 2019 12:03:55 -0400 Received: by mail-pl1-f179.google.com with SMTP id y5so3518585plk.8 for ; Wed, 27 Mar 2019 09:03:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QZrbwSiXW/iuRzMjIVeyxRnny31c5JnD9Xwt8Sn+1Lg=; b=XA/2U6CJ2BB1cf6jC6C9+2H0XV9UfcAMMoXVnLu16VmyKu5c1gcC5EoiqQYSKb7dZ7 lIMvyPtqldkqGuir0HjTkTHgVynzYyMQXipN2Gc0D2CCuUmEzHZCYVQlX/o5VQOIyyWa 4p4aFAf0RPFGjjAXjohu3dO5WVIYpKC8qC+TlBBeR67vDRUbpZAhUL2vtKGvUbs7mm4p HsgGyVxi4oTregqSJYSwiuAfPPg1ALZ1GtirmQZwl7CLVw7QFmRKviezmX68EIq73bV/ UNGkmO5Q7v0Yyi0ORIewp8sKHhybrRlDNulD6vrNxIcBcwAyXxE5p+06g7HxlWQyL+wk NXQw== X-Gm-Message-State: APjAAAXjtaceJNLywefvuuIK2NSqeejSiFMv12FgLzTPkc+irrhvpfoY Jl2DdV2rPOWRD8YVPNav3Z0= X-Google-Smtp-Source: APXvYqyiTul8erKR3tUsXfTTx0PNUGIevF2OSYyHcgA4PVwX5S7WDgcF6p06xbfcZNc2weHE6Kl+zA== X-Received: by 2002:a17:902:a5c3:: with SMTP id t3mr38095033plq.293.1553702635302; Wed, 27 Mar 2019 09:03:55 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:54 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 6/7] kernel-shark: Add automatic building of the NumPy interface Date: Wed, 27 Mar 2019 18:03:22 +0200 Message-Id: <20190327160323.31654-7-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP CMAKE will call the python script that build the NumPy interface. It will also takes care about cleaning. Signed-off-by: Yordan Karadzhov --- kernel-shark/build/py/pybuild.sh | 26 ++++++++++++++++++++++++++ kernel-shark/src/CMakeLists.txt | 20 ++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100755 kernel-shark/build/py/pybuild.sh diff --git a/kernel-shark/build/py/pybuild.sh b/kernel-shark/build/py/pybuild.sh new file mode 100755 index 0000000..978085c --- /dev/null +++ b/kernel-shark/build/py/pybuild.sh @@ -0,0 +1,26 @@ +#!/bin/bash + +python np_setup.py --pykslibdir=$1 \ + --trlibdir=$2 \ + --trincdir=$3 \ + --evlibdir=$4 \ + --evincdir=$5 \ + --jsnincdir=$6 \ + build_ext -i &> pybuild.log + +if grep -q 'error:' pybuild.log; then + cat pybuild.log +fi + +if grep -q 'Error:' pybuild.log; then + cat pybuild.log +fi + +WC=$(grep 'warning' pybuild.log | wc -l) +if ((WC > 2)); then + cat pybuild.log +fi + +if grep -q 'usage:' pybuild.log; then + cat pybuild.log +fi diff --git a/kernel-shark/src/CMakeLists.txt b/kernel-shark/src/CMakeLists.txt index a3c6743..886ddad 100644 --- a/kernel-shark/src/CMakeLists.txt +++ b/kernel-shark/src/CMakeLists.txt @@ -45,6 +45,26 @@ if (PYTHONLIBS_FOUND AND CYTHON_FOUND AND NUMPY_FOUND) ${CMAKE_DL_LIBS} ${JSONC_LIBRARIES}) + + add_custom_target(kshark_wrapper ALL DEPENDS pykshark libkshark-py.c + COMMENT "Generating libkshark_wrapper.c") + + add_custom_command(TARGET kshark_wrapper + PRE_BUILD + COMMAND ./pybuild.sh ${KS_DIR}/lib/ + ${TRACECMD_LIBRARY_DIR} ${TRACECMD_INCLUDE_DIR} + ${TRACEEVENT_LIBRARY_DIR} ${TRACEEVENT_INCLUDE_DIR} + ${JSONC_INCLUDE_DIR} + COMMAND cp libkshark_wrapper.so ${KS_DIR}/bin + WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/py + DEPENDS ${KS_DIR}/build/py/np_setup.py ${KS_DIR}/src/libkshark-py.c + ) + + set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES + "${KS_DIR}/bin/libkshark_wrapper.so" + "${KS_DIR}/build/py/libkshark_wrapper.so" + "${KS_DIR}/build/py/libkshark_wrapper.c") + endif (PYTHONLIBS_FOUND AND CYTHON_FOUND AND NUMPY_FOUND) if (Qt5Widgets_FOUND AND Qt5Network_FOUND) From patchwork Wed Mar 27 16:03:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yordan Karadzhov X-Patchwork-Id: 10873739 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D76E814DE for ; Wed, 27 Mar 2019 16:03:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3577287A7 for ; Wed, 27 Mar 2019 16:03:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B7B0E2882F; Wed, 27 Mar 2019 16:03:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FC40287A7 for ; Wed, 27 Mar 2019 16:03:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727574AbfC0QD6 (ORCPT ); Wed, 27 Mar 2019 12:03:58 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:38841 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727397AbfC0QD5 (ORCPT ); Wed, 27 Mar 2019 12:03:57 -0400 Received: by mail-pl1-f196.google.com with SMTP id g37so3524473plb.5 for ; Wed, 27 Mar 2019 09:03:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bzWrH+isx16ZgTWNl9NjPHRJcn9B+m5UVfnnt3zY+Qc=; b=A54I/py5o9Hk/zQVhFXYNuA3a6xPeZpCkPch0ZUwOhVIPiRXyiTkT0TqR8pAdfDaW8 G2EIZZRw2j2c92Jl3r0rHdV9vhMhPCJZvrjPWaucz+Ii5aJK5AtD38lpmJPhBYPJ8011 KaAZAgGAiJ3676AyqdeOcsrqpbGgqsCwqp5o0aLSiRbN1QaNRKUDBAyvHTbwLHjP4UpU YPNXomANX2wHK5R6qaalUP6BLTtyXGaObAJDtxc14XFprWc9FfofvG6FnwwAndU7dcQq xSi3ktJp4hJwGah+0hBVoMgigyn3+Rspz7jQpeVD+/F8pYCgeKFmNtrLzu0gPC7xUIh1 r93g== X-Gm-Message-State: APjAAAVqhaloVvlnGo9voi2pADGgGL3JmfPjnZYMVLVNGT0FAxLE6WFX +bwVpLEe4p3YTPET4RNUIbU= X-Google-Smtp-Source: APXvYqxE1tTSZmrbdAPDrd5ZUTgcSJQH47DnzghuI7kZlR9dgaIu3/E1jLk7U37P2AycUTc7tWtVjA== X-Received: by 2002:a17:902:7590:: with SMTP id j16mr3177289pll.98.1553702637271; Wed, 27 Mar 2019 09:03:57 -0700 (PDT) Received: from mamba.eng.vmware.com ([146.247.46.5]) by smtp.gmail.com with ESMTPSA id p88sm36150806pfa.80.2019.03.27.09.03.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 09:03:56 -0700 (PDT) From: Yordan Karadzhov To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org, y.karadz@gmail.com Subject: [RFC 7/7] kernel-shark: Add basic example demonstrating the NumPy interface Date: Wed, 27 Mar 2019 18:03:23 +0200 Message-Id: <20190327160323.31654-8-ykaradzhov@vmware.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190327160323.31654-1-ykaradzhov@vmware.com> References: <20190327160323.31654-1-ykaradzhov@vmware.com> MIME-Version: 1.0 Sender: linux-trace-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The example script plots the distribution (histogram) of the pulseaudio wake-up times and finds the biggest latency. The script also generates a KernelShark session descriptor file (JSON). The session descriptor file can be used by the KernelSherk GUI to open a session which will directly visualize the largest wake-up latency. Signed-off-by: Yordan Karadzhov --- kernel-shark/bin/sched_wakeup.py | 96 ++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100755 kernel-shark/bin/sched_wakeup.py diff --git a/kernel-shark/bin/sched_wakeup.py b/kernel-shark/bin/sched_wakeup.py new file mode 100755 index 0000000..79dbedd --- /dev/null +++ b/kernel-shark/bin/sched_wakeup.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python2 + +import matplotlib.pyplot as plt +import libkshark_wrapper as ks +import scipy.stats as st +import numpy as np +import array +import json +import sys + +fname = str(sys.argv[1]) + +ks.open_file(fname) +ofst, cpu, evt, pid, ts = ks.load_data() +tasks = ks.get_tasks() +task_pid = tasks["pulseaudio"] + +ss_eid = ks.event_id("sched", "sched_switch"); +w_eid = ks.event_id("sched", "sched_waking"); + +i = evt.size - 1 +dt = [] +delta_max = i_ss_max = i_sw_max = 0 + +while (i >= 0): + if (evt[i] == ss_eid): + next_pid = ks.read_event_field(offset=ofst[i], + sys="sched", + event="sched_switch", + field="next_pid") + + if (next_pid == task_pid): + time_ss = ts[i] + index_ss = i + + while(i >= 0): + i = i - 1 + if (evt[i] == w_eid): + waking_pid = ks.read_event_field(offset=ofst[i], + sys="sched", + event="sched_waking", + field="pid") + + if (waking_pid == task_pid): + delta = (time_ss - ts[i]) / 1000. + #print delta + dt.append(delta); + if (delta > delta_max): + print "lat. max: ", delta + i_ss_max = index_ss + i_sw_max = i + delta_max = delta + + break + + i = i - 1 + +desc = st.describe(np.array(dt)) +print desc + +fig, ax = plt.subplots(nrows=1, ncols=1) +fig.set_figheight(6) +fig.set_figwidth(7) + +rect = fig.patch +rect.set_facecolor('white') + +ax.set_xlabel('latency [$\mu$s]') +plt.yscale('log') +ax.hist(dt, bins=(200), histtype=u'step'); +plt.show() + +sname = 'sched.json' +ks.new_session(fname, sname) + +with open(sname, "r+") as s: + session = json.load(s) + session['TaskPlots'] = [task_pid] + session['CPUPlots'] = [int(cpu[i_sw_max]), int(cpu[i_ss_max])] + + delta = ts[i_ss_max] - ts[i_sw_max] + tmin = int(ts[i_sw_max] - delta) + tmax = int(ts[i_ss_max] + delta) + session['Model']['range'] = [tmin, tmax] + + session['Markers']['markA']['isSet'] = True + session['Markers']['markA']['row'] = int(i_sw_max) + + session['Markers']['markB']['isSet'] = True + session['Markers']['markB']['row'] = int(i_ss_max) + + session['ViewTop'] = int(i_sw_max) - 5 + + ks.save_session(session, s) + +ks.close()