From patchwork Tue Jun 28 12:16:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12898172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F43C43334 for ; Tue, 28 Jun 2022 12:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345193AbiF1MQ2 (ORCPT ); Tue, 28 Jun 2022 08:16:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345233AbiF1MQ1 (ORCPT ); Tue, 28 Jun 2022 08:16:27 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0174827B2B for ; Tue, 28 Jun 2022 05:16:26 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id fi2so25331953ejb.9 for ; Tue, 28 Jun 2022 05:16:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Bd4i6kukF74dEUHnWS2ndjqU6Fr27S6m9TzF1jL+l+M=; b=AeTMVqdP2DYaZeawF5OHdpeqzaKZl/pWypQ8DBk/bJP9XET6JSUy1SaZ6R5qgUazOD jGOdsERMmOSNDEy3kc2hMu7FCNUJnumLlCsFf4W5gvWNx4OltmGvnud305Ds91W0BT3P a+1HjNlumjH3VsC03ybDPdJTlQTI3B+D2u9VQu+bHf1mFpiHT3vQqOy4RvcSr5ngpa71 7BE9OT39IbddQTf5vsDeOEx6sH7vEfqgO78GRf5rNE/1cyGMtOpuKqsFqKIaaZYVX3MB ppmGH+LBqvjgF8+GSGH2E0j5ehwZ2qQ6HFqhqdEcTm9A9JcV5sSRZDLPvjCtOw5TJtVz ofzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bd4i6kukF74dEUHnWS2ndjqU6Fr27S6m9TzF1jL+l+M=; b=dZSfotbPiiY/6QkFkuxN7Asei9TmcUbw/K2dBZgjG0s8/Neja5T85XsSWnA5sGYI+v nesjJMX1wGbLSeJHuztOKVoyJddGFpjtNeki5EpqUcCB9azP/61jEzmka3TrQqI2qkf1 k9FGScI612BHzsi6V7dFVQ5zI3Lcd4Huiou0otoUTpG4D8qDQmQ4IfYqrtrAvNuy2KBF +qFpEoTjpTcE0uIZEHu11WlHZVk2TEz7CJUaKo+K7oPx6338Vr2RYzAr1IpM/e0ULYRA XyI0xW2POqDqi0TLfefrCpcUaMBDvgjAXf564DyhviD1HUmc0tK0cgiO36rsD2xyT4E/ 6B+g== X-Gm-Message-State: AJIora+i2qZYiYRlTOXlpiM9nHyufJtkgESwS03gJgpSFkr9yAX6XrTB vT8NP65I0WmBXJ6fHXteIlKC/Z5JYSHMtQ== X-Google-Smtp-Source: AGRyM1vDe9Y7WpIaWcTWbZxKJ2H8TExNWvVgFoDIa5CIIc+TdCDwNcoGQ1Hqhjq5ldbn7w9Dhc3wiA== X-Received: by 2002:a17:907:7207:b0:726:cc89:40ca with SMTP id dr7-20020a170907720700b00726cc8940camr3622637ejc.141.1656418584362; Tue, 28 Jun 2022 05:16:24 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id k14-20020aa7c04e000000b00431962fe5d4sm9512780edo.77.2022.06.28.05.16.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jun 2022 05:16:24 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: linux-trace-devel@vger.kernel.org Subject: [PATCH v4 1/3] trace-cruncher: ftrace uprobe raw API Date: Tue, 28 Jun 2022 15:16:19 +0300 Message-Id: <20220628121621.572506-2-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220628121621.572506-1-tz.stoyanov@gmail.com> References: <20220628121621.572506-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Trace-cruncher low level wrappers for tracefs library APIs for uprobe and uretprobe allocation. Signed-off-by: Tzvetomir Stoyanov (VMware) --- src/ftracepy-utils.c | 48 ++++++++++++++++++++++++++++++++++++++++++++ src/ftracepy-utils.h | 4 ++++ src/ftracepy.c | 10 +++++++++ 3 files changed, 62 insertions(+) diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c index e8411ae..b39459b 100644 --- a/src/ftracepy-utils.c +++ b/src/ftracepy-utils.c @@ -2449,6 +2449,54 @@ PyObject *PyFtrace_eprobe(PyObject *self, PyObject *args, PyObject *kwargs) return py_dyn; } +static PyObject *alloc_uprobe(PyObject *self, PyObject *args, PyObject *kwargs, bool pret) +{ + static char *kwlist[] = {"event", "file", "offset", "fetch_args", NULL}; + const char *event, *file, *fetchargs = NULL; + unsigned long long offset; + struct tracefs_dynevent *uprobe; + PyObject *py_dyn; + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "ssK|s", + kwlist, + &event, + &file, + &offset, + &fetchargs)) { + return NULL; + } + + if (pret) + uprobe = tracefs_uretprobe_alloc(TC_SYS, event, file, offset, fetchargs); + else + uprobe = tracefs_uprobe_alloc(TC_SYS, event, file, offset, fetchargs); + if (!uprobe) { + MEM_ERROR; + return NULL; + } + + py_dyn = PyDynevent_New(uprobe); + /* + * Here we only allocated and initializes a dynamic event object. + * However, no dynamic event is added to the system yet. Hence, + * there is no need to 'destroy' this event at exit. + */ + set_destroy_flag(py_dyn, false); + return py_dyn; +} + +PyObject *PyFtrace_uprobe(PyObject *self, PyObject *args, PyObject *kwargs) +{ + return alloc_uprobe(self, args, kwargs, false); +} + +PyObject *PyFtrace_uretprobe(PyObject *self, PyObject *args, PyObject *kwargs) +{ + return alloc_uprobe(self, args, kwargs, true); +} + static PyObject *set_filter(PyObject *args, PyObject *kwargs, struct tep_handle *tep, struct tep_event *event) diff --git a/src/ftracepy-utils.h b/src/ftracepy-utils.h index 9491b18..e6fab69 100644 --- a/src/ftracepy-utils.h +++ b/src/ftracepy-utils.h @@ -257,6 +257,10 @@ PyObject *PyFtrace_kretprobe(PyObject *self, PyObject *args, PyObject *kwargs); PyObject *PyFtrace_eprobe(PyObject *self, PyObject *args, PyObject *kwargs); +PyObject *PyFtrace_uprobe(PyObject *self, PyObject *args, PyObject *kwargs); + +PyObject *PyFtrace_uretprobe(PyObject *self, PyObject *args, PyObject *kwargs); + PyObject *PyFtrace_hist(PyObject *self, PyObject *args, PyObject *kwargs); diff --git a/src/ftracepy.c b/src/ftracepy.c index 2819da2..29d273f 100644 --- a/src/ftracepy.c +++ b/src/ftracepy.c @@ -483,6 +483,16 @@ static PyMethodDef ftracepy_methods[] = { METH_VARARGS | METH_KEYWORDS, PyFtrace_eprobe_doc, }, + {"uprobe", + (PyCFunction) PyFtrace_uprobe, + METH_VARARGS | METH_KEYWORDS, + "Define a uprobe." + }, + {"uretprobe", + (PyCFunction) PyFtrace_uretprobe, + METH_VARARGS | METH_KEYWORDS, + "Define a uretprobe." + }, {"hist", (PyCFunction) PyFtrace_hist, METH_VARARGS | METH_KEYWORDS, From patchwork Tue Jun 28 12:16:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12898175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F36AC433EF for ; Tue, 28 Jun 2022 12:16:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345380AbiF1MQ3 (ORCPT ); Tue, 28 Jun 2022 08:16:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345233AbiF1MQ3 (ORCPT ); Tue, 28 Jun 2022 08:16:29 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1719125C76 for ; Tue, 28 Jun 2022 05:16:27 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id e2so17308343edv.3 for ; Tue, 28 Jun 2022 05:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2Vk3nLOwiQa1o2bPQO7ijwuRYVGRHTjMJ79LJHnkMdo=; b=Jmro3QbmeBfVs76MRXiCL9J8bdNSSt/lvdoJ8G8eXBpL3YP5ZX4TvpSo47DaI5wWGI S6L/SdKhftcR0alo1ZC4mJodoZ/g+bTqLy7r9dQz+jxqCG8qq5gdLre8iOTEUGwKX9vA NS+KY1rOnqi1aeCx3BeGZziod+1/06ctkDLcSabOoYYmMkga3aVVoZZq2nEY/E8fDnco 021ORhZGzB2GUZetk8DInsIuLQHj8TIEk2ovkIczwus9CBFnMZKChAR4kd6Ch0iwUUVE H5GmVcLo8hxCf2BYSylobxhRIV7ET9Mqs7Uy7MDizlenI+rlnihl7A1ObUkSy2da4OFy GzNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2Vk3nLOwiQa1o2bPQO7ijwuRYVGRHTjMJ79LJHnkMdo=; b=aNv5wz22Mup2m1fvik2vjrL2fbFnBS9CQ8sQAmY9LqmNq1lXvrEvFNn4L7KD1GyLVl r71nGTsH4+IaOQoMyvD2xn48AfC4nPMoax0rPGujZPmtOWDx8NCx5h81bn1oXPaU65xM CiBpV2zJbzI4EbmzAQIhdN2vrBMQCKsUSYhhy+CssauAcFYNowztjsmhWCDH9HDpnsA+ ptb0BHuHhAA5pxsV0x4bCvTZ9qVrjvvVjW4IZClW6UMYXwAGolbYTO5NQT8T778ci/Fm Ha0fmtEvvS+5UVOOciyS/W9RPHz2zYqscWMAUceYiJQxXBE9nuwC5phuX3vlwCIpX9fD lWQw== X-Gm-Message-State: AJIora+wqTCj7b4G94vG8A9uKNxCwD6UWXffM9JkQ7GtZ7F0rpKreRjc +JtKYEC+o7FFhXAafkh+3cNVZA1cB+p+yQ== X-Google-Smtp-Source: AGRyM1tnlBdYT6Te7v3EgiT0k9QA9XFUwBMF3MMbxewPD1JZe3bIY2rpKER0ST16RRzdpn1EmSxlwg== X-Received: by 2002:a05:6402:4252:b0:437:6618:1738 with SMTP id g18-20020a056402425200b0043766181738mr23324289edb.259.1656418585120; Tue, 28 Jun 2022 05:16:25 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id k14-20020aa7c04e000000b00431962fe5d4sm9512780edo.77.2022.06.28.05.16.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jun 2022 05:16:24 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: linux-trace-devel@vger.kernel.org Subject: [PATCH v4 2/3] trace-cruncher: High level wrappers for ftrace uprobes Date: Tue, 28 Jun 2022 15:16:20 +0300 Message-Id: <20220628121621.572506-3-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220628121621.572506-1-tz.stoyanov@gmail.com> References: <20220628121621.572506-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Using uprobes requires finding the offset of a user function within the binary file, where this functions is compiled. This is not a trivial task, especially in the cases when a bunch of uprobes to user functions should be added. A high level trace-cruncher API allows adding multiple user functions as uprobes or uretprobes. It supports wildcards for function names and adding uprobes for library functions, used by the applications. Signed-off-by: Tzvetomir Stoyanov (VMware) --- setup.py | 4 +- src/ftracepy-utils.c | 698 +++++++++++++++++++++++++++++++++++++++++++ src/ftracepy-utils.h | 17 ++ src/ftracepy.c | 35 +++ 4 files changed, 752 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index aca634c..2145d81 100644 --- a/setup.py +++ b/setup.py @@ -70,8 +70,8 @@ def extension(name, sources, libraries): def main(): module_ft = extension(name='tracecruncher.ftracepy', - sources=['src/ftracepy.c', 'src/ftracepy-utils.c'], - libraries=['traceevent', 'tracefs']) + sources=['src/ftracepy.c', 'src/ftracepy-utils.c', 'src/trace-obj-debug.c'], + libraries=['traceevent', 'tracefs', 'bfd']) cythonize('src/npdatawrapper.pyx', language_level = 3) module_data = extension(name='tracecruncher.npdatawrapper', diff --git a/src/ftracepy-utils.c b/src/ftracepy-utils.c index b39459b..9d9fa5d 100644 --- a/src/ftracepy-utils.c +++ b/src/ftracepy-utils.c @@ -10,20 +10,51 @@ #endif // _GNU_SOURCE // C +#include #include #include #include #include #include +#include #include // trace-cruncher #include "ftracepy-utils.h" +#include "trace-obj-debug.h" PyObject *TFS_ERROR; PyObject *TEP_ERROR; PyObject *TRACECRUNCHER_ERROR; +#define UPROBES_SYSTEM "tc_uprobes" + +#define FTRACE_UPROBE 0x1 +#define FTRACE_URETPROBE 0x2 + +struct fprobes_list { + int size; + int count; + void **data; +}; + +struct utrace_func { + int type; + char *func_name; + char *func_args; +}; + +struct py_utrace_context { + pid_t pid; + char **cmd_argv; + char *usystem; + uint32_t trace_time; /* in msec */ + struct fprobes_list fretprobes; + struct fprobes_list ufuncs; + struct fprobes_list uevents; + struct dbg_trace_context *dbg; +}; + static char *kernel_version() { struct utsname uts; @@ -3508,3 +3539,670 @@ void PyFtrace_at_exit(void) if (seq.buffer) trace_seq_destroy(&seq); } + +#define UTRACE_LIST_INIT_SIZE 100 +static int utrace_list_add(struct fprobes_list *list, void *data) +{ + void **tmp; + int size; + + if (list->size <= list->count) { + if (!list->size) + size = UTRACE_LIST_INIT_SIZE; + else + size = 2 * list->size; + tmp = realloc(list->data, size * sizeof(void *)); + if (!tmp) + return -1; + list->data = tmp; + list->size = size; + } + + list->data[list->count] = data; + list->count++; + return list->count - 1; +} + +void py_utrace_free(struct py_utrace_context *utrace) +{ + struct utrace_func *f; + int i; + + if (!utrace) + return; + if (utrace->dbg) + dbg_trace_context_destroy(utrace->dbg); + + for (i = 0; i < utrace->ufuncs.count; i++) { + f = utrace->ufuncs.data[i]; + free(f->func_name); + free(f); + } + free(utrace->ufuncs.data); + if (utrace->cmd_argv) { + i = 0; + while (utrace->cmd_argv[i]) + free(utrace->cmd_argv[i++]); + free(utrace->cmd_argv); + } + + for (i = 0; i < utrace->uevents.count; i++) + tracefs_dynevent_free(utrace->uevents.data[i]); + free(utrace->uevents.data); + + free(utrace->usystem); + free(utrace); +} + +/* + * All strings, used as ftrace system or event name must contain only + * alphabetic characters, digits or underscores. + */ +static void fname_unify(char *fname) +{ + int i; + + for (i = 0; fname[i]; i++) + if (!isalnum(fname[i]) && fname[i] != '_') + fname[i] = '_'; +} + +int py_utrace_destroy(struct py_utrace_context *utrace) +{ + int i; + + for (i = 0; i < utrace->uevents.count; i++) + tracefs_dynevent_destroy(utrace->uevents.data[i], true); + + return 0; +} + +static unsigned long long str_hash(char *s) +{ + unsigned long long sum = 0, add; + int i, len = strlen(s); + + for (i = 0; i < len; i++) { + if (i + 8 < len) + add = *(long long *)(s+i); + else if (i + 4 < len) + add = *(long *)(s+i); + else + add = s[i]; + + sum += add; + } + + return sum; +} + +static struct py_utrace_context *utrace_new(pid_t pid, char **argv, bool libs) +{ + struct py_utrace_context *utrace; + + utrace = calloc(1, sizeof(*utrace)); + if (!utrace) + return NULL; + + if (argv) { + utrace->cmd_argv = argv; + utrace->dbg = dbg_trace_context_create_file(argv[0], libs); + if (!utrace->dbg) + goto error; + if (asprintf(&utrace->usystem, "%s_%llX", UPROBES_SYSTEM, str_hash(argv[0])) <= 0) + goto error; + } else { + utrace->pid = pid; + utrace->dbg = dbg_trace_context_create_pid(pid, libs); + if (!utrace->dbg) + goto error; + if (asprintf(&utrace->usystem, "%s_%d", UPROBES_SYSTEM, pid) <= 0) + goto error; + } + + fname_unify(utrace->usystem); + return utrace; + +error: + py_utrace_free(utrace); + return NULL; +} + +static int py_utrace_add_func(struct py_utrace_context *utrace, char *func, int type) +{ + struct utrace_func *p; + int ret; + int i; + + for (i = 0; i < utrace->ufuncs.count; i++) { + p = utrace->ufuncs.data[i]; + if (!strcmp(p->func_name, func)) { + p->type |= type; + return 0; + } + } + + p = calloc(1, sizeof(*p)); + if (!p) + return -1; + p->func_name = strdup(func); + if (!p->func_name) + goto error; + p->type = type; + + ret = utrace_list_add(&utrace->ufuncs, p); + if (ret < 0) + goto error; + + if (dbg_trace_add_resolve_symbol(utrace->dbg, 0, func, ret)) + goto error; + + return 0; + +error: + free(p->func_name); + free(p); + return -1; +} + +PyObject *PyUserTrace_add_function(PyUserTrace *self, PyObject *args, + PyObject *kwargs) +{ + struct py_utrace_context *utrace = self->ptrObj; + static char *kwlist[] = {"fname", NULL}; + char *fname; + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "s", + kwlist, + &fname)) { + return NULL; + } + + if (py_utrace_add_func(utrace, fname, FTRACE_UPROBE) < 0) { + MEM_ERROR + return NULL; + } + + Py_RETURN_NONE; +} + +PyObject *PyUserTrace_add_ret_function(PyUserTrace *self, PyObject *args, + PyObject *kwargs) +{ + struct py_utrace_context *utrace = self->ptrObj; + static char *kwlist[] = {"fname", NULL}; + char *fname; + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "s", + kwlist, + &fname)) { + return NULL; + } + + if (py_utrace_add_func(utrace, fname, FTRACE_URETPROBE) < 0) { + MEM_ERROR + return NULL; + } + + Py_RETURN_NONE; +} + +/* + * Max event name is 64 bytes, hard coded in the kernel. + * It can consist only of alphabetic characters, digits or underscores. + */ +#define FILENAME_TRUNCATE 10 +#define FUNCAME_TRUNCATE 50 +static char *uprobe_event_name(char *file, char *func, int type) +{ + char *event = NULL; + char *fname; + + fname = strrchr(file, '/'); + if (fname) + fname++; + if (!fname || *fname == '\0') + fname = file; + + asprintf(&event, "%s%.*s_%.*s", + type == FTRACE_URETPROBE ? "r_":"", + FILENAME_TRUNCATE, fname, FUNCAME_TRUNCATE, func); + if (event) + fname_unify(event); + + return event; +} + +/* + * Create uprobe based on function name, + * file name and function offset within the file. + */ +static int utrace_event_create(struct py_utrace_context *utrace, + struct dbg_trace_symbols *sym, char *fecthargs, + int type) +{ + struct tracefs_dynevent *uevent = NULL; + char *rname; + + /* Generate uprobe event name, according to ftrace name requirements. */ + rname = uprobe_event_name(sym->fname, sym->name, type); + if (!rname) + return -1; + + if (type == FTRACE_URETPROBE) + uevent = tracefs_uretprobe_alloc(utrace->usystem, rname, + sym->fname, sym->foffset, fecthargs); + else + uevent = tracefs_uprobe_alloc(utrace->usystem, rname, + sym->fname, sym->foffset, fecthargs); + + free(rname); + if (!uevent) + return -1; + + if (tracefs_dynevent_create(uevent)) { + tracefs_dynevent_free(uevent); + return -1; + } + + utrace_list_add(&utrace->uevents, uevent); + return 0; +} + +/* A callback, called on each resolved function. */ +static int symblos_walk(struct dbg_trace_symbols *sym, void *context) +{ + struct py_utrace_context *utrace = context; + struct utrace_func *ufunc; + + if (!sym->name || !sym->fname || !sym->foffset || + sym->cookie < 0 || sym->cookie >= utrace->ufuncs.count) + return 0; + + ufunc = utrace->ufuncs.data[sym->cookie]; + + if (ufunc->type & FTRACE_UPROBE) + utrace_event_create(utrace, sym, ufunc->func_args, FTRACE_UPROBE); + + if (ufunc->type & FTRACE_URETPROBE) + utrace_event_create(utrace, sym, ufunc->func_args, FTRACE_URETPROBE); + + return 0; +} + +static void py_utrace_generate_uprobes(struct py_utrace_context *utrace) +{ + /* Find the exact name and file offset of each user function that should be traced. */ + dbg_trace_resolve_symbols(utrace->dbg); + dbg_trace_walk_resolved_symbols(utrace->dbg, symblos_walk, utrace); +} + +static int uprobe_start_trace(struct py_utrace_context *utrace, struct tracefs_instance *instance) +{ + PyObject *pPid = PyLong_FromLong(utrace->pid); + bool ret; + + if (!pPid) + return -1; + + /* Filter the trace only on desired pid(s). */ + ret = hook2pid(instance, PyLong_FromLong(utrace->pid), true); + Py_DECREF(pPid); + if (!ret) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to set trace filter"); + return -1; + } + + /* Enable uprobes in the system. */ + if (tracefs_event_enable(instance, utrace->usystem, NULL)) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to enable trace events"); + return -1; + } + + return 0; +} + +#define PERF_EXEC_SYNC "/TC_PERF_SYNC_XXXXXX" +static int uprobe_exec_cmd(struct py_utrace_context *utrace, struct tracefs_instance *instance) +{ + char *envp[] = {NULL}; + char sname[strlen(PERF_EXEC_SYNC) + 1]; + sem_t *sem; + pid_t pid; + int ret; + + strcpy(sname, PERF_EXEC_SYNC); + mktemp(sname); + sem = sem_open(sname, O_CREAT | O_EXCL, 0644, 0); + sem_unlink(sname); + + pid = fork(); + if (pid < 0) { + PyErr_SetString(TRACECRUNCHER_ERROR, "Failed to fork"); + return -1; + } + if (pid == 0) { + sem_wait(sem); + execvpe(utrace->cmd_argv[0], utrace->cmd_argv, envp); + } else { + utrace->pid = pid; + uprobe_start_trace(utrace, instance); + sem_post(sem); + return ret; + } + + return 0; +} + +static int py_utrace_enable(struct py_utrace_context *utrace, struct tracefs_instance *instance) +{ + /* If uprobes on desired user functions are not yet generated, do it now. */ + if (!utrace->uevents.count) + py_utrace_generate_uprobes(utrace); + + /* No functions are found in the given program / pid. */ + if (!utrace->uevents.count) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Cannot find requested user functions"); + return -1; + } + + if (utrace->cmd_argv) + uprobe_exec_cmd(utrace, instance); + else + uprobe_start_trace(utrace, instance); + + return 0; +} + +static int py_utrace_disable(struct py_utrace_context *utrace, struct tracefs_instance *instance) +{ + /* Disable uprobes in the system. */ + if (tracefs_event_disable(instance, utrace->usystem, NULL)) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to disable trace events"); + return -1; + } + + return 0; +} + +static bool tracing_run; + +static void tracing_stop(int sig) +{ + tracing_run = false; +} + +static void tracing_timer(int sig, siginfo_t *si, void *uc) +{ + tracing_run = false; +} + +#define PID_WAIT_CHECK_USEC 500000 +#define TIMER_SEC_NANO 1000000000LL +static int utrace_wait_pid(struct py_utrace_context *utrace) +{ + struct itimerspec tperiod = {0}; + struct sigaction saction = {0}; + struct sigevent stime = {0}; + timer_t timer_id; + + if (utrace->pid == 0) + return -1; + + tracing_run = true; + signal(SIGINT, tracing_stop); + + if (utrace->trace_time) { + stime.sigev_notify = SIGEV_SIGNAL; + stime.sigev_signo = SIGRTMIN; + if (timer_create(CLOCK_MONOTONIC, &stime, &timer_id)) + return -1; + saction.sa_flags = SA_SIGINFO; + saction.sa_sigaction = tracing_timer; + sigemptyset(&saction.sa_mask); + if (sigaction(SIGRTMIN, &saction, NULL)) { + timer_delete(timer_id); + return -1; + } + /* Convert trace_time from msec to sec, nsec. */ + tperiod.it_value.tv_nsec = ((unsigned long long)utrace->trace_time * 1000000LL); + if (tperiod.it_value.tv_nsec >= TIMER_SEC_NANO) { + tperiod.it_value.tv_sec = tperiod.it_value.tv_nsec / TIMER_SEC_NANO; + tperiod.it_value.tv_nsec %= TIMER_SEC_NANO; + } + if (timer_settime(timer_id, 0, &tperiod, NULL)) + return -1; + } + + do { + if (utrace->cmd_argv) { /* Wait for a child. */ + if (waitpid(utrace->pid, NULL, WNOHANG) == (int)utrace->pid) { + utrace->pid = 0; + tracing_run = false; + } + } else { /* Not a child, check if still exist. */ + if (kill(utrace->pid, 0) == -1 && errno == ESRCH) { + utrace->pid = 0; + tracing_run = false; + } + } + usleep(PID_WAIT_CHECK_USEC); + } while (tracing_run); + + if (utrace->trace_time) + timer_delete(timer_id); + + signal(SIGINT, SIG_DFL); + + return 0; +} + +PyObject *PyUserTrace_enable(PyUserTrace *self, PyObject *args, PyObject *kwargs) +{ + struct py_utrace_context *utrace = self->ptrObj; + static char *kwlist[] = {"instance", "wait", "time", NULL}; + struct tracefs_instance *instance = NULL; + PyObject *py_inst = NULL; + int wait = false; + int ret; + + if (!utrace) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to get utrace context"); + return NULL; + } + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "|Ops", + kwlist, + &py_inst, + &wait, + &utrace->trace_time)) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to parse input arguments"); + return NULL; + } + + if (!get_optional_instance(py_inst, &instance)) + return NULL; + + ret = py_utrace_enable(utrace, instance); + if (ret) + return NULL; + + if (wait) { + utrace_wait_pid(utrace); + py_utrace_disable(utrace, instance); + } + + Py_RETURN_NONE; +} + +PyObject *PyUserTrace_disable(PyUserTrace *self, PyObject *args, PyObject *kwargs) +{ + struct py_utrace_context *utrace = self->ptrObj; + static char *kwlist[] = {"instance", NULL}; + struct tracefs_instance *instance = NULL; + PyObject *py_inst = NULL; + int ret; + + if (!utrace) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to get utrace context"); + return NULL; + } + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "|O", + kwlist, + &py_inst)) { + PyErr_SetString(TRACECRUNCHER_ERROR, + "Failed to parse input arguments"); + return NULL; + } + + if (!get_optional_instance(py_inst, &instance)) + return NULL; + + ret = py_utrace_disable(utrace, instance); + + if (ret) + return NULL; + + Py_RETURN_NONE; +} + +static char *find_in_path(const char *name) +{ + char *paths = strdup(getenv("PATH")); + char fullpath[PATH_MAX]; + bool found = false; + char *tmp = paths; + const char *item; + + if (!paths) + return NULL; + + while ((item = strsep(&paths, ":")) != NULL) { + snprintf(fullpath, PATH_MAX, "%s/%s", item, name); + if (access(fullpath, F_OK|X_OK) == 0) { + found = true; + break; + } + } + + free(tmp); + + if (found) + return strdup(fullpath); + + return NULL; +} + +static char *get_full_name(char *name) +{ + bool resolve = false; + char *fname, *tmp; + + if (!strchr(name, '/')) { + tmp = find_in_path(name); + if (!tmp) + return NULL; + resolve = true; + } else { + tmp = name; + } + + fname = realpath(tmp, NULL); + if (resolve) + free(tmp); + + return fname; +} + +PyObject *PyFtrace_user_trace(PyObject *self, PyObject *args, PyObject *kwargs) +{ + static char *kwlist[] = {"pid", "argv", "follow_libs", NULL}; + PyObject *py_utrace, *py_arg, *py_args = NULL; + struct py_utrace_context *utrace; + char **argv = NULL; + long long pid = 0; + int libs = 0; + int i, argc; + + if (!PyArg_ParseTupleAndKeywords(args, + kwargs, + "|KOp", + kwlist, + &pid, + &py_args, + &libs)) { + return NULL; + } + + if (pid <= 0 && !py_args) { + PyErr_Format(TFS_ERROR, + "Process ID or program name should be specified"); + return NULL; + } + if (pid > 0 && py_args) { + PyErr_Format(TFS_ERROR, + "Only one of Process ID or program name should be specified"); + return NULL; + } + + if (py_args) { + if (!PyList_CheckExact(py_args)) { + PyErr_Format(TFS_ERROR, "Failed to parse argv list"); + return NULL; + } + argc = PyList_Size(py_args); + argv = calloc(argc + 1, sizeof(char *)); + for (i = 0; i < argc; i++) { + py_arg = PyList_GetItem(py_args, i); + if (!PyUnicode_Check(py_arg)) + continue; + + if (i == 0) + argv[i] = get_full_name(PyUnicode_DATA(py_arg)); + else + argv[i] = strdup(PyUnicode_DATA(py_arg)); + + if (!argv[i]) { + if (i == 0) + PyErr_Format(TFS_ERROR, + "Failed to find program with name %s", + PyUnicode_DATA(py_arg)); + goto error; + } + } + argv[i] = NULL; + } + + utrace = utrace_new(pid, argv, libs); + if (!utrace) { + MEM_ERROR; + goto error; + } + + py_utrace = PyUserTrace_New(utrace); + return py_utrace; + +error: + if (argv) { + for (i = 0; i < argc; i++) + free(argv[i]); + free(argv); + } + return NULL; +} diff --git a/src/ftracepy-utils.h b/src/ftracepy-utils.h index e6fab69..3f9c906 100644 --- a/src/ftracepy-utils.h +++ b/src/ftracepy-utils.h @@ -34,6 +34,21 @@ C_OBJECT_WRAPPER_DECLARE(tracefs_synth, PySynthEvent) PyObject *PyTepRecord_time(PyTepRecord* self); +struct py_utrace_context; +void py_utrace_free(struct py_utrace_context *utrace); +int py_utrace_destroy(struct py_utrace_context *utrace); +C_OBJECT_WRAPPER_DECLARE(py_utrace_context, PyUserTrace); + +PyObject *PyUserTrace_add_function(PyUserTrace *self, PyObject *args, + PyObject *kwargs); + +PyObject *PyUserTrace_add_ret_function(PyUserTrace *self, PyObject *args, + PyObject *kwargs); + +PyObject *PyUserTrace_enable(PyUserTrace *self, PyObject *args, PyObject *kwargs); + +PyObject *PyUserTrace_disable(PyUserTrace *self, PyObject *args, PyObject *kwargs); + PyObject *PyTepRecord_cpu(PyTepRecord* self); PyObject *PyTepEvent_name(PyTepEvent* self); @@ -270,6 +285,8 @@ PyObject *PyFtrace_synth(PyObject *self, PyObject *args, PyObject *PyFtrace_set_ftrace_loglevel(PyObject *self, PyObject *args, PyObject *kwargs); +PyObject *PyFtrace_user_trace(PyObject *self, PyObject *args, PyObject *kwargs); + PyObject *PyFtrace_trace_process(PyObject *self, PyObject *args, PyObject *kwargs); diff --git a/src/ftracepy.c b/src/ftracepy.c index 29d273f..856cef0 100644 --- a/src/ftracepy.c +++ b/src/ftracepy.c @@ -317,6 +317,32 @@ C_OBJECT_WRAPPER(tracefs_synth, PySynthEvent, tracefs_synth_destroy, tracefs_synth_free) +static PyMethodDef PyUserTrace_methods[] = { + {"add_function", + (PyCFunction) PyUserTrace_add_function, + METH_VARARGS | METH_KEYWORDS, + "Add tracepoint on user function." + }, + {"add_ret_function", + (PyCFunction) PyUserTrace_add_ret_function, + METH_VARARGS | METH_KEYWORDS, + "Add tracepoint on user function return." + }, + {"enable", + (PyCFunction) PyUserTrace_enable, + METH_VARARGS | METH_KEYWORDS, + "Enable tracing of the configured user tracepoints." + }, + {"disable", + (PyCFunction) PyUserTrace_disable, + METH_VARARGS | METH_KEYWORDS, + "Disable tracing of the configured user tracepoints." + }, + {NULL, NULL, 0, NULL} +}; +C_OBJECT_WRAPPER(py_utrace_context, PyUserTrace, + py_utrace_destroy, py_utrace_free) + static PyMethodDef ftracepy_methods[] = { {"dir", (PyCFunction) PyFtrace_dir, @@ -503,6 +529,11 @@ static PyMethodDef ftracepy_methods[] = { METH_VARARGS | METH_KEYWORDS, PyFtrace_synth_doc, }, + {"user_trace", + (PyCFunction) PyFtrace_user_trace, + METH_VARARGS | METH_KEYWORDS, + "Create a context for tracing a user process using uprobes" + }, {"set_ftrace_loglevel", (PyCFunction) PyFtrace_set_ftrace_loglevel, METH_VARARGS | METH_KEYWORDS, @@ -577,6 +608,9 @@ PyMODINIT_FUNC PyInit_ftracepy(void) if (!PySynthEventTypeInit()) return NULL; + if (!PyUserTraceTypeInit()) + return NULL; + TFS_ERROR = PyErr_NewException("tracecruncher.ftracepy.tfs_error", NULL, NULL); @@ -595,6 +629,7 @@ PyMODINIT_FUNC PyInit_ftracepy(void) PyModule_AddObject(module, "tracefs_dynevent", (PyObject *) &PyDyneventType); PyModule_AddObject(module, "tracefs_hist", (PyObject *) &PyTraceHistType); PyModule_AddObject(module, "tracefs_synth", (PyObject *) &PySynthEventType); + PyModule_AddObject(module, "py_utrace_context", (PyObject *) &PyUserTraceType); PyModule_AddObject(module, "tfs_error", TFS_ERROR); PyModule_AddObject(module, "tep_error", TEP_ERROR); From patchwork Tue Jun 28 12:16:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tzvetomir Stoyanov (VMware)" X-Patchwork-Id: 12898174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C22BDCCA479 for ; Tue, 28 Jun 2022 12:16:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345382AbiF1MQ3 (ORCPT ); Tue, 28 Jun 2022 08:16:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345380AbiF1MQ2 (ORCPT ); Tue, 28 Jun 2022 08:16:28 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E27F27B2B for ; Tue, 28 Jun 2022 05:16:27 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id ge10so25354883ejb.7 for ; Tue, 28 Jun 2022 05:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=aQBlUNuDgqPvC7NX3xeLdCIV1BWkCfXjevdKpEbnUys=; b=eXEjBNP0otYk1MYy5nmqP1oACbp66nI5VnLoAyb7R4MNxubcXFTVvNAlvHd0HGBUgU NDQG95GmvFoeObR4uhCmPEEdaiYQPliZBd6+97eFDuOGhTPbCcPJAyahV7TRNDZlQzoi +Hct0YxkMzfmyLu3kuYyUfAwMvqiWPyrvIveeSE172gW/UQXZUhhOuoAVGQ1DK2Uyq/9 799i1WMSgsYvmJhV/eWGCumifPNL50Gp3FoX/GnBhCQZUlV9tA8xrvGYtONpeh3Xw5rC BlCLO/r3ZPPoFxY8xtQ0QXC37NV8W9I6kgpdVTg8U2VpTwXRZ222i33fgO4aY3NIjv9L v8vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aQBlUNuDgqPvC7NX3xeLdCIV1BWkCfXjevdKpEbnUys=; b=j6vvM6LSzZxRIfp4GvZxCdgs+LoDOs4coikylGZqxLjhztBxhJ8SDZlevu/KY6Qszc 6bDBc5AeX1JqLQXF1PK3T52r8XCY8Y6pS2nb7Fjtf9wvlJcPlaOMiT0ZCJjXxCsL39H+ VKb5ig1mpfgXPAj/FAXWqLiY2WyOUWhv2Ok9FndGGsoRIUpKsMTVTzdGbMgvjggxr572 WYrt+xIAoIhszsZppuOmkMWSRmwuZVG82gycmUGoOrpr0xglIgajZBOfSSm3F26o79Dj U9RJO6iYQojusiEwyslJ2qJlzNLKG4nkvCtqT37dE7lgw1xl3njNFieY6mWczquHhrbk NG3w== X-Gm-Message-State: AJIora9lz5qeDpHCbobIMASlSUsFvNBXbyNdSS5XxZSMk7iS5zLSJQxQ XvXXFDu9sTjoHTkC12W05URabJyB7qZE5g== X-Google-Smtp-Source: AGRyM1sWNg27JPYbj6O/Jt5GzbztVWUdzpNvDlUfpy6w74FylFHaebo4TjZ7J97fEg+zGfquAueoYg== X-Received: by 2002:a17:907:60c9:b0:726:d038:9968 with SMTP id hv9-20020a17090760c900b00726d0389968mr2799219ejc.468.1656418585792; Tue, 28 Jun 2022 05:16:25 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id k14-20020aa7c04e000000b00431962fe5d4sm9512780edo.77.2022.06.28.05.16.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jun 2022 05:16:25 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: linux-trace-devel@vger.kernel.org Subject: [PATCH v4 3/3] trace-cruncher: Example script for uprobes high level API Date: Tue, 28 Jun 2022 15:16:21 +0300 Message-Id: <20220628121621.572506-4-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220628121621.572506-1-tz.stoyanov@gmail.com> References: <20220628121621.572506-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Proposed example illustrates how to use uprobes high level API to trace all functions for user program. It can be attached to already running program, or run the program and trace its execution. Signed-off-by: Tzvetomir Stoyanov (VMware) --- examples/user_trace.py | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 examples/user_trace.py diff --git a/examples/user_trace.py b/examples/user_trace.py new file mode 100755 index 0000000..3fdef26 --- /dev/null +++ b/examples/user_trace.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 + +""" +SPDX-License-Identifier: CC-BY-4.0 + +Copyright 2022 VMware Inc, Tzvetomir Stoyanov (VMware) +""" + +import sys +import shutil + +import tracecruncher.ftracepy as ft + +if __name__ == "__main__": + if len(sys.argv) < 2: + print('Usage: ', sys.argv[0], ' [PROCESS or PID]') + sys.exit(1) + + # Create new Ftrace instance to work in. The tracing in this new instance + # is not going to be enabled yet. + inst = ft.create_instance(tracing_on=False) + + # Create a user tracing context for given process, exclude the libraries + if sys.argv[1].isdigit(): + utrace = ft.user_trace(pid=int(sys.argv[1]), follow_libs=False) + else: + utrace = ft.user_trace(argv=sys.argv[1:], follow_libs=False) + + # Trace execution of all available functions in the given process + utrace.add_function(fname="*") + + # Add trace points on functions return as well + utrace.add_ret_function(fname="*") + + # Start tracing in an instance + utrace.enable(instance=inst, wait=False) + + # Read the trace buffer during the trace until ctrl-c is pressed + ft.read_trace(instance=inst)