From patchwork Tue Jun 6 16:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74DFBC7EE29 for ; Tue, 6 Jun 2023 16:12:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237580AbjFFQMn (ORCPT ); Tue, 6 Jun 2023 12:12:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237579AbjFFQMm (ORCPT ); Tue, 6 Jun 2023 12:12:42 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD93CE8; Tue, 6 Jun 2023 09:12:39 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4A8D362E4C; Tue, 6 Jun 2023 16:12:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 72C63C433EF; Tue, 6 Jun 2023 16:12:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067958; bh=GqGwUJmK+00qm/XGLk0blCn7F+e5m9X1lkv94QKTFSM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IqhPAVMfhCXtct+mPWjrvHcF3b7ohhl3Wr2FXJFGrUMoy1N3ffJ0GRBK0HRbSrKJo 5h0JrBu+8lzGi/+5qlRwGhDB2vocuJYSF5FAmcmjycVX2FuiYHVbbTJi+kMtsB7CX6 NNvCCL0oNsAadrwmryRX+Akb0tajolAULc96qJ2ZjG5VUcYaqxnxOL1cnqRqHKGcAX jHmZ3tCUFTmj4PG0opnL1zZYiPH41Mypcr9xZ2TWm/YlOBBVn5d+SFtLl1nKOr+2mP WDaMoxD63wabSSv9nK1XA/cimh48cutAZLtGxkgowdrPtyG1WqqMOsYZLdFABvCNV/ tHzxJswW1f1Vw== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 01/11] rtla: Add -C cgroup support Date: Tue, 6 Jun 2023 18:12:15 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org The -C option sets a cgroup to the tracer's threads. If the -C option is passed without arguments, the tracer's thread will inherit rtla's cgroup. Otherwise, the threads will be placed on the cgroup passed to the option. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- Documentation/tools/rtla/common_options.rst | 4 + tools/tracing/rtla/src/osnoise_hist.c | 26 ++- tools/tracing/rtla/src/osnoise_top.c | 26 ++- tools/tracing/rtla/src/timerlat_hist.c | 27 ++- tools/tracing/rtla/src/timerlat_top.c | 26 ++- tools/tracing/rtla/src/utils.c | 185 ++++++++++++++++++++ tools/tracing/rtla/src/utils.h | 1 + 7 files changed, 286 insertions(+), 9 deletions(-) diff --git a/Documentation/tools/rtla/common_options.rst b/Documentation/tools/rtla/common_options.rst index af76df6205d4..ede07359d93c 100644 --- a/Documentation/tools/rtla/common_options.rst +++ b/Documentation/tools/rtla/common_options.rst @@ -42,6 +42,10 @@ - *f:prio* - use SCHED_FIFO with *prio*; - *d:runtime[us|ms|s]:period[us|ms|s]* - use SCHED_DEADLINE with *runtime* and *period* in nanoseconds. +**-C**, **--cgroup**\[*=cgroup*] + + Set a *cgroup* to the tracer's threads. If the **-C** option is passed without arguments, the tracer's thread will inherit **rtla**'s *cgroup*. Otherwise, the threads will be placed on the *cgroup* passed to the option. + **-h**, **--help** Print help menu. diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c index 13e1233690bb..076f4c6af3dd 100644 --- a/tools/tracing/rtla/src/osnoise_hist.c +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -19,6 +19,7 @@ struct osnoise_hist_params { char *cpus; char *monitored_cpus; char *trace_output; + char *cgroup_name; unsigned long long runtime; unsigned long long period; long long threshold; @@ -28,6 +29,7 @@ struct osnoise_hist_params { int duration; int set_sched; int output_divisor; + int cgroup; struct sched_attr sched_param; struct trace_events *events; @@ -433,7 +435,7 @@ static void osnoise_hist_usage(char *usage) " usage: rtla osnoise hist [-h] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\", " [-T us] [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] \\", " [-c cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] [--no-index] \\", - " [--with-zeros]", + " [--with-zeros] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit", @@ -443,6 +445,7 @@ static void osnoise_hist_usage(char *usage) " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", " -T/--threshold us: the minimum delta to be considered a noise", " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[s|m|h|d]: duration of the session", " -D/--debug: print debug info", " -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]", @@ -501,6 +504,7 @@ static struct osnoise_hist_params {"bucket-size", required_argument, 0, 'b'}, {"entries", required_argument, 0, 'E'}, {"cpus", required_argument, 0, 'c'}, + {"cgroup", optional_argument, 0, 'C'}, {"debug", no_argument, 0, 'D'}, {"duration", required_argument, 0, 'd'}, {"help", no_argument, 0, 'h'}, @@ -524,7 +528,7 @@ static struct osnoise_hist_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:b:d:e:E:Dhp:P:r:s:S:t::T:01234:5:", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhp:P:r:s:S:t::T:01234:5:", long_options, &option_index); /* detect the end of the options. */ @@ -554,6 +558,16 @@ static struct osnoise_hist_params osnoise_hist_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; break; + case 'C': + params->cgroup = 1; + if (!optarg) { + /* will inherit this cgroup */ + params->cgroup_name = NULL; + } else if (*optarg == '=') { + /* skip the = */ + params->cgroup_name = ++optarg; + } + break; case 'D': config_debug = 1; break; @@ -816,6 +830,14 @@ int osnoise_hist_main(int argc, char *argv[]) } } + if (params->cgroup) { + retval = set_comm_cgroup("timerlat/", params->cgroup_name); + if (!retval) { + err_msg("Failed to move threads to cgroup\n"); + goto out_free; + } + } + trace_instance_start(trace); if (params->trace_output) { diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index 562f2e4b18c5..139d8d392540 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -26,6 +26,7 @@ struct osnoise_top_params { char *cpus; char *monitored_cpus; char *trace_output; + char *cgroup_name; unsigned long long runtime; unsigned long long period; long long threshold; @@ -35,6 +36,7 @@ struct osnoise_top_params { int duration; int quiet; int set_sched; + int cgroup; struct sched_attr sched_param; struct trace_events *events; enum osnoise_mode mode; @@ -276,7 +278,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage) static const char * const msg[] = { " [-h] [-q] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\", " [-T us] [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] \\", - " [-c cpu-list] [-P priority]", + " [-c cpu-list] [-P priority] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit", @@ -286,6 +288,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage) " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", " -T/--threshold us: the minimum delta to be considered a noise", " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[s|m|h|d]: duration of the session", " -D/--debug: print debug info", " -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]", @@ -347,6 +350,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) static struct option long_options[] = { {"auto", required_argument, 0, 'a'}, {"cpus", required_argument, 0, 'c'}, + {"cgroup", optional_argument, 0, 'C'}, {"debug", no_argument, 0, 'D'}, {"duration", required_argument, 0, 'd'}, {"event", required_argument, 0, 'e'}, @@ -367,7 +371,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:d:De:hp:P:qr:s:S:t::T:0:1:", + c = getopt_long(argc, argv, "a:c:C::d:De:hp:P:qr:s:S:t::T:0:1:", long_options, &option_index); /* Detect the end of the options. */ @@ -392,6 +396,16 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) osnoise_top_usage(params, "\nInvalid -c cpu list\n"); params->cpus = optarg; break; + case 'C': + params->cgroup = 1; + if (!optarg) { + /* will inherit this cgroup */ + params->cgroup_name = NULL; + } else if (*optarg == '=') { + /* skip the = */ + params->cgroup_name = ++optarg; + } + break; case 'D': config_debug = 1; break; @@ -643,6 +657,14 @@ int osnoise_top_main(int argc, char **argv) } } + if (params->cgroup) { + retval = set_comm_cgroup("osnoise/", params->cgroup_name); + if (!retval) { + err_msg("Failed to move threads to cgroup\n"); + goto out_free; + } + } + trace_instance_start(trace); if (params->trace_output) { diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index 4b48af8a8309..459c159923e8 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -19,6 +19,7 @@ struct timerlat_hist_params { char *cpus; char *monitored_cpus; char *trace_output; + char *cgroup_name; unsigned long long runtime; long long stop_us; long long stop_total_us; @@ -29,9 +30,9 @@ struct timerlat_hist_params { int duration; int set_sched; int dma_latency; + int cgroup; struct sched_attr sched_param; struct trace_events *events; - char no_irq; char no_thread; char no_header; @@ -433,7 +434,7 @@ static void timerlat_hist_usage(char *usage) " usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\", " [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] \\", " [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\", - " [--no-index] [--with-zeros] [--dma-latency us]", + " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit", @@ -442,6 +443,7 @@ static void timerlat_hist_usage(char *usage) " -T/--thread us: stop trace if the thread latency is higher than the argument in us", " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", " -c/--cpus cpus: run the tracer only on the given cpus", + " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[m|h|d]: duration of the session in seconds", " -D/--debug: print debug info", " -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]", @@ -506,6 +508,7 @@ static struct timerlat_hist_params static struct option long_options[] = { {"auto", required_argument, 0, 'a'}, {"cpus", required_argument, 0, 'c'}, + {"cgroup", optional_argument, 0, 'C'}, {"bucket-size", required_argument, 0, 'b'}, {"debug", no_argument, 0, 'D'}, {"entries", required_argument, 0, 'E'}, @@ -534,7 +537,7 @@ static struct timerlat_hist_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:b:d:e:E:Dhi:np:P:s:t::T:0123456:7:8:", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhi:np:P:s:t::T:0123456:7:8:", long_options, &option_index); /* detect the end of the options. */ @@ -561,6 +564,16 @@ static struct timerlat_hist_params timerlat_hist_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; break; + case 'C': + params->cgroup = 1; + if (!optarg) { + /* will inherit this cgroup */ + params->cgroup_name = NULL; + } else if (*optarg == '=') { + /* skip the = */ + params->cgroup_name = ++optarg; + } + break; case 'b': params->bucket_size = get_llong_from_str(optarg); if ((params->bucket_size == 0) || (params->bucket_size >= 1000000)) @@ -840,6 +853,14 @@ int timerlat_hist_main(int argc, char *argv[]) } } + if (params->cgroup) { + retval = set_comm_cgroup("timerlat/", params->cgroup_name); + if (!retval) { + err_msg("Failed to move threads to cgroup\n"); + goto out_free; + } + } + if (params->dma_latency >= 0) { dma_latency_fd = set_cpu_dma_latency(params->dma_latency); if (dma_latency_fd < 0) { diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index 92c658c64f28..a19cbc2aa1f4 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -21,6 +21,7 @@ struct timerlat_top_params { char *cpus; char *monitored_cpus; char *trace_output; + char *cgroup_name; unsigned long long runtime; long long stop_us; long long stop_total_us; @@ -35,6 +36,7 @@ struct timerlat_top_params { int no_aa; int aa_only; int dump_tasks; + int cgroup; struct sched_attr sched_param; struct trace_events *events; }; @@ -285,7 +287,7 @@ static void timerlat_top_usage(char *usage) "", " usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\", " [[-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] \\", - " [-P priority] [--dma-latency us] [--aa-only us]", + " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit", @@ -295,6 +297,7 @@ static void timerlat_top_usage(char *usage) " -T/--thread us: stop trace if the thread latency is higher than the argument in us", " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", " -c/--cpus cpus: run the tracer only on the given cpus", + " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[m|h|d]: duration of the session in seconds", " -D/--debug: print debug info", " --dump-tasks: prints the task running on all CPUs if stop conditions are met (depends on !--no-aa)", @@ -352,6 +355,7 @@ static struct timerlat_top_params static struct option long_options[] = { {"auto", required_argument, 0, 'a'}, {"cpus", required_argument, 0, 'c'}, + {"cgroup", optional_argument, 0, 'C'}, {"debug", no_argument, 0, 'D'}, {"duration", required_argument, 0, 'd'}, {"event", required_argument, 0, 'e'}, @@ -376,7 +380,7 @@ static struct timerlat_top_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:d:De:hi:np:P:qs:t::T:0:1:2:345:", + c = getopt_long(argc, argv, "a:c:C::d:De:hi:np:P:qs:t::T:0:1:2:345:", long_options, &option_index); /* detect the end of the options. */ @@ -417,6 +421,16 @@ static struct timerlat_top_params timerlat_top_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; break; + case 'C': + params->cgroup = 1; + if (!optarg) { + /* will inherit this cgroup */ + params->cgroup_name = NULL; + } else if (*optarg == '=') { + /* skip the = */ + params->cgroup_name = ++optarg; + } + break; case 'D': config_debug = 1; break; @@ -694,6 +708,14 @@ int timerlat_top_main(int argc, char *argv[]) } } + if (params->cgroup) { + retval = set_comm_cgroup("timerlat/", params->cgroup_name); + if (!retval) { + err_msg("Failed to move threads to cgroup\n"); + goto out_free; + } + } + if (params->dma_latency >= 0) { dma_latency_fd = set_cpu_dma_latency(params->dma_latency); if (dma_latency_fd < 0) { diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c index 663a047f794d..bcc0a9f39cfe 100644 --- a/tools/tracing/rtla/src/utils.c +++ b/tools/tracing/rtla/src/utils.c @@ -529,3 +529,188 @@ int set_cpu_dma_latency(int32_t latency) return fd; } + +#define _STR(x) #x +#define STR(x) _STR(x) + +/* + * find_mount - find a the mount point of a given fs + * + * Returns 0 if mount is not found, otherwise return 1 and fill mp + * with the mount point. + */ +static const int find_mount(const char *fs, char *mp, int sizeof_mp) +{ + char mount_point[MAX_PATH]; + char type[100]; + int found; + FILE *fp; + + fp = fopen("/proc/mounts", "r"); + if (!fp) + return 0; + + while (fscanf(fp, "%*s %" STR(MAX_PATH) "s %99s %*s %*d %*d\n", mount_point, type) == 2) { + if (strcmp(type, fs) == 0) { + found = 1; + break; + } + } + fclose(fp); + + if (!found) + return 0; + + memset(mp, 0, sizeof_mp); + strncpy(mp, mount_point, sizeof_mp - 1); + + debug_msg("Fs %s found at %s\n", fs, mp); + return 1; +} + +/* + * get_self_cgroup - get the current thread cgroup path + * + * Parse /proc/$$/cgroup file to get the thread's cgroup. As an example of line to parse: + * + * 0::/user.slice/user-0.slice/session-3.scope'\n' + * + * This function is interested in the content after the second : and before the '\n'. + * + * Returns 1 if a string was found, 0 otherwise. + */ +static int get_self_cgroup(char *self_cg, int sizeof_self_cg) +{ + char path[MAX_PATH], *start; + int fd, retval; + + snprintf(path, MAX_PATH, "/proc/%d/cgroup", getpid()); + + fd = open(path, O_RDONLY); + if (fd < 0) + return 0; + + retval = read(fd, path, MAX_PATH); + + close(fd); + + if (retval <= 0) + return 0; + + start = path; + + start = strstr(start, ":"); + if (!start) + return 0; + + /* skip ":" */ + start++; + + start = strstr(start, ":"); + if (!start) + return 0; + + /* skip ":" */ + start++; + + if (strlen(start) >= sizeof_self_cg) + return 0; + + snprintf(self_cg, sizeof_self_cg, "%s", start); + + /* Swap '\n' with '\0' */ + start = strstr(self_cg, "\n"); + + /* there must be '\n' */ + if (!start) + return 0; + + /* ok, it found a string after the second : and before the \n */ + *start = '\0'; + + return 1; +} + +/** + * set_comm_cgroup - Set cgroup to threads starting with char *comm_prefix + * + * If cgroup argument is not NULL, the threads will move to the given cgroup. + * Otherwise, the cgroup of the calling, i.e., rtla, thread will be used. + * + * Supports cgroup v2. + * + * Returns 1 on success, 0 otherwise. + */ +int set_comm_cgroup(const char *comm_prefix, const char *cgroup) +{ + char cgroup_path[MAX_PATH - strlen("/cgroup.procs")]; + char cgroup_procs[MAX_PATH]; + struct dirent *proc_entry; + DIR *procfs; + int retval; + int cg_fd; + + if (strlen(comm_prefix) >= MAX_PATH) { + err_msg("Command prefix is too long: %d < strlen(%s)\n", + MAX_PATH, comm_prefix); + return 0; + } + + retval = find_mount("cgroup2", cgroup_path, sizeof(cgroup_path)); + if (!retval) { + err_msg("Did not find cgroupv2 mount point\n"); + return 0; + } + + if (!cgroup) { + retval = get_self_cgroup(&cgroup_path[strlen(cgroup_path)], + sizeof(cgroup_path) - strlen(cgroup_path)); + if (!retval) { + err_msg("Did not find self cgroup\n"); + return 0; + } + } else { + snprintf(&cgroup_path[strlen(cgroup_path)], + sizeof(cgroup_path) - strlen(cgroup_path), "%s/", cgroup); + } + + snprintf(cgroup_procs, MAX_PATH, "%s/cgroup.procs", cgroup_path); + + debug_msg("Using cgroup path at: %s\n", cgroup_procs); + + cg_fd = open(cgroup_procs, O_RDWR); + if (cg_fd < 0) + return 0; + + procfs = opendir("/proc"); + if (!procfs) { + err_msg("Could not open procfs\n"); + goto out_cg; + } + + while ((proc_entry = readdir(procfs))) { + + retval = procfs_is_workload_pid(comm_prefix, proc_entry); + if (!retval) + continue; + + retval = write(cg_fd, proc_entry->d_name, strlen(proc_entry->d_name)); + if (retval < 0) { + err_msg("Error setting cgroup attributes for pid:%s - %s\n", + proc_entry->d_name, strerror(errno)); + goto out_procfs; + } + + debug_msg("Set cgroup attributes for pid:%s\n", proc_entry->d_name); + } + + closedir(procfs); + close(cg_fd); + return 1; + +out_procfs: + closedir(procfs); +out_cg: + close(cg_fd); + return 0; +} diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h index 90e4f52a030b..42b6f099d10a 100644 --- a/tools/tracing/rtla/src/utils.h +++ b/tools/tracing/rtla/src/utils.h @@ -55,6 +55,7 @@ struct sched_attr { int parse_prio(char *arg, struct sched_attr *sched_param); int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr); +int set_comm_cgroup(const char *comm_prefix, const char *cgroup); int set_cpu_dma_latency(int32_t latency); #define ns_to_usf(x) (((double)x/1000)) From patchwork Tue Jun 6 16:12:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60A15C7EE24 for ; Tue, 6 Jun 2023 16:12:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237732AbjFFQMq (ORCPT ); Tue, 6 Jun 2023 12:12:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237642AbjFFQMo (ORCPT ); Tue, 6 Jun 2023 12:12:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4553F12D; Tue, 6 Jun 2023 09:12:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C0D5062DE6; Tue, 6 Jun 2023 16:12:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29267C4339E; Tue, 6 Jun 2023 16:12:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067961; bh=0ubVhPhyNbTPpZGTTaXHkR9+c1t8+bxgv6RbmWcdImM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QG0t8bS761E9ufHdGOFkJXyyoW7sPWgV7IQZ07FHx7qNty7L0mI0gBdlJK+T3Jg6K KwugnmHpXe+XbJshU7eqq/98ocBdZgzGelAAn7sSKUjsHCr/XNUBmeKisF5fgK9U74 gUX3EkPefQdYwYlc3LRRcNxdPI6rFKBMisLaCP7axbf9yHX/DOZyT9OPXOrGd+/Ycw 9xmCxfRAmcmYYPd7u6YgDz1sxjv1assdJ3hjiqXQyNAcO91myarc/Tg5kjnoGG8OLa tdB0O1leUW2+nUlJ7TH7y6zFyqQ1afKRtcbRUCocbBzoTARCF5ZgEoVa/Z9zS751+8 K5foC7BA6SnoQ== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 02/11] rtla: Add --house-keeping option Date: Tue, 6 Jun 2023 18:12:16 +0200 Message-Id: <6a6c78a579a96ba8b02ae67ee1e0ba2cb5e03c4a.1686066600.git.bristot@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org To avoid having rtla interfering with the measurement threads, add an option for the user to set the CPUs in which rtla should run. For instance: # rtla timerlat top -H 0 -C 1-7 Will place rtla in the CPU 0, while running the measurement threads in the CPU 1-7. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Suggested-by: Juri Lelli Signed-off-by: Daniel Bristot de Oliveira --- Documentation/tools/rtla/common_options.rst | 4 ++ tools/tracing/rtla/src/osnoise_hist.c | 29 +++++++++-- tools/tracing/rtla/src/osnoise_top.c | 27 +++++++++- tools/tracing/rtla/src/timerlat_hist.c | 27 +++++++++- tools/tracing/rtla/src/timerlat_top.c | 27 +++++++++- tools/tracing/rtla/src/utils.c | 58 +++++++++++++++++++++ tools/tracing/rtla/src/utils.h | 3 ++ 7 files changed, 166 insertions(+), 9 deletions(-) diff --git a/Documentation/tools/rtla/common_options.rst b/Documentation/tools/rtla/common_options.rst index ede07359d93c..aeb91ff3bd68 100644 --- a/Documentation/tools/rtla/common_options.rst +++ b/Documentation/tools/rtla/common_options.rst @@ -2,6 +2,10 @@ Set the osnoise tracer to run the sample threads in the cpu-list. +**-H**, **--house-keeping** *cpu-list* + + Run rtla control threads only on the given cpu-list. + **-d**, **--duration** *time[s|m|h|d]* Set the duration of the session. diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c index 076f4c6af3dd..d2b68177ffac 100644 --- a/tools/tracing/rtla/src/osnoise_hist.c +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -3,6 +3,7 @@ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira */ +#define _GNU_SOURCE #include #include #include @@ -11,6 +12,7 @@ #include #include #include +#include #include "utils.h" #include "osnoise.h" @@ -30,6 +32,8 @@ struct osnoise_hist_params { int set_sched; int output_divisor; int cgroup; + int hk_cpus; + cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; @@ -434,8 +438,8 @@ static void osnoise_hist_usage(char *usage) "", " usage: rtla osnoise hist [-h] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\", " [-T us] [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] \\", - " [-c cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] [--no-index] \\", - " [--with-zeros] [-C[=cgroup_name]]", + " [-c cpu-list] [-H cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] \\", + " [--no-index] [--with-zeros] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit", @@ -445,6 +449,7 @@ static void osnoise_hist_usage(char *usage) " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", " -T/--threshold us: the minimum delta to be considered a noise", " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -H/--house-keeping cpus: run rtla control threads only on the given cpus", " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[s|m|h|d]: duration of the session", " -D/--debug: print debug info", @@ -507,6 +512,7 @@ static struct osnoise_hist_params {"cgroup", optional_argument, 0, 'C'}, {"debug", no_argument, 0, 'D'}, {"duration", required_argument, 0, 'd'}, + {"house-keeping", required_argument, 0, 'H'}, {"help", no_argument, 0, 'h'}, {"period", required_argument, 0, 'p'}, {"priority", required_argument, 0, 'P'}, @@ -528,7 +534,7 @@ static struct osnoise_hist_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhp:P:r:s:S:t::T:01234:5:", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:p:P:r:s:S:t::T:01234:5:", long_options, &option_index); /* detect the end of the options. */ @@ -597,6 +603,14 @@ static struct osnoise_hist_params case '?': osnoise_hist_usage(NULL); break; + case 'H': + params->hk_cpus = 1; + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set); + if (retval) { + err_msg("Error parsing house keeping CPUs\n"); + exit(EXIT_FAILURE); + } + break; case 'p': params->period = get_llong_from_str(optarg); if (params->period > 10000000) @@ -732,6 +746,15 @@ osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params } } + if (params->hk_cpus) { + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set), + ¶ms->hk_cpu_set); + if (retval == -1) { + err_msg("Failed to set rtla to the house keeping CPUs\n"); + goto out_err; + } + } + return 0; out_err: diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index 139d8d392540..fcf6c14ce1bc 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -3,6 +3,7 @@ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira */ +#define _GNU_SOURCE #include #include #include @@ -10,6 +11,7 @@ #include #include #include +#include #include "osnoise.h" #include "utils.h" @@ -37,6 +39,8 @@ struct osnoise_top_params { int quiet; int set_sched; int cgroup; + int hk_cpus; + cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; enum osnoise_mode mode; @@ -278,7 +282,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage) static const char * const msg[] = { " [-h] [-q] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\", " [-T us] [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] \\", - " [-c cpu-list] [-P priority] [-C[=cgroup_name]]", + " [-c cpu-list] [-H cpu-list] [-P priority] [-C[=cgroup_name]]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit", @@ -288,6 +292,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage) " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", " -T/--threshold us: the minimum delta to be considered a noise", " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -H/--house-keeping cpus: run rtla control threads only on the given cpus", " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[s|m|h|d]: duration of the session", " -D/--debug: print debug info", @@ -354,6 +359,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) {"debug", no_argument, 0, 'D'}, {"duration", required_argument, 0, 'd'}, {"event", required_argument, 0, 'e'}, + {"house-keeping", required_argument, 0, 'H'}, {"help", no_argument, 0, 'h'}, {"period", required_argument, 0, 'p'}, {"priority", required_argument, 0, 'P'}, @@ -371,7 +377,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::d:De:hp:P:qr:s:S:t::T:0:1:", + c = getopt_long(argc, argv, "a:c:C::d:De:hH:p:P:qr:s:S:t::T:0:1:", long_options, &option_index); /* Detect the end of the options. */ @@ -430,6 +436,14 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) case '?': osnoise_top_usage(params, NULL); break; + case 'H': + params->hk_cpus = 1; + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set); + if (retval) { + err_msg("Error parsing house keeping CPUs\n"); + exit(EXIT_FAILURE); + } + break; case 'p': params->period = get_llong_from_str(optarg); if (params->period > 10000000) @@ -561,6 +575,15 @@ osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *p } } + if (params->hk_cpus) { + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set), + ¶ms->hk_cpu_set); + if (retval == -1) { + err_msg("Failed to set rtla to the house keeping CPUs\n"); + goto out_err; + } + } + return 0; out_err: diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index 459c159923e8..d48c05d238f9 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -3,6 +3,7 @@ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira */ +#define _GNU_SOURCE #include #include #include @@ -10,6 +11,7 @@ #include #include #include +#include #include "utils.h" #include "osnoise.h" @@ -31,6 +33,8 @@ struct timerlat_hist_params { int set_sched; int dma_latency; int cgroup; + int hk_cpus; + cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; char no_irq; @@ -432,7 +436,7 @@ static void timerlat_hist_usage(char *usage) char *msg[] = { "", " usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\", - " [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] \\", + " [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] [-H cpu-list]\\", " [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\", " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]", "", @@ -443,6 +447,7 @@ static void timerlat_hist_usage(char *usage) " -T/--thread us: stop trace if the thread latency is higher than the argument in us", " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", " -c/--cpus cpus: run the tracer only on the given cpus", + " -H/--house-keeping cpus: run rtla control threads only on the given cpus", " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[m|h|d]: duration of the session in seconds", " -D/--debug: print debug info", @@ -513,6 +518,7 @@ static struct timerlat_hist_params {"debug", no_argument, 0, 'D'}, {"entries", required_argument, 0, 'E'}, {"duration", required_argument, 0, 'd'}, + {"house-keeping", required_argument, 0, 'H'}, {"help", no_argument, 0, 'h'}, {"irq", required_argument, 0, 'i'}, {"nano", no_argument, 0, 'n'}, @@ -537,7 +543,7 @@ static struct timerlat_hist_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhi:np:P:s:t::T:0123456:7:8:", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:", long_options, &option_index); /* detect the end of the options. */ @@ -608,6 +614,14 @@ static struct timerlat_hist_params case '?': timerlat_hist_usage(NULL); break; + case 'H': + params->hk_cpus = 1; + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set); + if (retval) { + err_msg("Error parsing house keeping CPUs\n"); + exit(EXIT_FAILURE); + } + break; case 'i': params->stop_us = get_llong_from_str(optarg); break; @@ -755,6 +769,15 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param } } + if (params->hk_cpus) { + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set), + ¶ms->hk_cpu_set); + if (retval == -1) { + err_msg("Failed to set rtla to the house keeping CPUs\n"); + goto out_err; + } + } + return 0; out_err: diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index a19cbc2aa1f4..5395d1c5921e 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -3,6 +3,7 @@ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira */ +#define _GNU_SOURCE #include #include #include @@ -11,6 +12,7 @@ #include #include #include +#include #include "utils.h" #include "osnoise.h" @@ -37,6 +39,8 @@ struct timerlat_top_params { int aa_only; int dump_tasks; int cgroup; + int hk_cpus; + cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; }; @@ -286,7 +290,7 @@ static void timerlat_top_usage(char *usage) static const char *const msg[] = { "", " usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\", - " [[-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] \\", + " [[-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] [-H cpu-list]\\", " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]", "", " -h/--help: print this menu", @@ -297,6 +301,7 @@ static void timerlat_top_usage(char *usage) " -T/--thread us: stop trace if the thread latency is higher than the argument in us", " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", " -c/--cpus cpus: run the tracer only on the given cpus", + " -H/--house-keeping cpus: run rtla control threads only on the given cpus", " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[m|h|d]: duration of the session in seconds", " -D/--debug: print debug info", @@ -360,6 +365,7 @@ static struct timerlat_top_params {"duration", required_argument, 0, 'd'}, {"event", required_argument, 0, 'e'}, {"help", no_argument, 0, 'h'}, + {"house-keeping", required_argument, 0, 'H'}, {"irq", required_argument, 0, 'i'}, {"nano", no_argument, 0, 'n'}, {"period", required_argument, 0, 'p'}, @@ -380,7 +386,7 @@ static struct timerlat_top_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::d:De:hi:np:P:qs:t::T:0:1:2:345:", + c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:0:1:2:345:", long_options, &option_index); /* detect the end of the options. */ @@ -454,6 +460,14 @@ static struct timerlat_top_params case '?': timerlat_top_usage(NULL); break; + case 'H': + params->hk_cpus = 1; + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set); + if (retval) { + err_msg("Error parsing house keeping CPUs\n"); + exit(EXIT_FAILURE); + } + break; case 'i': params->stop_us = get_llong_from_str(optarg); break; @@ -598,6 +612,15 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params * } } + if (params->hk_cpus) { + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set), + ¶ms->hk_cpu_set); + if (retval == -1) { + err_msg("Failed to set rtla to the house keeping CPUs\n"); + goto out_err; + } + } + return 0; out_err: diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c index bcc0a9f39cfe..ee6fab09acae 100644 --- a/tools/tracing/rtla/src/utils.c +++ b/tools/tracing/rtla/src/utils.c @@ -3,6 +3,7 @@ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira */ +#define _GNU_SOURCE #include #include #include @@ -150,6 +151,63 @@ int parse_cpu_list(char *cpu_list, char **monitored_cpus) return 1; } +/* + * parse_cpu_set - parse a cpu_list filling cpu_set_t argument + * + * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set + * filling cpu_set_t argument. + * + * Returns 1 on success, 0 otherwise. + */ +int parse_cpu_set(char *cpu_list, cpu_set_t *set) +{ + const char *p; + int end_cpu; + int nr_cpus; + int cpu; + int i; + + CPU_ZERO(set); + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + for (p = cpu_list; *p; ) { + cpu = atoi(p); + if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus) + goto err; + + while (isdigit(*p)) + p++; + if (*p == '-') { + p++; + end_cpu = atoi(p); + if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus) + goto err; + while (isdigit(*p)) + p++; + } else + end_cpu = cpu; + + if (cpu == end_cpu) { + debug_msg("cpu_set: adding cpu %d\n", cpu); + CPU_SET(cpu, set); + } else { + for (i = cpu; i <= end_cpu; i++) { + debug_msg("cpu_set: adding cpu %d\n", i); + CPU_SET(i, set); + } + } + + if (*p == ',') + p++; + } + + return 0; +err: + debug_msg("Error parsing the cpu set %s\n", cpu_list); + return 1; +} + /* * parse_duration - parse duration with s/m/h/d suffix converting it to seconds */ diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h index 42b6f099d10a..9ab2f0d7bc1c 100644 --- a/tools/tracing/rtla/src/utils.h +++ b/tools/tracing/rtla/src/utils.h @@ -1,6 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 + #include #include +#include /* * '18446744073709551615\0' @@ -54,6 +56,7 @@ struct sched_attr { }; int parse_prio(char *arg, struct sched_attr *sched_param); +int parse_cpu_set(char *cpu_list, cpu_set_t *set); int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr); int set_comm_cgroup(const char *comm_prefix, const char *cgroup); int set_cpu_dma_latency(int32_t latency); From patchwork Tue Jun 6 16:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED9DFC7EE2A for ; Tue, 6 Jun 2023 16:12:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238047AbjFFQMw (ORCPT ); Tue, 6 Jun 2023 12:12:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237742AbjFFQMq (ORCPT ); Tue, 6 Jun 2023 12:12:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A316210D5; Tue, 6 Jun 2023 09:12:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3FEE162F5A; Tue, 6 Jun 2023 16:12:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E005C433A0; Tue, 6 Jun 2023 16:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067963; bh=ilvKubt4M+tI9BEkBeyxjS9nZpmPWWkvlzUOJ9vM89c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E4a4AkLCq6C4FCVXVO1OUfLvKy7MDnB6B2obdZC9bzb+/VsGUs+E6/wDiigpR85Sk WxzxGOAWlKSGppYAh5VeRgqxiTTzc1IQCtlquQmtXNd8MjDqNca5oGqpBmasW2/H8T Oi/usxLJLQjr82aGbMTw25/VXa+MAqd2EUyEizayXqyBa03aVLwFTbtfIg5hH12/6h XgIYu93bMRKH6Ze3rJwI2/cvO7FwAc+D6V9O7qPy9JYsUH5tQYk13TyM4JnoFUol7f zuN3xIf/ePJEhQZPrfjJ1/dhEvMgLidUnt5XtVwxL5F5GAV59N+lARuzNT9UaOZG4G NW7HZfzzK3GxA== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 03/11] rtla: Change monitored_cpus from char * to cpu_set_t Date: Tue, 6 Jun 2023 18:12:17 +0200 Message-Id: <54c46293261d13cb1042d0314486539eeb45fe5d.1686066600.git.bristot@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Use a cpumask instead of a char *, reducing memory footprint and code. No functional change, and in preparation for auto house-keeping. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise_hist.c | 18 ++++---- tools/tracing/rtla/src/osnoise_top.c | 6 +-- tools/tracing/rtla/src/timerlat_hist.c | 18 ++++---- tools/tracing/rtla/src/timerlat_top.c | 6 +-- tools/tracing/rtla/src/utils.c | 63 -------------------------- 5 files changed, 24 insertions(+), 87 deletions(-) diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c index d2b68177ffac..b616a72d5c0a 100644 --- a/tools/tracing/rtla/src/osnoise_hist.c +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -19,7 +19,7 @@ struct osnoise_hist_params { char *cpus; - char *monitored_cpus; + cpu_set_t monitored_cpus; char *trace_output; char *cgroup_name; unsigned long long runtime; @@ -274,7 +274,7 @@ static void osnoise_hist_header(struct osnoise_tool *tool) trace_seq_printf(s, "Index"); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -305,7 +305,7 @@ osnoise_print_summary(struct osnoise_hist_params *params, trace_seq_printf(trace->seq, "count:"); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -319,7 +319,7 @@ osnoise_print_summary(struct osnoise_hist_params *params, trace_seq_printf(trace->seq, "min: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -334,7 +334,7 @@ osnoise_print_summary(struct osnoise_hist_params *params, trace_seq_printf(trace->seq, "avg: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -352,7 +352,7 @@ osnoise_print_summary(struct osnoise_hist_params *params, trace_seq_printf(trace->seq, "max: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -387,7 +387,7 @@ osnoise_print_stats(struct osnoise_hist_params *params, struct osnoise_tool *too bucket * data->bucket_size); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -411,7 +411,7 @@ osnoise_print_stats(struct osnoise_hist_params *params, struct osnoise_tool *too trace_seq_printf(trace->seq, "over: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].count) @@ -559,7 +559,7 @@ static struct osnoise_hist_params osnoise_hist_usage("Bucket size needs to be > 0 and <= 1000000\n"); break; case 'c': - retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + retval = parse_cpu_set(optarg, ¶ms->monitored_cpus); if (retval) osnoise_hist_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index fcf6c14ce1bc..0e2f3b216d34 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -26,7 +26,7 @@ enum osnoise_mode { */ struct osnoise_top_params { char *cpus; - char *monitored_cpus; + cpu_set_t monitored_cpus; char *trace_output; char *cgroup_name; unsigned long long runtime; @@ -263,7 +263,7 @@ osnoise_print_stats(struct osnoise_top_params *params, struct osnoise_tool *top) osnoise_top_header(top); for (i = 0; i < nr_cpus; i++) { - if (params->cpus && !params->monitored_cpus[i]) + if (params->cpus && !CPU_ISSET(i, ¶ms->monitored_cpus)) continue; osnoise_top_print(top, i); } @@ -397,7 +397,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) break; case 'c': - retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + retval = parse_cpu_set(optarg, ¶ms->monitored_cpus); if (retval) osnoise_top_usage(params, "\nInvalid -c cpu list\n"); params->cpus = optarg; diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index d48c05d238f9..00287e96f22e 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -19,7 +19,7 @@ struct timerlat_hist_params { char *cpus; - char *monitored_cpus; + cpu_set_t monitored_cpus; char *trace_output; char *cgroup_name; unsigned long long runtime; @@ -227,7 +227,7 @@ static void timerlat_hist_header(struct osnoise_tool *tool) trace_seq_printf(s, "Index"); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -263,7 +263,7 @@ timerlat_print_summary(struct timerlat_hist_params *params, trace_seq_printf(trace->seq, "count:"); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -283,7 +283,7 @@ timerlat_print_summary(struct timerlat_hist_params *params, trace_seq_printf(trace->seq, "min: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -303,7 +303,7 @@ timerlat_print_summary(struct timerlat_hist_params *params, trace_seq_printf(trace->seq, "avg: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -331,7 +331,7 @@ timerlat_print_summary(struct timerlat_hist_params *params, trace_seq_printf(trace->seq, "max: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -371,7 +371,7 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t bucket * data->bucket_size); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -405,7 +405,7 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t trace_seq_printf(trace->seq, "over: "); for (cpu = 0; cpu < data->nr_cpus; cpu++) { - if (params->cpus && !params->monitored_cpus[cpu]) + if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus)) continue; if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count) @@ -565,7 +565,7 @@ static struct timerlat_hist_params break; case 'c': - retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + retval = parse_cpu_set(optarg, ¶ms->monitored_cpus); if (retval) timerlat_hist_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index 5395d1c5921e..920f2f6ef842 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -21,7 +21,7 @@ struct timerlat_top_params { char *cpus; - char *monitored_cpus; + cpu_set_t monitored_cpus; char *trace_output; char *cgroup_name; unsigned long long runtime; @@ -271,7 +271,7 @@ timerlat_print_stats(struct timerlat_top_params *params, struct osnoise_tool *to timerlat_top_header(top); for (i = 0; i < nr_cpus; i++) { - if (params->cpus && !params->monitored_cpus[i]) + if (params->cpus && !CPU_ISSET(i, ¶ms->monitored_cpus)) continue; timerlat_top_print(top, i); } @@ -422,7 +422,7 @@ static struct timerlat_top_params params->aa_only = 1; break; case 'c': - retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + retval = parse_cpu_set(optarg, ¶ms->monitored_cpus); if (retval) timerlat_top_usage("\nInvalid -c cpu list\n"); params->cpus = optarg; diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c index ee6fab09acae..8f9ad8f01e0f 100644 --- a/tools/tracing/rtla/src/utils.c +++ b/tools/tracing/rtla/src/utils.c @@ -88,69 +88,6 @@ void get_duration(time_t start_time, char *output, int output_size) tm_info->tm_sec); } -/* - * parse_cpu_list - parse a cpu_list filling a char vector with cpus set - * - * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set the char - * in the monitored_cpus. - * - * XXX: convert to a bitmask. - */ -int parse_cpu_list(char *cpu_list, char **monitored_cpus) -{ - char *mon_cpus; - const char *p; - int end_cpu; - int nr_cpus; - int cpu; - int i; - - nr_cpus = sysconf(_SC_NPROCESSORS_CONF); - - mon_cpus = calloc(nr_cpus, sizeof(char)); - if (!mon_cpus) - goto err; - - for (p = cpu_list; *p; ) { - cpu = atoi(p); - if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus) - goto err; - - while (isdigit(*p)) - p++; - if (*p == '-') { - p++; - end_cpu = atoi(p); - if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus) - goto err; - while (isdigit(*p)) - p++; - } else - end_cpu = cpu; - - if (cpu == end_cpu) { - debug_msg("cpu_list: adding cpu %d\n", cpu); - mon_cpus[cpu] = 1; - } else { - for (i = cpu; i <= end_cpu; i++) { - debug_msg("cpu_list: adding cpu %d\n", i); - mon_cpus[i] = 1; - } - } - - if (*p == ',') - p++; - } - - *monitored_cpus = mon_cpus; - - return 0; - -err: - debug_msg("Error parsing the cpu list %s", cpu_list); - return 1; -} - /* * parse_cpu_set - parse a cpu_list filling cpu_set_t argument * From patchwork Tue Jun 6 16:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2FA2C7EE29 for ; Tue, 6 Jun 2023 16:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237887AbjFFQM5 (ORCPT ); Tue, 6 Jun 2023 12:12:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237922AbjFFQMw (ORCPT ); Tue, 6 Jun 2023 12:12:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E20610F4; Tue, 6 Jun 2023 09:12:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B3CF360AF9; Tue, 6 Jun 2023 16:12:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DAF9C4339B; Tue, 6 Jun 2023 16:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067966; bh=I75jhz8De3NPiqlgBoHOa8UPruFRWJnxpiQkId7IjnI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SZE1i8HzRqmKNPoHMqabR5Z1PECmsMIX30uBE0fJPOWl4U7RmFKio+9xMCVQLzn48 +pJhM4fHsUUT/imhuR+qVSpGN5hNxECrU8IQC8hYl645uDCZ4TKu+53NkRRC37adkW FKAcPLKW2TNpPw0lDqHXaDFyH037A++5Vbev+go0v003wNY5j1Zx5Uh9cRXMbrwIMq IHbm5TLOV/asrCU8t+sxnPjtsEHIkRLE8r4F5tx2526Se3FWXm8WPSF7f7MUTcdO8+ YY8Wo6uhL6fCIBVHSfYNjBMdcMH5o5Mnv952TFfqG3u0XWhoqUMe+DZjzBwYh5kYbT Hzd12kz9Xxxjw== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 04/11] rtla: Automatically move rtla to a house-keeping cpu Date: Tue, 6 Jun 2023 18:12:18 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org When the user sets -c try to move rtla out of the , even without an -H option. This is useful to avoid having rtla interfering with the workload. This works by removing from rtla's current affinity. If rtla fails to move itself away it is not that of a problem as this is an automatic measure. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise_hist.c | 9 +++++ tools/tracing/rtla/src/osnoise_top.c | 9 +++++ tools/tracing/rtla/src/timerlat_hist.c | 9 +++++ tools/tracing/rtla/src/timerlat_top.c | 9 +++++ tools/tracing/rtla/src/utils.c | 50 ++++++++++++++++++++++++++ tools/tracing/rtla/src/utils.h | 1 + 6 files changed, 87 insertions(+) diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c index b616a72d5c0a..dfbcb5ca7ecb 100644 --- a/tools/tracing/rtla/src/osnoise_hist.c +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -753,6 +753,15 @@ osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params err_msg("Failed to set rtla to the house keeping CPUs\n"); goto out_err; } + } else if (params->cpus) { + /* + * Even if the user do not set a house-keeping CPU, try to + * move rtla to a CPU set different to the one where the user + * set the workload to run. + * + * No need to check results as this is an automatic attempt. + */ + auto_house_keeping(¶ms->monitored_cpus); } return 0; diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index 0e2f3b216d34..0833537bb2eb 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -582,6 +582,15 @@ osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *p err_msg("Failed to set rtla to the house keeping CPUs\n"); goto out_err; } + } else if (params->cpus) { + /* + * Even if the user do not set a house-keeping CPU, try to + * move rtla to a CPU set different to the one where the user + * set the workload to run. + * + * No need to check results as this is an automatic attempt. + */ + auto_house_keeping(¶ms->monitored_cpus); } return 0; diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index 00287e96f22e..1675d54cae81 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -776,6 +776,15 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param err_msg("Failed to set rtla to the house keeping CPUs\n"); goto out_err; } + } else if (params->cpus) { + /* + * Even if the user do not set a house-keeping CPU, try to + * move rtla to a CPU set different to the one where the user + * set the workload to run. + * + * No need to check results as this is an automatic attempt. + */ + auto_house_keeping(¶ms->monitored_cpus); } return 0; diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index 920f2f6ef842..f0c6d9735e2a 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -619,6 +619,15 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params * err_msg("Failed to set rtla to the house keeping CPUs\n"); goto out_err; } + } else if (params->cpus) { + /* + * Even if the user do not set a house-keeping CPU, try to + * move rtla to a CPU set different to the one where the user + * set the workload to run. + * + * No need to check results as this is an automatic attempt. + */ + auto_house_keeping(¶ms->monitored_cpus); } return 0; diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c index 8f9ad8f01e0f..3e25f0277fb9 100644 --- a/tools/tracing/rtla/src/utils.c +++ b/tools/tracing/rtla/src/utils.c @@ -709,3 +709,53 @@ int set_comm_cgroup(const char *comm_prefix, const char *cgroup) close(cg_fd); return 0; } + +/** + * auto_house_keeping - Automatically move rtla out of measurement threads + * + * Try to move rtla away from the tracer, if possible. + * + * Returns 1 on success, 0 otherwise. + */ +int auto_house_keeping(cpu_set_t *monitored_cpus) +{ + cpu_set_t rtla_cpus, house_keeping_cpus; + int retval; + + /* first get the CPUs in which rtla can actually run. */ + retval = sched_getaffinity(getpid(), sizeof(rtla_cpus), &rtla_cpus); + if (retval == -1) { + debug_msg("Could not get rtla affinity, rtla might run with the threads!\n"); + return 0; + } + + /* then check if the existing setup is already good. */ + CPU_AND(&house_keeping_cpus, &rtla_cpus, monitored_cpus); + if (!CPU_COUNT(&house_keeping_cpus)) { + debug_msg("rtla and the monitored CPUs do not share CPUs."); + debug_msg("Skipping auto house-keeping\n"); + return 1; + } + + /* remove the intersection */ + CPU_XOR(&house_keeping_cpus, &rtla_cpus, monitored_cpus); + + /* get only those that rtla can run */ + CPU_AND(&house_keeping_cpus, &house_keeping_cpus, &rtla_cpus); + + /* is there any cpu left? */ + if (!CPU_COUNT(&house_keeping_cpus)) { + debug_msg("Could not find any CPU for auto house-keeping\n"); + return 0; + } + + retval = sched_setaffinity(getpid(), sizeof(house_keeping_cpus), &house_keeping_cpus); + if (retval == -1) { + debug_msg("Could not set affinity for auto house-keeping\n"); + return 0; + } + + debug_msg("rtla automatically moved to an auto house-keeping cpu set\n"); + + return 1; +} diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h index 9ab2f0d7bc1c..dec59163cfbc 100644 --- a/tools/tracing/rtla/src/utils.h +++ b/tools/tracing/rtla/src/utils.h @@ -60,6 +60,7 @@ int parse_cpu_set(char *cpu_list, cpu_set_t *set); int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr); int set_comm_cgroup(const char *comm_prefix, const char *cgroup); int set_cpu_dma_latency(int32_t latency); +int auto_house_keeping(cpu_set_t *monitored_cpus); #define ns_to_usf(x) (((double)x/1000)) #define ns_to_per(total, part) ((part * 100) / (double)total) From patchwork Tue Jun 6 16:12:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49312C77B73 for ; Tue, 6 Jun 2023 16:13:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237871AbjFFQNL (ORCPT ); Tue, 6 Jun 2023 12:13:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238073AbjFFQM4 (ORCPT ); Tue, 6 Jun 2023 12:12:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4779170E; Tue, 6 Jun 2023 09:12:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2E08562F50; Tue, 6 Jun 2023 16:12:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92B65C433D2; Tue, 6 Jun 2023 16:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067968; bh=OOdhYxQnbsbsltTtXt6nP+lnhei8a/Ph8syUL4hrzGw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=esj+GlwLXQVj9xr3EOSateM48UDkLiEMU26mxwdfoT2Ts82BAfmdbh6Yg9HnAyyuF Xhbt/Arv7HrPX9hdYCpAWhB3eIhUdxTCHjd0ESdecSGFanrcGgNcmqmBpStlbr2LKJ slXS6axXnHYGtDGGSrPtbype+M/LC64VQQuer+mXp0V91GVQnQb6KI5BFCJkZAg4g5 kqKgbumIxpKTCzmRpO2U4qKJEubKjD+abH+ECV7sDpw/Zn/p7iWeezIsrpkDq6+/AQ Wbu8z7Ymro1skYJdH30jE18y5ReTHG1hHNw6G/LrLeh1bOpH5Jz5LUdPrTj0YkR+Fw uCQHM0lDmITXg== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 05/11] rtla/timerlat: Give timerlat auto analysis its own instance Date: Tue, 6 Jun 2023 18:12:19 +0200 Message-Id: <346b7168c1bae552a415715ec6d23c129a43bdb7.1686066600.git.bristot@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Currently, the auto-analysis is attached to the timerlat top instance. The idea was to avoid creating another instance just for that, so one instance could be reused. The drawback is that, by doing so, the auto-analysis run for the entire session, consuming CPU time. On my 24 box CPUs for timerlat with a 100 us period consumed 50 % with auto analysis, but only 16 % without. By creating an instance for auto-analysis, we can keep the processing stopped until a stop tracing condition is hit. Once it happens, timerlat auto-analysis can use its own trace instance to parse only the end of the trace. By doing so, auto-analysis stop consuming cpu time when it is not needed. If the --aa-only is passed, the timerlat top instance is reused for auto analysis. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/timerlat_aa.c | 35 +++++++++++++++++-- tools/tracing/rtla/src/timerlat_aa.h | 5 +-- tools/tracing/rtla/src/timerlat_top.c | 48 ++++++++++++++++++--------- 3 files changed, 67 insertions(+), 21 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat_aa.c b/tools/tracing/rtla/src/timerlat_aa.c index 1843fff66da5..e0ffe69c271c 100644 --- a/tools/tracing/rtla/src/timerlat_aa.c +++ b/tools/tracing/rtla/src/timerlat_aa.c @@ -8,6 +8,7 @@ #include "utils.h" #include "osnoise.h" #include "timerlat.h" +#include enum timelat_state { TIMERLAT_INIT = 0, @@ -233,7 +234,7 @@ static int timerlat_aa_thread_latency(struct timerlat_aa_data *taa_data, * * Returns 0 on success, -1 otherwise. */ -int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record, +static int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record, struct tep_event *event, void *context) { struct timerlat_aa_context *taa_ctx = timerlat_aa_get_ctx(); @@ -665,6 +666,25 @@ static void timerlat_thread_analysis(struct timerlat_aa_data *taa_data, int cpu, ns_to_usf(total)); } +static int timerlat_auto_analysis_collect_trace(struct timerlat_aa_context *taa_ctx) +{ + struct trace_instance *trace = &taa_ctx->tool->trace; + int retval; + + retval = tracefs_iterate_raw_events(trace->tep, + trace->inst, + NULL, + 0, + collect_registered_events, + trace); + if (retval < 0) { + err_msg("Error iterating on events\n"); + return 0; + } + + return 1; +} + /** * timerlat_auto_analysis - Analyze the collected data */ @@ -677,6 +697,8 @@ void timerlat_auto_analysis(int irq_thresh, int thread_thresh) struct tep_handle *tep; int cpu; + timerlat_auto_analysis_collect_trace(taa_ctx); + /* bring stop tracing to the ns scale */ irq_thresh = irq_thresh * 1000; thread_thresh = thread_thresh * 1000; @@ -838,6 +860,10 @@ static int timerlat_aa_init_seqs(struct timerlat_aa_context *taa_ctx) */ static void timerlat_aa_unregister_events(struct osnoise_tool *tool, int dump_tasks) { + + tep_unregister_event_handler(tool->trace.tep, -1, "ftrace", "timerlat", + timerlat_aa_handler, tool); + tracefs_event_disable(tool->trace.inst, "osnoise", NULL); tep_unregister_event_handler(tool->trace.tep, -1, "osnoise", "nmi_noise", @@ -875,6 +901,10 @@ static int timerlat_aa_register_events(struct osnoise_tool *tool, int dump_tasks { int retval; + tep_register_event_handler(tool->trace.tep, -1, "ftrace", "timerlat", + timerlat_aa_handler, tool); + + /* * register auto-analysis handlers. */ @@ -955,8 +985,9 @@ void timerlat_aa_destroy(void) * * Returns 0 on success, -1 otherwise. */ -int timerlat_aa_init(struct osnoise_tool *tool, int nr_cpus, int dump_tasks) +int timerlat_aa_init(struct osnoise_tool *tool, int dump_tasks) { + int nr_cpus = sysconf(_SC_NPROCESSORS_CONF); struct timerlat_aa_context *taa_ctx; int retval; diff --git a/tools/tracing/rtla/src/timerlat_aa.h b/tools/tracing/rtla/src/timerlat_aa.h index d4f6ca7e342a..cea4bb1531a8 100644 --- a/tools/tracing/rtla/src/timerlat_aa.h +++ b/tools/tracing/rtla/src/timerlat_aa.h @@ -3,10 +3,7 @@ * Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira */ -int timerlat_aa_init(struct osnoise_tool *tool, int nr_cpus, int dump_task); +int timerlat_aa_init(struct osnoise_tool *tool, int dump_task); void timerlat_aa_destroy(void); -int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record, - struct tep_event *event, void *context); - void timerlat_auto_analysis(int irq_thresh, int thread_thresh); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index f0c6d9735e2a..d6b5a382569e 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -156,9 +156,6 @@ timerlat_top_handler(struct trace_seq *s, struct tep_record *record, timerlat_top_update(top, cpu, thread, latency); } - if (!params->no_aa) - timerlat_aa_handler(s, record, event, context); - return 0; } @@ -644,7 +641,6 @@ static struct osnoise_tool { struct osnoise_tool *top; int nr_cpus; - int retval; nr_cpus = sysconf(_SC_NPROCESSORS_CONF); @@ -661,16 +657,6 @@ static struct osnoise_tool tep_register_event_handler(top->trace.tep, -1, "ftrace", "timerlat", timerlat_top_handler, top); - /* - * If no auto analysis, we are ready. - */ - if (params->no_aa) - return top; - - retval = timerlat_aa_init(top, nr_cpus, params->dump_tasks); - if (retval) - goto out_err; - return top; out_err: @@ -702,6 +688,7 @@ int timerlat_top_main(int argc, char *argv[]) struct timerlat_top_params *params; struct osnoise_tool *record = NULL; struct osnoise_tool *top = NULL; + struct osnoise_tool *aa = NULL; struct trace_instance *trace; int dma_latency_fd = -1; int return_value = 1; @@ -774,6 +761,35 @@ int timerlat_top_main(int argc, char *argv[]) trace_instance_start(&record->trace); } + if (!params->no_aa) { + if (params->aa_only) { + /* as top is not used for display, use it for aa */ + aa = top; + } else { + /* otherwise, a new instance is needed */ + aa = osnoise_init_tool("timerlat_aa"); + if (!aa) + goto out_top; + } + + retval = timerlat_aa_init(aa, params->dump_tasks); + if (retval) { + err_msg("Failed to enable the auto analysis instance\n"); + goto out_top; + } + + /* if it is re-using the main instance, there is no need to start it */ + if (aa != top) { + retval = enable_timerlat(&aa->trace); + if (retval) { + err_msg("Failed to enable timerlat tracer\n"); + goto out_top; + } + + trace_instance_start(&aa->trace); + } + } + top->start_time = time(NULL); timerlat_top_set_signals(params); @@ -829,13 +845,15 @@ int timerlat_top_main(int argc, char *argv[]) } out_top: + timerlat_aa_destroy(); if (dma_latency_fd >= 0) close(dma_latency_fd); trace_events_destroy(&record->trace, params->events); params->events = NULL; out_free: timerlat_free_top(top->data); - timerlat_aa_destroy(); + if (aa && aa != top) + osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(top); free(params); From patchwork Tue Jun 6 16:12:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E5D6C7EE24 for ; Tue, 6 Jun 2023 16:13:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238455AbjFFQNQ (ORCPT ); Tue, 6 Jun 2023 12:13:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238317AbjFFQNI (ORCPT ); Tue, 6 Jun 2023 12:13:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A63010D5; Tue, 6 Jun 2023 09:12:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9E38A62D99; Tue, 6 Jun 2023 16:12:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0FC68C433A1; Tue, 6 Jun 2023 16:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067971; bh=wrC0UjEKUIFXbnICT6UkqA3YCTd+/hzNT3Qa2DVLVho=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pqUxlUAAfHFBC3Son5rC5+xAa6jjgpsHSGxsg4M/WiYTt9pHprI4XaCAHrOG8o+/D 1XBX1YT//ifYVdvxF/OlNMImZ2737HUhKEfUOORPihYxc3ilmdXEbq9v3iwRRUcp8Z b30hYpzlxpAY25Mwa7/XA3kvPNOk6ly9Clt+zchhn0GHS5Jy3HZgaTMbfrTRpcjuuM 4r8eKpUrGtpebtoqpQFZSg3s9oNCvCtDM0vZu3YF5dgJmP/3e+5QKGjMeHGBpa+tRr 6ph3L08A5MUVe1na7AOi5dY677SKqIWruDdLp7NSUpplYqUlSVHxE/kThTW6buIQWv CejDQDWAFaGeg== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 06/11] rtla/timerlat_hist: Add auto-analysis support Date: Tue, 6 Jun 2023 18:12:20 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Add auto-analysis to timerlat hist, including the --no-aa option to reduce overhead and --dump-task. --aa-only was not added as it is already on timerlat top. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- .../tools/rtla/common_timerlat_aa.rst | 7 --- .../tools/rtla/rtla-timerlat-hist.rst | 7 ++- .../tools/rtla/rtla-timerlat-top.rst | 7 +++ tools/tracing/rtla/src/timerlat_hist.c | 51 ++++++++++++++++++- 4 files changed, 61 insertions(+), 11 deletions(-) diff --git a/Documentation/tools/rtla/common_timerlat_aa.rst b/Documentation/tools/rtla/common_timerlat_aa.rst index 795b9fbcbc6d..077029e6b289 100644 --- a/Documentation/tools/rtla/common_timerlat_aa.rst +++ b/Documentation/tools/rtla/common_timerlat_aa.rst @@ -5,10 +5,3 @@ **--no-aa** disable auto-analysis, reducing rtla timerlat cpu usage - -**--aa-only** *us* - - Set stop tracing conditions and run without collecting and displaying statistics. - Print the auto-analysis if the system hits the stop tracing condition. This option - is useful to reduce rtla timerlat CPU, enabling the debug without the overhead of - collecting the statistics. diff --git a/Documentation/tools/rtla/rtla-timerlat-hist.rst b/Documentation/tools/rtla/rtla-timerlat-hist.rst index 6bf7f0ca4556..057db78d4095 100644 --- a/Documentation/tools/rtla/rtla-timerlat-hist.rst +++ b/Documentation/tools/rtla/rtla-timerlat-hist.rst @@ -29,15 +29,18 @@ OPTIONS .. include:: common_options.rst +.. include:: common_timerlat_aa.rst + EXAMPLE ======= In the example below, **rtla timerlat hist** is set to run for *10* minutes, in the cpus *0-4*, *skipping zero* only lines. Moreover, **rtla timerlat hist** will change the priority of the *timerlat* threads to run under *SCHED_DEADLINE* priority, with a *10us* runtime every *1ms* period. The -*1ms* period is also passed to the *timerlat* tracer:: +*1ms* period is also passed to the *timerlat* tracer. Auto-analysis is disabled +to reduce overhead :: - [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms + [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms --no-aa # RTLA timerlat histogram # Time unit is microseconds (us) # Duration: 0 00:10:00 diff --git a/Documentation/tools/rtla/rtla-timerlat-top.rst b/Documentation/tools/rtla/rtla-timerlat-top.rst index 73799c1150ad..1b7cf4e3eafe 100644 --- a/Documentation/tools/rtla/rtla-timerlat-top.rst +++ b/Documentation/tools/rtla/rtla-timerlat-top.rst @@ -32,6 +32,13 @@ OPTIONS .. include:: common_timerlat_aa.rst +**--aa-only** *us* + + Set stop tracing conditions and run without collecting and displaying statistics. + Print the auto-analysis if the system hits the stop tracing condition. This option + is useful to reduce rtla timerlat CPU, enabling the debug without the overhead of + collecting the statistics. + EXAMPLE ======= diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index 1675d54cae81..e720c8908906 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -16,6 +16,7 @@ #include "utils.h" #include "osnoise.h" #include "timerlat.h" +#include "timerlat_aa.h" struct timerlat_hist_params { char *cpus; @@ -34,6 +35,8 @@ struct timerlat_hist_params { int dma_latency; int cgroup; int hk_cpus; + int no_aa; + int dump_tasks; cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; @@ -438,7 +441,7 @@ static void timerlat_hist_usage(char *usage) " usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\", " [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] [-H cpu-list]\\", " [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\", - " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]", + " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit", @@ -450,12 +453,14 @@ static void timerlat_hist_usage(char *usage) " -H/--house-keeping cpus: run rtla control threads only on the given cpus", " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited", " -d/--duration time[m|h|d]: duration of the session in seconds", + " --dump-tasks: prints the task running on all CPUs if stop conditions are met (depends on !--no-aa)", " -D/--debug: print debug info", " -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]", " -e/--event : enable the in the trace instance, multiple -e are allowed", " --filter : enable a trace event filter to the previous -e event", " --trigger : enable a trace event trigger to the previous -e event", " -n/--nano: display data in nanoseconds", + " --no-aa: disable auto-analysis, reducing rtla timerlat cpu usage", " -b/--bucket-size N: set the histogram bucket size (default 1)", " -E/--entries N: set the number of entries of the histogram (default 256)", " --no-irq: ignore IRQ latencies", @@ -537,13 +542,15 @@ static struct timerlat_hist_params {"trigger", required_argument, 0, '6'}, {"filter", required_argument, 0, '7'}, {"dma-latency", required_argument, 0, '8'}, + {"no-aa", no_argument, 0, '9'}, + {"dump-task", no_argument, 0, '\1'}, {0, 0, 0, 0} }; /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:9\1", long_options, &option_index); /* detect the end of the options. */ @@ -556,6 +563,7 @@ static struct timerlat_hist_params /* set thread stop to auto_thresh */ params->stop_total_us = auto_thresh; + params->stop_us = auto_thresh; /* get stack trace */ params->print_stack = auto_thresh; @@ -699,6 +707,12 @@ static struct timerlat_hist_params exit(EXIT_FAILURE); } break; + case '9': + params->no_aa = 1; + break; + case '\1': + params->dump_tasks = 1; + break; default: timerlat_hist_usage("Invalid option"); } @@ -715,6 +729,12 @@ static struct timerlat_hist_params if (params->no_index && !params->with_zeros) timerlat_hist_usage("no-index set with with-zeros is not set - it does not make sense"); + /* + * Auto analysis only happens if stop tracing, thus: + */ + if (!params->stop_us && !params->stop_total_us) + params->no_aa = 1; + return params; } @@ -848,6 +868,7 @@ int timerlat_hist_main(int argc, char *argv[]) struct timerlat_hist_params *params; struct osnoise_tool *record = NULL; struct osnoise_tool *tool = NULL; + struct osnoise_tool *aa = NULL; struct trace_instance *trace; int dma_latency_fd = -1; int return_value = 1; @@ -919,6 +940,26 @@ int timerlat_hist_main(int argc, char *argv[]) trace_instance_start(&record->trace); } + if (!params->no_aa) { + aa = osnoise_init_tool("timerlat_aa"); + if (!aa) + goto out_hist; + + retval = timerlat_aa_init(aa, params->dump_tasks); + if (retval) { + err_msg("Failed to enable the auto analysis instance\n"); + goto out_hist; + } + + retval = enable_timerlat(&aa->trace); + if (retval) { + err_msg("Failed to enable timerlat tracer\n"); + goto out_hist; + } + + trace_instance_start(&aa->trace); + } + tool->start_time = time(NULL); timerlat_hist_set_signals(params); @@ -946,6 +987,10 @@ int timerlat_hist_main(int argc, char *argv[]) if (trace_is_off(&tool->trace, &record->trace)) { printf("rtla timerlat hit stop tracing\n"); + + if (!params->no_aa) + timerlat_auto_analysis(params->stop_us, params->stop_total_us); + if (params->trace_output) { printf(" Saving trace to %s\n", params->trace_output); save_trace_to_file(record->trace.inst, params->trace_output); @@ -953,12 +998,14 @@ int timerlat_hist_main(int argc, char *argv[]) } out_hist: + timerlat_aa_destroy(); if (dma_latency_fd >= 0) close(dma_latency_fd); trace_events_destroy(&record->trace, params->events); params->events = NULL; out_free: timerlat_free_histogram(tool->data); + osnoise_destroy_tool(aa); osnoise_destroy_tool(record); osnoise_destroy_tool(tool); free(params); From patchwork Tue Jun 6 16:12:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 141B5C7EE29 for ; Tue, 6 Jun 2023 16:13:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238008AbjFFQNR (ORCPT ); Tue, 6 Jun 2023 12:13:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238005AbjFFQNJ (ORCPT ); Tue, 6 Jun 2023 12:13:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87E7610F7; Tue, 6 Jun 2023 09:12:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1DD4562F50; Tue, 6 Jun 2023 16:12:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 809E2C4339C; Tue, 6 Jun 2023 16:12:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067973; bh=Lo8FZAWJl35tTJ74PxG2G+/6DGoIbw32r9OQUCYExV4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WUTmYS4IjWAARaoZj47S3OMBdIkMIy4AFeXoGo1OQwm4USgCVNpDruk3/KcafRB5V F1r2C+m8R3jU3/tlhsD25e0kR5/p9BBls01cOqtCYm79QXkFaLKT5F2dPpffz28976 JXWa0v0WvyEPN/NrrpRPvvvB1pNgCIcUj9kxM9713cYw+Fc8LNGma1ppAhyPoCjy7A 0NRkeaQVhMc8Ytj3uoytQyIszLTLWyogentxYPJdY0HDvQ+vm5ZAJZvdGaa6Vrp8Uu tDeaPl8WlYy2E0L9apxGagMEvQaFPGza1Kw1WWh8M1wpQdCPIEwpCMW7aTUUVFdL98 s98jaj9lYILBA== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 07/11] rtla: Start the tracers after creating all instances Date: Tue, 6 Jun 2023 18:12:21 +0200 Message-Id: <67da7a703a56f75d7cd46568525145a65501a7e8.1686066600.git.bristot@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Group all start tracing after finishing creating all instances. The tracing instance starts first for the case of hitting a stop tracing while enabling other instances. The trace instance is the one with most valuable information. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise_hist.c | 14 +++++++++++--- tools/tracing/rtla/src/osnoise_top.c | 13 ++++++++++--- tools/tracing/rtla/src/timerlat_hist.c | 17 ++++++++++++----- tools/tracing/rtla/src/timerlat_top.c | 19 +++++++++++++------ 4 files changed, 46 insertions(+), 17 deletions(-) diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c index dfbcb5ca7ecb..8f81fa007364 100644 --- a/tools/tracing/rtla/src/osnoise_hist.c +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -870,8 +870,6 @@ int osnoise_hist_main(int argc, char *argv[]) } } - trace_instance_start(trace); - if (params->trace_output) { record = osnoise_init_trace_tool("osnoise"); if (!record) { @@ -885,9 +883,19 @@ int osnoise_hist_main(int argc, char *argv[]) goto out_hist; } - trace_instance_start(&record->trace); } + /* + * Start the tracer here, after having set all instances. + * + * Let the trace instance start first for the case of hitting a stop + * tracing while enabling other instances. The trace instance is the + * one with most valuable information. + */ + if (params->trace_output) + trace_instance_start(&record->trace); + trace_instance_start(trace); + tool->start_time = time(NULL); osnoise_hist_set_signals(params); diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index 0833537bb2eb..85abba568faf 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -697,8 +697,6 @@ int osnoise_top_main(int argc, char **argv) } } - trace_instance_start(trace); - if (params->trace_output) { record = osnoise_init_trace_tool("osnoise"); if (!record) { @@ -711,9 +709,18 @@ int osnoise_top_main(int argc, char **argv) if (retval) goto out_top; } + } + /* + * Start the tracer here, after having set all instances. + * + * Let the trace instance start first for the case of hitting a stop + * tracing while enabling other instances. The trace instance is the + * one with most valuable information. + */ + if (params->trace_output) trace_instance_start(&record->trace); - } + trace_instance_start(trace); tool->start_time = time(NULL); osnoise_top_set_signals(params); diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index e720c8908906..f431cf43246c 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -922,8 +922,6 @@ int timerlat_hist_main(int argc, char *argv[]) } } - trace_instance_start(trace); - if (params->trace_output) { record = osnoise_init_trace_tool("timerlat"); if (!record) { @@ -936,8 +934,6 @@ int timerlat_hist_main(int argc, char *argv[]) if (retval) goto out_hist; } - - trace_instance_start(&record->trace); } if (!params->no_aa) { @@ -956,9 +952,20 @@ int timerlat_hist_main(int argc, char *argv[]) err_msg("Failed to enable timerlat tracer\n"); goto out_hist; } + } + /* + * Start the tracers here, after having set all instances. + * + * Let the trace instance start first for the case of hitting a stop + * tracing while enabling other instances. The trace instance is the + * one with most valuable information. + */ + if (params->trace_output) + trace_instance_start(&record->trace); + if (!params->no_aa) trace_instance_start(&aa->trace); - } + trace_instance_start(trace); tool->start_time = time(NULL); timerlat_hist_set_signals(params); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index d6b5a382569e..02cff4948981 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -743,8 +743,6 @@ int timerlat_top_main(int argc, char *argv[]) } } - trace_instance_start(trace); - if (params->trace_output) { record = osnoise_init_trace_tool("timerlat"); if (!record) { @@ -757,8 +755,6 @@ int timerlat_top_main(int argc, char *argv[]) if (retval) goto out_top; } - - trace_instance_start(&record->trace); } if (!params->no_aa) { @@ -785,11 +781,22 @@ int timerlat_top_main(int argc, char *argv[]) err_msg("Failed to enable timerlat tracer\n"); goto out_top; } - - trace_instance_start(&aa->trace); } } + /* + * Start the tracers here, after having set all instances. + * + * Let the trace instance start first for the case of hitting a stop + * tracing while enabling other instances. The trace instance is the + * one with most valuable information. + */ + if (params->trace_output) + trace_instance_start(&record->trace); + if (!params->no_aa && aa != top) + trace_instance_start(&aa->trace); + trace_instance_start(trace); + top->start_time = time(NULL); timerlat_top_set_signals(params); From patchwork Tue Jun 6 16:12:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31E37C77B73 for ; Tue, 6 Jun 2023 16:13:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238363AbjFFQN2 (ORCPT ); Tue, 6 Jun 2023 12:13:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238358AbjFFQNK (ORCPT ); Tue, 6 Jun 2023 12:13:10 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CAE710F4; Tue, 6 Jun 2023 09:12:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 93CC662DE6; Tue, 6 Jun 2023 16:12:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00082C4339E; Tue, 6 Jun 2023 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067976; bh=NspQ5eSfzkW/s4ZKEIvi8KiViH9J4SR7mF8q0fY8gJ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SUtXBjQ4gi424FRAJAf2Lbg14t8Udt17Q1lNOb275daD7+epD2u4HVXHQakeqfdTx BmLXKKYW5L5TbtrrdFCyA/b5d6JOy4pd01YgfKbzz1yNgkTOcIrLvbMuPalTxWpbZ/ LRTytWxeT+b0Izt90iiS98ZXRun49fBSFhZNIkIclFuaDeQPQOWybL30eqzWhVGvWK Bn7IAnGAqSGUt3XuzeYcoGTFXQQrhim/8EyYQmPwJgu2DO880EQ/qI67GZKSi/wILd tF8/fh4Z0f32tydaH1C8CnEDAiZYnGjXYtTOSkI5da1PYbVwtHTJcoLAstoEzN94AH nF7jrOJ2hVnUA== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 08/11] rtla/hwnoise: Reduce runtime to 75% Date: Tue, 6 Jun 2023 18:12:22 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org osnoise runs 100% of time by default. It makes sense because osnoise is preemptive. hwnoise checks preemption once a second, so it reduces system progress. Reduce runtime to 75% to avoid problems by default. I added a Fixes as it might avoid problems for first time users as it lands on distros. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Fixes: 1f428356c38d ("rtla: Add hwnoise tool") Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise_top.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c index 85abba568faf..f7c959be8677 100644 --- a/tools/tracing/rtla/src/osnoise_top.c +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -348,8 +348,14 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) if (!params) exit(1); - if (strcmp(argv[0], "hwnoise") == 0) + if (strcmp(argv[0], "hwnoise") == 0) { params->mode = MODE_HWNOISE; + /* + * Reduce CPU usage for 75% to avoid killing the system. + */ + params->runtime = 750000; + params->period = 1000000; + } while (1) { static struct option long_options[] = { From patchwork Tue Jun 6 16:12:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5565C7EE24 for ; Tue, 6 Jun 2023 16:13:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238402AbjFFQNi (ORCPT ); Tue, 6 Jun 2023 12:13:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238410AbjFFQNM (ORCPT ); Tue, 6 Jun 2023 12:13:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E706E1982; Tue, 6 Jun 2023 09:12:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3D5D7634D8; Tue, 6 Jun 2023 16:12:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71C00C433D2; Tue, 6 Jun 2023 16:12:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067978; bh=FJeLVCepRy4b0eA6jvFIuUd6P33HsY/Ifs6R9TD+e3I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G83yNUXShpmOSaVme1AS4NIo6FG0ZPKggtUhHeFT+mYHgzrIsiSZpWMBVJDYCQMcZ 3KTg5cN9045RMpmXXxbgGYGMLl7rcp0FuPtmZRcgXPOgDAuTXZ00jhDXOqA7pH2KBT lw1g6EqOo6lZwD1sNvnEw0uoMvXpK8KGe7SE4dn1SVsXWsvRr/+/AKNd2BNAwBKCY5 hpJRl+vg4agVOnMDCP53iUf63uYuE+/C0fBEPhcRsLHKy/4h3AhbH+/ThyDrfNS+T8 /GluA0t9eVQOQSHVC/yxAt7EzvWkloKfKWo5sVn9yDG9V39VS8NtbH9MpR0/beXSmf jcglWCeivyZiQ== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 09/11] rtla/timerlat_top: Add timerlat user-space support Date: Tue, 6 Jun 2023 18:12:23 +0200 Message-Id: <758ad2292a0a1d884138d08219e1a0f572d257a2.1686066600.git.bristot@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Add the support for running timerlat threads in user-space. In this mode, enabled with -u/--user-threads, timerlat dispatches user-space processes that will loop in the timerlat_fd, measuring the overhead for going to user-space and then returning to the kernel - in addition to the existing measurements. Here is one example of the tool's output with -u enabled: $ sudo timerlat top -u -d 600 -q Timer Latency 0 00:10:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) | Ret user Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max | cur min avg max 0 #600001 | 0 0 0 3 | 2 1 2 9 | 3 2 3 15 1 #600001 | 0 0 0 2 | 2 1 2 13 | 2 2 3 18 2 #600001 | 0 0 0 10 | 2 1 2 16 | 3 2 3 20 3 #600001 | 0 0 0 7 | 2 1 2 10 | 3 2 3 11 4 #600000 | 0 0 0 16 | 2 1 2 41 | 3 2 3 58 5 #600000 | 0 0 0 3 | 2 1 2 10 | 3 2 3 13 6 #600000 | 0 0 0 5 | 2 1 2 7 | 3 2 3 10 7 #600000 | 0 0 0 1 | 2 1 2 7 | 3 2 3 10 The tuning setup like -p or -C work for the user-space threads as well. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise.c | 65 ++++++++ tools/tracing/rtla/src/osnoise.h | 5 + tools/tracing/rtla/src/timerlat_top.c | 108 ++++++++++++- tools/tracing/rtla/src/timerlat_u.c | 224 ++++++++++++++++++++++++++ tools/tracing/rtla/src/timerlat_u.h | 18 +++ tools/tracing/rtla/src/utils.c | 58 +++++++ tools/tracing/rtla/src/utils.h | 2 + 7 files changed, 474 insertions(+), 6 deletions(-) create mode 100644 tools/tracing/rtla/src/timerlat_u.c create mode 100644 tools/tracing/rtla/src/timerlat_u.h diff --git a/tools/tracing/rtla/src/osnoise.c b/tools/tracing/rtla/src/osnoise.c index 3ca7a3853943..245e9344932b 100644 --- a/tools/tracing/rtla/src/osnoise.c +++ b/tools/tracing/rtla/src/osnoise.c @@ -841,6 +841,67 @@ static void osnoise_put_irq_disable(struct osnoise_context *context) context->orig_opt_irq_disable = OSNOISE_OPTION_INIT_VAL; } +static int osnoise_get_workload(struct osnoise_context *context) +{ + if (context->opt_workload != OSNOISE_OPTION_INIT_VAL) + return context->opt_workload; + + if (context->orig_opt_workload != OSNOISE_OPTION_INIT_VAL) + return context->orig_opt_workload; + + context->orig_opt_workload = osnoise_options_get_option("OSNOISE_WORKLOAD"); + + return context->orig_opt_workload; +} + +int osnoise_set_workload(struct osnoise_context *context, bool onoff) +{ + int opt_workload = osnoise_get_workload(context); + int retval; + + if (opt_workload == OSNOISE_OPTION_INIT_VAL) + return -1; + + if (opt_workload == onoff) + return 0; + + retval = osnoise_options_set_option("OSNOISE_WORKLOAD", onoff); + if (retval < 0) + return -1; + + context->opt_workload = onoff; + + return 0; +} + +static void osnoise_restore_workload(struct osnoise_context *context) +{ + int retval; + + if (context->orig_opt_workload == OSNOISE_OPTION_INIT_VAL) + return; + + if (context->orig_opt_workload == context->opt_workload) + goto out_done; + + retval = osnoise_options_set_option("OSNOISE_WORKLOAD", context->orig_opt_workload); + if (retval < 0) + err_msg("Could not restore original OSNOISE_WORKLOAD option\n"); + +out_done: + context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL; +} + +static void osnoise_put_workload(struct osnoise_context *context) +{ + osnoise_restore_workload(context); + + if (context->orig_opt_workload == OSNOISE_OPTION_INIT_VAL) + return; + + context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL; +} + /* * enable_osnoise - enable osnoise tracer in the trace_instance */ @@ -908,6 +969,9 @@ struct osnoise_context *osnoise_context_alloc(void) context->orig_opt_irq_disable = OSNOISE_OPTION_INIT_VAL; context->opt_irq_disable = OSNOISE_OPTION_INIT_VAL; + context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL; + context->opt_workload = OSNOISE_OPTION_INIT_VAL; + osnoise_get_context(context); return context; @@ -935,6 +999,7 @@ void osnoise_put_context(struct osnoise_context *context) osnoise_put_print_stack(context); osnoise_put_tracing_thresh(context); osnoise_put_irq_disable(context); + osnoise_put_workload(context); free(context); } diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h index 4dcf22ccd704..555f4f4903cc 100644 --- a/tools/tracing/rtla/src/osnoise.h +++ b/tools/tracing/rtla/src/osnoise.h @@ -42,6 +42,10 @@ struct osnoise_context { /* -1 as init value because 0 is off */ int orig_opt_irq_disable; int opt_irq_disable; + + /* -1 as init value because 0 is off */ + int orig_opt_workload; + int opt_workload; }; /* @@ -84,6 +88,7 @@ int osnoise_set_print_stack(struct osnoise_context *context, long long print_stack); int osnoise_set_irq_disable(struct osnoise_context *context, bool onoff); +int osnoise_set_workload(struct osnoise_context *context, bool onoff); /* * osnoise_tool - osnoise based tool definition. diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c index 02cff4948981..1640f121baca 100644 --- a/tools/tracing/rtla/src/timerlat_top.c +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -13,11 +13,13 @@ #include #include #include +#include #include "utils.h" #include "osnoise.h" #include "timerlat.h" #include "timerlat_aa.h" +#include "timerlat_u.h" struct timerlat_top_params { char *cpus; @@ -40,6 +42,7 @@ struct timerlat_top_params { int dump_tasks; int cgroup; int hk_cpus; + int user_top; cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; @@ -48,6 +51,7 @@ struct timerlat_top_params { struct timerlat_top_cpu { int irq_count; int thread_count; + int user_count; unsigned long long cur_irq; unsigned long long min_irq; @@ -58,6 +62,11 @@ struct timerlat_top_cpu { unsigned long long min_thread; unsigned long long sum_thread; unsigned long long max_thread; + + unsigned long long cur_user; + unsigned long long min_user; + unsigned long long sum_user; + unsigned long long max_user; }; struct timerlat_top_data { @@ -98,6 +107,7 @@ static struct timerlat_top_data *timerlat_alloc_top(int nr_cpus) for (cpu = 0; cpu < nr_cpus; cpu++) { data->cpu_data[cpu].min_irq = ~0; data->cpu_data[cpu].min_thread = ~0; + data->cpu_data[cpu].min_user = ~0; } return data; @@ -124,12 +134,18 @@ timerlat_top_update(struct osnoise_tool *tool, int cpu, update_min(&cpu_data->min_irq, &latency); update_sum(&cpu_data->sum_irq, &latency); update_max(&cpu_data->max_irq, &latency); - } else { + } else if (thread == 1) { cpu_data->thread_count++; cpu_data->cur_thread = latency; update_min(&cpu_data->min_thread, &latency); update_sum(&cpu_data->sum_thread, &latency); update_max(&cpu_data->max_thread, &latency); + } else { + cpu_data->user_count++; + cpu_data->cur_user = latency; + update_min(&cpu_data->min_user, &latency); + update_sum(&cpu_data->sum_user, &latency); + update_max(&cpu_data->max_user, &latency); } } @@ -172,15 +188,25 @@ static void timerlat_top_header(struct osnoise_tool *top) trace_seq_printf(s, "\033[2;37;40m"); trace_seq_printf(s, " Timer Latency "); + if (params->user_top) + trace_seq_printf(s, " "); trace_seq_printf(s, "\033[0;0;0m"); trace_seq_printf(s, "\n"); - trace_seq_printf(s, "%-6s | IRQ Timer Latency (%s) | Thread Timer Latency (%s)\n", duration, + trace_seq_printf(s, "%-6s | IRQ Timer Latency (%s) | Thread Timer Latency (%s)", duration, params->output_divisor == 1 ? "ns" : "us", params->output_divisor == 1 ? "ns" : "us"); + if (params->user_top) { + trace_seq_printf(s, " | Ret user Timer Latency (%s)", + params->output_divisor == 1 ? "ns" : "us"); + } + + trace_seq_printf(s, "\n"); trace_seq_printf(s, "\033[2;30;47m"); trace_seq_printf(s, "CPU COUNT | cur min avg max | cur min avg max"); + if (params->user_top) + trace_seq_printf(s, " | cur min avg max"); trace_seq_printf(s, "\033[0;0;0m"); trace_seq_printf(s, "\n"); } @@ -233,7 +259,27 @@ static void timerlat_top_print(struct osnoise_tool *top, int cpu) trace_seq_printf(s, "%9llu ", cpu_data->min_thread / divisor); trace_seq_printf(s, "%9llu ", (cpu_data->sum_thread / cpu_data->thread_count) / divisor); - trace_seq_printf(s, "%9llu\n", cpu_data->max_thread / divisor); + trace_seq_printf(s, "%9llu", cpu_data->max_thread / divisor); + } + + if (!params->user_top) { + trace_seq_printf(s, "\n"); + return; + } + + trace_seq_printf(s, " |"); + + if (!cpu_data->user_count) { + trace_seq_printf(s, " - "); + trace_seq_printf(s, " - "); + trace_seq_printf(s, " - "); + trace_seq_printf(s, " -\n"); + } else { + trace_seq_printf(s, "%9llu ", cpu_data->cur_user / divisor); + trace_seq_printf(s, "%9llu ", cpu_data->min_user / divisor); + trace_seq_printf(s, "%9llu ", + (cpu_data->sum_user / cpu_data->user_count) / divisor); + trace_seq_printf(s, "%9llu\n", cpu_data->max_user / divisor); } } @@ -288,7 +334,7 @@ static void timerlat_top_usage(char *usage) "", " usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\", " [[-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] [-H cpu-list]\\", - " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]", + " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]] [-u]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit", @@ -317,6 +363,7 @@ static void timerlat_top_usage(char *usage) " f:prio - use SCHED_FIFO with prio", " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", " in nanoseconds", + " -u/--user-threads: use rtla user-space threads instead of in-kernel timerlat threads", NULL, }; @@ -371,6 +418,7 @@ static struct timerlat_top_params {"stack", required_argument, 0, 's'}, {"thread", required_argument, 0, 'T'}, {"trace", optional_argument, 0, 't'}, + {"user-threads", no_argument, 0, 'u'}, {"trigger", required_argument, 0, '0'}, {"filter", required_argument, 0, '1'}, {"dma-latency", required_argument, 0, '2'}, @@ -383,7 +431,7 @@ static struct timerlat_top_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:0:1:2:345:", + c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:u0:1:2:345:", long_options, &option_index); /* detect the end of the options. */ @@ -498,6 +546,9 @@ static struct timerlat_top_params else params->trace_output = "timerlat_trace.txt"; + break; + case 'u': + params->user_top = true; break; case '0': /* trigger */ if (params->events) { @@ -563,6 +614,7 @@ static int timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *params) { int retval; + int i; if (!params->sleep_time) params->sleep_time = 1; @@ -573,6 +625,9 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params * err_msg("Failed to apply CPUs config\n"); goto out_err; } + } else { + for (i = 0; i < sysconf(_SC_NPROCESSORS_CONF); i++) + CPU_SET(i, ¶ms->monitored_cpus); } if (params->stop_us) { @@ -627,6 +682,14 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params * auto_house_keeping(¶ms->monitored_cpus); } + if (params->user_top) { + retval = osnoise_set_workload(top->context, 0); + if (retval) { + err_msg("Failed to set OSNOISE_WORKLOAD option\n"); + goto out_err; + } + } + return 0; out_err: @@ -687,10 +750,12 @@ int timerlat_top_main(int argc, char *argv[]) { struct timerlat_top_params *params; struct osnoise_tool *record = NULL; + struct timerlat_u_params params_u; struct osnoise_tool *top = NULL; struct osnoise_tool *aa = NULL; struct trace_instance *trace; int dma_latency_fd = -1; + pthread_t timerlat_u; int return_value = 1; char *max_lat; int retval; @@ -727,7 +792,7 @@ int timerlat_top_main(int argc, char *argv[]) } } - if (params->cgroup) { + if (params->cgroup && !params->user_top) { retval = set_comm_cgroup("timerlat/", params->cgroup_name); if (!retval) { err_msg("Failed to move threads to cgroup\n"); @@ -800,6 +865,25 @@ int timerlat_top_main(int argc, char *argv[]) top->start_time = time(NULL); timerlat_top_set_signals(params); + if (params->user_top) { + /* rtla asked to stop */ + params_u.should_run = 1; + /* all threads left */ + params_u.stopped_running = 0; + + params_u.set = ¶ms->monitored_cpus; + if (params->set_sched) + params_u.sched_param = ¶ms->sched_param; + else + params_u.sched_param = NULL; + + params_u.cgroup_name = params->cgroup_name; + + retval = pthread_create(&timerlat_u, NULL, timerlat_u_dispatcher, ¶ms_u); + if (retval) + err_msg("Error creating timerlat user-space threads\n"); + } + while (!stop_tracing) { sleep(params->sleep_time); @@ -823,6 +907,18 @@ int timerlat_top_main(int argc, char *argv[]) if (trace_is_off(&top->trace, &record->trace)) break; + /* is there still any user-threads ? */ + if (params->user_top) { + if (params_u.stopped_running) { + debug_msg("timerlat user space threads stopped!\n"); + break; + } + } + } + + if (params->user_top && !params_u.stopped_running) { + params_u.should_run = 0; + sleep(1); } timerlat_print_stats(params, top); diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c new file mode 100644 index 000000000000..05e310696dd5 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat_u.c @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "utils.h" +#include "timerlat_u.h" + +/* + * This is the user-space main for the tool timerlatu/ threads. + * + * It is as simple as this: + * - set affinity + * - set priority + * - open tracer fd + * - spin + * - close + */ +static int timerlat_u_main(int cpu, struct timerlat_u_params *params) +{ + struct sched_param sp = { .sched_priority = 95 }; + char buffer[1024]; + int timerlat_fd; + cpu_set_t set; + int retval; + + /* + * This all is only setting up the tool. + */ + CPU_ZERO(&set); + CPU_SET(cpu, &set); + + retval = sched_setaffinity(gettid(), sizeof(set), &set); + if (retval == -1) { + err_msg("Error setting user thread affinity\n"); + exit(1); + } + + if (!params->sched_param) { + retval = sched_setscheduler(0, SCHED_FIFO, &sp); + if (retval < 0) { + err_msg("Error setting timerlat u default priority: %s\n", strerror(errno)); + exit(1); + } + } else { + retval = __set_sched_attr(getpid(), params->sched_param); + if (retval) { + /* __set_sched_attr prints an error message, so */ + exit(0); + } + } + + if (params->cgroup_name) { + retval = set_pid_cgroup(gettid(), params->cgroup_name); + if (!retval) { + err_msg("Error setting timerlat u cgroup pid\n"); + pthread_exit(&retval); + } + } + + /* + * This is the tool's loop. If you want to use as base for your own tool... + * go ahead. + */ + snprintf(buffer, sizeof(buffer), "osnoise/per_cpu/cpu%d/timerlat_fd", cpu); + + timerlat_fd = tracefs_instance_file_open(NULL, buffer, O_RDONLY); + if (timerlat_fd < 0) { + err_msg("Error opening %s:%s\n", buffer, strerror(errno)); + exit(1); + } + + debug_msg("User-space timerlat pid %d on cpu %d\n", gettid(), cpu); + + /* add should continue with a signal handler */ + while (true) { + retval = read(timerlat_fd, buffer, 1024); + if (retval < 0) + break; + } + + close(timerlat_fd); + + debug_msg("Leaving timerlat pid %d on cpu %d\n", gettid(), cpu); + exit(0); +} + +/* + * timerlat_u_send_kill - send a kill signal for all processes + * + * Return the number of processes that received the kill. + */ +static int timerlat_u_send_kill(pid_t *procs, int nr_cpus) +{ + int killed = 0; + int i, retval; + + for (i = 0; i < nr_cpus; i++) { + if (!procs[i]) + continue; + retval = kill(procs[i], SIGKILL); + if (!retval) + killed++; + else + err_msg("Error killing child process %d\n", procs[i]); + } + + return killed; +} + +/** + * timerlat_u_dispatcher - dispatch one timerlatu/ process per monitored CPU + * + * This is a thread main that will fork one new process for each monitored + * CPU. It will wait for: + * + * - rtla to tell to kill the child processes + * - some child process to die, and the cleanup all the processes + * + * whichever comes first. + * + */ +void *timerlat_u_dispatcher(void *data) +{ + int nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + struct timerlat_u_params *params = data; + char proc_name[128]; + int procs_count = 0; + int retval = 1; + pid_t *procs; + int wstatus; + pid_t pid; + int i; + + debug_msg("Dispatching timerlat u procs\n"); + + procs = calloc(nr_cpus, sizeof(pid_t)); + if (!procs) + pthread_exit(&retval); + + for (i = 0; i < nr_cpus; i++) { + if (params->set && !CPU_ISSET(i, params->set)) + continue; + + pid = fork(); + + /* child */ + if (!pid) { + + /* + * rename the process + */ + snprintf(proc_name, sizeof(proc_name), "timerlatu/%d", i); + pthread_setname_np(pthread_self(), proc_name); + prctl(PR_SET_NAME, (unsigned long)proc_name, 0, 0, 0); + + timerlat_u_main(i, params); + /* timerlat_u_main should exit()! Anyways... */ + pthread_exit(&retval); + } + + /* parent */ + if (pid == -1) { + timerlat_u_send_kill(procs, nr_cpus); + debug_msg("Failed to create child processes"); + pthread_exit(&retval); + } + + procs_count++; + procs[i] = pid; + } + + while (params->should_run) { + /* check if processes died */ + pid = waitpid(-1, &wstatus, WNOHANG); + if (pid != 0) { + for (i = 0; i < nr_cpus; i++) { + if (procs[i] == pid) { + procs[i] = 0; + procs_count--; + } + } + break; + } + + sleep(1); + } + + timerlat_u_send_kill(procs, nr_cpus); + + while (procs_count) { + pid = waitpid(-1, &wstatus, 0); + if (pid == -1) { + err_msg("Failed to monitor child processes"); + pthread_exit(&retval); + } + for (i = 0; i < nr_cpus; i++) { + if (procs[i] == pid) { + procs[i] = 0; + procs_count--; + } + } + } + + params->stopped_running = 1; + + free(procs); + retval = 0; + pthread_exit(&retval); + +} diff --git a/tools/tracing/rtla/src/timerlat_u.h b/tools/tracing/rtla/src/timerlat_u.h new file mode 100644 index 000000000000..661511908957 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat_u.h @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira + */ + +struct timerlat_u_params { + /* timerlat -> timerlat_u: user-space threads can keep running */ + int should_run; + /* timerlat_u -> timerlat: all timerlat_u threads left, no reason to continue */ + int stopped_running; + + /* threads config */ + cpu_set_t *set; + char *cgroup_name; + struct sched_attr *sched_param; +}; + +void *timerlat_u_dispatcher(void *data); diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c index 3e25f0277fb9..623a38908ed5 100644 --- a/tools/tracing/rtla/src/utils.c +++ b/tools/tracing/rtla/src/utils.c @@ -626,6 +626,64 @@ static int get_self_cgroup(char *self_cg, int sizeof_self_cg) return 1; } +/* + * set_comm_cgroup - Set cgroup to pid_t pid + * + * If cgroup argument is not NULL, the threads will move to the given cgroup. + * Otherwise, the cgroup of the calling, i.e., rtla, thread will be used. + * + * Supports cgroup v2. + * + * Returns 1 on success, 0 otherwise. + */ +int set_pid_cgroup(pid_t pid, const char *cgroup) +{ + char cgroup_path[MAX_PATH - strlen("/cgroup.procs")]; + char cgroup_procs[MAX_PATH]; + char pid_str[24]; + int retval; + int cg_fd; + + retval = find_mount("cgroup2", cgroup_path, sizeof(cgroup_path)); + if (!retval) { + err_msg("Did not find cgroupv2 mount point\n"); + return 0; + } + + if (!cgroup) { + retval = get_self_cgroup(&cgroup_path[strlen(cgroup_path)], + sizeof(cgroup_path) - strlen(cgroup_path)); + if (!retval) { + err_msg("Did not find self cgroup\n"); + return 0; + } + } else { + snprintf(&cgroup_path[strlen(cgroup_path)], + sizeof(cgroup_path) - strlen(cgroup_path), "%s/", cgroup); + } + + snprintf(cgroup_procs, MAX_PATH, "%s/cgroup.procs", cgroup_path); + + debug_msg("Using cgroup path at: %s\n", cgroup_procs); + + cg_fd = open(cgroup_procs, O_RDWR); + if (cg_fd < 0) + return 0; + + snprintf(pid_str, sizeof(pid_str), "%d\n", pid); + + retval = write(cg_fd, pid_str, strlen(pid_str)); + if (retval < 0) + err_msg("Error setting cgroup attributes for pid:%s - %s\n", + pid_str, strerror(errno)); + else + debug_msg("Set cgroup attributes for pid:%s\n", pid_str); + + close(cg_fd); + + return (retval >= 0); +} + /** * set_comm_cgroup - Set cgroup to threads starting with char *comm_prefix * diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h index dec59163cfbc..04ed1e650495 100644 --- a/tools/tracing/rtla/src/utils.h +++ b/tools/tracing/rtla/src/utils.h @@ -57,8 +57,10 @@ struct sched_attr { int parse_prio(char *arg, struct sched_attr *sched_param); int parse_cpu_set(char *cpu_list, cpu_set_t *set); +int __set_sched_attr(int pid, struct sched_attr *attr); int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr); int set_comm_cgroup(const char *comm_prefix, const char *cgroup); +int set_pid_cgroup(pid_t pid, const char *cgroup); int set_cpu_dma_latency(int32_t latency); int auto_house_keeping(cpu_set_t *monitored_cpus); From patchwork Tue Jun 6 16:12:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C0B4C77B73 for ; Tue, 6 Jun 2023 16:13:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238429AbjFFQNl (ORCPT ); Tue, 6 Jun 2023 12:13:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238434AbjFFQNP (ORCPT ); Tue, 6 Jun 2023 12:13:15 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A83FD19A6; Tue, 6 Jun 2023 09:13:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AF98462F4C; Tue, 6 Jun 2023 16:13:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B57CC433EF; Tue, 6 Jun 2023 16:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067981; bh=vmyL9MyUu2hxplL8j6cHadIp1XaH4bM3tg8Zfm4bAdM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eeMHycdW6LhGqBao2IbokXLHuYuPGYFiMSLoG8g33V3cxeGHTPbv5gA5HMGVxJ9IU Gjvz7F+22V33YFYGw9TYSKlSllpDMGk8ETQO3QhptkBbzjZ9P2Xf7GgNymZ8UAnaxp dmqrdURrkXayeULN93tEOBQizC3ckn72uty/xVd/WxB70D9llaDVszzlioU7iodz8v E6ecg3yIoAdFemEvMIW87+AeIXTbMe/wlwrd/3bqFCboP4thdBD0tF6HhHvXj58koA /X+5uLUPVYHRLKuo45eZzzyFQ6C0XoyH4lbb71hpmx9z+q3s90/lSUJ4Jqa8ZXZD4G DuB3W/EFmWk7w== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 10/11] rtla/timerlat_hist: Add timerlat user-space support Date: Tue, 6 Jun 2023 18:12:24 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Add the support for running timerlat threads in user-space. In this mode, enabled with -u/--user-threads, timerlat dispatches user-space processes that will loop in the timerlat_fd, measuring the overhead for going to user-space and then returning to the kernel - in addition to the existing measurements. Here is one example of the tool's output with -u enabled: $ sudo timerlat hist -u -c 1-3 -d 600 # RTLA timerlat histogram # Time unit is microseconds (us) # Duration: 0 00:10:01 Index IRQ-001 Thr-001 Usr-001 IRQ-002 Thr-002 Usr-002 IRQ-003 Thr-003 Usr-003 0 477555 0 0 425287 0 0 474357 0 0 1 122385 7998 0 174616 1921 0 125412 3138 0 2 47 587376 492150 89 594717 447830 147 593463 454872 3 11 2549 101930 7 2682 145580 64 2530 138680 4 3 1954 2833 1 463 4917 11 548 4656 5 0 60 1037 0 138 1117 6 179 1130 6 0 26 1837 0 38 277 1 76 339 7 0 15 143 0 28 147 2 37 156 8 0 10 23 0 11 75 0 12 80 9 0 7 17 0 0 26 0 11 42 10 0 2 11 0 0 18 0 2 20 11 0 0 7 0 1 8 0 2 12 12 0 0 6 0 1 4 0 2 8 13 0 1 3 0 0 0 0 0 1 14 0 1 0 0 0 1 0 0 2 15 0 1 0 0 0 0 0 0 2 16 0 1 2 0 0 0 0 0 0 17 0 0 1 0 0 0 0 0 0 19 0 0 1 0 0 0 0 0 0 over: 0 0 0 0 0 0 0 0 0 count: 600001 600001 600001 600000 600000 600000 600000 600000 600000 min: 0 1 2 0 1 2 0 1 2 avg: 0 1 2 0 2 2 0 2 2 max: 4 16 19 4 12 14 7 12 15 The tuning setup like -p or -C work for the user-space threads as well. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/timerlat_hist.c | 129 ++++++++++++++++++++++--- 1 file changed, 118 insertions(+), 11 deletions(-) diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c index f431cf43246c..47d3d8b53cb2 100644 --- a/tools/tracing/rtla/src/timerlat_hist.c +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -12,11 +12,13 @@ #include #include #include +#include #include "utils.h" #include "osnoise.h" #include "timerlat.h" #include "timerlat_aa.h" +#include "timerlat_u.h" struct timerlat_hist_params { char *cpus; @@ -37,6 +39,7 @@ struct timerlat_hist_params { int hk_cpus; int no_aa; int dump_tasks; + int user_hist; cpu_set_t hk_cpu_set; struct sched_attr sched_param; struct trace_events *events; @@ -53,9 +56,11 @@ struct timerlat_hist_params { struct timerlat_hist_cpu { int *irq; int *thread; + int *user; int irq_count; int thread_count; + int user_count; unsigned long long min_irq; unsigned long long sum_irq; @@ -64,6 +69,10 @@ struct timerlat_hist_cpu { unsigned long long min_thread; unsigned long long sum_thread; unsigned long long max_thread; + + unsigned long long min_user; + unsigned long long sum_user; + unsigned long long max_user; }; struct timerlat_hist_data { @@ -88,6 +97,10 @@ timerlat_free_histogram(struct timerlat_hist_data *data) if (data->hist[cpu].thread) free(data->hist[cpu].thread); + + if (data->hist[cpu].user) + free(data->hist[cpu].user); + } /* one set of histograms per CPU */ @@ -124,15 +137,21 @@ static struct timerlat_hist_data data->hist[cpu].irq = calloc(1, sizeof(*data->hist->irq) * (entries + 1)); if (!data->hist[cpu].irq) goto cleanup; + data->hist[cpu].thread = calloc(1, sizeof(*data->hist->thread) * (entries + 1)); if (!data->hist[cpu].thread) goto cleanup; + + data->hist[cpu].user = calloc(1, sizeof(*data->hist->user) * (entries + 1)); + if (!data->hist[cpu].user) + goto cleanup; } /* set the min to max */ for (cpu = 0; cpu < nr_cpus; cpu++) { data->hist[cpu].min_irq = ~0; data->hist[cpu].min_thread = ~0; + data->hist[cpu].min_user = ~0; } return data; @@ -147,7 +166,7 @@ static struct timerlat_hist_data */ static void timerlat_hist_update(struct osnoise_tool *tool, int cpu, - unsigned long long thread, + unsigned long long context, unsigned long long latency) { struct timerlat_hist_params *params = tool->params; @@ -162,18 +181,24 @@ timerlat_hist_update(struct osnoise_tool *tool, int cpu, if (data->bucket_size) bucket = latency / data->bucket_size; - if (!thread) { + if (!context) { hist = data->hist[cpu].irq; data->hist[cpu].irq_count++; update_min(&data->hist[cpu].min_irq, &latency); update_sum(&data->hist[cpu].sum_irq, &latency); update_max(&data->hist[cpu].max_irq, &latency); - } else { + } else if (context == 1) { hist = data->hist[cpu].thread; data->hist[cpu].thread_count++; update_min(&data->hist[cpu].min_thread, &latency); update_sum(&data->hist[cpu].sum_thread, &latency); update_max(&data->hist[cpu].max_thread, &latency); + } else { /* user */ + hist = data->hist[cpu].user; + data->hist[cpu].user_count++; + update_min(&data->hist[cpu].min_user, &latency); + update_sum(&data->hist[cpu].sum_user, &latency); + update_max(&data->hist[cpu].max_user, &latency); } if (bucket < entries) @@ -190,16 +215,16 @@ timerlat_hist_handler(struct trace_seq *s, struct tep_record *record, struct tep_event *event, void *data) { struct trace_instance *trace = data; - unsigned long long thread, latency; + unsigned long long context, latency; struct osnoise_tool *tool; int cpu = record->cpu; tool = container_of(trace, struct osnoise_tool, trace); - tep_get_field_val(s, event, "context", record, &thread, 1); + tep_get_field_val(s, event, "context", record, &context, 1); tep_get_field_val(s, event, "timer_latency", record, &latency, 1); - timerlat_hist_update(tool, cpu, thread, latency); + timerlat_hist_update(tool, cpu, context, latency); return 0; } @@ -241,6 +266,9 @@ static void timerlat_hist_header(struct osnoise_tool *tool) if (!params->no_thread) trace_seq_printf(s, " Thr-%03d", cpu); + + if (params->user_hist) + trace_seq_printf(s, " Usr-%03d", cpu); } trace_seq_printf(s, "\n"); @@ -279,6 +307,10 @@ timerlat_print_summary(struct timerlat_hist_params *params, if (!params->no_thread) trace_seq_printf(trace->seq, "%9d ", data->hist[cpu].thread_count); + + if (params->user_hist) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].user_count); } trace_seq_printf(trace->seq, "\n"); @@ -299,6 +331,10 @@ timerlat_print_summary(struct timerlat_hist_params *params, if (!params->no_thread) trace_seq_printf(trace->seq, "%9llu ", data->hist[cpu].min_thread); + + if (params->user_hist) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].min_user); } trace_seq_printf(trace->seq, "\n"); @@ -323,7 +359,15 @@ timerlat_print_summary(struct timerlat_hist_params *params, if (!params->no_thread) { if (data->hist[cpu].thread_count) trace_seq_printf(trace->seq, "%9llu ", - data->hist[cpu].sum_thread / data->hist[cpu].thread_count); + data->hist[cpu].sum_thread / data->hist[cpu].thread_count); + else + trace_seq_printf(trace->seq, " - "); + } + + if (params->user_hist) { + if (data->hist[cpu].user_count) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].sum_user / data->hist[cpu].user_count); else trace_seq_printf(trace->seq, " - "); } @@ -347,6 +391,10 @@ timerlat_print_summary(struct timerlat_hist_params *params, if (!params->no_thread) trace_seq_printf(trace->seq, "%9llu ", data->hist[cpu].max_thread); + + if (params->user_hist) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].max_user); } trace_seq_printf(trace->seq, "\n"); trace_seq_do_printf(trace->seq); @@ -392,6 +440,12 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t data->hist[cpu].thread[bucket]); } + if (params->user_hist) { + total += data->hist[cpu].user[bucket]; + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].user[bucket]); + } + } if (total == 0 && !params->with_zeros) { @@ -421,6 +475,10 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t if (!params->no_thread) trace_seq_printf(trace->seq, "%9d ", data->hist[cpu].thread[data->entries]); + + if (params->user_hist) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].user[data->entries]); } trace_seq_printf(trace->seq, "\n"); trace_seq_do_printf(trace->seq); @@ -441,7 +499,7 @@ static void timerlat_hist_usage(char *usage) " usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\", " [-t[=file]] [-e sys[:event]] [--filter ] [--trigger ] [-c cpu-list] [-H cpu-list]\\", " [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\", - " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task]", + " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task] [-u]", "", " -h/--help: print this menu", " -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit", @@ -476,6 +534,7 @@ static void timerlat_hist_usage(char *usage) " f:prio - use SCHED_FIFO with prio", " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", " in nanoseconds", + " -u/--user-threads: use rtla user-space threads instead of in-kernel timerlat threads", NULL, }; @@ -532,6 +591,7 @@ static struct timerlat_hist_params {"stack", required_argument, 0, 's'}, {"thread", required_argument, 0, 'T'}, {"trace", optional_argument, 0, 't'}, + {"user-threads", no_argument, 0, 'u'}, {"event", required_argument, 0, 'e'}, {"no-irq", no_argument, 0, '0'}, {"no-thread", no_argument, 0, '1'}, @@ -550,7 +610,7 @@ static struct timerlat_hist_params /* getopt_long stores the option index here. */ int option_index = 0; - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:9\1", + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:u0123456:7:8:9\1", long_options, &option_index); /* detect the end of the options. */ @@ -660,6 +720,9 @@ static struct timerlat_hist_params else params->trace_output = "timerlat_trace.txt"; break; + case 'u': + params->user_hist = 1; + break; case '0': /* no irq */ params->no_irq = 1; break; @@ -744,7 +807,7 @@ static struct timerlat_hist_params static int timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_params *params) { - int retval; + int retval, i; if (!params->sleep_time) params->sleep_time = 1; @@ -755,6 +818,9 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param err_msg("Failed to apply CPUs config\n"); goto out_err; } + } else { + for (i = 0; i < sysconf(_SC_NPROCESSORS_CONF); i++) + CPU_SET(i, ¶ms->monitored_cpus); } if (params->stop_us) { @@ -807,6 +873,14 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param auto_house_keeping(¶ms->monitored_cpus); } + if (params->user_hist) { + retval = osnoise_set_workload(tool->context, 0); + if (retval) { + err_msg("Failed to set OSNOISE_WORKLOAD option\n"); + goto out_err; + } + } + return 0; out_err: @@ -867,11 +941,13 @@ int timerlat_hist_main(int argc, char *argv[]) { struct timerlat_hist_params *params; struct osnoise_tool *record = NULL; + struct timerlat_u_params params_u; struct osnoise_tool *tool = NULL; struct osnoise_tool *aa = NULL; struct trace_instance *trace; int dma_latency_fd = -1; int return_value = 1; + pthread_t timerlat_u; int retval; params = timerlat_hist_parse_args(argc, argv); @@ -906,7 +982,7 @@ int timerlat_hist_main(int argc, char *argv[]) } } - if (params->cgroup) { + if (params->cgroup && !params->user_hist) { retval = set_comm_cgroup("timerlat/", params->cgroup_name); if (!retval) { err_msg("Failed to move threads to cgroup\n"); @@ -970,6 +1046,25 @@ int timerlat_hist_main(int argc, char *argv[]) tool->start_time = time(NULL); timerlat_hist_set_signals(params); + if (params->user_hist) { + /* rtla asked to stop */ + params_u.should_run = 1; + /* all threads left */ + params_u.stopped_running = 0; + + params_u.set = ¶ms->monitored_cpus; + if (params->set_sched) + params_u.sched_param = ¶ms->sched_param; + else + params_u.sched_param = NULL; + + params_u.cgroup_name = params->cgroup_name; + + retval = pthread_create(&timerlat_u, NULL, timerlat_u_dispatcher, ¶ms_u); + if (retval) + err_msg("Error creating timerlat user-space threads\n"); + } + while (!stop_tracing) { sleep(params->sleep_time); @@ -986,6 +1081,18 @@ int timerlat_hist_main(int argc, char *argv[]) if (trace_is_off(&tool->trace, &record->trace)) break; + + /* is there still any user-threads ? */ + if (params->user_hist) { + if (params_u.stopped_running) { + debug_msg("timerlat user-space threads stopped!\n"); + break; + } + } + } + if (params->user_hist && !params_u.stopped_running) { + params_u.should_run = 0; + sleep(1); } timerlat_print_stats(params, tool); From patchwork Tue Jun 6 16:12:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 13269429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E12F4C7EE29 for ; Tue, 6 Jun 2023 16:13:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238544AbjFFQNz (ORCPT ); Tue, 6 Jun 2023 12:13:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237776AbjFFQN1 (ORCPT ); Tue, 6 Jun 2023 12:13:27 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92E9719BD; Tue, 6 Jun 2023 09:13:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2DF4462DE6; Tue, 6 Jun 2023 16:13:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8CEEBC4339E; Tue, 6 Jun 2023 16:13:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686067983; bh=SNMZc0L/b6GLmVD1yPrxAZQaDWAwMEPVbWJyNUZPapU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ISrLLMAQMED7jumB7rUxj4NLoUAL1EUyBok+B70FXLeTV8RbWw2YEyhDwpUUt2W+8 BSxK7qj0Xsy1sBuOEiYEIDscWsMI8kOhRR5UpQ1uYg9B2NN7A3Omlc1BfNFBZY0dvi 4brp+zhcc4iOG+YQwY906r240v+Ox0bb7YrQqkZA2AgpiYnVG/xEO7OM3wnAUf3T1a tC7uJTlisTSrjMOg93qS/eQVXTnpCtKJ4KNPzc+VxwQ0RwYF7suOc5KDTQFRGDOVrk M4xRJ0dhiRl0bQyu6vIGQpuwUPKNMeKGpLwXJPqYVcPlyuNBp7fyJKxA9iF59EasAc mSJfqaSQpNEEQ== From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Steven Rostedt Cc: linux-doc@vger.kernel.org, Juri Lelli , William White , Daniel Bristot de Oliveira , Masami Hiramatsu , Jonathan Corbet Subject: [PATCH V3 11/11] Documentation: Add tools/rtla timerlat -u option documentation Date: Tue, 6 Jun 2023 18:12:25 +0200 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Add the -u/--user-thread option documentation for timerlat top/hist. Cc: Steven Rostedt Cc: Daniel Bristot de Oliveira Cc: Masami Hiramatsu Cc: Jonathan Corbet Signed-off-by: Daniel Bristot de Oliveira --- Documentation/tools/rtla/common_timerlat_options.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Documentation/tools/rtla/common_timerlat_options.rst b/Documentation/tools/rtla/common_timerlat_options.rst index bacdea6de7a3..88506b397c2d 100644 --- a/Documentation/tools/rtla/common_timerlat_options.rst +++ b/Documentation/tools/rtla/common_timerlat_options.rst @@ -26,3 +26,10 @@ Set the /dev/cpu_dma_latency to *us*, aiming to bound exit from idle latencies. *cyclictest* sets this value to *0* by default, use **--dma-latency** *0* to have similar results. + +**-u**, **--user-threads** + + Set timerlat to run without a workload, and then dispatches user-space workloads + to wait on the timerlat_fd. Once the workload is awakes, it goes to sleep again + adding so the measurement for the kernel-to-user and user-to-kernel to the tracer + output.