From patchwork Mon Oct 11 14:35:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37637C433F5 for ; Mon, 11 Oct 2021 14:36:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2122160BD3 for ; Mon, 11 Oct 2021 14:36:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238509AbhJKOir convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:38:47 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:52527 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237686AbhJKOir (ORCPT ); Mon, 11 Oct 2021 10:38:47 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-283-7oLUnXY2OdugmSW903-5_A-1; Mon, 11 Oct 2021 10:36:42 -0400 X-MC-Unique: 7oLUnXY2OdugmSW903-5_A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2A56618125D2; Mon, 11 Oct 2021 14:36:41 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 57A7919C59; Mon, 11 Oct 2021 14:36:37 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 01/19] trace/osnoise: Do not follow tracing_cpumask Date: Mon, 11 Oct 2021 16:35:49 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In preparation to support multiple instances, decople the osnoise/timelat workload from instance specific tracing_cpumask. Different instances can have conflicing cpumasks, making osnoise workload management needlessly complex. Osnoise already have its global cpu mask. I also thought about using the first instance mask, but the "first" instance could be removed before the others. This also fixes the problem that changing the tracing_mask was not re-starting the trace. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index ce053619f289..7b1f8187764c 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1553,13 +1553,10 @@ static int start_per_cpu_kthreads(struct trace_array *tr) cpus_read_lock(); /* - * Run only on CPUs in which trace and osnoise are allowed to run. + * Run only on online CPUs in which trace and osnoise are allowed to + * run. */ - cpumask_and(current_mask, tr->tracing_cpumask, &osnoise_cpumask); - /* - * And the CPU is online. - */ - cpumask_and(current_mask, cpu_online_mask, current_mask); + cpumask_and(current_mask, cpu_online_mask, &osnoise_cpumask); for_each_possible_cpu(cpu) per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL; @@ -1580,10 +1577,8 @@ static int start_per_cpu_kthreads(struct trace_array *tr) #ifdef CONFIG_HOTPLUG_CPU static void osnoise_hotplug_workfn(struct work_struct *dummy) { - struct trace_array *tr = osnoise_trace; unsigned int cpu = smp_processor_id(); - mutex_lock(&trace_types_lock); if (!osnoise_busy) @@ -1595,9 +1590,6 @@ static void osnoise_hotplug_workfn(struct work_struct *dummy) if (!cpumask_test_cpu(cpu, &osnoise_cpumask)) goto out_unlock; - if (!cpumask_test_cpu(cpu, tr->tracing_cpumask)) - goto out_unlock; - start_kthread(cpu); out_unlock: @@ -1700,13 +1692,10 @@ static void osnoise_tracer_stop(struct trace_array *tr); * interface to the osnoise trace. By default, it lists all CPUs, * in this way, allowing osnoise threads to run on any online CPU * of the system. It serves to restrict the execution of osnoise to the - * set of CPUs writing via this interface. Note that osnoise also - * respects the "tracing_cpumask." Hence, osnoise threads will run only - * on the set of CPUs allowed here AND on "tracing_cpumask." Why not - * have just "tracing_cpumask?" Because the user might be interested - * in tracing what is running on other CPUs. For instance, one might - * run osnoise in one HT CPU while observing what is running on the - * sibling HT CPU. + * set of CPUs writing via this interface. Why not use "tracing_cpumask"? + * Because the user might be interested in tracing what is running on + * other CPUs. For instance, one might run osnoise in one HT CPU + * while observing what is running on the sibling HT CPU. */ static ssize_t osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, From patchwork Mon Oct 11 14:35:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4356EC433EF for ; Mon, 11 Oct 2021 14:36:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CACD60E74 for ; Mon, 11 Oct 2021 14:36:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238751AbhJKOiv convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:38:51 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:41097 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238606AbhJKOiu (ORCPT ); Mon, 11 Oct 2021 10:38:50 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-404-yIryLqSjMKKHUiMv7OYBdA-1; Mon, 11 Oct 2021 10:36:46 -0400 X-MC-Unique: yIryLqSjMKKHUiMv7OYBdA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50FB71882FD8; Mon, 11 Oct 2021 14:36:45 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 88ED927073; Mon, 11 Oct 2021 14:36:41 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 02/19] trace/osnoise: Split workload start from the tracer start Date: Mon, 11 Oct 2021 16:35:50 +0200 Message-Id: <338d0210b154ffeb775ea71fed63f7b10e1f51cf.1633958325.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In preparation from supporting multiple trace instances, create workload start/stop specific functions. No functional change. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 53 ++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 7b1f8187764c..0a04a9a4c85a 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1545,7 +1545,7 @@ static int start_kthread(unsigned int cpu) * This starts the kernel thread that will look for osnoise on many * cpus. */ -static int start_per_cpu_kthreads(struct trace_array *tr) +static int start_per_cpu_kthreads(void) { struct cpumask *current_mask = &save_cpumask; int retval = 0; @@ -1678,8 +1678,8 @@ osnoise_cpus_read(struct file *filp, char __user *ubuf, size_t count, return count; } -static void osnoise_tracer_start(struct trace_array *tr); -static void osnoise_tracer_stop(struct trace_array *tr); +static int osnoise_workload_start(void); +static void osnoise_workload_stop(void); /* * osnoise_cpus_write - Write function for "cpus" entry @@ -1701,7 +1701,6 @@ static ssize_t osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, loff_t *ppos) { - struct trace_array *tr = osnoise_trace; cpumask_var_t osnoise_cpumask_new; int running, err; char buf[256]; @@ -1726,7 +1725,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_lock(&trace_types_lock); running = osnoise_busy; if (running) - osnoise_tracer_stop(tr); + osnoise_workload_stop(); mutex_lock(&interface_lock); /* @@ -1740,7 +1739,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_unlock(&interface_lock); if (running) - osnoise_tracer_start(tr); + osnoise_workload_start(); mutex_unlock(&trace_types_lock); free_cpumask_var(osnoise_cpumask_new); @@ -1921,7 +1920,10 @@ static int osnoise_hook_events(void) return -EINVAL; } -static int __osnoise_tracer_start(struct trace_array *tr) +/* + * osnoise_workload_start - start the workload and hook to events + */ +static int osnoise_workload_start(void) { int retval; @@ -1936,7 +1938,7 @@ static int __osnoise_tracer_start(struct trace_array *tr) barrier(); trace_osnoise_callback_enabled = true; - retval = start_per_cpu_kthreads(tr); + retval = start_per_cpu_kthreads(); if (retval) { unhook_irq_events(); return retval; @@ -1947,6 +1949,26 @@ static int __osnoise_tracer_start(struct trace_array *tr) return 0; } +/* + * osnoise_workload_stop - stop the workload and unhook the events + */ +static void osnoise_workload_stop(void) +{ + if (!osnoise_busy) + return; + + trace_osnoise_callback_enabled = false; + barrier(); + + stop_per_cpu_kthreads(); + + unhook_irq_events(); + unhook_softirq_events(); + unhook_thread_events(); + + osnoise_busy = false; +} + static void osnoise_tracer_start(struct trace_array *tr) { int retval; @@ -1954,7 +1976,7 @@ static void osnoise_tracer_start(struct trace_array *tr) if (osnoise_busy) return; - retval = __osnoise_tracer_start(tr); + retval = osnoise_workload_start(); if (retval) pr_err(BANNER "Error starting osnoise tracer\n"); @@ -1965,16 +1987,7 @@ static void osnoise_tracer_stop(struct trace_array *tr) if (!osnoise_busy) return; - trace_osnoise_callback_enabled = false; - barrier(); - - stop_per_cpu_kthreads(); - - unhook_irq_events(); - unhook_softirq_events(); - unhook_thread_events(); - - osnoise_busy = false; + osnoise_workload_stop(); } static int osnoise_tracer_init(struct trace_array *tr) @@ -2017,7 +2030,7 @@ static void timerlat_tracer_start(struct trace_array *tr) osnoise_data.timerlat_tracer = 1; - retval = __osnoise_tracer_start(tr); + retval = osnoise_workload_start(); if (retval) goto out_err; From patchwork Mon Oct 11 14:35:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1E08C433F5 for ; Mon, 11 Oct 2021 14:36:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BDEAC60E98 for ; Mon, 11 Oct 2021 14:36:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236716AbhJKOi4 convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:38:56 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:34063 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238643AbhJKOiy (ORCPT ); Mon, 11 Oct 2021 10:38:54 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-601-HH6fUSyrPV6vIf6jR9RQsg-1; Mon, 11 Oct 2021 10:36:51 -0400 X-MC-Unique: HH6fUSyrPV6vIf6jR9RQsg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 43A63824FA8; Mon, 11 Oct 2021 14:36:49 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 867341F6; Mon, 11 Oct 2021 14:36:45 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 03/19] trace/osnoise: Use start/stop_per_cpu_kthreads() on osnoise_cpus_write() Date: Mon, 11 Oct 2021 16:35:51 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When writing a new CPU mask via osnoise/cpus, if the tracer is running, the workload is restarted to follow the new cpumask. The restart is currently done using osnoise_workload_start/stop(), which disables the workload *and* the instrumentation. However, disabling the instrumentation is not necessary. Calling start/stop_per_cpu_kthreads() is enough to apply the new osnoise/cpus config. No functional change. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 0a04a9a4c85a..9b9e6bc889e9 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1678,9 +1678,6 @@ osnoise_cpus_read(struct file *filp, char __user *ubuf, size_t count, return count; } -static int osnoise_workload_start(void); -static void osnoise_workload_stop(void); - /* * osnoise_cpus_write - Write function for "cpus" entry * @filp: The active open file structure @@ -1725,7 +1722,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_lock(&trace_types_lock); running = osnoise_busy; if (running) - osnoise_workload_stop(); + stop_per_cpu_kthreads(); mutex_lock(&interface_lock); /* @@ -1739,7 +1736,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_unlock(&interface_lock); if (running) - osnoise_workload_start(); + start_per_cpu_kthreads(); mutex_unlock(&trace_types_lock); free_cpumask_var(osnoise_cpumask_new); From patchwork Mon Oct 11 14:35:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B5E1C433EF for ; Mon, 11 Oct 2021 14:37:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F315C60E98 for ; Mon, 11 Oct 2021 14:37:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238912AbhJKOjC convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:02 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:52881 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238822AbhJKOi7 (ORCPT ); Mon, 11 Oct 2021 10:38:59 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-412-kUe5sNkvPCSZZ3nYkrQvRg-1; Mon, 11 Oct 2021 10:36:56 -0400 X-MC-Unique: kUe5sNkvPCSZZ3nYkrQvRg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 89BD31906810; Mon, 11 Oct 2021 14:36:53 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9DD7B19C59; Mon, 11 Oct 2021 14:36:49 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 04/19] trace/osnoise: Support a list of trace_array *tr Date: Mon, 11 Oct 2021 16:35:52 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org osnoise/timerlat were built to run a single instance, and for this, a single variable is enough to store the current struct trace_array *tr with information about the tracing instance. This is done via the *osnoise_trace variable. A trace_array represents a trace instance. In preparation to support multiple instances, replace the *osnoise_trace variable with an RCU protected list of instances. The operations that refer to an instance now propagate to all elements of the list (all instances). Also, replace the osnoise_busy variable with a check if the list has elements (busy). No functional change is expected with this patch, i.e., only one instance is allowed yet. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 253 +++++++++++++++++++++++++---------- 1 file changed, 184 insertions(+), 69 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 9b9e6bc889e9..a6d8f514bd7c 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -38,8 +38,6 @@ #define CREATE_TRACE_POINTS #include -static struct trace_array *osnoise_trace; - /* * Default values. */ @@ -50,6 +48,73 @@ static struct trace_array *osnoise_trace; #define DEFAULT_TIMERLAT_PERIOD 1000 /* 1ms */ #define DEFAULT_TIMERLAT_PRIO 95 /* FIFO 95 */ +/* + * trace_array of the enabled osnoise/timerlat instances. + */ +struct osnoise_instance { + struct list_head list; + struct trace_array *tr; +}; +struct list_head osnoise_instances; + +static bool osnoise_has_registered_instances(void) +{ + return !!list_first_or_null_rcu(&osnoise_instances, + struct osnoise_instance, + list); +} + +/* + * osnoise_register_instance - register a new trace instance + * + * Register a trace_array *tr in the list of instances running + * osnoise/timerlat tracers. + */ +static int osnoise_register_instance(struct trace_array *tr) +{ + struct osnoise_instance *inst; + + inst = kmalloc(sizeof(*inst), GFP_KERNEL); + if (!inst) + return -ENOMEM; + + INIT_LIST_HEAD_RCU(&inst->list); + inst->tr = tr; + list_add_tail_rcu(&inst->list, &osnoise_instances); + + return 0; +} + +/* + * osnoise_unregister_instance - unregister a registered trace instance + * + * Remove the trace_array *tr from the list of instances running + * osnoise/timerlat tracers. + */ +static void osnoise_unregister_instance(struct trace_array *tr) +{ + struct osnoise_instance *inst; + int found = 0; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + if (inst->tr == tr) { + list_del_rcu(&inst->list); + found = 1; + break; + } + } + rcu_read_unlock(); + + if (!found) { + WARN("osnoise unregister did not find tr %s\n", tr->name); + return; + } + + synchronize_rcu(); + kfree(inst); +} + /* * NMI runtime info. */ @@ -248,11 +313,6 @@ static struct osnoise_data { #endif }; -/* - * Boolean variable used to inform that the tracer is currently sampling. - */ -static bool osnoise_busy; - #ifdef CONFIG_PREEMPT_RT /* * Print the osnoise header info. @@ -315,19 +375,24 @@ static void print_osnoise_headers(struct seq_file *s) * osnoise_taint - report an osnoise error. */ #define osnoise_taint(msg) ({ \ - struct trace_array *tr = osnoise_trace; \ + struct osnoise_instance *inst; \ + struct trace_buffer *buffer; \ \ - trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \ + rcu_read_lock(); \ + list_for_each_entry_rcu(inst, &osnoise_instances, list) { \ + buffer = inst->tr->array_buffer.buffer; \ + trace_array_printk_buf(buffer, _THIS_IP_, msg); \ + } \ + rcu_read_unlock(); \ osnoise_data.tainted = true; \ }) /* * Record an osnoise_sample into the tracer buffer. */ -static void trace_osnoise_sample(struct osnoise_sample *sample) +static void +__trace_osnoise_sample(struct osnoise_sample *sample, struct trace_buffer *buffer) { - struct trace_array *tr = osnoise_trace; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct trace_event_call *call = &event_osnoise; struct ring_buffer_event *event; struct osnoise_entry *entry; @@ -350,6 +415,22 @@ static void trace_osnoise_sample(struct osnoise_sample *sample) trace_buffer_unlock_commit_nostack(buffer, event); } +/* + * Record an osnoise_sample on all osnoise instances. + */ +static void trace_osnoise_sample(struct osnoise_sample *sample) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __trace_osnoise_sample(sample, buffer); + } + rcu_read_unlock(); +} + #ifdef CONFIG_TIMERLAT_TRACER /* * Print the timerlat header info. @@ -387,14 +468,10 @@ static void print_timerlat_headers(struct seq_file *s) } #endif /* CONFIG_PREEMPT_RT */ -/* - * Record an timerlat_sample into the tracer buffer. - */ -static void trace_timerlat_sample(struct timerlat_sample *sample) +static void +__trace_timerlat_sample(struct timerlat_sample *sample, struct trace_buffer *buffer) { - struct trace_array *tr = osnoise_trace; struct trace_event_call *call = &event_osnoise; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct ring_buffer_event *event; struct timerlat_entry *entry; @@ -411,6 +488,22 @@ static void trace_timerlat_sample(struct timerlat_sample *sample) trace_buffer_unlock_commit_nostack(buffer, event); } +/* + * Record an timerlat_sample into the tracer buffer. + */ +static void trace_timerlat_sample(struct timerlat_sample *sample) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __trace_timerlat_sample(sample, buffer); + } + rcu_read_unlock(); +} + #ifdef CONFIG_STACKTRACE #define MAX_CALLS 256 @@ -450,29 +543,18 @@ static void timerlat_save_stack(int skip) return; } -/* - * timerlat_dump_stack - dump a stack trace previously saved - * - * Dump a saved stack trace into the trace buffer. - */ -static void timerlat_dump_stack(void) + +static void +__timerlat_dump_stack(struct trace_buffer *buffer, struct trace_stack *fstack, unsigned int size) { struct trace_event_call *call = &event_osnoise; - struct trace_array *tr = osnoise_trace; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct ring_buffer_event *event; - struct trace_stack *fstack; struct stack_entry *entry; - unsigned int size; - - preempt_disable_notrace(); - fstack = this_cpu_ptr(&trace_stack); - size = fstack->stack_size; event = trace_buffer_lock_reserve(buffer, TRACE_STACK, sizeof(*entry) + size, tracing_gen_ctx()); if (!event) - goto out; + return; entry = ring_buffer_event_data(event); @@ -481,8 +563,29 @@ static void timerlat_dump_stack(void) if (!call_filter_check_discard(call, entry, buffer, event)) trace_buffer_unlock_commit_nostack(buffer, event); +} -out: +/* + * timerlat_dump_stack - dump a stack trace previously saved + */ +static void timerlat_dump_stack(void) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + struct trace_stack *fstack; + unsigned int size; + + preempt_disable_notrace(); + fstack = this_cpu_ptr(&trace_stack); + size = fstack->stack_size; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __timerlat_dump_stack(buffer, fstack, size); + + } + rcu_read_unlock(); preempt_enable_notrace(); } #else @@ -1077,12 +1180,37 @@ diff_osn_sample_stats(struct osnoise_variables *osn_var, struct osnoise_sample * */ static __always_inline void osnoise_stop_tracing(void) { - struct trace_array *tr = osnoise_trace; + struct osnoise_instance *inst; + struct trace_array *tr; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + tr = inst->tr; + trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, + "stop tracing hit on cpu %d\n", smp_processor_id()); + + tracer_tracing_off(tr); + } + rcu_read_unlock(); +} - trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, - "stop tracing hit on cpu %d\n", smp_processor_id()); +/* + * notify_new_max_latency - Notify a new max latency via fsnotify interface. + */ +void notify_new_max_latency(u64 latency) +{ + struct osnoise_instance *inst; + struct trace_array *tr; - tracer_tracing_off(tr); + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + tr = inst->tr; + if (tr->max_latency < latency) { + tr->max_latency = latency; + latency_fsnotify(tr); + } + } + rcu_read_unlock(); } /* @@ -1096,7 +1224,6 @@ static __always_inline void osnoise_stop_tracing(void) static int run_osnoise(void) { struct osnoise_variables *osn_var = this_cpu_osn_var(); - struct trace_array *tr = osnoise_trace; u64 start, sample, last_sample; u64 last_int_count, int_count; s64 noise = 0, max_noise = 0; @@ -1231,11 +1358,7 @@ static int run_osnoise(void) trace_osnoise_sample(&s); - /* Keep a running maximum ever recorded osnoise "latency" */ - if (max_noise > tr->max_latency) { - tr->max_latency = max_noise; - latency_fsnotify(tr); - } + notify_new_max_latency(max_noise); if (osnoise_data.stop_tracing_total) if (s.noise > osnoise_data.stop_tracing_total) @@ -1293,7 +1416,6 @@ static int osnoise_main(void *data) static enum hrtimer_restart timerlat_irq(struct hrtimer *timer) { struct osnoise_variables *osn_var = this_cpu_osn_var(); - struct trace_array *tr = osnoise_trace; struct timerlat_variables *tlat; struct timerlat_sample s; u64 now; @@ -1364,11 +1486,7 @@ static enum hrtimer_restart timerlat_irq(struct hrtimer *timer) trace_timerlat_sample(&s); - /* Keep a running maximum ever recorded os noise "latency" */ - if (diff > tr->max_latency) { - tr->max_latency = diff; - latency_fsnotify(tr); - } + notify_new_max_latency(diff); if (osnoise_data.stop_tracing) if (time_to_us(diff) >= osnoise_data.stop_tracing) @@ -1581,7 +1699,7 @@ static void osnoise_hotplug_workfn(struct work_struct *dummy) mutex_lock(&trace_types_lock); - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) goto out_unlock_trace; mutex_lock(&interface_lock); @@ -1716,11 +1834,10 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, goto err_free; /* - * trace_types_lock is taken to avoid concurrency on start/stop - * and osnoise_busy. + * trace_types_lock is taken to avoid concurrency on start/stop. */ mutex_lock(&trace_types_lock); - running = osnoise_busy; + running = osnoise_has_registered_instances(); if (running) stop_per_cpu_kthreads(); @@ -1941,8 +2058,6 @@ static int osnoise_workload_start(void) return retval; } - osnoise_busy = true; - return 0; } @@ -1951,7 +2066,7 @@ static int osnoise_workload_start(void) */ static void osnoise_workload_stop(void) { - if (!osnoise_busy) + if (osnoise_has_registered_instances()) return; trace_osnoise_callback_enabled = false; @@ -1962,28 +2077,28 @@ static void osnoise_workload_stop(void) unhook_irq_events(); unhook_softirq_events(); unhook_thread_events(); - - osnoise_busy = false; } static void osnoise_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_busy) + if (osnoise_has_registered_instances()) return; retval = osnoise_workload_start(); if (retval) pr_err(BANNER "Error starting osnoise tracer\n"); + osnoise_register_instance(tr); } static void osnoise_tracer_stop(struct trace_array *tr) { - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) return; + osnoise_unregister_instance(tr); osnoise_workload_stop(); } @@ -1991,14 +2106,12 @@ static int osnoise_tracer_init(struct trace_array *tr) { /* Only allow one instance to enable this */ - if (osnoise_busy) + if (osnoise_has_registered_instances()) return -EBUSY; - osnoise_trace = tr; tr->max_latency = 0; osnoise_tracer_start(tr); - return 0; } @@ -2022,7 +2135,7 @@ static void timerlat_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_busy) + if (osnoise_has_registered_instances()) return; osnoise_data.timerlat_tracer = 1; @@ -2031,6 +2144,8 @@ static void timerlat_tracer_start(struct trace_array *tr) if (retval) goto out_err; + osnoise_register_instance(tr); + return; out_err: pr_err(BANNER "Error starting timerlat tracer\n"); @@ -2040,7 +2155,7 @@ static void timerlat_tracer_stop(struct trace_array *tr) { int cpu; - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) return; for_each_online_cpu(cpu) @@ -2054,11 +2169,9 @@ static void timerlat_tracer_stop(struct trace_array *tr) static int timerlat_tracer_init(struct trace_array *tr) { /* Only allow one instance to enable this */ - if (osnoise_busy) + if (osnoise_has_registered_instances()) return -EBUSY; - osnoise_trace = tr; - tr->max_latency = 0; timerlat_tracer_start(tr); @@ -2105,6 +2218,8 @@ __init static int init_osnoise_tracer(void) #endif osnoise_init_hotplug_support(); + INIT_LIST_HEAD_RCU(&osnoise_instances); + init_tracefs(); return 0; From patchwork Mon Oct 11 14:35:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B790FC433FE for ; Mon, 11 Oct 2021 14:37:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A406661078 for ; Mon, 11 Oct 2021 14:37:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238922AbhJKOjC convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:02 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:59446 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238864AbhJKOjB (ORCPT ); Mon, 11 Oct 2021 10:39:01 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-269-7B8wISHsPaC8UiLWm0lnYA-1; Mon, 11 Oct 2021 10:36:59 -0400 X-MC-Unique: 7B8wISHsPaC8UiLWm0lnYA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B70611922962; Mon, 11 Oct 2021 14:36:57 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id E855019C59; Mon, 11 Oct 2021 14:36:53 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 05/19] trace/osnoise: Allow multiple instances of the same tracer Date: Mon, 11 Oct 2021 16:35:53 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Allow more than one instance of the same tracer. The workload will start when the first trace_array (instance) is registered and will stop when the last instance is unregistered. Osnoise and timerlat are still mutually exclusive because of the different behavior of the osnoise: tracepoints. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- kernel/trace/trace_osnoise.c | 106 ++++++++++++++++++++++++++--------- 1 file changed, 80 insertions(+), 26 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index a6d8f514bd7c..f913964f6861 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -64,6 +64,24 @@ static bool osnoise_has_registered_instances(void) list); } +/* + * osnoise_instance_registered - check if a tr is already registered + */ +static int osnoise_instance_registered(struct trace_array *tr) +{ + struct osnoise_instance *inst; + int found = 0; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + if (inst->tr == tr) + found = 1; + } + rcu_read_unlock(); + + return found; +} + /* * osnoise_register_instance - register a new trace instance * @@ -106,10 +124,8 @@ static void osnoise_unregister_instance(struct trace_array *tr) } rcu_read_unlock(); - if (!found) { - WARN("osnoise unregister did not find tr %s\n", tr->name); + if (!found) return; - } synchronize_rcu(); kfree(inst); @@ -1190,6 +1206,7 @@ static __always_inline void osnoise_stop_tracing(void) "stop tracing hit on cpu %d\n", smp_processor_id()); tracer_tracing_off(tr); + } rcu_read_unlock(); } @@ -2041,6 +2058,16 @@ static int osnoise_workload_start(void) { int retval; + /* + * Instances need to be registered after calling workload + * start. Hence, if there is already an instance, the + * workload was already registered. Otherwise, this + * code is on the way to register the first instance, + * and the workload will start. + */ + if (osnoise_has_registered_instances()) + return 0; + osn_var_reset_all(); retval = osnoise_hook_events(); @@ -2066,6 +2093,13 @@ static int osnoise_workload_start(void) */ static void osnoise_workload_stop(void) { + /* + * Instances need to be unregistered before calling + * stop. Hence, if there is a registered instance, more + * than one instance is running, and the workload will not + * yet stop. Otherwise, this code is on the way to disable + * the last instance, and the workload can stop. + */ if (osnoise_has_registered_instances()) return; @@ -2083,7 +2117,11 @@ static void osnoise_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_has_registered_instances()) + /* + * If the instance is already registered, there is no need to + * register it again. + */ + if (osnoise_instance_registered(tr)) return; retval = osnoise_workload_start(); @@ -2095,18 +2133,17 @@ static void osnoise_tracer_start(struct trace_array *tr) static void osnoise_tracer_stop(struct trace_array *tr) { - if (!osnoise_has_registered_instances()) - return; - osnoise_unregister_instance(tr); osnoise_workload_stop(); } static int osnoise_tracer_init(struct trace_array *tr) { - - /* Only allow one instance to enable this */ - if (osnoise_has_registered_instances()) + /* + * Only allow osnoise tracer if timerlat tracer is not running + * already. + */ + if (osnoise_data.timerlat_tracer) return -EBUSY; tr->max_latency = 0; @@ -2135,45 +2172,55 @@ static void timerlat_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_has_registered_instances()) + /* + * If the instance is already registered, there is no need to + * register it again. + */ + if (osnoise_instance_registered(tr)) return; - osnoise_data.timerlat_tracer = 1; - retval = osnoise_workload_start(); if (retval) - goto out_err; + pr_err(BANNER "Error starting timerlat tracer\n"); osnoise_register_instance(tr); return; -out_err: - pr_err(BANNER "Error starting timerlat tracer\n"); } static void timerlat_tracer_stop(struct trace_array *tr) { int cpu; - if (!osnoise_has_registered_instances()) - return; - - for_each_online_cpu(cpu) - per_cpu(per_cpu_osnoise_var, cpu).sampling = 0; + osnoise_unregister_instance(tr); - osnoise_tracer_stop(tr); + /* + * Instruct the threads to stop only if this is the last instance. + */ + if (!osnoise_has_registered_instances()) { + for_each_online_cpu(cpu) + per_cpu(per_cpu_osnoise_var, cpu).sampling = 0; + } - osnoise_data.timerlat_tracer = 0; + osnoise_workload_stop(); } static int timerlat_tracer_init(struct trace_array *tr) { - /* Only allow one instance to enable this */ - if (osnoise_has_registered_instances()) + /* + * Only allow timerlat tracer if osnoise tracer is not running already. + */ + if (osnoise_has_registered_instances() && !osnoise_data.timerlat_tracer) return -EBUSY; - tr->max_latency = 0; + /* + * If this is the first instance, set timerlat_tracer to block + * osnoise tracer start. + */ + if (!osnoise_has_registered_instances()) + osnoise_data.timerlat_tracer = 1; + tr->max_latency = 0; timerlat_tracer_start(tr); return 0; @@ -2182,6 +2229,13 @@ static int timerlat_tracer_init(struct trace_array *tr) static void timerlat_tracer_reset(struct trace_array *tr) { timerlat_tracer_stop(tr); + + /* + * If this is the last instance, reset timerlat_tracer allowing + * osnoise to be started. + */ + if (!osnoise_has_registered_instances()) + osnoise_data.timerlat_tracer = 0; } static struct tracer timerlat_tracer __read_mostly = { From patchwork Mon Oct 11 14:35:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 068ACC433EF for ; Mon, 11 Oct 2021 14:37:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E723A603E9 for ; Mon, 11 Oct 2021 14:37:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239029AbhJKOjJ convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:09 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:45803 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238606AbhJKOjH (ORCPT ); Mon, 11 Oct 2021 10:39:07 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-360-S5KDYCjiNCO33Sest1YstA-1; Mon, 11 Oct 2021 10:37:03 -0400 X-MC-Unique: S5KDYCjiNCO33Sest1YstA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 16C2EBD525; Mon, 11 Oct 2021 14:37:02 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D56419C59; Mon, 11 Oct 2021 14:36:57 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 06/19] rtla: Real-Time Linux Analysis tool Date: Mon, 11 Oct 2021 16:35:54 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results. rtla --help works and provide information about the available options. This is just the "main" and the Makefile, no function yet. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/Makefile | 59 ++++++++++++++++++++++++++++ tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+) create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/rtla.c diff --git a/tools/tracing/rtla/Makefile b/tools/tracing/rtla/Makefile new file mode 100644 index 000000000000..b3ddd5138829 --- /dev/null +++ b/tools/tracing/rtla/Makefile @@ -0,0 +1,59 @@ +NAME := rtla +VERSION := 0.1 + +INSTALL = install +CC := gcc +FOPTS := -flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \ + -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection +MOPTS := -m64 -mtune=generic +WOPTS := -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized + +TRACEFS_HEADERS := $$(pkg-config --cflags libtracefs) + +CFLAGS := -O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS) +LDFLAGS := -ggdb +LIBS := -ltracefs -ltraceevent -lpthread -lprocps + +SRC := $(wildcard src/*.c) +HDR := $(wildcard src/*.h) +OBJ := $(SRC:.c=.o) +DIRS := src +FILES := Makefile +CEXT := bz2 +TARBALL := $(NAME)-$(VERSION).tar.$(CEXT) +TAROPTS := -cvjf $(TARBALL) +BINDIR := /usr/bin +DATADIR := /usr/share +DOCDIR := $(DATADIR)/doc +MANDIR := $(DATADIR)/man +LICDIR := $(DATADIR)/licenses + +.PHONY: all tests + +all: rtla + +rtla: $(OBJ) + $(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS) + +static: $(OBJ) + $(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl + +.PHONY: install +install: + $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) + @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise + ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise + +.PHONY: clean tarball push +clean: + @test ! -f rtla || rm rtla + @test ! -f rtla-static || rm rtla-static + @test ! -f src/rtla.o || rm src/rtla.o + @test ! -f $(TARBALL) || rm -f $(TARBALL) + @rm -rf *~ $(OBJ) *.tar.$(CEXT) + +tarball: clean + rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION) + cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION) + tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION) + rm -rf $(NAME)-$(VERSION) diff --git a/tools/tracing/rtla/src/rtla.c b/tools/tracing/rtla/src/rtla.c new file mode 100644 index 000000000000..5f326dcebc6d --- /dev/null +++ b/tools/tracing/rtla/src/rtla.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include + +/* + * rtla_usage - print rtla usage + */ +static void rtla_usage(void) +{ + int i; + + char *msg[] = { + "", + "rtla version " VERSION, + "", + " usage: rtla COMMAND ...", + "", + " commands:", + "", + NULL, + }; + + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +/* + * run_command - try to run a rtla tool command + * + * It returns 0 if it fails. The tool's main will generally not + * return as they should call exit(). + */ +int run_command(int argc, char **argv, int start_position) +{ + return 0; +} + +int main(int argc, char *argv[]) +{ + int retval; + + /* is it an alias? */ + retval = run_command(argc, argv, 0); + if (retval) + exit(0); + + if (argc < 2) + goto usage; + + if (strcmp(argv[1], "-h") == 0) { + rtla_usage(); + exit(0); + } else if (strcmp(argv[1], "--help") == 0) { + rtla_usage(); + exit(0); + } + + retval = run_command(argc, argv, 1); + if (retval) + exit(0); + +usage: + rtla_usage(); + exit(1); +} From patchwork Mon Oct 11 14:35:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94520C433EF for ; Mon, 11 Oct 2021 14:37:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7BBC260BD3 for ; Mon, 11 Oct 2021 14:37:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239107AbhJKOjN convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:13 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:25442 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239953AbhJKOjM (ORCPT ); Mon, 11 Oct 2021 10:39:12 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-439-m6SeDcvDPCSU8goWvCIjDw-1; Mon, 11 Oct 2021 10:37:09 -0400 X-MC-Unique: m6SeDcvDPCSU8goWvCIjDw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3D4D51922960; Mon, 11 Oct 2021 14:37:07 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id D75F719C59; Mon, 11 Oct 2021 14:37:02 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 07/19] rtla: Helper functions for rtla Date: Mon, 11 Oct 2021 16:35:55 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org This is a set of utils and tracer helper functions. They are used by rtla mostly to parse config, display data and some trace operations that are not part of libtracefs (because they are only useful it for this case). Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-of-by: Daniel Bristot de Oliveira Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/trace.c | 220 +++++++++++++++++ tools/tracing/rtla/src/trace.h | 26 ++ tools/tracing/rtla/src/utils.c | 433 +++++++++++++++++++++++++++++++++ tools/tracing/rtla/src/utils.h | 55 +++++ 4 files changed, 734 insertions(+) create mode 100644 tools/tracing/rtla/src/trace.c create mode 100644 tools/tracing/rtla/src/trace.h create mode 100644 tools/tracing/rtla/src/utils.c create mode 100644 tools/tracing/rtla/src/utils.h diff --git a/tools/tracing/rtla/src/trace.c b/tools/tracing/rtla/src/trace.c new file mode 100644 index 000000000000..6ea148c75403 --- /dev/null +++ b/tools/tracing/rtla/src/trace.c @@ -0,0 +1,220 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include + +#include "trace.h" +#include "utils.h" + +/* + * enable_tracer_by_name - enable a tracer on the given instance + */ +int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer) +{ + enum tracefs_tracers t; + int retval; + + t = TRACEFS_TRACER_CUSTOM; + + debug_msg("enabling %s tracer\n", tracer); + + retval = tracefs_tracer_set(inst, t, tracer); + if (retval < 0) { + if (errno == ENODEV) + err_msg("tracer %s not found!\n", tracer); + + err_msg("failed to enable the tracer %s\n", tracer); + return -1; + } + + return 0; +} + +/* + * disable_tracer - set nop tracer to the insta + */ +void disable_tracer(struct tracefs_instance *inst) +{ + enum tracefs_tracers t = TRACEFS_TRACER_NOP; + int retval; + + retval = tracefs_tracer_set(inst, t); + if (retval < 0) + err_msg("oops, error disabling tracer\n"); + + return; +} + +/* + * create_instance - create a trace instance with *instance_name + */ +struct tracefs_instance *create_instance(char *instance_name) +{ + return tracefs_instance_create(instance_name); +} + +/* + * destroy_instance - remove a trace instance and free the data + */ +void destroy_instance(struct tracefs_instance *inst) +{ + tracefs_instance_destroy(inst); + tracefs_instance_free(inst); +} + +/* + * save_trace_to_file - save the trace output of the instance to the file + */ +int save_trace_to_file(struct tracefs_instance *inst, const char *filename) +{ + mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH; + const char *file = "trace"; + char *buffer[4096]; + int out_fd, in_fd; + int retval = -1; + + in_fd = tracefs_instance_file_open(inst, file, O_RDONLY); + if (in_fd < 0) { + err_msg("Failed to open trace file\n"); + return -1; + } + + out_fd = creat(filename, mode); + if (out_fd < 0) { + err_msg("Failed to create output file %s\n", filename); + goto out_close; + } + + do { + retval = read(in_fd, buffer, sizeof(buffer)); + if (read <= 0) + goto out_close; + + retval = write(out_fd, buffer, retval); + if (retval < 0) + goto out_close; + } while (retval > 0); + + retval = 0; + close(out_fd); +out_close: + close(in_fd); + return retval; +} + +struct tracefs_instance *stop_inst = NULL; +static void stop_trace_pipe(int sig) +{ + tracefs_trace_pipe_stop(stop_inst); +} + +/* + * save_trace_pipe_to_file - save the trace output from a pipe to the file + * + * it works until receiving a signal to stop. + */ +int save_trace_pipe_to_file(struct tracefs_instance *inst, const char *filename) +{ + mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH; + int retval; + int fd; + + fd = creat(filename, mode); + + stop_inst = inst; + signal(SIGINT, stop_trace_pipe); + retval = tracefs_trace_pipe_stream(fd, inst, SPLICE_F_NONBLOCK); + stop_inst = NULL; + + close(fd); + return retval; +} + +/* + * collect_registered_events - call the existing callback function for the event + * + * If an event has a registered callback function, call it. + * Otherwise, ignore the event. + */ +int +collect_registered_events(struct tep_event *event, struct tep_record *record, + int cpu, void *context) +{ + struct trace_instance *trace = context; + struct trace_seq *s = trace->seq; + + if (!event->handler) + return 0; + + event->handler(s, record, event, context); + + return 0; +} + +/* + * trace_instance_destroy - destroy and free a rtla trace instance + */ +void trace_instance_destroy(struct trace_instance *trace) +{ + if (trace->inst) { + disable_tracer(trace->inst); + destroy_instance(trace->inst); + } + + if (trace->seq) + free(trace->seq); + + if (trace->tep) + tep_free(trace->tep); +} + +/* + * trace_instance_init - create an rtla trace instance + * + * It is more than the tracefs instance, as it contains other + * things required for the tracing, such as the local events and + * a seq file. + * + * Note that the trace instance is returned disabled. This allows + * the tool to apply some other configs, like setting priority + * to the kernel threads, before starting generating trace entires. + */ +int trace_instance_init(struct trace_instance *trace, char *tool_name) +{ + trace->seq = calloc(1, sizeof(*trace->seq)); + if (!trace->seq) + goto out_err; + + trace_seq_init(trace->seq); + + trace->inst = create_instance(tool_name); + if (!trace->inst) + goto out_err; + + trace->tep = tracefs_local_events(NULL); + if (!trace->tep) + goto out_err; + + /* + * Let the main enable the record after setting some other + * things such as the priority of the tracer's threads. + */ + tracefs_trace_off(trace->inst); + + return 0; + +out_err: + trace_instance_destroy(trace); + return 1; +} + +/* + * trace_instance_start - start tracing a given rtla instance + */ +int trace_instance_start(struct trace_instance *trace) +{ + return tracefs_trace_on(trace->inst); +} diff --git a/tools/tracing/rtla/src/trace.h b/tools/tracing/rtla/src/trace.h new file mode 100644 index 000000000000..c961e23b2909 --- /dev/null +++ b/tools/tracing/rtla/src/trace.h @@ -0,0 +1,26 @@ +#include +#include + +struct trace_instance { + struct tracefs_instance *inst; + struct tep_handle *tep; + struct trace_seq *seq; +}; + +int trace_instance_init(struct trace_instance *trace, char *tool_name); +int trace_instance_start(struct trace_instance *trace); +void trace_instance_destroy(struct trace_instance *trace); + +struct trace_seq *get_trace_seq(void); +int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer); +void disable_tracer(struct tracefs_instance *inst); + +int enable_osnoise(struct trace_instance *trace); +int enable_timerlat(struct trace_instance *trace); + +struct tracefs_instance *create_instance(char *instance_name); +void destroy_instance(struct tracefs_instance *inst); + +int save_trace_to_file(struct tracefs_instance *inst, const char *filename); +int collect_registered_events(struct tep_event *tep, struct tep_record *record, + int cpu, void *context); diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c new file mode 100644 index 000000000000..20900e4bb194 --- /dev/null +++ b/tools/tracing/rtla/src/utils.c @@ -0,0 +1,433 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "utils.h" + +#define MAX_MSG_LENGTH 1024 +int config_debug = 0; + +/* + * err_msg - print an error message to the stderr + */ +void err_msg(const char *fmt, ...) +{ + char message[MAX_MSG_LENGTH]; + va_list ap; + + va_start(ap, fmt); + vsnprintf(message, sizeof(message), fmt, ap); + va_end(ap); + + fprintf(stderr, "%s", message); +} + +/* + * debug_msg - print a debug message to stderr if debug is set + */ +void debug_msg(const char *fmt, ...) +{ + char message[MAX_MSG_LENGTH]; + va_list ap; + + if (!config_debug) + return; + + va_start(ap, fmt); + vsnprintf(message, sizeof(message), fmt, ap); + va_end(ap); + + fprintf(stderr, "%s", message); +} + +/* + * get_long_from_str - get a long int from a string + */ +long get_long_from_str(char *start) +{ + long value; + char *end; + + errno = 0; + value = strtol(start, &end, 10); + if (errno || start == end) + return -1; + + return value; +} + +/* + * get_duration - fill output with a human readable duration since start_time + */ +void get_duration(time_t start_time, char *output, int output_size) +{ + time_t now = time(NULL); + struct tm *tm_info; + time_t duration; + + duration = difftime(now, start_time); + tm_info = localtime(&duration); + + snprintf(output, output_size, "%3d %02d:%02d:%02d", + tm_info->tm_yday, + tm_info->tm_hour - 1, + tm_info->tm_min, + tm_info->tm_sec); + + return; +} + +/* + * parse_cpu_list - parse a cpu_list filling a char vector with cpus set + * + * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set the char + * in the monitored_cpus. + * + * XXX: convert to a bitmask. + */ +int parse_cpu_list(char *cpu_list, char **monitored_cpus) +{ + char *mon_cpus; + const char *p; + int end_cpu; + int nr_cpus; + int cpu; + int i; + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + mon_cpus = malloc(nr_cpus * sizeof(char)); + memset(mon_cpus, 0, (nr_cpus * sizeof(char))); + + for (p = cpu_list; *p; ) { + cpu = atoi(p); + if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus) + goto err; + + while (isdigit(*p)) + p++; + if (*p == '-') { + p++; + end_cpu = atoi(p); + if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus) + goto err; + while (isdigit(*p)) + p++; + } else + end_cpu = cpu; + + if (cpu == end_cpu) { + debug_msg("cpu_list: adding cpu %d\n", cpu); + mon_cpus[cpu] = 1; + } else { + for (i = cpu; i <= end_cpu; i++) { + debug_msg("cpu_list: adding cpu %d\n", i); + mon_cpus[i] = 1; + } + } + + if (*p == ',') + p++; + } + + *monitored_cpus = mon_cpus; + + return 0; + +err: + debug_msg("Error parsing the cpu list %s", cpu_list); + return 1; +} + +/* + * parse_duration - parse duration with s/m/h/d suffix converting it to seconds + */ +long parse_seconds_duration(char *val) +{ + char *end; + long t; + t = strtol(val, &end, 10); + + if (end) { + switch (*end) { + case 's': + case 'S': + break; + case 'm': + case 'M': + t *= 60; + break; + case 'h': + case 'H': + t *= 60 * 60; + break; + + case 'd': + case 'D': + t *= 24 * 60 * 60; + break; + } + } + + return t; +} + +/* + * parse_ns_duration - parse duration with ns/us/ms/s converting it to nanoseconds + */ +long parse_ns_duration(char *val) +{ + char *end; + long t; + + t = strtol(val, &end, 10); + + if (end) { + if (!strncmp(end, "ns", 2)) { + return t; + } else if (!strncmp(end, "us", 2)) { + t *= 1000; + return t; + } else if (!strncmp(end, "ms", 2)) { + t *= 1000 * 1000; + return t; + } else if (!strncmp(end, "s", 1)) { + t *= 1000 * 1000 *100; + return t; + } + return -1; + } + + return t; +} + +/* + * This is a set of helper functions to use SCHED_DEADLINE. + */ +#ifdef __x86_64__ +# define __NR_sched_setattr 314 +# define __NR_sched_getattr 315 +#elif __i386__ +# define __NR_sched_setattr 351 +# define __NR_sched_getattr 352 +#elif __arm__ +# define __NR_sched_setattr 380 +# define __NR_sched_getattr 381 +#elif __aarch64__ +# define __NR_sched_setattr 274 +# define __NR_sched_getattr 275 +#elif __powerpc__ +# define __NR_sched_setattr 355 +# define __NR_sched_getattr 356 +#elif __s390x__ +# define __NR_sched_setattr 345 +# define __NR_sched_getattr 346 +#endif + +#define SCHED_DEADLINE 6 + +static inline int sched_setattr(pid_t pid, const struct sched_attr *attr, + unsigned int flags) { + return syscall(__NR_sched_setattr, pid, attr, flags); +} + +static inline int sched_getattr(pid_t pid, struct sched_attr *attr, + unsigned int size, unsigned int flags) +{ + return syscall (__NR_sched_getattr, pid , attr, size, flags); +} + +int __set_sched_attr(int pid, struct sched_attr *attr) +{ + int flags = 0; + int retval; + + retval = sched_setattr(pid, attr, flags); + if (retval < 0) { + err_msg("boost_with_deadline failed to boost pid %d: %s\n", + pid, strerror(errno)); + return 1; + } + + return 0; +} +/* + * set_comm_sched_attr - set sched params to threads starting with char *comm + * + * This function uses procps to list the currently running threads and then + * set the sched_attr *attr to the threads that start with char *comm. It is + * mainly used to set the priority to the kernel threads created by the + * tracers. + */ +int set_comm_sched_attr(const char *comm, struct sched_attr *attr) +{ + int flags = PROC_FILLCOM | PROC_FILLSTAT; + PROCTAB *ptp; + proc_t task; + int retval; + + ptp = openproc(flags); + if (!ptp) { + err_msg("error openproc()\n"); + return -ENOSYS; + } + + memset(&task, 0, sizeof (task)); + while (readproc(ptp, &task)) { + retval = strncmp(comm, task.cmd, strlen(comm)); + if (retval) + continue; + retval = __set_sched_attr(task.tid, attr); + if (retval) + goto out_err; + } + + closeproc(ptp); + return 0; + +out_err: + closeproc(ptp); + return 1; +} + +#define INVALID_VAL (~0L) +static long get_long_ns_after_colon(char *start) +{ + long val = INVALID_VAL; + + /* find the ":" */ + start = strstr(start, ":"); + if (!start) + return -1; + + /* skip ":" */ + start++; + val = parse_ns_duration(start); + + return val; +} + +static long get_long_after_colon(char *start) +{ + long val = INVALID_VAL; + + /* find the ":" */ + start = strstr(start, ":"); + if (!start) + return -1; + + /* skip ":" */ + start++; + val = get_long_from_str(start); + + return val; +} + +/* + * parse priority in the format: + * SCHED_OTHER: + * o: + * O: + * SCHED_RR: + * r: + * R: + * SCHED_FIFO: + * f: + * F: + * SCHED_DEADLINE: + * d:runtime:period + * R:runtime:period + */ +int parse_prio(char *arg, struct sched_attr *sched_param) +{ + long prio; + long runtime; + long period; + + memset(sched_param, 0, sizeof(*sched_param)); + sched_param->size = sizeof(*sched_param); + + switch (arg[0]) { + case 'd': + case 'D': + /* d:runtime:period */ + if (strlen(arg) < 4) + return -1; + + runtime = get_long_ns_after_colon(arg); + if (runtime == INVALID_VAL) + return -1; + + period = get_long_ns_after_colon(&arg[2]); + if (period == INVALID_VAL) + return -1; + + if (runtime > period) + return -1; + + sched_param->sched_policy = SCHED_DEADLINE; + sched_param->sched_runtime = runtime; + sched_param->sched_deadline = period; + sched_param->sched_period = period; + break; + case 'f': + case 'F': + /* f:prio */ + prio = get_long_after_colon(arg); + if (prio == INVALID_VAL) + return -1; + + if (prio < sched_get_priority_min(SCHED_FIFO)) + return -1; + if (prio > sched_get_priority_max(SCHED_FIFO)) + return -1; + + sched_param->sched_policy = SCHED_FIFO; + sched_param->sched_priority = prio; + break; + case 'r': + case 'R': + /* r:prio */ + prio = get_long_after_colon(arg); + if (prio == INVALID_VAL) + return -1; + + if (prio < sched_get_priority_min(SCHED_RR)) + return -1; + if (prio > sched_get_priority_max(SCHED_RR)) + return -1; + + sched_param->sched_policy = SCHED_RR; + sched_param->sched_priority = prio; + break; + case 'o': + case 'O': + /* o:prio */ + prio = get_long_after_colon(arg); + if (prio == INVALID_VAL) + return -1; + + if (prio < sched_get_priority_min(SCHED_OTHER)) + return -1; + if (prio > sched_get_priority_max(SCHED_OTHER)) + return -1; + + sched_param->sched_policy = SCHED_OTHER; + sched_param->sched_priority = prio; + break; + default: + return -1; + } + return 0; +} diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h new file mode 100644 index 000000000000..31a7ce1bfa9a --- /dev/null +++ b/tools/tracing/rtla/src/utils.h @@ -0,0 +1,55 @@ +#include +#include + +/* + * '18446744073709551615\0' + */ +#define BUFF_U64_STR_SIZE 24 + +#define container_of(ptr, type, member) ({ \ + const typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (type *)( (char *)__mptr - offsetof(type,member) );}) + +extern int config_debug; +void debug_msg(const char *fmt, ...); +void err_msg(const char *fmt, ...); + +long parse_seconds_duration(char *val); +void get_duration(time_t start_time, char *output, int output_size); + +int parse_cpu_list(char *cpu_list, char **monitored_cpus); +long get_long_from_str(char *start); + +static inline void +update_min(unsigned long long *a, unsigned long long *b) +{ + if (*a > *b) + *a = *b; +} + +static inline void +update_max(unsigned long long *a, unsigned long long *b) +{ + if (*a < *b) + *a = *b; +} + +static inline void +update_sum(unsigned long long *a, unsigned long long *b) +{ + *a += *b; +} + +struct sched_attr { + uint32_t size; + uint32_t sched_policy; + uint64_t sched_flags; + int32_t sched_nice; + uint32_t sched_priority; + uint64_t sched_runtime; + uint64_t sched_deadline; + uint64_t sched_period; +}; + +int parse_prio(char *arg, struct sched_attr *sched_param); +int set_comm_sched_attr(const char *comm, struct sched_attr *attr); From patchwork Mon Oct 11 14:35:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 635FCC433EF for ; Mon, 11 Oct 2021 14:37:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4BAFB606A5 for ; Mon, 11 Oct 2021 14:37:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242997AbhJKOjn convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:43 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:45088 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241834AbhJKOjQ (ORCPT ); Mon, 11 Oct 2021 10:39:16 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-427-5Sdszm0FMjaqCsBalpY4CQ-1; Mon, 11 Oct 2021 10:37:13 -0400 X-MC-Unique: 5Sdszm0FMjaqCsBalpY4CQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C6E18800482; Mon, 11 Oct 2021 14:37:11 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8D35A19723; Mon, 11 Oct 2021 14:37:07 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 08/19] rtla: Add osnoise tool Date: Mon, 11 Oct 2021 16:35:56 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included. The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored. It also includes some other helper functions for managing osnoise tracer sessions. With these bits and pieces in place, we can start adding some functionality to rtla. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise.c | 824 +++++++++++++++++++++++++++++++ tools/tracing/rtla/src/osnoise.h | 84 ++++ tools/tracing/rtla/src/rtla.c | 10 + 3 files changed, 918 insertions(+) create mode 100644 tools/tracing/rtla/src/osnoise.c create mode 100644 tools/tracing/rtla/src/osnoise.h diff --git a/tools/tracing/rtla/src/osnoise.c b/tools/tracing/rtla/src/osnoise.c new file mode 100644 index 000000000000..65577ff99b5d --- /dev/null +++ b/tools/tracing/rtla/src/osnoise.c @@ -0,0 +1,824 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "osnoise.h" +#include "utils.h" + +/* + * osnoise_set_cpus - configure osnoise to run on *cpus + * + * "osnoise/cpus" file is used to set the cpus in which osnoise/timerlat + * will run. This function opens this file, saves the current value, + * and set the cpus passed as argument. + */ +int osnoise_set_cpus(struct osnoise_context *context, char *cpus) +{ + char *osnoise_cpus = tracefs_get_tracing_file("osnoise/cpus"); + char curr_cpus[1024]; + int retval; + + context->cpus_fd = open(osnoise_cpus, O_RDWR); + if (!context->cpus_fd) + goto out_err; + + retval = read(context->cpus_fd, &curr_cpus, sizeof(curr_cpus)); + if (!retval) + goto out_close; + + context->orig_cpus = strdup(curr_cpus); + if (!context->orig_cpus) + goto out_err; + + retval = write(context->cpus_fd, cpus, strlen(cpus) + 1); + if (!retval) + goto out_err; + + tracefs_put_tracing_file(osnoise_cpus); + + return 0; + +out_close: + close(context->cpus_fd); + context->cpus_fd = -1; +out_err: + tracefs_put_tracing_file(osnoise_cpus); + return 1; +} + +/* + * osnoise_restore_cpus - restore the original "osnoise/cpus" + * + * osnoise_set_cpus() saves the original data for the "osnoise/cpus" + * file. This function restore the original config it was previously + * modified. + */ +void osnoise_restore_cpus(struct osnoise_context *context) +{ + int retval; + + if (!context->orig_cpus) + return; + + retval = write(context->cpus_fd, context->orig_cpus, strlen(context->orig_cpus)); + if (!retval) + err_msg("could not restore original osnoise cpus\n"); + + free(context->orig_cpus); + return; +} + +/* + * osnoise_get_runtime - return the original "osnoise/runtime_us" value + * + * It also saves the value to be restored. + */ +unsigned long long osnoise_get_runtime(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *runtime_path; + int retval; + + if (context->runtime_us) + return context->runtime_us; + + runtime_path = tracefs_get_tracing_file("osnoise/runtime_us"); + + context->runtime_fd = open(runtime_path, O_RDWR); + if (!context->runtime_fd) + goto out_err; + + retval = read(context->runtime_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->runtime_us = get_long_from_str(buffer); + if (!context->runtime_us) + goto out_close; + + context->orig_runtime_us = context->runtime_us; + + tracefs_put_tracing_file(runtime_path); + + return context->runtime_us; + +out_close: + close(context->runtime_fd); + context->runtime_fd = -1; +out_err: + tracefs_put_tracing_file(runtime_path); + return 0; +} + +/* + * osnoise_get_period - return the original "osnoise/period_us" value + * + * It also saves the value to be restored. + */ +unsigned long long osnoise_get_period(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *period_path; + int retval; + + if (context->period_us) + return context->period_us; + + period_path = tracefs_get_tracing_file("osnoise/period_us"); + + context->period_fd = open(period_path, O_RDWR); + if (!context->period_fd) + goto out_err; + + retval = read(context->period_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->period_us = get_long_from_str(buffer); + if (!context->period_us) + goto out_close; + + context->orig_period_us = context->period_us; + + tracefs_put_tracing_file(period_path); + + return context->period_us; + +out_close: + close(context->period_fd); + context->period_fd = -1; +out_err: + tracefs_put_tracing_file(period_path); + return 0; +} + +static int __osnoise_write_runtime(struct osnoise_context *context, + unsigned long long runtime) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->runtime_fd == -1) { + err_msg("tried to write osnoise runtime before getting it\n"); + return 0; + } + + snprintf(buffer, sizeof(buffer), "%llu\n", runtime); + + retval = write(context->runtime_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->runtime_us = runtime; + return 0; +} + +static int __osnoise_write_period(struct osnoise_context *context, + unsigned long long period) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->period_fd == -1) + return 0; + + snprintf(buffer, sizeof(buffer), "%llu\n", period); + + retval = write(context->period_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->period_us = period; + return 0; +} + +/* + * osnoise_set_runtime_period - set osnoise runtime and period + * + * Osnoise's runtime and period are related as runtime <= period. + * Thus, this function saves the original values, and then tries + * to set the runtime and period if they are != 0. + */ +int osnoise_set_runtime_period(struct osnoise_context *context, + unsigned long long runtime, + unsigned long long period) +{ + unsigned long long curr_runtime_us; + unsigned long long curr_period_us; + int retval; + + if (!period && !runtime) + return 0; + + curr_runtime_us = osnoise_get_runtime(context); + curr_period_us = osnoise_get_period(context); + + if (!curr_period_us || !curr_runtime_us) + return -1; + + if (!period) { + if (runtime > curr_period_us) + return -1; + return __osnoise_write_runtime(context, runtime); + } else if (!runtime) { + if (period < curr_runtime_us) + return -1; + return __osnoise_write_period(context, period); + } + + if (runtime > curr_period_us) { + retval = __osnoise_write_period(context, period); + if (retval) + return -1; + retval = __osnoise_write_runtime(context, runtime); + if (retval) + return -1; + } else { + retval = __osnoise_write_runtime(context, runtime); + if (retval) + return -1; + retval = __osnoise_write_period(context, period); + if (retval) + return -1; + } + + return 0; +} + +/* + * osnoise_restore_runtime_period - restore the original runtime and period + */ +void osnoise_restore_runtime_period(struct osnoise_context *context) +{ + unsigned long long runtime = context->orig_runtime_us; + unsigned long long period = context->orig_period_us; + int retval; + + if (context->runtime_fd < 0 && context->period_fd < 0) + return; + + retval = osnoise_set_runtime_period(context, runtime, period); + if (retval) + err_msg("Could not restore original osnoise runtime/period\n"); +} + +/* + * osnoise_get_stop_us - read and save the original "stop_tracing_us" + */ +static long long +osnoise_get_stop_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *stop_path; + int retval; + + if (context->stop_us_fd > 0) + return context->stop_us; + + stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_us"); + + context->stop_us_fd = open(stop_path, O_RDWR); + if (!context->stop_us_fd) + goto out_err; + + retval = read(context->stop_us_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->stop_us = get_long_from_str(buffer); + context->orig_stop_us = context->stop_us; + + tracefs_put_tracing_file(stop_path); + + return context->stop_us; + +out_close: + close(context->stop_us_fd); + context->stop_us_fd = -1; +out_err: + tracefs_put_tracing_file(stop_path); + return -1; +} + +/* + * osnoise_set_stop_us - set "stop_tracing_us" + */ +int osnoise_set_stop_us(struct osnoise_context *context, long long stop_us) +{ + long long curr_stop_us = osnoise_get_stop_us(context); + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (curr_stop_us < 0) + return -1; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", stop_us); + + retval = write(context->stop_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->stop_us = stop_us; + + return 0; +} + +/* + * osnoise_restore_stop_us - restore the original stop_tracing_us + */ +void osnoise_restore_stop_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->stop_us_fd < 0) + return; + + if (context->orig_stop_us == context->stop_us) + return; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_stop_us); + + retval = write(context->stop_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + err_msg("Could not restore original osnoise stop_us\n"); + + return; +} + +/* + * osnoise_get_stop_us - read and save the original "stop_tracing_total_us" + */ +static long long +osnoise_get_stop_total_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *stop_path; + int retval; + + if (context->stop_total_us_fd > 0) + return context->stop_total_us; + + stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_total_us"); + + context->stop_total_us_fd = open(stop_path, O_RDWR); + if (!context->stop_total_us_fd) + goto out_err; + + retval = read(context->stop_total_us_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->stop_total_us = get_long_from_str(buffer); + context->orig_stop_total_us = context->stop_total_us; + + tracefs_put_tracing_file(stop_path); + + return context->stop_total_us; + +out_close: + close(context->stop_total_us_fd); + context->stop_total_us_fd = -1; +out_err: + tracefs_put_tracing_file(stop_path); + return -1; +} + +/* + * osnoise_set_stop_us - set "stop_tracing_total_us" + */ +int osnoise_set_stop_total_us(struct osnoise_context *context, long long stop_total_us) +{ + long long curr_stop_total_us = osnoise_get_stop_total_us(context); + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (curr_stop_total_us < 0) + return -1; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", stop_total_us); + + retval = write(context->stop_total_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->stop_total_us = stop_total_us; + + return 0; +} + +/* + * osnoise_restore_stop_us - restore the original stop_tracing_us + */ +void osnoise_restore_stop_total_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->stop_total_us_fd < 0) + return; + + if (context->orig_stop_total_us == context->stop_total_us) + return; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_stop_total_us); + + retval = write(context->stop_total_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + err_msg("Could not restore original osnoise stop_total_us\n"); + return; +} + +/* + * osnoise_get_timerlat_period_us - read and save the original "timerlat_period_us" + */ +static long long +osnoise_get_timerlat_period_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *stop_path; + int retval; + + if (context->timerlat_period_us_fd > 0) + return context->timerlat_period_us; + + stop_path = tracefs_get_tracing_file("osnoise/timerlat_period_us"); + + context->timerlat_period_us_fd = open(stop_path, O_RDWR); + if (!context->timerlat_period_us_fd) + goto out_err; + + retval = read(context->timerlat_period_us_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->timerlat_period_us = get_long_from_str(buffer); + context->orig_timerlat_period_us = context->timerlat_period_us; + + tracefs_put_tracing_file(stop_path); + + return context->timerlat_period_us; + +out_close: + close(context->timerlat_period_us_fd); + context->timerlat_period_us_fd = -1; +out_err: + tracefs_put_tracing_file(stop_path); + return -1; +} + +/* + * osnoise_set_timerlat_period_us - set "timerlat_period_us" + */ +int osnoise_set_timerlat_period_us(struct osnoise_context *context, long long timerlat_period_us) +{ + long long curr_timerlat_period_us = osnoise_get_timerlat_period_us(context); + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (curr_timerlat_period_us < 0) + return -1; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", timerlat_period_us); + + retval = write(context->timerlat_period_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->timerlat_period_us = timerlat_period_us; + + return 0; +} + +/* + * osnoise_restore_timerlat_period_us - restore "timerlat_period_us" + */ +void osnoise_restore_timerlat_period_us(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->timerlat_period_us_fd < 0) + return; + + if (context->orig_timerlat_period_us == context->timerlat_period_us) + return; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_timerlat_period_us); + + retval = write(context->timerlat_period_us_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + err_msg("Could not restore original osnoise timerlat_period_us\n"); + + return; +} + +/* + * osnoise_get_print_stack - read and save the original "print_stack" + */ +static long long +osnoise_get_print_stack(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + char *stop_path; + int retval; + + if (context->print_stack_fd > 0) + return context->print_stack; + + stop_path = tracefs_get_tracing_file("osnoise/print_stack"); + + context->print_stack_fd = open(stop_path, O_RDWR); + if (!context->print_stack_fd) + goto out_err; + + retval = read(context->print_stack_fd, &buffer, sizeof(buffer)); + if (!retval) + goto out_close; + + context->print_stack = get_long_from_str(buffer); + context->orig_print_stack = context->print_stack; + + tracefs_put_tracing_file(stop_path); + + return context->print_stack; + +out_close: + close(context->print_stack_fd); + context->print_stack_fd = -1; +out_err: + tracefs_put_tracing_file(stop_path); + return -1; +} + +/* + * osnoise_set_print_stack - set "print_stack" + */ +int osnoise_set_print_stack(struct osnoise_context *context, long long print_stack) +{ + long long curr_print_stack = osnoise_get_print_stack(context); + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (curr_print_stack < 0) + return -1; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", print_stack); + + retval = write(context->print_stack_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + return -1; + + context->print_stack = print_stack; + + return 0; +} + +/* + * osnoise_restore_print_stack - restore the original "print_stack" + */ +void osnoise_restore_print_stack(struct osnoise_context *context) +{ + char buffer[BUFF_U64_STR_SIZE]; + int retval; + + if (context->print_stack_fd < 0) + return; + + if (context->orig_print_stack == context->print_stack) + return; + + snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_print_stack); + + retval = write(context->print_stack_fd, buffer, strlen(buffer) + 1); + if (retval < 0) + err_msg("Could not restore original osnoise print_stack\n"); + + return; +} + +/* + * enable_osnoise - enable osnoise tracer in the trace_instance + */ +int enable_osnoise(struct trace_instance *trace) +{ + return enable_tracer_by_name(trace->inst, "osnoise"); +} + +/* + * enable_timerlat - enable timerlat tracer in the trace_instance + */ +int enable_timerlat(struct trace_instance *trace) +{ + return enable_tracer_by_name(trace->inst, "timerlat"); +} + +enum { + FLAG_CONTEXT_NEWLY_CREATED = (1 << 0), + FLAG_CONTEXT_DELETED = (1 << 1), +}; + +/* + * osnoise_get_context - increase the usage of a context and return it + */ +int osnoise_get_context(struct osnoise_context *context) +{ + int ret; + + pthread_mutex_lock(&context->lock); + if (context->flags & FLAG_CONTEXT_DELETED) { + ret = -1; + } else { + context->ref++; + ret = 0; + } + pthread_mutex_unlock(&context->lock); + + return ret; +} + +/* + * osnoise_context_alloc - alloc an osnoise_context + * + * The osnoise context contains the information of the "osnoise/" configs. + * It is used to set and restore the config. + */ +struct osnoise_context *osnoise_context_alloc(void) +{ + struct osnoise_context *context; + + context = calloc(1, sizeof(*context)); + if (!context) + goto out_err; + + if (pthread_mutex_init(&context->lock, NULL) < 0) + goto out_err; + + context->cpus_fd = -1; + context->runtime_fd = -1; + context->period_fd = -1; + context->stop_us_fd = -1; + context->stop_total_us_fd = -1; + + osnoise_get_context(context); + + return context; +out_err: + if (context) { + free(context); + } + return NULL; +} + +/* + * osnoise_put_context - put the osnoise_put_context + * + * If there is no other user for the context, the original data + * is restored. + */ +void osnoise_put_context(struct osnoise_context *context) +{ + pthread_mutex_lock(&context->lock); + if (--context->ref < 1) + context->flags |= FLAG_CONTEXT_DELETED; + pthread_mutex_unlock(&context->lock); + + if (!(context->flags & FLAG_CONTEXT_DELETED)) + return; + + osnoise_restore_cpus(context); + if (context->cpus_fd >= 0) + close(context->cpus_fd); + + osnoise_restore_runtime_period(context); + if (context->runtime_fd >= 0) + close(context->runtime_fd); + if (context->period_fd >= 0) + close(context->period_fd); + + osnoise_restore_stop_us(context); + osnoise_restore_stop_total_us(context); + osnoise_restore_print_stack(context); + osnoise_restore_timerlat_period_us(context); + + pthread_mutex_destroy(&context->lock); + free(context); +} + +/* + * osnoise_destroy_tool - disable trace, restore configs and free data + */ +void osnoise_destroy_tool(struct osnoise_tool *top) +{ + trace_instance_destroy(&top->trace); + + if (top->context) + osnoise_put_context(top->context); + + free(top); +} + +/* + * osnoise_init_tool - init an osnoise tool + * + * It allocs data, create a context to store data and + * creates a new trace instance for the tool. + */ +struct osnoise_tool *osnoise_init_tool(char *tool_name) +{ + struct osnoise_tool *top; + int retval; + + top = calloc(1, sizeof(*top)); + if (!top) + return NULL; + + top->context = osnoise_context_alloc(); + if (!top->context) + goto out_err; + + retval = trace_instance_init(&top->trace, tool_name); + if (retval) + goto out_err; + + return top; +out_err: + osnoise_destroy_tool(top); + return NULL; +} + +/* + * osnoise_init_trace_tool - init a tracer instance to trace osnoise events + */ +struct osnoise_tool *osnoise_init_trace_tool(char *tracer) +{ + struct osnoise_tool *trace; + int retval; + + trace = osnoise_init_tool("osnoise_trace"); + if (!trace) + return NULL; + + retval = tracefs_event_enable(trace->trace.inst, "osnoise", NULL); + if (retval < 0 && !errno) { + err_msg("Could not find osnoise events\n"); + goto out_err; + } + + retval = enable_tracer_by_name(trace->trace.inst, tracer); + if (retval) { + err_msg("Could not enable osnoiser tracer for tracing\n"); + goto out_err; + } + + return trace; +out_err: + osnoise_destroy_tool(trace); + return NULL; +} + +static void osnoise_usage(void) +{ + int i; + + char *msg[] = { + "", + "osnoise version " VERSION, + "", + " usage: [rtla] osnoise [MODE] ...", + "", + " modes:", + "", + NULL, + }; + + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +int osnoise_main(int argc, char *argv[]) +{ + if (argc <= 1) + goto usage; + + if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) { + osnoise_usage(); + exit(0); + } + +usage: + osnoise_usage(); + exit(1); +} diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h new file mode 100644 index 000000000000..f1dd9a2bcbf8 --- /dev/null +++ b/tools/tracing/rtla/src/osnoise.h @@ -0,0 +1,84 @@ +#include "trace.h" + +/* + * osnoise_context - read, store, write, restore osnoise configs. + */ +struct osnoise_context { + int flags; + int ref; + pthread_mutex_t lock; + + int cpus_fd; + int runtime_fd; + int period_fd; + int stop_us_fd; + int stop_total_us_fd; + int timerlat_period_us_fd; + int print_stack_fd; + + char *curr_cpus; + char *orig_cpus; + + unsigned long long orig_runtime_us; + unsigned long long runtime_us; + + unsigned long long orig_period_us; + unsigned long long period_us; + + long long orig_stop_us; + long long stop_us; + + long long orig_stop_total_us; + long long stop_total_us; + + long long orig_timerlat_period_us; + long long timerlat_period_us; + + long long orig_print_stack; + long long print_stack; +}; + +struct osnoise_context *osnoise_context_alloc(void); +int osnoise_get_context(struct osnoise_context *context); +void osnoise_put_context(struct osnoise_context *context); + +int osnoise_set_cpus(struct osnoise_context *context, char *cpus); +void osnoise_restore_cpus(struct osnoise_context *context); + +int osnoise_set_runtime_period(struct osnoise_context *context, + unsigned long long runtime, + unsigned long long period); +void osnoise_restore_runtime_period(struct osnoise_context *context); + +int osnoise_set_stop_us(struct osnoise_context *context, + long long stop_us); +void osnoise_restore_stop_us(struct osnoise_context *context); + +int osnoise_set_stop_total_us(struct osnoise_context *context, + long long stop_total_us); +void osnoise_restore_stop_total_us(struct osnoise_context *context); + +int osnoise_set_timerlat_period_us(struct osnoise_context *context, + long long timerlat_period_us); +void osnoise_restore_timerlat_period_us(struct osnoise_context *context); + +void osnoise_restore_print_stack(struct osnoise_context *context); +int osnoise_set_print_stack(struct osnoise_context *context, + long long print_stack); + +/* + * osnoise_tool - osnoise based tool definition. + */ +struct osnoise_tool { + struct trace_instance trace; + struct osnoise_context *context; + void *data; + void *params; + time_t start_time; +}; + +void osnoise_destroy_tool(struct osnoise_tool *top); +struct osnoise_tool *osnoise_init_tool(char *tool_name); +struct osnoise_tool *osnoise_init_trace_tool(char *tracer); + +int osnoise_main(int argc, char **argv); diff --git a/tools/tracing/rtla/src/rtla.c b/tools/tracing/rtla/src/rtla.c index 5f326dcebc6d..9427cab04724 100644 --- a/tools/tracing/rtla/src/rtla.c +++ b/tools/tracing/rtla/src/rtla.c @@ -8,6 +8,8 @@ #include #include +#include "osnoise.h" + /* * rtla_usage - print rtla usage */ @@ -22,6 +24,7 @@ static void rtla_usage(void) " usage: rtla COMMAND ...", "", " commands:", + " osnoise - gives information about the operating system noise (osnoise)", "", NULL, }; @@ -39,7 +42,14 @@ static void rtla_usage(void) */ int run_command(int argc, char **argv, int start_position) { + if (strcmp(argv[start_position], "osnoise") == 0) { + osnoise_main(argc-start_position, &argv[start_position]); + goto ran; + } + return 0; +ran: + return 1; } int main(int argc, char *argv[]) From patchwork Mon Oct 11 14:35:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D564FC4332F for ; Mon, 11 Oct 2021 14:37:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C059F60BD3 for ; Mon, 11 Oct 2021 14:37:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242323AbhJKOjs convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:48 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:43381 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242285AbhJKOjV (ORCPT ); Mon, 11 Oct 2021 10:39:21 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-128-MMC4vfAfNWKWCebZmUixog-1; Mon, 11 Oct 2021 10:37:18 -0400 X-MC-Unique: MMC4vfAfNWKWCebZmUixog-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3F59B8042C3; Mon, 11 Oct 2021 14:37:16 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2A31519D9B; Mon, 11 Oct 2021 14:37:11 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 09/19] rtla/osnoise: Add osnoise top mode Date: Mon, 11 Oct 2021 16:35:57 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The rtla osnoise tool is an interface for the osnoise tracer. The osnoise tracer dispatches a kernel thread per-cpu. These threads read the time in a loop while with preemption, softirqs and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise threads take note of the entry and exit point of any source of interferences, increasing a per-cpu interference counter. The osnoise tracer also saves an interference counter for each source of interference. The rtla osnoise top mode displays information about the periodic summary from the osnoise tracer. One example of rtla osnoise top output is: [root@alien ~]# rtla osnoise top -c 0-3 -d 1m -q -r 900000 -P F:1 Operating System Noise duration: 0 00:01:00 | time is in us CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI IRQ Softirq Thread 0 #58 52200000 1031 99.99802 91 60 0 0 52285 0 101 1 #59 53100000 5 99.99999 5 5 0 9 53122 0 18 2 #59 53100000 7 99.99998 7 7 0 8 53115 0 18 3 #59 53100000 8274 99.98441 277 23 0 9 53778 0 660 "rtla osnoise top --help" works and provide information about the available options. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise.c | 20 +- tools/tracing/rtla/src/osnoise.h | 1 + tools/tracing/rtla/src/osnoise_top.c | 566 +++++++++++++++++++++++++++ 3 files changed, 586 insertions(+), 1 deletion(-) create mode 100644 tools/tracing/rtla/src/osnoise_top.c diff --git a/tools/tracing/rtla/src/osnoise.c b/tools/tracing/rtla/src/osnoise.c index 65577ff99b5d..7b66b9b6ba9a 100644 --- a/tools/tracing/rtla/src/osnoise.c +++ b/tools/tracing/rtla/src/osnoise.c @@ -799,7 +799,9 @@ static void osnoise_usage(void) " usage: [rtla] osnoise [MODE] ...", "", " modes:", + " top - prints the summary from osnoise tracer", "", + "if no MODE is given, the top mode is called, passing the arguments", NULL, }; @@ -810,12 +812,28 @@ static void osnoise_usage(void) int osnoise_main(int argc, char *argv[]) { - if (argc <= 1) + if (argc == 0) goto usage; + /* + * if osnoise was called without any argument, run the + * default cmdline. + */ + if (argc == 1) { + osnoise_top_main(argc, argv); + exit(0); + } + if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) { osnoise_usage(); exit(0); + } else if (strncmp(argv[1], "-", 1) == 0) { + /* the user skipped the tool, call the default one */ + osnoise_top_main(argc, argv); + exit(0); + } else if (strcmp(argv[1], "top") == 0) { + osnoise_top_main(argc-1, &argv[1]); + exit(0); } usage: diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h index f1dd9a2bcbf8..b4c1369d743d 100644 --- a/tools/tracing/rtla/src/osnoise.h +++ b/tools/tracing/rtla/src/osnoise.h @@ -81,4 +81,5 @@ void osnoise_destroy_tool(struct osnoise_tool *top); struct osnoise_tool *osnoise_init_tool(char *tool_name); struct osnoise_tool *osnoise_init_trace_tool(char *tracer); +int osnoise_top_main(int argc, char **argv); int osnoise_main(int argc, char **argv); diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c new file mode 100644 index 000000000000..e1a9d0d64440 --- /dev/null +++ b/tools/tracing/rtla/src/osnoise_top.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "osnoise.h" +#include "utils.h" + +/* + * osnoise top parameters + */ +struct osnoise_top_params { + char *cpus; + char *monitored_cpus; + char *trace_output; + unsigned long long runtime; + unsigned long long period; + long long stop_us; + long long stop_total_us; + int sleep_time; + int duration; + int quiet; + int set_sched; + struct sched_attr sched_param; +}; + +struct osnoise_top_cpu { + unsigned long long sum_runtime; + unsigned long long sum_noise; + unsigned long long max_noise; + unsigned long long max_sample; + + unsigned long long hw_count; + unsigned long long nmi_count; + unsigned long long irq_count; + unsigned long long softirq_count; + unsigned long long thread_count; + + int sum_cycles; +}; + +struct osnoise_top_data { + struct osnoise_top_cpu *cpu_data; + int nr_cpus; +}; + +/* + * osnoise_free_top - free runtime data + */ +static void +osnoise_free_top(struct osnoise_top_data *data) +{ + free(data->cpu_data); + free(data); +} + +/* + * osnoise_alloc_histogram - alloc runtime data + */ +static struct osnoise_top_data *osnoise_alloc_top(int nr_cpus) +{ + struct osnoise_top_data *data; + + data = calloc(1, sizeof(*data)); + if (!data) + return NULL; + + data->nr_cpus = nr_cpus; + + /* one set of histograms per CPU */ + data->cpu_data = calloc(1, sizeof(*data->cpu_data) * nr_cpus); + if (!data->cpu_data) + goto cleanup; + + return data; + +cleanup: + osnoise_free_top(data); + return NULL; +} + +/* + * osnoise_top_handler - this is the handler for osnoise tracer events + */ +static int +osnoise_top_handler(struct trace_seq *s, struct tep_record *record, + struct tep_event *event, void *context) +{ + struct trace_instance *trace = context; + struct osnoise_tool *tool; + unsigned long long val; + struct osnoise_top_cpu *cpu_data; + struct osnoise_top_data *data; + int cpu = record->cpu; + + tool = container_of(trace, struct osnoise_tool, trace); + + data = tool->data; + cpu_data = &data->cpu_data[cpu]; + + cpu_data->sum_cycles++; + + tep_get_field_val(s, event, "runtime", record, &val, 1); + update_sum(&cpu_data->sum_runtime, &val); + + tep_get_field_val(s, event, "noise", record, &val, 1); + update_max(&cpu_data->max_noise, &val); + update_sum(&cpu_data->sum_noise, &val); + + tep_get_field_val(s, event, "max_sample", record, &val, 1); + update_max(&cpu_data->max_sample, &val); + + tep_get_field_val(s, event, "hw_count", record, &val, 1); + update_sum(&cpu_data->hw_count, &val); + + tep_get_field_val(s, event, "nmi_count", record, &val, 1); + update_sum(&cpu_data->nmi_count, &val); + + tep_get_field_val(s, event, "irq_count", record, &val, 1); + update_sum(&cpu_data->irq_count, &val); + + tep_get_field_val(s, event, "softirq_count", record, &val, 1); + update_sum(&cpu_data->softirq_count, &val); + + tep_get_field_val(s, event, "thread_count", record, &val, 1); + update_sum(&cpu_data->thread_count, &val); + + return 0; +} + +/* + * osnoise_top_header - print the header of the tool output + */ +static void osnoise_top_header(struct osnoise_tool *top) +{ + struct trace_seq *s = top->trace.seq; + char duration[26]; + + get_duration(top->start_time, duration, sizeof(duration)); + + trace_seq_printf(s, "\033[2;37;40m"); + trace_seq_printf(s, " Operating System Noise"); + trace_seq_printf(s, " "); + trace_seq_printf(s, " "); + trace_seq_printf(s, "\033[0;0;0m"); + trace_seq_printf(s, "\n"); + + trace_seq_printf(s, "duration: %9s | time is in us\n" , duration); + + trace_seq_printf(s, "\033[2;30;47m"); + trace_seq_printf(s, "CPU Period Runtime "); + trace_seq_printf(s, " Noise "); + trace_seq_printf(s, " %% CPU Aval "); + trace_seq_printf(s, " Max Noise Max Single "); + trace_seq_printf(s, " HW NMI IRQ Softirq Thread"); + trace_seq_printf(s, "\033[0;0;0m"); + trace_seq_printf(s, "\n"); +} + +/* + * clear_terminal - clears the output terminal + */ +static void clear_terminal(struct trace_seq *seq) +{ + if (!config_debug) + trace_seq_printf(seq, "\033c"); +} + +/* + * osnoise_top_print - prints the output of a given CPU + */ +static void osnoise_top_print(struct osnoise_tool *tool, int cpu) +{ + struct trace_seq *s = tool->trace.seq; + struct osnoise_top_cpu *cpu_data; + struct osnoise_top_data *data; + int percentage; + int decimal; + + data = tool->data; + cpu_data = &data->cpu_data[cpu]; + + if (!cpu_data->sum_runtime) + return; + + percentage = ((cpu_data->sum_runtime - cpu_data->sum_noise) * 10000000) + / cpu_data->sum_runtime; + decimal = percentage % 100000; + percentage = percentage / 100000; + + trace_seq_printf(s, "%3d #%-6d %12llu ", cpu, cpu_data->sum_cycles, cpu_data->sum_runtime); + trace_seq_printf(s, "%12llu ", cpu_data->sum_noise); + trace_seq_printf(s, " %3d.%05d", percentage, decimal); + trace_seq_printf(s, "%12llu %12llu", cpu_data->max_noise, cpu_data->max_sample); + + trace_seq_printf(s, "%12llu ", cpu_data->hw_count); + trace_seq_printf(s, "%12llu ", cpu_data->nmi_count); + trace_seq_printf(s, "%12llu ", cpu_data->irq_count); + trace_seq_printf(s, "%12llu ", cpu_data->softirq_count); + trace_seq_printf(s, "%12llu\n", cpu_data->thread_count); +} + +/* + * osnoise_print_stats - print data for all cpus + */ +static void +osnoise_print_stats(struct osnoise_top_params *params, struct osnoise_tool *top) +{ + struct trace_instance *trace = &top->trace; + static int nr_cpus = -1; + int i; + + if (nr_cpus == -1) + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + if (!params->quiet) + clear_terminal(trace->seq); + + osnoise_top_header(top); + + for (i = 0; i < nr_cpus; i++) { + if (params->cpus && !params->monitored_cpus[i]) + continue; + osnoise_top_print(top, i); + } + + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); +} + +/* + * osnoise_top_usage - prints osnoise top usage message + */ +void osnoise_top_usage(char *usage) +{ + int i; + + char *msg[] = { + "" + " usage: rtla osnoise [top] [-h] [-q] [-p us] [-r us] [-s us] [-S us] [-T[=file]] \\", + " [-c cpu-list] -P priority", + "", + " -h/--help: print this menu", + " -p/--period us: osnoise period in us", + " -r/--runtime us: osnoise runtime in us", + " -s/--stop us: stop trace if a single sample is higher than the argument in us", + " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", + " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -d/--duration time[s|m|h|d]: duration of the session", + " -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]", + " -q/--quiet print only a summary at the end", + " -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters", + " o:prio - use SCHED_OTHER with prio", + " r:prio - use SCHED_RR with prio", + " f:prio - use SCHED_FIFO with prio", + " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", + " in nanoseconds", + NULL, + }; + + if (usage) + fprintf(stderr, "%s\n", usage); + + fprintf(stderr, "rtla osnoise top: display a per-cpu summary of the os noise (version %s)\n", VERSION); + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +/* + * osnoise_top_parse_args - allocs, parse and fill the cmd line parameters + */ +struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv) +{ + struct osnoise_top_params *params; + int retval; + int c; + + params = calloc(1, sizeof(*params)); + if (!params) + exit(1); + + while (1) { + static struct option long_options[] = { + {"cpus", required_argument, 0, 'c'}, + {"debug", no_argument, 0, 'D'}, + {"duration", required_argument, 0, 'd'}, + {"help", no_argument, 0, 'h'}, + {"period", required_argument, 0, 'p'}, + {"priority", required_argument, 0, 'P'}, + {"quiet", no_argument, 0, 'q'}, + {"runtime", required_argument, 0, 'r'}, + {"stop", required_argument, 0, 's'}, + {"stop-total", required_argument, 0, 'S'}, + {"trace", optional_argument, 0, 't'}, + {0, 0, 0, 0} + }; + + /* getopt_long stores the option index here. */ + int option_index = 0; + + c = getopt_long(argc, argv, "c:d:Dhp:P:qr:s:S:t::", + long_options, &option_index); + + /* Detect the end of the options. */ + if (c == -1) + break; + + switch (c) { + case 'c': + retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + if (retval) + osnoise_top_usage("\nInvalid -c cpu list\n"); + params->cpus = optarg; + debug_msg("-c for %s\n", params->cpus); + break; + case 'D': + config_debug = 1; + break; + case 'd': + params->duration = parse_seconds_duration(optarg); + if (!params->duration) + osnoise_top_usage("Invalid -D duration\n"); + break; + case 'h': + case '?': + osnoise_top_usage(NULL); + break; + case 'p': + params->period = get_long_from_str(optarg); + if (params->period > 10000000) + osnoise_top_usage("Period longer than 10 s\n"); + break; + case 'P': + retval = parse_prio(optarg, ¶ms->sched_param); + if (retval == -1) + osnoise_top_usage("Invalid -P priority"); + params->set_sched = 1; + break; + case 'q': + params->quiet = 1; + break; + case 'r': + params->runtime = get_long_from_str(optarg); + if (params->runtime < 100) + osnoise_top_usage("Runtime shorter than 100 us\n"); + break; + case 's': + params->stop_us = get_long_from_str(optarg); + break; + case 'S': + params->stop_total_us = get_long_from_str(optarg); + break; + case 't': + if (optarg) + /* skip = */ + params->trace_output = &optarg[1]; + else + params->trace_output = "osnoise_trace.txt"; + break; + default: + osnoise_top_usage("Invalid option"); + } + } + + if (geteuid()) { + err_msg("osnoise needs root permission\n"); + exit(EXIT_FAILURE); + } + + return params; +} + +/* + * osnoise_top_apply_config - apply the top configs to the initialized tool + */ +static int +osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *params) +{ + int retval; + + if (!params->sleep_time) + params->sleep_time = 1; + + if (params->cpus) { + retval = osnoise_set_cpus(tool->context, params->cpus); + if (retval) { + err_msg("Failed to apply CPUs config\n"); + goto out_err; + } + } + + if (params->runtime || params->period) { + retval = osnoise_set_runtime_period(tool->context, + params->runtime, + params->period); + if (retval) { + err_msg("Failed to set runtime and/or period\n"); + goto out_err; + } + } + + if (params->stop_us) + osnoise_set_stop_us(tool->context, params->stop_us); + + if (params->stop_total_us) + osnoise_set_stop_total_us(tool->context, params->stop_total_us); + + return 0; + +out_err: + return -1; +} + +/* + * osnoise_init_top - initialize a osnoise top tool with parameters + */ +struct osnoise_tool *osnoise_init_top(struct osnoise_top_params *params) +{ + struct osnoise_tool *tool; + int nr_cpus; + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + tool = osnoise_init_tool("osnoise_top"); + if (!tool) + return NULL; + + tool->data = osnoise_alloc_top(nr_cpus); + if (!tool->data) + goto out_err; + + tool->params = params; + + tep_register_event_handler(tool->trace.tep, -1, "ftrace", "osnoise", + osnoise_top_handler, NULL); + + return tool; + +out_err: + osnoise_free_top(tool->data); + osnoise_destroy_tool(tool); + return NULL; +} + +static int stop_tracing = 0; +static void stop_top(int sig) +{ + stop_tracing = 1; +} + +/* + * osnoise_top_set_signals - handles the signal to stop the tool + */ +static void osnoise_top_set_signals(struct osnoise_top_params *params) +{ + signal(SIGINT, stop_top); + if (params->duration) { + signal(SIGALRM, stop_top); + alarm(params->duration); + } +} + +int osnoise_top_main(int argc, char **argv) +{ + struct osnoise_top_params *params; + struct trace_instance *trace; + struct osnoise_tool *record; + struct osnoise_tool *tool; + int return_value = 1; + int retval; + + params = osnoise_top_parse_args(argc, argv); + if (!params) + exit(1); + + tool = osnoise_init_top(params); + if (!tool) { + err_msg("Could not init osnoise top\n"); + goto out_exit; + } + + retval = osnoise_top_apply_config(tool, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_exit; + } + + trace = &tool->trace; + + retval = enable_osnoise(trace); + if (retval) { + err_msg("Failed to enable osnoise tracer\n"); + goto out_top; + } + + if (params->set_sched) { + retval = set_comm_sched_attr("osnoise/", ¶ms->sched_param); + if (retval) + osnoise_top_usage("Failed to set sched parameters\n"); + } + + trace_instance_start(trace); + + if (params->trace_output) { + record = osnoise_init_trace_tool("osnoise"); + if (!record) { + err_msg("Failed to enable the trace instance\n"); + goto out_top; + } + trace_instance_start(&record->trace); + } + + tool->start_time = time(NULL); + osnoise_top_set_signals(params); + + do { + sleep(params->sleep_time); + + retval = tracefs_iterate_raw_events(trace->tep, + trace->inst, + NULL, + 0, + collect_registered_events, + trace); + if (retval < 0) { + err_msg("Error iterating on events\n"); + goto out_top; + } + + if (!params->quiet) + osnoise_print_stats(params, tool); + + if (!tracefs_trace_is_on(trace->inst)) + break; + + } while(!stop_tracing); + + osnoise_print_stats(params, tool); + + return_value = 0; + + if (!tracefs_trace_is_on(trace->inst)) { + printf("osnoise hit stop tracing\n"); + if (params->trace_output) { + printf(" Saving trace to %s\n", params->trace_output); + save_trace_to_file(record->trace.inst, params->trace_output); + } + } + +out_top: + osnoise_free_top(tool->data); + osnoise_destroy_tool(tool); + if (params->trace_output) + osnoise_destroy_tool(record); +out_exit: + exit(return_value); +} From patchwork Mon Oct 11 14:35:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 644F7C433EF for ; Mon, 11 Oct 2021 14:37:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B9A760E98 for ; Mon, 11 Oct 2021 14:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243625AbhJKOjy convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:39:54 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:48623 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242419AbhJKOjX (ORCPT ); Mon, 11 Oct 2021 10:39:23 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-98-iutX6OY9O3yejg6Jzr25jw-1; Mon, 11 Oct 2021 10:37:22 -0400 X-MC-Unique: iutX6OY9O3yejg6Jzr25jw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 79F22100D680; Mon, 11 Oct 2021 14:37:20 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7F1119C59; Mon, 11 Oct 2021 14:37:16 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 10/19] rtla/osnoise: Add the hist mode Date: Mon, 11 Oct 2021 16:35:58 +0200 Message-Id: <58bc6a3be529674923de46bbe3861fc47708a753.1633958325.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The rtla osnoise hist tool collects all osnoise:sample_threshold occurrence in a histogram, displaying the results in a user-friendly way. The tool also allows many configurations of the osnoise tracer and the collection of the tracer output. Here is one example of the rtla osnoise hist tool output: ---------- %< ---------- [root@f34 ~]# rtla osnoise hist --bucket-size 10 --entries 100 --skip-zeros -c 0-8 -d 1M -r 9000 -P F:1 # RTLA osnoise histogram # Time unit is microseconds (us) # Duration: 0 00:01:00 Index CPU-000 CPU-001 CPU-002 CPU-003 CPU-004 CPU-005 CPU-006 CPU-007 CPU-008 0 430 434 352 455 440 463 467 436 484 10 88 88 92 141 120 100 126 166 100 20 19 7 12 22 8 8 13 13 16 30 6 0 2 0 1 2 2 1 0 50 0 0 0 0 0 0 1 0 0 over: 0 0 0 0 0 0 0 0 0 count: 543 529 458 618 569 573 609 616 600 min: 0 0 0 0 0 0 0 0 0 avg: 0 0 0 0 0 0 0 0 0 max: 30 20 30 20 30 30 50 30 20 ---------- >% ---------- Running - rtla osnoise hist --help provides information about the available options. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/osnoise.c | 6 +- tools/tracing/rtla/src/osnoise.h | 1 + tools/tracing/rtla/src/osnoise_hist.c | 860 ++++++++++++++++++++++++++ 3 files changed, 866 insertions(+), 1 deletion(-) create mode 100644 tools/tracing/rtla/src/osnoise_hist.c diff --git a/tools/tracing/rtla/src/osnoise.c b/tools/tracing/rtla/src/osnoise.c index 7b66b9b6ba9a..bc668faf4b15 100644 --- a/tools/tracing/rtla/src/osnoise.c +++ b/tools/tracing/rtla/src/osnoise.c @@ -799,7 +799,8 @@ static void osnoise_usage(void) " usage: [rtla] osnoise [MODE] ...", "", " modes:", - " top - prints the summary from osnoise tracer", + " top - prints the summary from osnoise tracer", + " hist - prints a histogram of osnoise samples", "", "if no MODE is given, the top mode is called, passing the arguments", NULL, @@ -834,6 +835,9 @@ int osnoise_main(int argc, char *argv[]) } else if (strcmp(argv[1], "top") == 0) { osnoise_top_main(argc-1, &argv[1]); exit(0); + } else if (strcmp(argv[1], "hist") == 0) { + osnoise_hist_main(argc-1, &argv[1]); + exit(0); } usage: diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h index b4c1369d743d..c6d517445f08 100644 --- a/tools/tracing/rtla/src/osnoise.h +++ b/tools/tracing/rtla/src/osnoise.h @@ -81,5 +81,6 @@ void osnoise_destroy_tool(struct osnoise_tool *top); struct osnoise_tool *osnoise_init_tool(char *tool_name); struct osnoise_tool *osnoise_init_trace_tool(char *tracer); +int osnoise_hist_main(int argc, char *argv[]); int osnoise_top_main(int argc, char **argv); int osnoise_main(int argc, char **argv); diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c new file mode 100644 index 000000000000..19b6fa200c35 --- /dev/null +++ b/tools/tracing/rtla/src/osnoise_hist.c @@ -0,0 +1,860 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "utils.h" +#include "osnoise.h" + +struct osnoise_hist_params { + char *cpus; + char *monitored_cpus; + char *trace_output; + unsigned long long runtime; + unsigned long long period; + long long stop_us; + long long stop_total_us; + int sleep_time; + int duration; + int set_sched; + int output_divisor; + struct sched_attr sched_param; + + char no_header; + char no_summary; + char no_index; + char skip_zeroes; + int bucket_size; + int entries; +}; + +struct osnoise_hist_cpu { + int *samples; + int *thread; + + int count; + int thread_count; + + unsigned long long min_sample; + unsigned long long sum_sample; + unsigned long long max_sample; + + unsigned long long min_thread; + unsigned long long sum_thread; + unsigned long long max_thread; +}; + +struct osnoise_hist_data { + struct tracefs_hist *trace_hist; + struct osnoise_hist_cpu *hist; + int entries; + int bucket_size; + int nr_cpus; +}; + +/* + * osnoise_free_histogram - free runtime data + */ +static void +osnoise_free_histogram(struct osnoise_hist_data *data) +{ + int cpu; + + /* one histogram for IRQ and one for thread, per CPU */ + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (data->hist[cpu].samples) + free(data->hist[cpu].samples); + + if (data->hist[cpu].thread) + free(data->hist[cpu].thread); + } + + /* one set of histograms per CPU */ + if (data->hist) + free(data->hist); + + free(data); +} + +/* + * osnoise_alloc_histogram - alloc runtime data + */ +static struct osnoise_hist_data +*osnoise_alloc_histogram(int nr_cpus, int entries, int bucket_size) +{ + struct osnoise_hist_data *data; + int cpu; + + data = calloc(1, sizeof(*data)); + if (!data) + return NULL; + + data->entries = entries; + data->bucket_size = bucket_size; + data->nr_cpus = nr_cpus; + + /* one set of histograms per CPU */ + data->hist = calloc(1, sizeof(*data->hist) * nr_cpus); + if (!data->hist) + goto cleanup; + + /* one histogram for IRQ and one for thread, per cpu */ + for (cpu = 0; cpu < nr_cpus; cpu++) { + data->hist[cpu].samples = calloc(1, sizeof(*data->hist) * (entries + 1)); + if (!data->hist[cpu].samples) + goto cleanup; + data->hist[cpu].thread = calloc(1, sizeof(*data->hist) * (entries + 1)); + if (!data->hist[cpu].thread) + goto cleanup; + } + + /* set the min to max */ + for (cpu = 0; cpu < nr_cpus; cpu++) { + data->hist[cpu].min_sample = ~0; + data->hist[cpu].min_thread = ~0; + } + + return data; + +cleanup: + osnoise_free_histogram(data); + return NULL; +} + +/* + * osnoise_hist_update - record a new osnoise occurrence on cpu, updating data + */ +static void +osnoise_hist_update(struct osnoise_tool *tool, int cpu, + unsigned long long duration) +{ + struct osnoise_hist_params *params = tool->params; + struct osnoise_hist_data *data = tool->data; + int entries = data->entries; + int bucket; + int *hist; + + if (params->output_divisor) + duration = duration / params->output_divisor; + + if (data->bucket_size) + bucket = duration / data->bucket_size; + + hist = data->hist[cpu].samples; + data->hist[cpu].count++; + update_min(&data->hist[cpu].min_sample, &duration); + update_sum(&data->hist[cpu].sum_sample, &duration); + update_max(&data->hist[cpu].max_sample, &duration); + + if (bucket < entries) + hist[bucket]++; + else + hist[entries]++; +} + +static void osnoise_hist_update_multiple(struct osnoise_tool *tool, int cpu, + unsigned long long duration, int count) +{ + struct osnoise_hist_params *params = tool->params; + struct osnoise_hist_data *data = tool->data; + int entries = data->entries; + int bucket; + int *hist; + + if (params->output_divisor) + duration = duration / params->output_divisor; + + if (data->bucket_size) + bucket = duration / data->bucket_size; + + hist = data->hist[cpu].samples; + data->hist[cpu].count += count; + update_min(&data->hist[cpu].min_sample, &duration); + update_sum(&data->hist[cpu].sum_sample, &duration); + update_max(&data->hist[cpu].max_sample, &duration); + + if (bucket < entries) + hist[bucket] += count; + else + hist[entries] += count; +} + +/* + * osnoise_destroy_trace_hist - disable events used to collect histogram + */ +static void osnoise_destroy_trace_hist(struct osnoise_tool *tool) +{ + struct osnoise_hist_data *data = tool->data; + + tracefs_event_disable(tool->trace.inst, "osnoise", "sample_threshold"); + tracefs_hist_pause(tool->trace.inst, data->trace_hist); + tracefs_hist_destroy(tool->trace.inst, data->trace_hist); + return; +} + +/* + * osnoise_init_trace_hist - enable events used to collect histogram + */ +static int osnoise_init_trace_hist(struct osnoise_tool *tool) +{ + struct osnoise_hist_params *params = tool->params; + struct osnoise_hist_data *data = tool->data; + int bucket_size, overrun; + char buff[128]; + int retval = 0; + + /* + * Set the size of the bucket. + */ + bucket_size = params->output_divisor * params->bucket_size; + snprintf(buff, sizeof(buff), "duration.buckets=%d", bucket_size); + + data->trace_hist = tracefs_hist_alloc(tool->trace.tep, "osnoise", "sample_threshold", + buff, TRACEFS_HIST_KEY_NORMAL); + if (!data->trace_hist) + return 1; + + retval = tracefs_hist_add_key(data->trace_hist, "cpu", 0); + if (retval) + goto out_err; + + retval = tracefs_hist_start(tool->trace.inst, data->trace_hist); + if (retval) + goto out_err; + + /* + * The event still needs to be hooked to collect values higher than the + * histogram can store. By default, histogram allocs 2048 buckets, so: + */ + overrun = 2048 * bucket_size; + + /* + * Filter the event to print only values higher than the overrun value. + */ + snprintf(buff, sizeof(buff), "duration > %d", overrun); + retval = tracefs_event_file_write(tool->trace.inst, "osnoise", + "sample_threshold", + "filter", buff); + if (!retval) + goto out_err; + + retval = tracefs_event_enable(tool->trace.inst, "osnoise", "sample_threshold"); + if (retval < 0 && !errno) { + err_msg("Could not find osnoise events\n"); + goto out_err; + } + + return 0; + +out_err: + osnoise_destroy_trace_hist(tool); + return 1; +} + +/* + * osnoise_hist_handler - this is the handler for osnoise histogram overrun events + */ +static int +osnoise_hist_handler(struct trace_seq *s, struct tep_record *record, + struct tep_event *event, void *data) +{ + struct trace_instance *trace = data; + unsigned long long duration; + struct osnoise_tool *tool; + int cpu = record->cpu; + + tool = container_of(trace, struct osnoise_tool, trace); + + tep_get_field_val(s, event, "duration", record, &duration, 1); + + osnoise_hist_update(tool, cpu, duration); + + return 0; +} + +/* + * osnoise_read_trace_hist - parse histogram file and file osnoise histogram + */ +static void osnoise_read_trace_hist(struct osnoise_tool *tool) +{ + struct osnoise_hist_data *data = tool->data; + long cpu, counter, duration; + char *content, *position; + + tracefs_hist_pause(tool->trace.inst, data->trace_hist); + + content = tracefs_event_file_read(tool->trace.inst, "osnoise", + "sample_threshold", + "hist", NULL); + if (!content) + return; + + position = content; + while (true) { + position = strstr(position ,"duration: ~"); + if (!position) + break; + position += strlen("duration: ~"); + duration = get_long_from_str(position); + if (duration == -1) + err_msg("error reading duration from histogram\n"); + + position = strstr(position, "cpu:"); + if (!position) + break; + position += strlen("cpu: "); + cpu = get_long_from_str(position); + if (cpu == -1) + err_msg("error reading cpu from histogram\n"); + + position = strstr(position, "hitcount:"); + if (!position) + break; + position += strlen("hitcount: "); + counter = get_long_from_str(position); + if (counter == -1) + err_msg("error reading counter from histogram\n"); + + osnoise_hist_update_multiple(tool, cpu, duration, counter); + } + free(content); +} + +/* + * osnoise_hist_header - print the header of the tracer to the output + */ +static void osnoise_hist_header(struct osnoise_tool *tool) +{ + struct osnoise_hist_params *params = tool->params; + struct osnoise_hist_data *data = tool->data; + struct trace_seq *s = tool->trace.seq; + char duration[26]; + int cpu; + + if (params->no_header) + return; + + get_duration(tool->start_time, duration, sizeof(duration)); + trace_seq_printf(s, "# RTLA osnoise histogram\n"); + trace_seq_printf(s, "# Time unit is %s (%s) \n", + params->output_divisor == 1 ? "nanoseconds" : "microseconds", + params->output_divisor == 1 ? "ns" : "us"); + + trace_seq_printf(s, "# Duration: %s\n", duration); + + if (!params->no_index) + trace_seq_printf(s, "Index"); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + trace_seq_printf(s, " CPU-%03d", cpu); + } + trace_seq_printf(s, "\n"); + + trace_seq_do_printf(s); + trace_seq_reset(s); +} + +/* + * osnoise_print_summary - print the summary of the hist data to the output + */ +static void +osnoise_print_summary(struct osnoise_hist_params *params, + struct trace_instance *trace, + struct osnoise_hist_data *data) +{ + int cpu; + + if (params->no_summary) + return; + + if (!params->no_index) + trace_seq_printf(trace->seq, "count:" ); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + trace_seq_printf(trace->seq, "%9d ", data->hist[cpu].count); + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "min: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + trace_seq_printf(trace->seq, "%9llu ", data->hist[cpu].min_sample); + + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "avg: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (data->hist[cpu].count) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].sum_sample / data->hist[cpu].count); + else + trace_seq_printf(trace->seq, " - "); + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "max: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + trace_seq_printf(trace->seq, "%9llu ", data->hist[cpu].max_sample); + + } + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); +} + +/* + * osnoise_print_stats - print data for all CPUs + */ +static void +osnoise_print_stats(struct osnoise_hist_params *params, struct osnoise_tool *tool) +{ + struct osnoise_hist_data *data = tool->data; + struct trace_instance *trace = &tool->trace; + int bucket, cpu; + int total; + + osnoise_hist_header(tool); + + for (bucket = 0; bucket < data->entries; bucket++) { + total = 0; + + if (!params->no_index) + trace_seq_printf(trace->seq, "%-6d", + bucket * data->bucket_size); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + total += data->hist[cpu].samples[bucket]; + trace_seq_printf(trace->seq, "%9d ", data->hist[cpu].samples[bucket]); + } + + if (params->skip_zeroes && total == 0) { + trace_seq_reset(trace->seq); + continue; + } + + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); + } + + if (!params->no_index) + trace_seq_printf(trace->seq, "over: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].samples[data->entries]); + } + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); + + osnoise_print_summary(params, trace, data); +} + +/* + * osnoise_hist_usage - prints osnoise hist usage message + */ +static void osnoise_hist_usage(char *usage) +{ + int i; + + char *msg[] = { + "", + " usage: rtla osnoise hist [-h] [-p us] [-r us] [-s us] [-S us] [-T[=file]] \\", + " [-c cpu-list] [-P priority] [-b N] [-e N] [--no-header] [--no-summary] \\", + " [--no-index] [--skip-zeros]", + "", + " -h/--help: print this menu", + " -p/--period us: osnoise period in us", + " -r/--runtime us: osnoise runtime in us", + " -s/--stop us: stop trace if a single sample is higher than the argument in us", + " -S/--stop-total us: stop trace if the total sample is higher than the argument in us", + " -c/--cpus cpu-list: list of cpus to run osnoise threads", + " -d/--duration time[s|m|h|d]: duration of the session", + " -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]", + " -b/--bucket-size N: set the histogram bucket size (default 1)", + " -e/--entries N: set the number of entries of the histogram (default 256)", + " --no-header: do not print header", + " --no-summary: do not print summary", + " --no-index: do not print index", + " --skip-zeros: skip zero only entries", + " -P/--priority o:prio|r:prio|f:prio|d:runtime:period: set scheduling parameters", + " o:prio - use SCHED_OTHER with prio", + " r:prio - use SCHED_RR with prio", + " f:prio - use SCHED_FIFO with prio", + " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", + " in nanoseconds", + NULL, + }; + + if (usage) + fprintf(stderr, "%s\n", usage); + + fprintf(stderr, "rtla osnoise hist: display a per-cpu histogram of the operating system noise (version %s)\n", VERSION); + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +/* + * osnoise_hist_parse_args - allocs, parse and fill the cmd line parameters + */ +static struct osnoise_hist_params +*osnoise_hist_parse_args(int argc, char *argv[]) +{ + struct osnoise_hist_params *params; + int retval; + int c; + + params = calloc(1, sizeof(*params)); + if (!params) + exit(1); + + /* display data in microseconds */ + params->output_divisor = 1000; + params->bucket_size = 1; + params->entries = 256; + + while (1) { + static struct option long_options[] = { + {"bucket-size", required_argument, 0, 'b'}, + {"entries", required_argument, 0, 'e'}, + {"cpus", required_argument, 0, 'c'}, + {"debug", no_argument, 0, 'D'}, + {"duration", required_argument, 0, 'd'}, + {"help", no_argument, 0, 'h'}, + {"period", required_argument, 0, 'p'}, + {"priority", required_argument, 0, 'P'}, + {"runtime", required_argument, 0, 'r'}, + {"stop", required_argument, 0, 's'}, + {"stop-total", required_argument, 0, 'S'}, + {"trace", optional_argument, 0, 't'}, + {"no-header", no_argument, 0, '0'}, + {"no-summary", no_argument, 0, '1'}, + {"no-index", no_argument, 0, '2'}, + {"skip-zeros", no_argument, 0, '3'}, + {0, 0, 0, 0} + }; + + /* getopt_long stores the option index here. */ + int option_index = 0; + + c = getopt_long(argc, argv, "c:b:d:e:Dhp:P:r:s:S:t::0123", + long_options, &option_index); + + /* detect the end of the options. */ + if (c == -1) + break; + + switch (c) { + case 'b': + params->bucket_size = get_long_from_str(optarg); + if ((params->bucket_size == 0) || (params->bucket_size >= 1000000)) + osnoise_hist_usage("Bucket size needs to be > 0 and <= 1000000\n"); + break; + case 'c': + retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + if (retval) + osnoise_hist_usage("\nInvalid -c cpu list\n"); + params->cpus = optarg; + debug_msg("-c for %s\n", params->cpus); + break; + case 'D': + config_debug = 1; + break; + case 'd': + params->duration = parse_seconds_duration(optarg); + if (!params->duration) + osnoise_hist_usage("Invalid -D duration\n"); + break; + case 'e': + params->entries = get_long_from_str(optarg); + if ((params->entries < 10) || (params->entries > 9999999)) + osnoise_hist_usage("Entries must be > 10 and < 9999999\n"); + break; + case 'h': + case '?': + osnoise_hist_usage(NULL); + break; + case 'p': + params->period = get_long_from_str(optarg); + if (params->period > 10000000) + osnoise_hist_usage("Period longer than 10 s\n"); + break; + case 'P': + retval = parse_prio(optarg, ¶ms->sched_param); + if (retval == -1) + osnoise_hist_usage("Invalid -P priority"); + params->set_sched = 1; + break; + case 'r': + params->runtime = get_long_from_str(optarg); + if (params->runtime < 100) + osnoise_hist_usage("Runtime shorter than 100 us\n"); + break; + case 's': + params->stop_us = get_long_from_str(optarg); + break; + case 'S': + params->stop_total_us = get_long_from_str(optarg); + break; + case 't': + if (optarg) + /* skip = */ + params->trace_output = &optarg[1]; + else + params->trace_output = "osnoise_trace.txt"; + break; + case '0': /* no header */ + params->no_header = 1; + break; + case '1': /* no summary */ + params->no_summary = 1; + break; + case '2': /* no index */ + params->no_index = 1; + break; + case '3': /* skip zeroes */ + params->skip_zeroes = 1; + break; + default: + osnoise_hist_usage("Invalid option"); + } + } + + if (geteuid()) { + err_msg("rtla needs root permission\n"); + exit(EXIT_FAILURE); + } + + if (params->no_index && params->skip_zeroes) + osnoise_hist_usage("no-index and skip-zeros set - it does not make sense"); + + return params; +} + +/* + * osnoise_hist_apply_config - apply the hist configs to the initialized tool + */ +static int +osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params *params) +{ + int retval; + + if (!params->sleep_time) + params->sleep_time = 1; + + if (params->cpus) { + retval = osnoise_set_cpus(tool->context, params->cpus); + if (retval) { + err_msg("Failed to apply CPUs config\n"); + goto out_err; + } + } + + if (params->runtime || params->period) { + retval = osnoise_set_runtime_period(tool->context, + params->runtime, + params->period); + if (retval) { + err_msg("Failed to set runtime and/or period\n"); + goto out_err; + } + } + + if (params->stop_us) + osnoise_set_stop_us(tool->context, params->stop_us); + + if (params->stop_total_us) + osnoise_set_stop_total_us(tool->context, params->stop_total_us); + + return 0; + +out_err: + return -1; +} + +/* + * osnoise_init_hist - initialize a osnoise hist tool with parameters + */ +static struct osnoise_tool +*osnoise_init_hist(struct osnoise_hist_params *params) +{ + struct osnoise_tool *tool; + int nr_cpus; + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + tool = osnoise_init_tool("osnoise_hist"); + if (!tool) + return NULL; + + tool->data = osnoise_alloc_histogram(nr_cpus, params->entries, params->bucket_size); + if (!tool->data) + goto out_err; + + tool->params = params; + + tep_register_event_handler(tool->trace.tep, -1, "osnoise", "sample_threshold", + osnoise_hist_handler, tool); + + return tool; + +out_err: + osnoise_destroy_tool(tool); + return NULL; +} + +static int stop_tracing = 0; +static void stop_hist(int sig) +{ + stop_tracing = 1; +} + +/* + * osnoise_hist_set_signals - handles the signal to stop the tool + */ +static void +osnoise_hist_set_signals(struct osnoise_hist_params *params) +{ + signal(SIGINT, stop_hist); + if (params->duration) { + signal(SIGALRM, stop_hist); + alarm(params->duration); + } +} + +int osnoise_hist_main(int argc, char *argv[]) +{ + struct osnoise_hist_params *params; + struct trace_instance *trace; + struct osnoise_tool *record; + struct osnoise_tool *tool; + int return_value = 1; + int retval; + + params = osnoise_hist_parse_args(argc, argv); + if (!params) + exit(1); + + tool = osnoise_init_hist(params); + if (!tool) { + err_msg("Could not init osnoise hist\n"); + goto out_exit; + } + + retval = osnoise_hist_apply_config(tool, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_exit; + } + + trace = &tool->trace; + + retval = enable_osnoise(trace); + if (retval) { + err_msg("Failed to enable osnoise tracer\n"); + goto out_destroy; + } + + retval = osnoise_init_trace_hist(tool); + if (retval) + goto out_destroy; + + if (params->set_sched) { + retval = set_comm_sched_attr("osnoise/", ¶ms->sched_param); + if (retval) + osnoise_hist_usage("Failed to set sched parameters\n"); + } + + trace_instance_start(trace); + + if (params->trace_output) { + record = osnoise_init_trace_tool("osnoise"); + if (!record) { + err_msg("Failed to enable the trace instance\n"); + goto out_hist; + } + trace_instance_start(&record->trace); + } + + tool->start_time = time(NULL); + osnoise_hist_set_signals(params); + + while (!stop_tracing) { + sleep(params->sleep_time); + + retval = tracefs_iterate_raw_events(trace->tep, + trace->inst, + NULL, + 0, + collect_registered_events, + trace); + if (retval < 0) { + err_msg("Error iterating on events\n"); + goto out_hist; + } + + if (!tracefs_trace_is_on(trace->inst)) + break; + }; + + osnoise_read_trace_hist(tool); + + osnoise_print_stats(params, tool); + + return_value = 0; + + if (!tracefs_trace_is_on(trace->inst)) { + printf("rtla timelat hit stop tracing\n"); + if (params->trace_output) { + printf(" Saving trace to %s\n", params->trace_output); + save_trace_to_file(record->trace.inst, params->trace_output); + } + } + +out_hist: + osnoise_free_histogram(tool->data); +out_destroy: + osnoise_destroy_tool(tool); + if (params->trace_output) + osnoise_destroy_tool(record); + free(params); +out_exit: + exit(return_value); +} From patchwork Mon Oct 11 14:35:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E190C433F5 for ; Mon, 11 Oct 2021 14:38:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 12BA760BD3 for ; Mon, 11 Oct 2021 14:38:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242578AbhJKOkG convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:40:06 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:45592 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242714AbhJKOjg (ORCPT ); Mon, 11 Oct 2021 10:39:36 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-575-mW_pcPAjNwacw_KGiCBxGQ-1; Mon, 11 Oct 2021 10:37:30 -0400 X-MC-Unique: mW_pcPAjNwacw_KGiCBxGQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5DCAE138CA90; Mon, 11 Oct 2021 14:37:28 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6EA819723; Mon, 11 Oct 2021 14:37:20 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 11/19] rtla: Add timerlat tool and timelart top mode Date: Mon, 11 Oct 2021 16:35:59 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency. The timerlat tracer outputs information in two ways. It periodically prints the timer latency at the timer IRQ handler and the Thread handler. It also provides information for each noise via the osnoise tracepoints. The rtla timerlat top mode displays a summary of the periodic output from the timerlat tracer. Here is one example of the rtla timerlat tool output: ---------- %< ---------- [root@alien ~]# rtla timerlat top -c 0-3 -d 1m Timer Latency 0 00:01:00 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 0 #60001 | 0 0 0 3 | 1 1 1 6 1 #60001 | 0 0 0 3 | 2 1 1 5 2 #60001 | 0 0 1 6 | 1 1 2 7 3 #60001 | 0 0 0 7 | 1 1 1 11 ---------- >% ---------- Running: # rtla timerlat --help # rtla timerlat top --help provides information about the available options. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/Makefile | 2 + tools/tracing/rtla/src/rtla.c | 5 + tools/tracing/rtla/src/timerlat.c | 68 +++ tools/tracing/rtla/src/timerlat.h | 2 + tools/tracing/rtla/src/timerlat_top.c | 578 ++++++++++++++++++++++++++ 5 files changed, 655 insertions(+) create mode 100644 tools/tracing/rtla/src/timerlat.c create mode 100644 tools/tracing/rtla/src/timerlat.h create mode 100644 tools/tracing/rtla/src/timerlat_top.c diff --git a/tools/tracing/rtla/Makefile b/tools/tracing/rtla/Makefile index b3ddd5138829..580a33db6645 100644 --- a/tools/tracing/rtla/Makefile +++ b/tools/tracing/rtla/Makefile @@ -43,6 +43,8 @@ install: $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise + @test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat + ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat .PHONY: clean tarball push clean: diff --git a/tools/tracing/rtla/src/rtla.c b/tools/tracing/rtla/src/rtla.c index 9427cab04724..dfc7e08200d1 100644 --- a/tools/tracing/rtla/src/rtla.c +++ b/tools/tracing/rtla/src/rtla.c @@ -9,6 +9,7 @@ #include #include "osnoise.h" +#include "timerlat.h" /* * rtla_usage - print rtla usage @@ -25,6 +26,7 @@ static void rtla_usage(void) "", " commands:", " osnoise - gives information about the operating system noise (osnoise)", + " timerlat - measures the timer irq and thread latency", "", NULL, }; @@ -45,6 +47,9 @@ int run_command(int argc, char **argv, int start_position) if (strcmp(argv[start_position], "osnoise") == 0) { osnoise_main(argc-start_position, &argv[start_position]); goto ran; + } else if (strcmp(argv[start_position], "timerlat") == 0) { + timerlat_main(argc-start_position, &argv[start_position]); + goto ran; } return 0; diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c new file mode 100644 index 000000000000..bdd7ab55bf91 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "timerlat.h" + +static void timerlat_usage(void) +{ + int i; + + char *msg[] = { + "", + "timerlat version " VERSION, + "", + " usage: [rtla] timerlat [MODE] ...", + "", + " modes:", + " top - prints the summary from timerlat tracer", + "", + "if no MODE is given, the top mode is called, passing the arguments", + NULL, + }; + + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +int timerlat_main(int argc, char *argv[]) +{ + if (argc == 0) + goto usage; + + /* + * if timerlat was called without any argument, run the + * default cmdline. + */ + if (argc == 1) { + timerlat_top_main(argc, argv); + exit(0); + } + + if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) { + timerlat_usage(); + exit(0); + } else if (strncmp(argv[1], "-", 1) == 0) { + /* the user skipped the tool, call the default one */ + timerlat_top_main(argc, argv); + exit(0); + } else if (strcmp(argv[1], "top") == 0) { + timerlat_top_main(argc-1, &argv[1]); + exit(0); + } + +usage: + timerlat_usage(); + exit(1); +} diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h new file mode 100644 index 000000000000..4f0f912b3b81 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat.h @@ -0,0 +1,2 @@ +int timerlat_top_main(int argc, char *argv[]); +int timerlat_main(int argc, char *argv[]); diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c new file mode 100644 index 000000000000..258eaf796658 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat_top.c @@ -0,0 +1,578 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "utils.h" +#include "osnoise.h" +#include "timerlat.h" + +struct timerlat_top_params { + char *cpus; + char *monitored_cpus; + char *trace_output; + unsigned long long runtime; + long long stop_us; + long long stop_total_us; + long long timerlat_period_us; + long long print_stack; + int sleep_time; + int output_divisor; + int duration; + int quiet; + int set_sched; + struct sched_attr sched_param; +}; + +struct timerlat_top_cpu { + int irq_count; + int thread_count; + + unsigned long long cur_irq; + unsigned long long min_irq; + unsigned long long sum_irq; + unsigned long long max_irq; + + unsigned long long cur_thread; + unsigned long long min_thread; + unsigned long long sum_thread; + unsigned long long max_thread; +}; + +struct timerlat_top_data { + struct timerlat_top_cpu *cpu_data; + int nr_cpus; +}; + +/* + * timerlat_free_top - free runtime data + */ +static void +timerlat_free_top(struct timerlat_top_data *data) +{ + free(data->cpu_data); + free(data); +} + +/* + * timerlat_alloc_histogram - alloc runtime data + */ +static struct timerlat_top_data *timerlat_alloc_top(int nr_cpus) +{ + struct timerlat_top_data *data; + int cpu; + + data = calloc(1, sizeof(*data)); + if (!data) + return NULL; + + data->nr_cpus = nr_cpus; + + /* one set of histograms per CPU */ + data->cpu_data = calloc(1, sizeof(*data->cpu_data) * nr_cpus); + if (!data->cpu_data) + goto cleanup; + + /* set the min to max */ + for (cpu = 0; cpu < nr_cpus; cpu++) { + data->cpu_data[cpu].min_irq = ~0; + data->cpu_data[cpu].min_thread = ~0; + } + + return data; + +cleanup: + timerlat_free_top(data); + return NULL; +} + +/* + * timerlat_hist_update - record a new timerlat occurent on cpu, updating data + */ +static void +timerlat_top_update(struct osnoise_tool *tool, int cpu, + unsigned long long thread, + unsigned long long latency) +{ + struct timerlat_top_data *data = tool->data; + struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu]; + + if (!thread) { + cpu_data->irq_count++; + cpu_data->cur_irq = latency; + update_min(&cpu_data->min_irq, &latency); + update_sum(&cpu_data->sum_irq, &latency); + update_max(&cpu_data->max_irq, &latency); + } else { + cpu_data->thread_count++; + cpu_data->cur_thread = latency; + update_min(&cpu_data->min_thread, &latency); + update_sum(&cpu_data->sum_thread, &latency); + update_max(&cpu_data->max_thread, &latency); + } +} + +/* + * timerlat_top_handler - this is the handler for timerlat tracer events + */ +static int +timerlat_top_handler(struct trace_seq *s, struct tep_record *record, + struct tep_event *event, void *context) +{ + struct trace_instance *trace = context; + unsigned long long latency, thread; + struct osnoise_tool *top; + int cpu = record->cpu; + + top = container_of(trace, struct osnoise_tool, trace); + + tep_get_field_val(s, event, "context", record, &thread, 1); + tep_get_field_val(s, event, "timer_latency", record, &latency, 1); + + timerlat_top_update(top, cpu, thread, latency); + + return 0; +} + +/* + * timerlat_top_header - print the header of the tool output + */ +static void timerlat_top_header(struct osnoise_tool *top) +{ + struct timerlat_top_params *params = top->params; + struct trace_seq *s = top->trace.seq; + char duration[26]; + + get_duration(top->start_time, duration, sizeof(duration)); + + trace_seq_printf(s, "\033[2;37;40m"); + trace_seq_printf(s, " Timer Latency "); + trace_seq_printf(s, "\033[0;0;0m"); + trace_seq_printf(s, "\n"); + + trace_seq_printf(s, "%-6s | IRQ Timer Latency (%s) | Thread Timer Latency (%s) \n", duration, + params->output_divisor == 1 ? "ns" : "us", + params->output_divisor == 1 ? "ns" : "us"); + + trace_seq_printf(s, "\033[2;30;47m"); + trace_seq_printf(s, "CPU COUNT | cur min avg max | cur min avg max"); + trace_seq_printf(s, "\033[0;0;0m"); + trace_seq_printf(s, "\n"); +} + +/* + * timerlat_top_print - prints the output of a given CPU + */ +static void timerlat_top_print(struct osnoise_tool *top, int cpu) +{ + + struct timerlat_top_params *params = top->params; + struct timerlat_top_data *data = top->data; + struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu]; + int divisor = params->output_divisor; + struct trace_seq *s = top->trace.seq; + + if (divisor == 0) + return; + + trace_seq_printf(s, "%3d #%-9d |", cpu, cpu_data->irq_count); + + trace_seq_printf(s, "%9llu ", cpu_data->cur_irq / params->output_divisor); + trace_seq_printf(s, "%9llu ", cpu_data->min_irq / params->output_divisor); + + if (!cpu_data->irq_count) + trace_seq_printf(s, " "); + else + trace_seq_printf(s, "%9llu ", (cpu_data->sum_irq / cpu_data->irq_count) / divisor); + + trace_seq_printf(s, "%9llu |", cpu_data->max_irq / divisor); + + trace_seq_printf(s, "%9llu ", cpu_data->cur_thread / divisor); + trace_seq_printf(s, "%9llu ", cpu_data->min_thread / divisor); + + if (!cpu_data->thread_count) + trace_seq_printf(s, " "); + else + trace_seq_printf(s, "%9llu ", (cpu_data->sum_thread / cpu_data->thread_count) / divisor); + + trace_seq_printf(s, "%9llu\n", cpu_data->max_thread / divisor); +} + +/* + * clear_terminal - clears the output terminal + */ +static void clear_terminal(struct trace_seq *seq) +{ + if (!config_debug) + trace_seq_printf(seq, "\033c"); +} + +/* + * timerlat_print_stats - print data for all cpus + */ +static void +timerlat_print_stats(struct timerlat_top_params *params, struct osnoise_tool *top) +{ + struct trace_instance *trace = &top->trace; + static int nr_cpus = -1; + int i; + + if (nr_cpus == -1) + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + if (!params->quiet) + clear_terminal(trace->seq); + + timerlat_top_header(top); + + for (i = 0; i < nr_cpus; i++) { + if (params->cpus && !params->monitored_cpus[i]) + continue; + timerlat_top_print(top, i); + } + + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); +} + +/* + * timerlat_top_usage - prints timerlat top usage message + */ +static void timerlat_top_usage(char *usage) +{ + int i; + + char *msg[] = { + "", + " usage: rtla timerlat [top] [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\", + " [-c cpu-list] [-P priority]", + "", + " -h/--help: print this menu", + " -p/--period us: timerlat period in us", + " -i/--irq us: stop trace if the irq latency is higher than the argument in us", + " -T/--thread us: stop trace if the thread latency is higher than the argument in us", + " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", + " -c/--cpus cpus: run the tracer only on the given cpus" + " -d/--duration time[m|h|d]: duration of the session in seconds", + " -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]", + " -n/--nano: display data in nanoseconds", + " -q/--quiet print only a summary at the end", + " -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters", + " o:prio - use SCHED_OTHER with prio", + " r:prio - use SCHED_RR with prio", + " f:prio - use SCHED_FIFO with prio", + " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", + " in nanoseconds", + NULL, + }; + + if (usage) + fprintf(stderr, "%s\n", usage); + + fprintf(stderr, "rtla timerlat top: display a per-cpu summary of the timer latency (version %s)\n", VERSION); + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +/* + * timerlat_top_parse_args - allocs, parse and fill the cmd line parameters + */ +static struct timerlat_top_params +*timerlat_top_parse_args(int argc, char **argv) +{ + struct timerlat_top_params *params; + int retval; + int c; + + params = calloc(1, sizeof(*params)); + if (!params) + exit(1); + + /* display data in microseconds */ + params->output_divisor = 1000; + + while (1) { + static struct option long_options[] = { + {"cpus", required_argument, 0, 'c'}, + {"debug", no_argument, 0, 'D'}, + {"duration", required_argument, 0, 'd'}, + {"help", no_argument, 0, 'h'}, + {"irq", required_argument, 0, 'i'}, + {"nano", no_argument, 0, 'n'}, + {"period", required_argument, 0, 'p'}, + {"priority", required_argument, 0, 'P'}, + {"quiet", no_argument, 0, 'q'}, + {"stack", required_argument, 0, 's'}, + {"thread", required_argument, 0, 'T'}, + {"trace", optional_argument, 0, 't'}, + {0, 0, 0, 0} + }; + + /* getopt_long stores the option index here. */ + int option_index = 0; + + c = getopt_long(argc, argv, "c:d:Dhi:np:P:qs:t::T:", + long_options, &option_index); + + /* detect the end of the options. */ + if (c == -1) + break; + + switch (c) { + case 'c': + retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + if (retval) + timerlat_top_usage("\nInvalid -c cpu list\n"); + params->cpus = optarg; + debug_msg("-c for %s\n", params->cpus); + break; + case 'D': + config_debug = 1; + break; + case 'd': + params->duration = parse_seconds_duration(optarg); + if (!params->duration) + timerlat_top_usage("Invalid -D duration\n"); + break; + case 'h': + case '?': + timerlat_top_usage(NULL); + break; + case 'i': + params->stop_us = get_long_from_str(optarg); + break; + case 'n': + params->output_divisor = 1; + break; + case 'p': + params->timerlat_period_us = get_long_from_str(optarg); + if (params->timerlat_period_us > 1000000) + timerlat_top_usage("Period longer than 1 s\n"); + break; + case 'P': + retval = parse_prio(optarg, ¶ms->sched_param); + if (retval == -1) + timerlat_top_usage("Invalid -P priority"); + params->set_sched = 1; + break; + case 'q': + params->quiet = 1; + break; + case 's': + params->print_stack = get_long_from_str(optarg); + break; + case 'T': + params->stop_total_us = get_long_from_str(optarg); + break; + case 't': + if (optarg) + /* skip = */ + params->trace_output = &optarg[1]; + else + params->trace_output = "timerlat_trace.txt"; + break; + default: + timerlat_top_usage("Invalid option"); + } + } + + if (geteuid()) { + err_msg("rtla needs root permission\n"); + exit(EXIT_FAILURE); + } + + return params; +} + +/* + * timerlat_top_apply_config - apply the top configs to the initialized tool + */ +static int +timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *params) +{ + int retval; + + if (!params->sleep_time) + params->sleep_time = 1; + + if (params->cpus) { + retval = osnoise_set_cpus(top->context, params->cpus); + if (retval) { + err_msg("Failed to apply CPUs config\n"); + goto out_err; + } + } + + if (params->stop_us) + osnoise_set_stop_us(top->context, params->stop_us); + + if (params->stop_total_us) + osnoise_set_stop_total_us(top->context, params->stop_total_us); + + if (params->timerlat_period_us) + osnoise_set_timerlat_period_us(top->context, params->timerlat_period_us); + + if (params->print_stack) + osnoise_set_print_stack(top->context, params->print_stack); + + return 0; + +out_err: + return -1; +} + +/* + * timerlat_init_top - initialize a timerlat top tool with parameters + */ +static struct osnoise_tool +*timerlat_init_top(struct timerlat_top_params *params) +{ + struct osnoise_tool *top; + int nr_cpus; + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + top = osnoise_init_tool("timerlat_top"); + if (!top) + return NULL; + + top->data = timerlat_alloc_top(nr_cpus); + if (!top->data) + goto out_err; + + top->params = params; + + tep_register_event_handler(top->trace.tep, -1, "ftrace", "timerlat", + timerlat_top_handler, top); + + return top; + +out_err: + osnoise_destroy_tool(top); + return NULL; +} + +static int stop_tracing = 0; +static void stop_top(int sig) +{ + stop_tracing = 1; +} + +/* + * timerlat_top_set_signals - handles the signal to stop the tool + */ +static void +timerlat_top_set_signals(struct timerlat_top_params *params) +{ + signal(SIGINT, stop_top); + if (params->duration) { + signal(SIGALRM, stop_top); + alarm(params->duration); + } +} + +int timerlat_top_main(int argc, char *argv[]) +{ + struct timerlat_top_params *params; + struct trace_instance *trace; + struct osnoise_tool *record; + struct osnoise_tool *top; + int return_value = 1; + int retval; + + params = timerlat_top_parse_args(argc, argv); + if (!params) + exit(1); + + top = timerlat_init_top(params); + if (!top) { + err_msg("Could not init osnoise top\n"); + goto out_exit; + } + + retval = timerlat_top_apply_config(top, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_exit; + } + + trace = &top->trace; + + retval = enable_timerlat(trace); + if (retval) { + err_msg("Failed to enable timerlat tracer\n"); + goto out_top; + } + + if (params->set_sched) { + retval = set_comm_sched_attr("timerlat/", ¶ms->sched_param); + if (retval) + timerlat_top_usage("Failed to set sched parameters\n"); + } + + trace_instance_start(trace); + + if (params->trace_output) { + record = osnoise_init_trace_tool("timerlat"); + if (!record) { + err_msg("Failed to enable the trace instance\n"); + goto out_top; + } + trace_instance_start(&record->trace); + } + + top->start_time = time(NULL); + timerlat_top_set_signals(params); + + while (!stop_tracing) { + sleep(params->sleep_time); + + retval = tracefs_iterate_raw_events(trace->tep, + trace->inst, + NULL, + 0, + collect_registered_events, + trace); + if (retval < 0) { + err_msg("Error iterating on events\n"); + goto out_top; + } + + if (!params->quiet) + timerlat_print_stats(params, top); + + if (!tracefs_trace_is_on(trace->inst)) + break; + + }; + + timerlat_print_stats(params, top); + + return_value = 0; + + if (!tracefs_trace_is_on(trace->inst)) { + printf("rtla timelat hit stop tracing\n"); + if (params->trace_output) { + printf(" Saving trace to %s\n", params->trace_output); + save_trace_to_file(record->trace.inst, params->trace_output); + } + } + +out_top: + timerlat_free_top(top->data); + osnoise_destroy_tool(top); + if (params->trace_output) + osnoise_destroy_tool(record); + free(params); +out_exit: + exit(return_value); +} From patchwork Mon Oct 11 14:36:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0EFEC43219 for ; Mon, 11 Oct 2021 14:38:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A105360F23 for ; Mon, 11 Oct 2021 14:38:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242974AbhJKOkJ convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:40:09 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:34648 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242790AbhJKOjk (ORCPT ); Mon, 11 Oct 2021 10:39:40 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-368-6-_21mfLO1e5HCkC0lKiuQ-1; Mon, 11 Oct 2021 10:37:36 -0400 X-MC-Unique: 6-_21mfLO1e5HCkC0lKiuQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50775801A93; Mon, 11 Oct 2021 14:37:35 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE42619C59; Mon, 11 Oct 2021 14:37:28 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 12/19] rtla/timerlat: Add timerlat hist mode Date: Mon, 11 Oct 2021 16:36:00 +0200 Message-Id: <0682253da669e30aec759dde7dc0c72062ff8f47.1633958325.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org The rtla hist hist mode displays a histogram of each tracer event occurrence, both for IRQ and timer latencies. The tool also allows many configurations of the timerlat tracer and the collection of the tracer output. Here is one example of the rtla timerlat hist mode output: ---------- %< ---------- [root@alien ~]# rtla timerlat hist --skip-zeros -c 0-3 -d 1M # RTLA timerlat histogram # Time unit is microseconds (us) # Duration: 0 00:01:00 Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 0 58572 0 59373 0 58691 0 58895 0 1 1422 57021 628 57241 1310 56160 1102 56805 2 6 2931 0 2695 0 3567 4 3031 3 1 40 0 53 0 260 0 142 4 0 7 0 5 0 6 0 17 5 0 2 0 5 0 7 0 4 6 0 0 0 2 0 1 0 1 8 0 0 0 0 0 0 0 1 over: 0 0 0 0 0 0 0 0 count: 60001 60001 60001 60001 60001 60001 60001 60001 min: 0 1 0 1 0 1 0 1 avg: 0 1 0 1 0 1 0 1 max: 3 5 1 6 1 6 2 8 ---------- >% ---------- Running - rtla timerlat hist --help provides information about the available options. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/src/timerlat.c | 6 +- tools/tracing/rtla/src/timerlat.h | 1 + tools/tracing/rtla/src/timerlat_hist.c | 778 +++++++++++++++++++++++++ 3 files changed, 784 insertions(+), 1 deletion(-) create mode 100644 tools/tracing/rtla/src/timerlat_hist.c diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c index bdd7ab55bf91..06ad66902b3d 100644 --- a/tools/tracing/rtla/src/timerlat.c +++ b/tools/tracing/rtla/src/timerlat.c @@ -25,7 +25,8 @@ static void timerlat_usage(void) " usage: [rtla] timerlat [MODE] ...", "", " modes:", - " top - prints the summary from timerlat tracer", + " top - prints the summary from timerlat tracer", + " hist - prints a histogram of timer latencies", "", "if no MODE is given, the top mode is called, passing the arguments", NULL, @@ -60,6 +61,9 @@ int timerlat_main(int argc, char *argv[]) } else if (strcmp(argv[1], "top") == 0) { timerlat_top_main(argc-1, &argv[1]); exit(0); + } else if (strcmp(argv[1], "hist") == 0) { + timerlat_hist_main(argc-1, &argv[1]); + exit(0); } usage: diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h index 4f0f912b3b81..3ae3c1d94903 100644 --- a/tools/tracing/rtla/src/timerlat.h +++ b/tools/tracing/rtla/src/timerlat.h @@ -1,2 +1,3 @@ +int timerlat_hist_main(int argc, char *argv[]); int timerlat_top_main(int argc, char *argv[]); int timerlat_main(int argc, char *argv[]); diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c new file mode 100644 index 000000000000..6a6a6a5bb629 --- /dev/null +++ b/tools/tracing/rtla/src/timerlat_hist.c @@ -0,0 +1,778 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "utils.h" +#include "osnoise.h" +#include "timerlat.h" + +struct timerlat_hist_params { + char *cpus; + char *monitored_cpus; + char *trace_output; + unsigned long long runtime; + long long stop_us; + long long stop_total_us; + long long timerlat_period_us; + long long print_stack; + int sleep_time; + int output_divisor; + int duration; + int set_sched; + struct sched_attr sched_param; + + char no_irq; + char no_thread; + char no_header; + char no_summary; + char no_index; + char skip_zeroes; + int bucket_size; + int entries; +}; + +struct timerlat_hist_cpu { + int *irq; + int *thread; + + int irq_count; + int thread_count; + + unsigned long long min_irq; + unsigned long long sum_irq; + unsigned long long max_irq; + + unsigned long long min_thread; + unsigned long long sum_thread; + unsigned long long max_thread; +}; + +struct timerlat_hist_data { + struct timerlat_hist_cpu *hist; + int entries; + int bucket_size; + int nr_cpus; +}; + +/* + * timerlat_free_histogram - free runtime data + */ +static void +timerlat_free_histogram(struct timerlat_hist_data *data) +{ + int cpu; + + /* one histogram for IRQ and one for thread, per CPU */ + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (data->hist[cpu].irq) + free(data->hist[cpu].irq); + + if (data->hist[cpu].thread) + free(data->hist[cpu].thread); + } + + /* one set of histograms per CPU */ + if (data->hist) + free(data->hist); + + free(data); +} + +/* + * timerlat_alloc_histogram - alloc runtime data + */ +static struct timerlat_hist_data +*timerlat_alloc_histogram(int nr_cpus, int entries, int bucket_size) +{ + struct timerlat_hist_data *data; + int cpu; + + data = calloc(1, sizeof(*data)); + if (!data) + return NULL; + + data->entries = entries; + data->bucket_size = bucket_size; + data->nr_cpus = nr_cpus; + + /* one set of histograms per CPU */ + data->hist = calloc(1, sizeof(*data->hist) * nr_cpus); + if (!data->hist) + goto cleanup; + + /* one histogram for IRQ and one for thread, per cpu */ + for (cpu = 0; cpu < nr_cpus; cpu++) { + data->hist[cpu].irq = calloc(1, sizeof(*data->hist) * (entries + 1)); + if (!data->hist[cpu].irq) + goto cleanup; + data->hist[cpu].thread = calloc(1, sizeof(*data->hist) * (entries + 1)); + if (!data->hist[cpu].thread) + goto cleanup; + } + + /* set the min to max */ + for (cpu = 0; cpu < nr_cpus; cpu++) { + data->hist[cpu].min_irq = ~0; + data->hist[cpu].min_thread = ~0; + } + + return data; + +cleanup: + timerlat_free_histogram(data); + return NULL; +} + +/* + * timerlat_hist_update - record a new timerlat occurent on cpu, updating data + */ +static void +timerlat_hist_update(struct osnoise_tool *tool, int cpu, + unsigned long long thread, + unsigned long long latency) +{ + struct timerlat_hist_params *params = tool->params; + struct timerlat_hist_data *data = tool->data; + int entries = data->entries; + int bucket; + int *hist; + + if (params->output_divisor) + latency = latency / params->output_divisor; + + if (data->bucket_size) + bucket = latency / data->bucket_size; + + if (!thread) { + hist = data->hist[cpu].irq; + data->hist[cpu].irq_count++; + update_min(&data->hist[cpu].min_irq, &latency); + update_sum(&data->hist[cpu].sum_irq, &latency); + update_max(&data->hist[cpu].max_irq, &latency); + } else { + hist = data->hist[cpu].thread; + data->hist[cpu].thread_count++; + update_min(&data->hist[cpu].min_thread, &latency); + update_sum(&data->hist[cpu].sum_thread, &latency); + update_max(&data->hist[cpu].max_thread, &latency); + } + + if (bucket < entries) + hist[bucket]++; + else + hist[entries]++; +} + +/* + * timerlat_hist_handler - this is the handler for timerlat tracer events + */ +static int +timerlat_hist_handler(struct trace_seq *s, struct tep_record *record, + struct tep_event *event, void *data) +{ + struct trace_instance *trace = data; + unsigned long long thread, latency; + struct osnoise_tool *tool; + int cpu = record->cpu; + + tool = container_of(trace, struct osnoise_tool, trace); + + tep_get_field_val(s, event, "context", record, &thread, 1); + tep_get_field_val(s, event, "timer_latency", record, &latency, 1); + + timerlat_hist_update(tool, cpu, thread, latency); + + return 0; +} + +/* + * timerlat_hist_header - print the header of the tracer to the output + */ +static void timerlat_hist_header(struct osnoise_tool *tool) +{ + struct timerlat_hist_params *params = tool->params; + struct timerlat_hist_data *data = tool->data; + struct trace_seq *s = tool->trace.seq; + char duration[26]; + int cpu; + + if (params->no_header) + return; + + get_duration(tool->start_time, duration, sizeof(duration)); + trace_seq_printf(s, "# RTLA timerlat histogram\n"); + trace_seq_printf(s, "# Time unit is %s (%s) \n", + params->output_divisor == 1 ? "nanoseconds" : "microseconds", + params->output_divisor == 1 ? "ns" : "us"); + + trace_seq_printf(s, "# Duration: %s\n", duration); + + if (!params->no_index) + trace_seq_printf(s, "Index"); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) + trace_seq_printf(s, " IRQ-%03d", cpu); + + if (!params->no_thread) + trace_seq_printf(s, " Thr-%03d", cpu); + } + trace_seq_printf(s, "\n"); + + + trace_seq_do_printf(s); + trace_seq_reset(s); +} + +/* + * timerlat_print_summary - print the summary of the hist data to the output + */ +static void +timerlat_print_summary(struct timerlat_hist_params *params, + struct trace_instance *trace, + struct timerlat_hist_data *data) +{ + int cpu; + + if (params->no_summary) + return; + + if (!params->no_index) + trace_seq_printf(trace->seq, "count:" ); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].irq_count); + + if (!params->no_thread) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].thread_count); + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "min: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].min_irq); + + if (!params->no_thread) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].min_thread); + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "avg: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) { + if (data->hist[cpu].irq_count) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].sum_irq / data->hist[cpu].irq_count); + else + trace_seq_printf(trace->seq, " - "); + } + + if (!params->no_thread) { + if (data->hist[cpu].thread_count) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].sum_thread / data->hist[cpu].thread_count); + else + trace_seq_printf(trace->seq, " - "); + } + } + trace_seq_printf(trace->seq, "\n"); + + if (!params->no_index) + trace_seq_printf(trace->seq, "max: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].max_irq); + + if (!params->no_thread) + trace_seq_printf(trace->seq, "%9llu ", + data->hist[cpu].max_thread); + } + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); +} + +/* + * timerlat_print_stats - print data for all CPUs + */ +static void +timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *tool) +{ + struct timerlat_hist_data *data = tool->data; + struct trace_instance *trace = &tool->trace; + int bucket, cpu; + int total; + + timerlat_hist_header(tool); + + for (bucket = 0; bucket < data->entries; bucket++) { + total = 0; + + if (!params->no_index) + trace_seq_printf(trace->seq, "%-6d", + bucket * data->bucket_size); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + + if (!params->no_irq) { + total += data->hist[cpu].irq[bucket]; + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].irq[bucket]); + } + + if (!params->no_thread) { + total += data->hist[cpu].thread[bucket]; + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].thread[bucket]); + } + + } + + if (params->skip_zeroes && total == 0) { + trace_seq_reset(trace->seq); + continue; + } + + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); + } + + if (!params->no_index) + trace_seq_printf(trace->seq, "over: "); + + for (cpu = 0; cpu < data->nr_cpus; cpu++) { + if (params->cpus && !params->monitored_cpus[cpu]) + continue; + + if (!params->no_irq) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].irq[data->entries]); + + if (!params->no_thread) + trace_seq_printf(trace->seq, "%9d ", + data->hist[cpu].thread[data->entries]); + } + trace_seq_printf(trace->seq, "\n"); + trace_seq_do_printf(trace->seq); + trace_seq_reset(trace->seq); + + timerlat_print_summary(params, trace, data); +} + +/* + * timerlat_hist_usage - prints timerlat top usage message + */ +static void timerlat_hist_usage(char *usage) +{ + int i; + + char *msg[] = { + "", + " usage: [rtla] timerlat hist [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\", + " [-c cpu-list] [-P priority] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\", + " [--no-index] [--skip-zeros]", + "", + " -h/--help: print this menu", + " -p/--period us: timerlat period in us", + " -i/--irq us: stop trace if the irq latency is higher than the argument in us", + " -T/--thread us: stop trace if the thread latency is higher than the argument in us", + " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us", + " -c/--cpus cpus: run the tracer only on the given cpus", + " -d/--duration time[m|h|d]: duration of the session in seconds", + " -T/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]", + " -n/--nano: display data in nanoseconds", + " -b/--bucket-size N: set the histogram bucket size (default 1)", + " -e/--entries N: set the number of entries of the histogram (default 256)", + " --no-irq: ignore IRQ latencies", + " --no-thread: ignore thread latencies", + " --no-header: do not print header", + " --no-summary: do not print summary", + " --no-index: do not print index", + " --skip-zeros: skip zero only entries", + " -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters", + " o:prio - use SCHED_OTHER with prio", + " r:prio - use SCHED_RR with prio", + " f:prio - use SCHED_FIFO with prio", + " d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period", + " in nanoseconds", + NULL, + }; + + if (usage) + fprintf(stderr, "%s\n", usage); + + fprintf(stderr, "rtla timerlat hist: display a per-cpu histogram of the timer latency (version %s)\n", VERSION); + for(i = 0; msg[i]; i++) + fprintf(stderr, "%s\n", msg[i]); + exit(1); +} + +/* + * timerlat_hist_parse_args - allocs, parse and fill the cmd line parameters + */ +static struct timerlat_hist_params +*timerlat_hist_parse_args(int argc, char *argv[]) +{ + struct timerlat_hist_params *params; + int retval; + int c; + + params = calloc(1, sizeof(*params)); + if (!params) + exit(1); + + /* display data in microseconds */ + params->output_divisor = 1000; + params->bucket_size = 1; + params->entries = 256; + + while (1) { + static struct option long_options[] = { + {"cpus", required_argument, 0, 'c'}, + {"bucket-size", required_argument, 0, 'b'}, + {"debug", no_argument, 0, 'D'}, + {"entries", required_argument, 0, 'e'}, + {"duration", required_argument, 0, 'd'}, + {"help", no_argument, 0, 'h'}, + {"irq", required_argument, 0, 'i'}, + {"nano", no_argument, 0, 'n'}, + {"period", required_argument, 0, 'p'}, + {"priority", required_argument, 0, 'P'}, + {"stack", required_argument, 0, 's'}, + {"thread", required_argument, 0, 'T'}, + {"trace", optional_argument, 0, 't'}, + {"no-irq", no_argument, 0, '0'}, + {"no-thread", no_argument, 0, '1'}, + {"no-header", no_argument, 0, '2'}, + {"no-summary", no_argument, 0, '3'}, + {"no-index", no_argument, 0, '4'}, + {"skip-zeros", no_argument, 0, '5'}, + {0, 0, 0, 0} + }; + + /* getopt_long stores the option index here. */ + int option_index = 0; + + c = getopt_long(argc, argv, "c:b:d:e:Dhi:np:P:s:t::T:012345", + long_options, &option_index); + + /* detect the end of the options. */ + if (c == -1) + break; + + switch (c) { + case 'c': + retval = parse_cpu_list(optarg, ¶ms->monitored_cpus); + if (retval) + timerlat_hist_usage("\nInvalid -c cpu list\n"); + params->cpus = optarg; + debug_msg("-c for %s\n", params->cpus); + break; + case 'b': + params->bucket_size = get_long_from_str(optarg); + if ((params->bucket_size == 0) || (params->bucket_size >= 1000000)) + timerlat_hist_usage("Bucket size needs to be > 0 and <= 1000000\n"); + break; + case 'D': + config_debug = 1; + break; + case 'd': + params->duration = parse_seconds_duration(optarg); + if (!params->duration) + timerlat_hist_usage("Invalid -D duration\n"); + break; + case 'e': + params->entries = get_long_from_str(optarg); + if ((params->entries < 10) || (params->entries > 9999999)) + timerlat_hist_usage("Entries must be > 10 and < 9999999\n"); + break; + case 'h': + case '?': + timerlat_hist_usage(NULL); + break; + case 'i': + params->stop_us = get_long_from_str(optarg); + break; + case 'n': + params->output_divisor = 1; + break; + case 'p': + params->timerlat_period_us = get_long_from_str(optarg); + if (params->timerlat_period_us > 1000000) + timerlat_hist_usage("Period longer than 1 s\n"); + break; + case 'P': + retval = parse_prio(optarg, ¶ms->sched_param); + if (retval == -1) + timerlat_hist_usage("Invalid -P priority"); + params->set_sched = 1; + break; + case 's': + params->print_stack = get_long_from_str(optarg); + break; + case 'T': + params->stop_total_us = get_long_from_str(optarg); + break; + case 't': + if (optarg) + /* skip = */ + params->trace_output = &optarg[1]; + else + params->trace_output = "timerlat_trace.txt"; + break; + case '0': /* no irq */ + params->no_irq = 1; + break; + case '1': /* no thread */ + params->no_thread = 1; + break; + case '2': /* no header */ + params->no_header = 1; + break; + case '3': /* no summary */ + params->no_summary = 1; + break; + case '4': /* no index */ + params->no_index = 1; + break; + case '5': /* skip zeroes */ + params->skip_zeroes = 1; + break; + default: + timerlat_hist_usage("Invalid option"); + } + } + + if (geteuid()) { + err_msg("rtla needs root permission\n"); + exit(EXIT_FAILURE); + } + + if (params->no_irq && params->no_thread) + timerlat_hist_usage("no-irq and no-thread set, there is nothing to do here"); + + if (params->no_index && params->skip_zeroes) + timerlat_hist_usage("no-index and skip-zeros set - it does not make sense"); + + return params; +} + +/* + * timerlat_hist_apply_config - apply the hist configs to the initialized tool + */ +static int +timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_params *params) +{ + int retval; + + if (!params->sleep_time) + params->sleep_time = 1; + + if (params->cpus) { + retval = osnoise_set_cpus(tool->context, params->cpus); + if (retval) { + err_msg("Failed to apply CPUs config\n"); + goto out_err; + } + } + + if (params->stop_us) + osnoise_set_stop_us(tool->context, params->stop_us); + + if (params->stop_total_us) + osnoise_set_stop_total_us(tool->context, params->stop_total_us); + + if (params->timerlat_period_us) + osnoise_set_timerlat_period_us(tool->context, params->timerlat_period_us); + + if (params->print_stack) + osnoise_set_print_stack(tool->context, params->print_stack); + + return 0; + +out_err: + return -1; +} + +/* + * timerlat_init_hist - initialize a timerlat hist tool with parameters + */ +static struct osnoise_tool +*timerlat_init_hist(struct timerlat_hist_params *params) +{ + struct osnoise_tool *tool; + int nr_cpus; + + nr_cpus = sysconf(_SC_NPROCESSORS_CONF); + + tool = osnoise_init_tool("timerlat_hist"); + if (!tool) + return NULL; + + tool->data = timerlat_alloc_histogram(nr_cpus, params->entries, params->bucket_size); + if (!tool->data) + goto out_err; + + tool->params = params; + + tep_register_event_handler(tool->trace.tep, -1, "ftrace", "timerlat", + timerlat_hist_handler, tool); + + return tool; + +out_err: + osnoise_destroy_tool(tool); + return NULL; +} + +static int stop_tracing = 0; +static void stop_hist(int sig) +{ + stop_tracing = 1; +} + +/* + * timerlat_hist_set_signals - handles the signal to stop the tool + */ +static void +timerlat_hist_set_signals(struct timerlat_hist_params *params) +{ + signal(SIGINT, stop_hist); + if (params->duration) { + signal(SIGALRM, stop_hist); + alarm(params->duration); + } +} + +int timerlat_hist_main(int argc, char *argv[]) +{ + struct timerlat_hist_params *params; + struct trace_instance *trace; + struct osnoise_tool *record; + struct osnoise_tool *tool; + int return_value = 1; + int retval; + + params = timerlat_hist_parse_args(argc, argv); + if (!params) + exit(1); + + tool = timerlat_init_hist(params); + if (!tool) { + err_msg("Could not init osnoise hist\n"); + goto out_exit; + } + + retval = timerlat_hist_apply_config(tool, params); + if (retval) { + err_msg("Could not apply config\n"); + goto out_exit; + } + + trace = &tool->trace; + + retval = enable_timerlat(trace); + if (retval) { + err_msg("Failed to enable timerlat tracer\n"); + goto out_hist; + } + + if (params->set_sched) { + retval = set_comm_sched_attr("timerlat/", ¶ms->sched_param); + if (retval) + timerlat_hist_usage("Failed to set sched parameters\n"); + } + + trace_instance_start(trace); + + if (params->trace_output) { + record = osnoise_init_trace_tool("timerlat"); + if (!record) { + err_msg("Failed to enable the trace instance\n"); + goto out_hist; + } + trace_instance_start(&record->trace); + } + + tool->start_time = time(NULL); + timerlat_hist_set_signals(params); + + while (!stop_tracing) { + sleep(params->sleep_time); + + retval = tracefs_iterate_raw_events(trace->tep, + trace->inst, + NULL, + 0, + collect_registered_events, + trace); + if (retval < 0) { + err_msg("Error iterating on events\n"); + goto out_hist; + } + + if (!tracefs_trace_is_on(trace->inst)) + break; + }; + + timerlat_print_stats(params, tool); + + return_value = 0; + + if (!tracefs_trace_is_on(trace->inst)) { + printf("rtla timelat hit stop tracing\n"); + if (params->trace_output) { + printf(" Saving trace to %s\n", params->trace_output); + save_trace_to_file(record->trace.inst, params->trace_output); + } + } + +out_hist: + timerlat_free_histogram(tool->data); + osnoise_destroy_tool(tool); + if (params->trace_output) + osnoise_destroy_tool(record); + free(params); +out_exit: + exit(return_value); +} From patchwork Mon Oct 11 14:36:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22073C433EF for ; Mon, 11 Oct 2021 14:39:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 060A160EB6 for ; Mon, 11 Oct 2021 14:39:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243332AbhJKOk7 convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:40:59 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:32311 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243446AbhJKOjs (ORCPT ); Mon, 11 Oct 2021 10:39:48 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-587-g4c8S0ZnMdaxuB_U2p6_pQ-1; Mon, 11 Oct 2021 10:37:41 -0400 X-MC-Unique: g4c8S0ZnMdaxuB_U2p6_pQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7562B1922967; Mon, 11 Oct 2021 14:37:39 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9ADB419C59; Mon, 11 Oct 2021 14:37:35 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 13/19] rtla: Add Documentation Date: Mon, 11 Oct 2021 16:36:01 +0200 Message-Id: <8d5e1eac4e90a0bad5a9a169d9e0e7972223f694.1633958325.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Adds the basis for rtla documentation. It is based on libtracefs Documentation as suggested by Steven Rostedt. This patch also includes the rtla(1) man page. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Suggested-by: Steven Rostedt Signed-off-by: Daniel Bristot de Oliveira --- tools/tracing/rtla/Documentation/Makefile | 223 ++++++++++++++++++ .../tracing/rtla/Documentation/asciidoc.conf | 118 +++++++++ .../rtla/Documentation/manpage-base.xsl | 35 +++ .../rtla/Documentation/manpage-normal.xsl | 13 + tools/tracing/rtla/Documentation/rtla.txt | 56 +++++ tools/tracing/rtla/Documentation/utils.mk | 144 +++++++++++ tools/tracing/rtla/Makefile | 20 +- 7 files changed, 604 insertions(+), 5 deletions(-) create mode 100644 tools/tracing/rtla/Documentation/Makefile create mode 100644 tools/tracing/rtla/Documentation/asciidoc.conf create mode 100644 tools/tracing/rtla/Documentation/manpage-base.xsl create mode 100644 tools/tracing/rtla/Documentation/manpage-normal.xsl create mode 100644 tools/tracing/rtla/Documentation/rtla.txt create mode 100644 tools/tracing/rtla/Documentation/utils.mk diff --git a/tools/tracing/rtla/Documentation/Makefile b/tools/tracing/rtla/Documentation/Makefile new file mode 100644 index 000000000000..790e5c1afaeb --- /dev/null +++ b/tools/tracing/rtla/Documentation/Makefile @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: LGPL-2.1 + +include utils.mk + +# This Makefile and manpage XSL files were taken from libtracefs +# and modified + + +MAN1_TXT= \ + $(wildcard rtla-*.txt) \ + rtla.txt + +MAN_TXT = $(MAN1_TXT) +_MAN_XML=$(patsubst %.txt,%.xml,$(MAN_TXT)) +_MAN_HTML=$(patsubst %.txt,%.html,$(MAN_TXT)) +_DOC_MAN1=$(patsubst %.txt,%.m,$(MAN1_TXT)) + +MAN_XML=$(addprefix $(OUTPUT),$(_MAN_XML)) +MAN_HTML=$(addprefix $(OUTPUT),$(_MAN_HTML)) +DOC_MAN1=$(addprefix $(OUTPUT),$(_DOC_MAN1)) + +# Make the path relative to DESTDIR, not prefix +ifndef DESTDIR +prefix?=$(HOME) +endif +bindir?=$(prefix)/bin +htmldir?=$(prefix)/share/doc/rtla-doc +pdfdir?=$(prefix)/share/doc/rtla-doc +mandir?=$(prefix)/share/man +man1dir=$(mandir)/man1 + +ASCIIDOC=asciidoc +ASCIIDOC_EXTRA = --unsafe -f asciidoc.conf +ASCIIDOC_HTML = xhtml11 +MANPAGE_XSL = manpage-normal.xsl +XMLTO_EXTRA = --skip-validation +INSTALL?=install +RM ?= rm -f + +ifdef USE_ASCIIDOCTOR +ASCIIDOC = asciidoctor +ASCIIDOC_EXTRA = -a compat-mode +ASCIIDOC_EXTRA += -I. -rasciidoctor-extensions +ASCIIDOC_EXTRA += -a mansource="rtla" -a manmanual="rtla Manual" +ASCIIDOC_HTML = xhtml5 +endif + +ASCIIDOC_INSTALLED := $(shell command -v $(ASCIIDOC) 2> /dev/null) +ifndef ASCIIDOC_INSTALLED + missing_tools += $(ASCIIDOC) +endif + +XMLTO=xmlto +XMLTO_INSTALLED := $(shell command -v $(XMLTO) 2> /dev/null) +ifndef XMLTO_INSTALLED + missing_tools += $(XMLTO) +endif + +# +# For asciidoc ... +# -7.1.2, no extra settings are needed. +# 8.0-, set ASCIIDOC8. +# + +# +# For docbook-xsl ... +# -1.68.1, set ASCIIDOC_NO_ROFF? (based on changelog from 1.73.0) +# 1.69.0, no extra settings are needed? +# 1.69.1-1.71.0, set DOCBOOK_SUPPRESS_SP? +# 1.71.1, no extra settings are needed? +# 1.72.0, set DOCBOOK_XSL_172. +# 1.73.0-, set ASCIIDOC_NO_ROFF +# + +# +# If you had been using DOCBOOK_XSL_172 in an attempt to get rid +# of 'the ".ft C" problem' in your generated manpages, and you +# instead ended up with weird characters around callouts, try +# using ASCIIDOC_NO_ROFF instead (it works fine with ASCIIDOC8). +# + +ifdef ASCIIDOC8 +ASCIIDOC_EXTRA += -a asciidoc7compatible +endif +ifdef DOCBOOK_XSL_172 +ASCIIDOC_EXTRA += -a rtla-asciidoc-no-roff +MANPAGE_XSL = manpage-1.72.xsl +else + ifdef ASCIIDOC_NO_ROFF + # docbook-xsl after 1.72 needs the regular XSL, but will not + # pass-thru raw roff codes from asciidoc.conf, so turn them off. + ASCIIDOC_EXTRA += -a rtla-asciidoc-no-roff + endif +endif +ifdef MAN_BOLD_LITERAL +XMLTO_EXTRA += -m manpage-bold-literal.xsl +endif +ifdef DOCBOOK_SUPPRESS_SP +XMLTO_EXTRA += -m manpage-suppress-sp.xsl +endif + +SHELL_PATH ?= $(SHELL) +# Shell quote; +SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH)) + +DESTDIR ?= +DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))' + +export DESTDIR DESTDIR_SQ + +QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir +QUIET_SUBDIR1 = + +ifneq ($(findstring $(MAKEFLAGS),w),w) +PRINT_DIR = --no-print-directory +else # "make -w" +NO_SUBDIR = : +endif + +ifneq ($(findstring $(MAKEFLAGS),s),s) +ifneq ($(V),1) + QUIET_ASCIIDOC = @echo ' ASCIIDOC '$@; + QUIET_XMLTO = @echo ' XMLTO '$@; + QUIET_SUBDIR0 = +@subdir= + QUIET_SUBDIR1 = ;$(NO_SUBDIR) \ + echo ' SUBDIR ' $$subdir; \ + $(MAKE) $(PRINT_DIR) -C $$subdir + export V +endif +endif + +all: check-man-tools html man1 + +man: man1 +man1: $(DOC_MAN1) + +html: $(MAN_HTML) $(MAN1_HTML) + +$(MAN_HTML) $(MAN1_HTML) $(DOC_MAN1): asciidoc.conf + +install: check-man-tools install-man install-html + +check-man-tools: +ifdef missing_tools + $(error "You need to install $(missing_tools) for man pages") +endif + +install-%.1: $(OUTPUT)%.1 + $(Q)$(call do_install,$<,$(man1dir),644); + +do-install-man: man $(addprefix install-,$(wildcard $(OUTPUT)*.1)) + +install-man: man + $(Q)$(MAKE) -C . do-install-man + +install-%.txt: $(OUTPUT)%.html + $(Q)$(call do_install,$<,$(htmldir),644); + +install-%.txt.1: $(OUTPUT)%.html + $(Q)$(call do_install,$<,$(htmldir),644); + +do-install-html: html $(addprefix install-,$(wildcard *.txt)) $(addprefix install-,$(wildcard *.txt.1)) + +install-html: html do-install-html + +uninstall: uninstall-man uninstall-html + +uninstall-man: + $(Q)$(RM) $(addprefix $(DESTDIR)$(man1dir)/,$(DOC_MAN1)) + +uninstall-html: + $(Q)$(RM) $(addprefix $(DESTDIR)$(htmldir)/,$(MAN_HTML)) $(addprefix $(DESTDIR)$(htmldir)/,$(MAN1_HTML)) + +ifdef missing_tools + DO_INSTALL_MAN = $(warning Please install $(missing_tools) to have the man pages installed) +else + DO_INSTALL_MAN = do-install-man +endif + +CLEAN_FILES = \ + $(MAN_XML) $(addsuffix +,$(MAN_XML)) \ + $(MAN_HTML) $(addsuffix +,$(MAN_HTML)) \ + $(DOC_MAN1) *.1 *.m + +clean: + $(Q) $(RM) $(CLEAN_FILES) + +ifdef USE_ASCIIDOCTOR +$(OUTPUT)%.1 : $(OUTPUT)%.txt.1 + $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ + $(ASCIIDOC) -b manpage -d manpage \ + $(ASCIIDOC_EXTRA) -artla_version=$(TRACEFS_VERSION) -o $@+ $< && \ + mv $@+ $@ +endif + +$(OUTPUT)%.m : $(OUTPUT)%.xml + $(QUIET_XMLTO)$(RM) $@ && \ + $(XMLTO) -o $(OUTPUT). -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<; \ + touch $@ + +$(OUTPUT)%.xml : %.txt + $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ + $(ASCIIDOC) -b docbook -d manpage \ + $(ASCIIDOC_EXTRA) -artla_version=$(TRACEFS_VERSION) -o $@+ $< && \ + mv $@+ $@ + +$(OUTPUT)%.xml : %.txt.1 + $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ + $(ASCIIDOC) -b docbook -d manpage \ + $(ASCIIDOC_EXTRA) -artla_version=$(TRACEFS_VERSION) -o $@+ $< && \ + mv $@+ $@ + +$(MAN_HTML): $(OUTPUT)%.html : %.txt + $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ + $(ASCIIDOC) -b $(ASCIIDOC_HTML) -d manpage \ + $(ASCIIDOC_EXTRA) -artla_version=$(TRACEFS_VERSION) -o $@+ $< && \ + mv $@+ $@ + +$(MAN1_HTML): $(OUTPUT)%.html : %.txt.1 + $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \ + $(ASCIIDOC) -b $(ASCIIDOC_HTML) -d manpage \ + $(ASCIIDOC_EXTRA) -artla_version=$(TRACEFS_VERSION) -o $@+ $< && \ + mv $@+ $@ diff --git a/tools/tracing/rtla/Documentation/asciidoc.conf b/tools/tracing/rtla/Documentation/asciidoc.conf new file mode 100644 index 000000000000..40678f1ab901 --- /dev/null +++ b/tools/tracing/rtla/Documentation/asciidoc.conf @@ -0,0 +1,118 @@ +## linktep: macro +# +# Usage: linktep:command[manpage-section] +# +# Note, {0} is the manpage section, while {target} is the command. +# +# Show TEP link as: (
); if section is defined, else just show +# the command. + +[macros] +(?su)[\\]?(?Plinktep):(?P\S*?)\[(?P.*?)\]= + +[attributes] +asterisk=* +plus=+ +caret=^ +startsb=[ +endsb=] +tilde=~ + +ifdef::backend-docbook[] +[linktep-inlinemacro] +{0%{target}} +{0#} +{0#{target}{0}} +{0#} +endif::backend-docbook[] + +ifdef::backend-docbook[] +ifndef::tep-asciidoc-no-roff[] +# "unbreak" docbook-xsl v1.68 for manpages. v1.69 works with or without this. +# v1.72 breaks with this because it replaces dots not in roff requests. +[listingblock] +{title} + +ifdef::doctype-manpage[] +endif::doctype-manpage[] +| +ifdef::doctype-manpage[] +endif::doctype-manpage[] + +{title#} +endif::tep-asciidoc-no-roff[] + +ifdef::tep-asciidoc-no-roff[] +ifdef::doctype-manpage[] +# The following two small workarounds insert a simple paragraph after screen +[listingblock] +{title} + +| + +{title#} + +[verseblock] +{title} +{title%} +{title#} +| + +{title#} +{title%} +endif::doctype-manpage[] +endif::tep-asciidoc-no-roff[] +endif::backend-docbook[] + +ifdef::doctype-manpage[] +ifdef::backend-docbook[] +[header] +template::[header-declarations] + + +{mantitle} +{manvolnum} +rtla +{rtla_version} +RTLA Manual + + + {manname1} + {manname2} + {manname3} + {manname4} + {manname5} + {manname6} + {manname7} + {manname8} + {manname9} + {manname10} + {manname11} + {manname12} + {manname13} + {manname14} + {manname15} + {manname16} + {manname17} + {manname18} + {manname19} + {manname20} + {manname21} + {manname22} + {manname23} + {manname24} + {manname25} + {manname26} + {manname27} + {manname28} + {manname29} + {manname30} + {manpurpose} + +endif::backend-docbook[] +endif::doctype-manpage[] + +ifdef::backend-xhtml11[] +[linktep-inlinemacro] +{target}{0?({0})} +endif::backend-xhtml11[] diff --git a/tools/tracing/rtla/Documentation/manpage-base.xsl b/tools/tracing/rtla/Documentation/manpage-base.xsl new file mode 100644 index 000000000000..a264fa616093 --- /dev/null +++ b/tools/tracing/rtla/Documentation/manpage-base.xsl @@ -0,0 +1,35 @@ + + + + + + + + + + + + + + sp + + + + + + + + br + + + diff --git a/tools/tracing/rtla/Documentation/manpage-normal.xsl b/tools/tracing/rtla/Documentation/manpage-normal.xsl new file mode 100644 index 000000000000..a48f5b11f3dc --- /dev/null +++ b/tools/tracing/rtla/Documentation/manpage-normal.xsl @@ -0,0 +1,13 @@ + + + + + + +\ +. + + diff --git a/tools/tracing/rtla/Documentation/rtla.txt b/tools/tracing/rtla/Documentation/rtla.txt new file mode 100644 index 000000000000..bc02aeb35d04 --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla.txt @@ -0,0 +1,56 @@ +rtla(1) +======= + +NAME +---- +rtla - Real-time Linux Analysis tool + +SYNOPSIS +-------- +*rtla* COMMAND [OPTIONS] + +DESCRIPTION +----------- +The rtla(1) is a meta-tool that includes a set of commands that aims to +analyze the real-time properties of Linux. But instead of testing Linux +as a black box, rtla leverages kernel tracing capabilities to provide +precise information about the properties and root causes of unexpected +results. + +COMMANDS +-------- +*osnoise*:: +Gives information about the operating system noise (osnoise). +*timerlat*:: +Measures the irq and thread timerlat latency. + +OPTIONS +------- +*-h*, --help + Display the help text. + +For other options, see the man page for the corresponding command. + +SEE ALSO +-------- +_rtla-osnoise(1)_, _rtla-timerlat(1)_. + +AUTHOR +------ +Daniel Bristot de Oliveira . + +REPORTING BUGS +-------------- +Report bugs to . + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2. + +RESOURCES +--------- + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). diff --git a/tools/tracing/rtla/Documentation/utils.mk b/tools/tracing/rtla/Documentation/utils.mk new file mode 100644 index 000000000000..32a9b3c5b0e2 --- /dev/null +++ b/tools/tracing/rtla/Documentation/utils.mk @@ -0,0 +1,144 @@ +# SPDX-License-Identifier: LGPL-2.1 + +# Utils + + GOBJ = $(notdir $(strip $@)) + + +ifeq ($(VERBOSE),1) + Q = + S = +else + Q = @ + S = -s +endif + +# Use empty print_* macros if either SILENT or VERBOSE. +ifeq ($(findstring 1,$(SILENT)$(VERBOSE)),1) + print_compile = + print_app_build = + print_fpic_compile = + print_shared_lib_compile = + print_plugin_obj_compile = + print_plugin_build = + print_install = + print_uninstall = + print_update = +else + print_compile = echo ' COMPILE '$(GOBJ); + print_app_build = echo ' BUILD '$(GOBJ); + print_fpic_compile = echo ' COMPILE FPIC '$(GOBJ); + print_shared_lib_compile = echo ' COMPILE SHARED LIB '$(GOBJ); + print_plugin_obj_compile = echo ' COMPILE PLUGIN OBJ '$(GOBJ); + print_plugin_build = echo ' BUILD PLUGIN '$(GOBJ); + print_static_lib_build = echo ' BUILD STATIC LIB '$(GOBJ); + print_install = echo ' INSTALL '$1' to $(DESTDIR_SQ)$2'; + print_uninstall = echo ' UNINSTALL $(DESTDIR_SQ)$1'; + print_update = echo ' UPDATE '$(GOBJ); +endif + +do_fpic_compile = \ + ($(print_fpic_compile) \ + $(CC) -c $(CPPFLAGS) $(CFLAGS) $(EXT) -fPIC $< -o $@) + +do_compile = \ + ($(if $(GENERATE_PIC), $(do_fpic_compile), \ + $(print_compile) \ + $(CC) -c $(CPPFLAGS) $(CFLAGS) $(EXT) $< -o $@)) + +do_app_build = \ + ($(print_app_build) \ + $(CC) $^ -rdynamic -o $@ $(LDFLAGS) $(CONFIG_LIBS) $(LIBS)) + +do_build_static_lib = \ + ($(print_static_lib_build) \ + $(RM) $@; $(AR) rcs $@ $^) + +do_compile_shared_library = \ + ($(print_shared_lib_compile) \ + $(CC) --shared $^ '-Wl,-soname,$(1),-rpath=$$ORIGIN' -o $@ $(LIBS)) + +do_compile_plugin_obj = \ + ($(print_plugin_obj_compile) \ + $(CC) -c $(CPPFLAGS) $(CFLAGS) -fPIC -o $@ $<) + +do_plugin_build = \ + ($(print_plugin_build) \ + $(CC) $(CFLAGS) $(LDFLAGS) -shared -nostartfiles -o $@ $<) + +do_compile_python_plugin_obj = \ + ($(print_plugin_obj_compile) \ + $(CC) -c $(CPPFLAGS) $(CFLAGS) $(PYTHON_DIR_SQ) $(PYTHON_INCLUDES) -fPIC -o $@ $<) + +do_python_plugin_build = \ + ($(print_plugin_build) \ + $(CC) $< -shared $(LDFLAGS) $(PYTHON_LDFLAGS) -o $@) + +define make_version.h + (echo '/* This file is automatically generated. Do not modify. */'; \ + echo \#define VERSION_CODE $(shell \ + expr $(VERSION) \* 256 + $(PATCHLEVEL)); \ + echo '#define EXTRAVERSION ' $(EXTRAVERSION); \ + echo '#define VERSION_STRING "'$(VERSION).$(PATCHLEVEL).$(EXTRAVERSION)'"'; \ + echo '#define FILE_VERSION '$(FILE_VERSION); \ + if [ -d $(src)/.git ]; then \ + d=`git diff`; \ + x=""; \ + if [ ! -z "$$d" ]; then x="+"; fi; \ + echo '#define VERSION_GIT "'$(shell \ + git log -1 --pretty=format:"%H" 2>/dev/null)$$x'"'; \ + else \ + echo '#define VERSION_GIT "not-a-git-repo"'; \ + fi \ + ) > $1 +endef + +define update_version.h + ($(call make_version.h, $@.tmp); \ + if [ -r $@ ] && cmp -s $@ $@.tmp; then \ + rm -f $@.tmp; \ + else \ + $(print_update) \ + mv -f $@.tmp $@; \ + fi); +endef + +define update_dir + (echo $1 > $@.tmp; \ + if [ -r $@ ] && cmp -s $@ $@.tmp; then \ + rm -f $@.tmp; \ + else \ + $(print_update) \ + mv -f $@.tmp $@; \ + fi); +endef + +define build_prefix + (echo $1 > $@.tmp; \ + if [ -r $@ ] && cmp -s $@ $@.tmp; then \ + rm -f $@.tmp; \ + else \ + $(print_update) \ + mv -f $@.tmp $@; \ + fi); +endef + +define do_install_mkdir + if [ ! -d '$(DESTDIR_SQ)$1' ]; then \ + $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \ + fi +endef + +define do_install + $(print_install) \ + $(call do_install_mkdir,$2); \ + $(INSTALL) $(if $3,-m $3,) $1 '$(DESTDIR_SQ)$2' +endef + +define do_install_pkgconfig_file + if [ -n "${pkgconfig_dir}" ]; then \ + $(call do_install,$(PKG_CONFIG_FILE),$(pkgconfig_dir),644); \ + else \ + (echo Failed to locate pkg-config directory) 1>&2; \ + fi +endef diff --git a/tools/tracing/rtla/Makefile b/tools/tracing/rtla/Makefile index 580a33db6645..9a914391e507 100644 --- a/tools/tracing/rtla/Makefile +++ b/tools/tracing/rtla/Makefile @@ -17,7 +17,7 @@ LIBS := -ltracefs -ltraceevent -lpthread -lprocps SRC := $(wildcard src/*.c) HDR := $(wildcard src/*.h) OBJ := $(SRC:.c=.o) -DIRS := src +DIRS := src Documentation FILES := Makefile CEXT := bz2 TARBALL := $(NAME)-$(VERSION).tar.$(CEXT) @@ -27,19 +27,20 @@ DATADIR := /usr/share DOCDIR := $(DATADIR)/doc MANDIR := $(DATADIR)/man LICDIR := $(DATADIR)/licenses +SRCTREE := $(if $(BUILD_SRC),$(BUILD_SRC),$(CURDIR)) .PHONY: all tests all: rtla -rtla: $(OBJ) +rtla: $(OBJ) doc $(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS) static: $(OBJ) $(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl .PHONY: install -install: +install: doc_install $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise @@ -47,15 +48,24 @@ install: ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat .PHONY: clean tarball push -clean: +clean: doc_clean @test ! -f rtla || rm rtla @test ! -f rtla-static || rm rtla-static @test ! -f src/rtla.o || rm src/rtla.o @test ! -f $(TARBALL) || rm -f $(TARBALL) @rm -rf *~ $(OBJ) *.tar.$(CEXT) -tarball: clean +tarball: clean rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION) cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION) tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION) rm -rf $(NAME)-$(VERSION) + +doc: + $(MAKE) -C $(SRCTREE)/Documentation all + +doc_clean: + $(MAKE) -C $(SRCTREE)/Documentation clean + +doc_install: + $(MAKE) -C $(SRCTREE)/Documentation install From patchwork Mon Oct 11 14:36:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06114C433F5 for ; Mon, 11 Oct 2021 14:39:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E596560E98 for ; Mon, 11 Oct 2021 14:39:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243740AbhJKOlM convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:41:12 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:32098 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243768AbhJKOkF (ORCPT ); Mon, 11 Oct 2021 10:40:05 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-115-OkDu8Ek4MRSdTpPui3RX0g-1; Mon, 11 Oct 2021 10:38:01 -0400 X-MC-Unique: OkDu8Ek4MRSdTpPui3RX0g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7495280292B; Mon, 11 Oct 2021 14:37:59 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id C9C8F19C59; Mon, 11 Oct 2021 14:37:39 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 14/19] rtla: Add rtla osnoise man page Date: Mon, 11 Oct 2021 16:36:02 +0200 Message-Id: <1b5a5183af9fbc5a64721620e2d344fed1d0f85d.1633958325.git.bristot@kernel.org> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla osnoise command. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-osnoise.txt | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-osnoise.txt diff --git a/tools/tracing/rtla/Documentation/rtla-osnoise.txt b/tools/tracing/rtla/Documentation/rtla-osnoise.txt new file mode 100644 index 000000000000..2da49246b645 --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-osnoise.txt @@ -0,0 +1,68 @@ +rtla-osnoise(1) +=============== + +NAME +---- +rtla-osnoise - Measure the operating system noise + +SYNOPSIS +-------- +*rtla osnoise* [MODE] ... + +DESCRIPTION +----------- +The rtla-osnoise(1) tool is an interface for the osnoise tracer. The osnoise +tracer dispatches a kernel thread per-cpu. These threads read the time in +a loop while with preemption, softirqs and IRQs enabled, thus allowing all +the sources of osnoise during its execution. The osnoise threads take note +of the entry and exit point of any source of interferences, increasing a +per-cpu interference counter. The osnoise tracer also saves an interference +counter for each source of interference. + +The osnoise tracer outputs information in two ways. It periodically prints +a summary of the noise of the operating system, including the counters of +the occurrence of the source of interference. It also provides information +for each noise via the osnoise tracepoints. The rtla-osnoise-top(1) mode +display information about the periodic summary from the osnoise tracer. +The rtla-osnoise-hist(1) mode display information about the noise using +the *osnoise:* tracepoints. For further details, please refer to the +respective man page. + +MODES +----- +*top*:: + Prints the summary from osnoise tracer. +*hist*:: + Prints a histogram of osnoise samples. + +If no MODE is given, the top mode is called, passing the arguments. + +OPTIONS +------- +*-h*, *--help*:: +Display the help text. + +For other options, see the man page for the corresponding mode. + +SEE ALSO +-------- +_rtla-osnoise-top(1)_, _rtla-osnoise-hist(1)_ + +Osnoise tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). From patchwork Mon Oct 11 14:36:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F747C433EF for ; Mon, 11 Oct 2021 14:39:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6901E60BD3 for ; Mon, 11 Oct 2021 14:39:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243863AbhJKOlV convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:41:21 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:22853 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243911AbhJKOkL (ORCPT ); Mon, 11 Oct 2021 10:40:11 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-195-9GvSpm-sOSOS87zmdjCx7w-1; Mon, 11 Oct 2021 10:38:07 -0400 X-MC-Unique: 9GvSpm-sOSOS87zmdjCx7w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A1757835DE0; Mon, 11 Oct 2021 14:38:05 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id A9C8219D9B; Mon, 11 Oct 2021 14:37:59 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 15/19] rtla: Add rtla osnoise top documentation Date: Mon, 11 Oct 2021 16:36:03 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla osnoise top mode. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-osnoise-top.txt | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-osnoise-top.txt diff --git a/tools/tracing/rtla/Documentation/rtla-osnoise-top.txt b/tools/tracing/rtla/Documentation/rtla-osnoise-top.txt new file mode 100644 index 000000000000..c7c0e7983c6f --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-osnoise-top.txt @@ -0,0 +1,98 @@ +rtla-osnoise-top(1) +=================== + +NAME +---- +rtla-osnoise-top - Display a summary of the operating system noise + +SYNOPSIS +-------- +*rtla osnoise top* ['OPTIONS'] + +DESCRIPTION +----------- +The rtla-osnoise-top(1) tool is an interface for the osnoise tracer. The +osnoise tracer dispatches a kernel thread per-cpu. These threads read the +time in a loop while with preemption, softirq and IRQs enabled, thus +allowing all the sources of osnoise during its execution. The osnoise's +tracer threads take note of the delta between each time read, along with +an interference counter for each source of interference. At the end of +each period, the osnoise tracer displays a summary of the results. + +*rtla osnoise top* collects the periodic summary from the osnoise tracer, +including the counters of the occurrence of the interference source, +displaying the results in a user-friendly format. + +The tool also allows many configurations of the osnoise tracer and the +collection of the tracer output. + +OPTIONS +------- +*-h*, *--help*:: +Print help menu. +*-p*, *--period* 'us':: +Set the osnoise tracer period in microseconds. +*-r*, *--runtime* 'us':: +Set the osnoise tracer runtime in microseconds. +*-s*, *--stop* 'us':: +Stop the trace if a single sample is higher than the argument in microseconds. +If -T is set, it will also save the trace to the output. +*-S*, *--stop-total* 'us':: +Stop the trace if the total sample is higher than the argument in microseconds. +If -T is set, it will also save the trace to the output. +*-c*, *--cpus* 'cpu-list':: +Set the osnoise tracer to run the sample threads in the cpu-list. +*-d*, *--duration* 'time[s|m|h|d]':: +Set the duration of the session. +*-t*, *--trace*['=file']:: +Save the stopped trace to ['file|osnoise_trace.txt']. +*-q*, *--quiet*:: +Print only a summary at the end of the session. +*-P*, *--priority* 'o:prio|r:prio|f:prio|d:runtime:period':: +Set scheduling parameters to the osnoise tracer threads, the format to set the priority are: + - 'o:prio' - use SCHED_OTHER with 'prio'; + - 'r:prio' - use SCHED_RR with 'prio'; + - 'f:prio' - use SCHED_FIFO with 'prio'; + - 'd:runtime[us|ms|s]:period[us|ms|s]' - use SCHED_DEADLINE with 'runtime' and 'period' in nanoseconds. + +EXAMPLE +------- +In the example below, the rtla osnoise top tool is set to run with a +real-time priority 'FIFO:1', on CPUs '0-3', for '900ms' at each period +('1s' by default). The reason for reducing the runtime is to avoid starving +the rtla tool. The tool is also set to run for 'one minute' and to display +a summary of the report at the end of the session. + +------------------------------------------------------- +[root@f34 ~]# rtla osnoise top -P F:1 -c 0-3 -r 900000 -d 1M -q + Operating System Noise +duration: 0 00:01:00 | time is in us +CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI IRQ Softirq Thread + 0 #59 53100000 304896 99.42580 6978 56 549 0 53111 1590 13 + 1 #59 53100000 338339 99.36282 8092 24 399 0 53130 1448 31 + 2 #59 53100000 290842 99.45227 6582 39 855 0 53110 1406 12 + 3 #59 53100000 204935 99.61405 6251 33 290 0 53156 1460 12 +------------------------------------------------------- + +SEE ALSO +-------- +_rtla-osnoise(1)_, _rtla-osnoise-hist_(1) + +Osnoise tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). From patchwork Mon Oct 11 14:36:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DBD2C4332F for ; Mon, 11 Oct 2021 14:39:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA96E603E9 for ; Mon, 11 Oct 2021 14:39:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243046AbhJKOlY convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:41:24 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:58675 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243952AbhJKOkP (ORCPT ); Mon, 11 Oct 2021 10:40:15 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-288-mstXSbDvNYuh71aS25uLYQ-1; Mon, 11 Oct 2021 10:38:11 -0400 X-MC-Unique: mstXSbDvNYuh71aS25uLYQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 192D1801A92; Mon, 11 Oct 2021 14:38:10 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0780419C59; Mon, 11 Oct 2021 14:38:05 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 16/19] rtla: Add rtla osnoise hist documentation Date: Mon, 11 Oct 2021 16:36:04 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla osnoise hist mode. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-osnoise-hist.txt | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-osnoise-hist.txt diff --git a/tools/tracing/rtla/Documentation/rtla-osnoise-hist.txt b/tools/tracing/rtla/Documentation/rtla-osnoise-hist.txt new file mode 100644 index 000000000000..e5e6aac8d4c0 --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-osnoise-hist.txt @@ -0,0 +1,117 @@ +rtla-osnoise-hist(1) +=================== + +NAME +---- +rtla-osnoise-hist - Display a histogram of the osnoise tracer samples + +SYNOPSIS +-------- +*rtla osnoise hist* ['OPTIONS'] + +DESCRIPTION +----------- +The *rtla-osnoise-hist(1)* tool is an interface for the osnoise tracer. The +osnoise tracer dispatches a kernel thread per-cpu. These threads read the +time in a loop while with preemption, SoftIRQs and IRQs enabled, thus +allowing all the sources of osnoise during its execution. The osnoise threads +take note of the delta between each time read. Anytime the delta between two +consecutive reads of the timer is higher than a 'threshold,' an +*osnoise:sample_threshold* event is generated reporting the detected noise. + +The *rtla-osnoise-hist(1)* tool collects all *osnoise:sample_threshold* +occurrence in a histogram, displaying the results in a user-friendly way. +The tool also allows many configurations of the osnoise tracer and the +collection of the tracer output. + +OPTIONS +------- +*-h*, *--help*:: +Print help menu. +*-p*, *--period* 'us':: +Set the osnoise tracer period in microseconds. +*-r*, *--runtime* 'us':: +Set the osnoise tracer runtime in microseconds. +*-s*, *--stop* 'us':: +Stop the trace if a single sample is higher than the argument in microseconds. +If -t is set, it will also save the trace to the output. +*-S*, *--stop-total* 'us':: +Stop the trace if the total sample is higher than the argument in microseconds. +If -t is set, it will also save the trace to the output. +*-c*, *--cpus* 'cpu-list':: +Set the osnoise tracer to run the sample threads in the cpu-list. +*-d*, *--duration* 'time[s|m|h|d]':: +Set the duration of the session. +*-t*, *--trace*['=file']:: +Save the stopped trace to ['file|osnoise_trace.txt']. +*-P*, *--priority* 'o:prio|r:prio|f:prio|d:runtime:period':: +Set scheduling parameters to the osnoise tracer threads, the format to +set the priority are: + - 'o:prio' - use SCHED_OTHER with 'prio'; + - 'r:prio' - use SCHED_RR with 'prio'; + - 'f:prio' - use SCHED_FIFO with 'prio'; + - 'd:runtime[us|ms|s]:period[us|ms|s]' - use SCHED_DEADLINE with +'runtime' and 'period' in nanoseconds. + +*-b*, --bucket-size 'N':: +Set the histogram bucket size (default 1). +*-e*, --entries 'N':: +Set the number of entries of the histogram (default 256). +*--no-header*:: +Do not print header. +*--no-summary*:: +Do not print summary. +*--no-index*:: +Do not print index. +*--skip-zeros*:: +Skip zero only entries. + +EXAMPLE +------- +In the example below, osnoise tracer threads are set to run with real-time +priority 'FIFO:1', on CPUs '0-11', for '900ms' at each period ('1s' by +default). The reason for reducing the runtime is to avoid starving the rtla +tool. The tool is also set to run for 'one minute.' The output histogram is +set to group outputs in buckets of '10 us' and '25' entries. + +------------------------------------------------------- +[root@f34 ~/]# rtla osnoise hist -P F:1 -c 0-11 -r 900000 -d 1M --skip-zeros -b 10 -e 25 +# RTLA osnoise histogram +# Time unit is microseconds (us) +# Duration: 0 00:01:00 +Index CPU-000 CPU-001 CPU-002 CPU-003 CPU-004 CPU-005 CPU-006 CPU-007 CPU-008 CPU-009 CPU-010 CPU-011 +0 42982 46287 51779 53740 52024 44817 49898 36500 50408 50128 49523 52377 +10 12224 8356 2912 878 2667 10155 4573 18894 4214 4836 5708 2413 +20 8 5 12 2 13 24 20 41 29 53 39 39 +30 1 1 0 0 10 3 6 19 15 31 30 38 +40 0 0 0 0 0 4 2 7 2 3 8 11 +50 0 0 0 0 0 0 0 0 0 1 1 2 +over: 0 0 0 0 0 0 0 0 0 0 0 0 +count: 55215 54649 54703 54620 54714 55003 54499 55461 54668 55052 55309 54880 +min: 0 0 0 0 0 0 0 0 0 0 0 0 +avg: 0 0 0 0 0 0 0 0 0 0 0 0 +max: 30 30 20 20 30 40 40 40 40 50 50 50 +------------------------------------------------------- + +SEE ALSO +-------- +_rtla-osnoise(1)_, _rtla-osnoise-top_(1) + +Osnoise tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). From patchwork Mon Oct 11 14:36:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D8FC433EF for ; Mon, 11 Oct 2021 14:40:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 63B3F606A5 for ; Mon, 11 Oct 2021 14:40:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244139AbhJKOmJ convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:42:09 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:22008 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244287AbhJKOkp (ORCPT ); Mon, 11 Oct 2021 10:40:45 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-547-_hVP4DPjMGSgqQ_yLohEHg-1; Mon, 11 Oct 2021 10:38:40 -0400 X-MC-Unique: _hVP4DPjMGSgqQ_yLohEHg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 80D80192296A; Mon, 11 Oct 2021 14:38:38 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 642C119C59; Mon, 11 Oct 2021 14:38:10 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 17/19] rtla: Add rtla timerlat documentation Date: Mon, 11 Oct 2021 16:36:05 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla timerlat tool. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-timerlat.txt | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-timerlat.txt diff --git a/tools/tracing/rtla/Documentation/rtla-timerlat.txt b/tools/tracing/rtla/Documentation/rtla-timerlat.txt new file mode 100644 index 000000000000..2b5ef9a9d6b0 --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-timerlat.txt @@ -0,0 +1,65 @@ +rtla-timerlat(1) +================ + +NAME +---- +rtla-timerlat - Measures the operating system timer latency + +SYNOPSIS +-------- +*rtla timerlat* [MODE] ... + +DESCRIPTION +----------- +The rtla-timerlat(1) tool is an interface for the timerlat tracer. The +timerlat tracer dispatches a kernel thread per-cpu. These threads set +a periodic timer to wake themselves up and go back to sleep. After the +wakeup, they collect and generate useful information for the debugging +of operating system timer latency. + +The timerlat tracer outputs information in two ways. It periodically +prints the timer latency at the timer 'IRQ' handler and the 'Thread' handler. +It also provides information for each noise via the osnoise tracepoints. +The rtla-timerlat-top(1) mode displays a summary of the periodic output +from the timerlat tracer. The rtla-hist-hist(1) mode displays a histogram of +each tracer event occurrence. For further details, please refer to the +respective man page. + +MODES +----- +*top*:: + Prints the summary from timerlat tracer. +*hist*:: + Prints a histogram of timerlat samples. + +If no MODE is given, the top mode is called, passing the arguments. + +OPTIONS +------- +*-h*, *--help*:: + Display the help text. + +For other options, see the man page for the corresponding mode. + +SEE ALSO +-------- +_rtla-timerlat-top(1)_, _rtla-timerlat-hist(1)_ + +Timerlat tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). From patchwork Mon Oct 11 14:36:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D391C433EF for ; Mon, 11 Oct 2021 14:40:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F3A360EB6 for ; Mon, 11 Oct 2021 14:40:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238715AbhJKOmL convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:42:11 -0400 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:33885 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244304AbhJKOkq (ORCPT ); Mon, 11 Oct 2021 10:40:46 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-155-0_u4_U2LM2uk8Qz7o0uXIQ-1; Mon, 11 Oct 2021 10:38:45 -0400 X-MC-Unique: 0_u4_U2LM2uk8Qz7o0uXIQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9FA20801B3C; Mon, 11 Oct 2021 14:38:43 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id DDDC719C59; Mon, 11 Oct 2021 14:38:38 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 18/19] rtla: Add rtla timerlat top documentation Date: Mon, 11 Oct 2021 16:36:06 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla timerlat top mode. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-timerlat-top.txt | 181 ++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-timerlat-top.txt diff --git a/tools/tracing/rtla/Documentation/rtla-timerlat-top.txt b/tools/tracing/rtla/Documentation/rtla-timerlat-top.txt new file mode 100644 index 000000000000..59bc8f2ea6cd --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-timerlat-top.txt @@ -0,0 +1,181 @@ +rtla-timerlat-top(1) +==================== + +NAME +---- +rtla-timerlat-top - Measures the operating system timer latency + +SYNOPSIS +-------- +*rtla timerlat top* ['OPTIONS'] ... + +DESCRIPTION +The rtla-timerlat-top(1) mode displays a summary of the periodic output +from the timerlat tracer. + +The timerlat tracer dispatches a kernel thread per-cpu. These threads +set a periodic timer to wake themselves up and go back to sleep. After +the wakeup, they collect and generate useful information for the +debugging of operating system timer latency. + +The timerlat tracer outputs information in two ways. It periodically +prints the timer latency at the timer 'IRQ' handler and the 'Thread' handler. +It also provides information for each noise via the osnoise tracepoints +that can be seem with the option -T. + +OPTIONS +------- +*-h*, *--help*:: +Print help menu. +*-p*, *--period* 'us':: +Set the timerlat tracer period in microseconds. +*-i*, *--irq* 'us':: +Stop trace if the irq latency is higher than the argument in us. +*-T*, *--thread* 'us':: +Stop trace if the thread latency is higher than the argument in us. +*-s*, *--stack* 'us':: +Save the stack trace at the IRQ if a thread latency is higher than the +argument in us. +*-c*, *--cpus* 'cpu-list':: +Set the timerlat tracer to run the sample threads in the cpu-list. +*-d*, *--duration* 'time[s|m|h|d]':: +Set the duration of the session. +*-T*, *--trace*['=file']:: +Save the stopped trace to ['file|timerlat_trace.txt']. +*-q*, *--quiet*:: +Print only a summary at the end of the session. +*-P*, *--priority* 'o:prio|r:prio|f:prio|d:runtime:period':: +Set scheduling parameters to the timerlat tracer threads, the format to set +the priority are: + - 'o:prio' - use SCHED_OTHER with 'prio'; + - 'r:prio' - use SCHED_RR with 'prio'; + - 'f:prio' - use SCHED_FIFO with 'prio'; + - 'd:runtime[us|ms|s]:period[us|ms|s]' - use SCHED_DEADLINE with 'runtime' +and 'period' in nanoseconds. + +EXAMPLE +------- + +In the example below, the timerlat tracer is set to capture the stack trace at +the IRQ handler, printing it to the buffer if the *Thread Timer Latency* is +higher than _30 us_. It is also set to stop the session if a *Thread Timer +Latency* higher than _30 us_ is hit. Finally, it is set to save the trace +buffer if the stop condition is hit. + +------------------------------------------------------- +[root@alien ~]# rtla timerlat top -s 30 -t 30 -T + Timer Latency + 0 00:00:59 | IRQ Timer Latency (us) | Thread Timer Latency (us) +CPU COUNT | cur min avg max | cur min avg max + 0 #58634 | 1 0 1 10 | 11 2 10 23 + 1 #58634 | 1 0 1 9 | 12 2 9 23 + 2 #58634 | 0 0 1 11 | 10 2 9 23 + 3 #58634 | 1 0 1 11 | 11 2 9 24 + 4 #58634 | 1 0 1 10 | 11 2 9 26 + 5 #58634 | 1 0 1 8 | 10 2 9 25 + 6 #58634 | 12 0 1 12 | 30 2 10 30 <--- CPU with spike + 7 #58634 | 1 0 1 9 | 11 2 9 23 + 8 #58633 | 1 0 1 9 | 11 2 9 26 + 9 #58633 | 1 0 1 9 | 10 2 9 26 + 10 #58633 | 1 0 1 13 | 11 2 9 28 + 11 #58633 | 1 0 1 13 | 12 2 9 24 + 12 #58633 | 1 0 1 8 | 10 2 9 23 + 13 #58633 | 1 0 1 10 | 10 2 9 22 + 14 #58633 | 1 0 1 18 | 12 2 9 27 + 15 #58633 | 1 0 1 10 | 11 2 9 28 + 16 #58633 | 0 0 1 11 | 7 2 9 26 + 17 #58633 | 1 0 1 13 | 10 2 9 24 + 18 #58633 | 1 0 1 9 | 13 2 9 22 + 19 #58633 | 1 0 1 10 | 11 2 9 23 + 20 #58633 | 1 0 1 12 | 11 2 9 28 + 21 #58633 | 1 0 1 14 | 11 2 9 24 + 22 #58633 | 1 0 1 8 | 11 2 9 22 + 23 #58633 | 1 0 1 10 | 11 2 9 27 +timerlat hit stop tracing +saving trace to timerlat_trace.txt +[root@alien bristot]# tail -60 timerlat_trace.txt +[...] + timerlat/5-79755 [005] ....... 426.271226: #58634 context thread timer_latency 10823 ns + sh-109404 [006] dnLh213 426.271247: #58634 context irq timer_latency 12505 ns + sh-109404 [006] dNLh313 426.271258: irq_noise: local_timer:236 start 426.271245463 duration 12553 ns + sh-109404 [006] d...313 426.271263: thread_noise: sh:109404 start 426.271245853 duration 4769 ns + timerlat/6-79756 [006] ....... 426.271264: #58634 context thread timer_latency 30328 ns + timerlat/6-79756 [006] ....1.. 426.271265: + => timerlat_irq + => __hrtimer_run_queues + => hrtimer_interrupt + => __sysvec_apic_timer_interrupt + => sysvec_apic_timer_interrupt + => asm_sysvec_apic_timer_interrupt + => _raw_spin_unlock_irqrestore <---- spinlock that disabled interrupt. + => try_to_wake_up + => autoremove_wake_function + => __wake_up_common + => __wake_up_common_lock + => ep_poll_callback + => __wake_up_common + => __wake_up_common_lock + => fsnotify_add_event + => inotify_handle_inode_event + => fsnotify + => __fsnotify_parent + => __fput + => task_work_run + => exit_to_user_mode_prepare + => syscall_exit_to_user_mode + => do_syscall_64 + => entry_SYSCALL_64_after_hwframe + => 0x7265000001378c + => 0x10000cea7 + => 0x25a00000204a + => 0x12e302d00000000 + => 0x19b51010901b6 + => 0x283ce00726500 + => 0x61ea308872 + => 0x00000fe3 + bash-109109 [007] d..h... 426.271265: #58634 context irq timer_latency 1211 ns + timerlat/6-79756 [006] ....... 426.271267: timerlat_main: stop tracing hit on cpu 6 +---------------------------------------- + +In the trace, it is possible the notice that the *IRQ Timer Latency* was +already high, accounting *12505 ns*. The IRQ delay was caused by the +'bash-109109' process that disabled IRQs in the wake-up path +('_try_to_wake_up()' function). The duration of the IRQ handler that woke +up the timerlat thread, informed with the *irq_noise* event, was also high +and added more *12553 ns* to the Thread latency. Finally, the *thread_noise* +added by the currently running thread (including the scheduling overhead) +added more *4769 ns*. Summing up these values, the *Thread Timer Latency* +accounted for *30328 ns*. + +The primary reason for this high value is the wake-up path that was hit +twice during this case: when the 'bash-109109' was waking up a thread +and then when the 'timerlat' thread was awakened. This information can +then be used as the starting point of a more fine-grained analysis. + +Note that timerlat was dispatched without changing timerlat threads' priority. +That is generally not needed because the thread has priority FIFO:95 by +default, which is a common priority used by real-time kernel developers to +analyze scheduling delays. + +SEE ALSO +-------- +_rtla-timerlat(1)_, _rtla-timerlat-hist(1)_ + +Timerlat tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL). From patchwork Mon Oct 11 14:36:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12550211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6910C4332F for ; Mon, 11 Oct 2021 14:40:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9474F60E98 for ; Mon, 11 Oct 2021 14:40:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243419AbhJKOmW convert rfc822-to-8bit (ORCPT ); Mon, 11 Oct 2021 10:42:22 -0400 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:55507 "EHLO us-smtp-delivery-44.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244362AbhJKOk5 (ORCPT ); Mon, 11 Oct 2021 10:40:57 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-274-4AUeVWkkNAO7dA1OGZ6sAw-1; Mon, 11 Oct 2021 10:38:54 -0400 X-MC-Unique: 4AUeVWkkNAO7dA1OGZ6sAw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A2508BD521; Mon, 11 Oct 2021 14:38:52 +0000 (UTC) Received: from x1.bristot.me.homenet.telecomitalia.it (unknown [10.22.17.206]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0724919C59; Mon, 11 Oct 2021 14:38:43 +0000 (UTC) From: Daniel Bristot de Oliveira To: Steven Rostedt Cc: Jonathan Corbet , Kate Carcia , John Kacur , Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , Juri Lelli , Clark Williams , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , linux-rt-users@vger.kernel.org, linux-trace-devel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 19/19] rtla: Add rtla timerlat hist documentation Date: Mon, 11 Oct 2021 16:36:07 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=bristot@kernel.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Man page for rtla timerlat hist mode. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Juri Lelli Cc: Clark Williams Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Sebastian Andrzej Siewior Cc: Daniel Bristot de Oliveira Cc: linux-rt-users@vger.kernel.org Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira --- .../rtla/Documentation/rtla-timerlat-hist.txt | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt diff --git a/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt b/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt new file mode 100644 index 000000000000..fbbc981aa82d --- /dev/null +++ b/tools/tracing/rtla/Documentation/rtla-timerlat-hist.txt @@ -0,0 +1,162 @@ +rtla-timerlat-hist(1) +===================== + +NAME +---- +rtla-timerlat-hist - Histograms of the operating system timer latency + +SYNOPSIS +-------- +*rtla timerlat hist* ['OPTIONS'] ... + +DESCRIPTION +----------- +The rtla-hist-hist(1) mode displays a histogram of each tracer event occurrence. + +The rtla-timerlat(1) tool is an interface for the timerlat tracer. The +timerlat tracer dispatches a kernel thread per-cpu. These threads set a +periodic timer to wake themselves up and go back to sleep. After the wakeup, +they collect and generate useful information for the debugging of operating +system timer latency. + +The timerlat tracer outputs information in two ways. It periodically prints +the timer latency at the timer 'IRQ' handler and the 'Thread' handler. It +also provides information for each noise via the osnoise tracepoints. This +tool uses the periodic information, and the osnoise tracepoints are enabled +when using the -T option. + +OPTIONS +------- +*-h*, *--help*:: +Print help menu. +*-p*, *--period* 'us':: +Set the timerlat tracer period in microseconds. +*-i*, *--irq* 'us':: +Stop trace if the irq latency is higher than the argument in us. +*-T*, *--thread* 'us':: +Stop trace if the thread latency is higher than the argument in us. +*-s*, *--stack* 'us':: +Save the stack trace at the IRQ if a thread latency is higher than the +argument in us. +*-c*, *--cpus* 'cpu-list':: +Set the timerlat tracer to run the sample threads in the cpu-list. +*-d*, *--duration* 'time[s|m|h|d]':: +Set the duration of the session. +*-T*, *--trace*['=file']:: +Save the stopped trace to ['file|timerlat_trace.txt']. +*-P*, *--priority* 'o:prio|r:prio|f:prio|d:runtime:period':: +Set scheduling parameters to the timerlat tracer threads, the format to +set the priority are: + - 'o:prio' - use SCHED_OTHER with 'prio'; + - 'r:prio' - use SCHED_RR with 'prio'; + - 'f:prio' - use SCHED_FIFO with 'prio'; + - 'd:runtime[us|ms|s]:period[us|ms|s]' - use SCHED_DEADLINE with 'runtime' and +'period' in nanoseconds. + +*-b*, *--bucket-size* 'N':: +Set the histogram bucket size (default 1). +*-e*, *--entries* 'N':: +Set the number of entries of the histogram (default 256). +*--no-irq*:: +Ignore IRQ latencies. +*--no-thread*:: +Ignore thread latencies. +*--no-header*:: +So not print header. +*--no-summary*:: +Do not print summary. +*--no-index*:: +So not print index. +*--skip-zeros*:: +Skip zero only entries. + +EXAMPLE +------- +In the example below, *rtla timerlat hist* is set to run for '10' minutes, +in the cpus '0-4', 'skipping zero' only lines. Moreover, *rtla timerlat +hist* will change the priority of the timelat threads to run under +'SCHED_DEADLINE' priority, with a '10 us' runtime every '1 ms' period. The +'1ms' period is also passed to the timerlat tracer. + +------------------------------------------ +[root@alien ~]# timerlat hist -d 10m -c 0-4 --skip-zeros -P d:100us:1ms -p 1ms +# RTLA timerlat histogram +# Time unit is microseconds (us) +# Duration: 0 00:10:00 +Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 IRQ-004 Thr-004 +0 276489 0 206089 0 466018 0 481102 0 205546 0 +1 318327 35487 388149 30024 94531 48382 83082 71078 388026 55730 +2 3282 122584 4019 126527 28231 109012 23311 89309 4568 98739 +3 940 11815 837 9863 6209 16227 6895 17196 910 9780 +4 444 17287 424 11574 2097 38443 2169 36736 462 13476 +5 206 43291 255 25581 1223 101908 1304 101137 236 28913 +6 132 101501 96 64584 635 213774 757 215471 99 73453 +7 74 169347 65 124758 350 57466 441 53639 69 148573 +8 53 85183 31 156751 229 9052 306 9026 39 139907 +9 22 10387 12 42762 161 2554 225 2689 19 26192 +10 13 1898 8 5770 114 1247 128 1405 13 3772 +11 9 560 9 924 71 686 76 765 8 713 +12 4 256 2 360 50 411 64 474 3 278 +13 2 167 2 172 43 256 53 350 4 180 +14 1 88 1 116 15 198 42 223 0 115 +15 2 63 3 94 11 139 20 150 0 58 +16 2 37 0 56 5 78 10 102 0 39 +17 0 18 0 28 4 57 8 80 0 15 +18 0 8 0 17 2 50 6 56 0 12 +19 0 9 0 5 0 19 0 48 0 18 +20 0 4 0 8 0 11 2 27 0 4 +21 0 2 0 3 1 9 1 18 0 6 +22 0 1 0 3 1 7 0 3 0 5 +23 0 2 0 4 0 2 0 7 0 2 +24 0 2 0 2 1 3 0 3 0 5 +25 0 0 0 1 0 1 0 1 0 3 +26 0 1 0 0 0 2 0 2 0 0 +27 0 0 0 3 0 1 0 0 0 1 +28 0 0 0 3 0 0 0 1 0 0 +29 0 0 0 2 0 2 0 1 0 3 +30 0 1 0 0 0 0 0 0 0 0 +31 0 1 0 0 0 0 0 2 0 2 +32 0 0 0 1 0 2 0 0 0 0 +33 0 0 0 2 0 0 0 0 0 1 +34 0 0 0 0 0 0 0 0 0 2 +35 0 1 0 1 0 0 0 0 0 1 +36 0 1 0 0 0 1 0 1 0 0 +37 0 0 0 1 0 0 0 0 0 0 +40 0 0 0 0 0 1 0 1 0 0 +41 0 0 0 0 0 0 0 0 0 1 +42 0 0 0 0 0 0 0 0 0 1 +44 0 0 0 0 0 1 0 0 0 0 +46 0 0 0 0 0 0 0 1 0 0 +47 0 0 0 0 0 0 0 0 0 1 +50 0 0 0 0 0 0 0 0 0 1 +54 0 0 0 1 0 0 0 0 0 0 +58 0 0 0 1 0 0 0 0 0 0 +over: 0 0 0 0 0 0 0 0 0 0 +count: 600002 600002 600002 600002 600002 600002 600002 600002 600002 600002 +min: 0 1 0 1 0 1 0 1 0 1 +avg: 0 5 0 5 0 4 0 4 0 5 +max: 16 36 15 58 24 44 21 46 13 50 +------------------------------------------ + +SEE ALSO +-------- +_rtla-timerlat(1)_, _rtla-timerlat-top(1)_ + +Timerlat tracer documentation: + +AUTHOR +------ +Written by Daniel Bristot de Oliveira + +REPORTING BUGS +-------------- +Report bugs to + +LICENSE +------- +rtla is Free Software licensed under the GNU GPLv2 + +COPYING +------- +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL).