From patchwork Thu Aug 12 13:57:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12433683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7692C4338F for ; Thu, 12 Aug 2021 14:20:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B4439610A3 for ; Thu, 12 Aug 2021 14:20:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237386AbhHLOUk (ORCPT ); Thu, 12 Aug 2021 10:20:40 -0400 Received: from gateway30.websitewelcome.com ([192.185.144.21]:27857 "EHLO gateway30.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237341AbhHLOUj (ORCPT ); Thu, 12 Aug 2021 10:20:39 -0400 Received: from cm11.websitewelcome.com (cm11.websitewelcome.com [100.42.49.5]) by gateway30.websitewelcome.com (Postfix) with ESMTP id D292911F19 for ; Thu, 12 Aug 2021 08:57:36 -0500 (CDT) Received: from gator4132.hostgator.com ([192.185.4.144]) by cmsmtp with SMTP id EBDEmBXkmK61iEBDEm9bX7; Thu, 12 Aug 2021 08:57:36 -0500 X-Authority-Reason: nr=8 Received: from host-79-56-204-120.retail.telecomitalia.it ([79.56.204.120]:54910 helo=f34.bristot.me) by gator4132.hostgator.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mEBDD-004IPl-Vw; Thu, 12 Aug 2021 08:57:36 -0500 From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , linux-trace-devel@vger.kernel.org Subject: [PATCH 1/5] trace/osnoise: Do not follow tracing_cpumask Date: Thu, 12 Aug 2021 15:57:20 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4132.hostgator.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kernel.org X-BWhitelist: no X-Source-IP: 79.56.204.120 X-Source-L: No X-Exim-ID: 1mEBDD-004IPl-Vw X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: host-79-56-204-120.retail.telecomitalia.it (f34.bristot.me) [79.56.204.120]:54910 X-Source-Auth: kernel@bristot.me X-Email-Count: 8 X-Source-Cap: YnJpc3RvdG1lO2JyaXN0b3RtZTtnYXRvcjQxMzIuaG9zdGdhdG9yLmNvbQ== X-Local-Domain: no Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In preparation to support multiple instances, decople the osnoise/timelat workload from instance specific tracing_cpumask. Different instances can have conflicing cpumasks, making osnoise workload management needlessly complex. Osnoise already have its global cpu mask. I also thought about using the first instance mask, but the "first" instance could be removed before the others. This also fixes the problem that changing the tracing_mask was not re-starting the trace. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Daniel Bristot de Oliveira Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira (Red Hat) --- kernel/trace/trace_osnoise.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index b576e96d9741..aadb0d110155 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1553,13 +1553,10 @@ static int start_per_cpu_kthreads(struct trace_array *tr) get_online_cpus(); /* - * Run only on CPUs in which trace and osnoise are allowed to run. + * Run only on online CPUs in which trace and osnoise are allowed to + * run. */ - cpumask_and(current_mask, tr->tracing_cpumask, &osnoise_cpumask); - /* - * And the CPU is online. - */ - cpumask_and(current_mask, cpu_online_mask, current_mask); + cpumask_and(current_mask, cpu_online_mask, &osnoise_cpumask); for_each_possible_cpu(cpu) per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL; @@ -1580,10 +1577,8 @@ static int start_per_cpu_kthreads(struct trace_array *tr) #ifdef CONFIG_HOTPLUG_CPU static void osnoise_hotplug_workfn(struct work_struct *dummy) { - struct trace_array *tr = osnoise_trace; unsigned int cpu = smp_processor_id(); - mutex_lock(&trace_types_lock); if (!osnoise_busy) @@ -1595,9 +1590,6 @@ static void osnoise_hotplug_workfn(struct work_struct *dummy) if (!cpumask_test_cpu(cpu, &osnoise_cpumask)) goto out_unlock; - if (!cpumask_test_cpu(cpu, tr->tracing_cpumask)) - goto out_unlock; - start_kthread(cpu); out_unlock: @@ -1700,13 +1692,10 @@ static void osnoise_tracer_stop(struct trace_array *tr); * interface to the osnoise trace. By default, it lists all CPUs, * in this way, allowing osnoise threads to run on any online CPU * of the system. It serves to restrict the execution of osnoise to the - * set of CPUs writing via this interface. Note that osnoise also - * respects the "tracing_cpumask." Hence, osnoise threads will run only - * on the set of CPUs allowed here AND on "tracing_cpumask." Why not - * have just "tracing_cpumask?" Because the user might be interested - * in tracing what is running on other CPUs. For instance, one might - * run osnoise in one HT CPU while observing what is running on the - * sibling HT CPU. + * set of CPUs writing via this interface. Why not use "tracing_cpumask"? + * Because the user might be interested in tracing what is running on + * other CPUs. For instance, one might run osnoise in one HT CPU + * while observing what is running on the sibling HT CPU. */ static ssize_t osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, From patchwork Thu Aug 12 13:57:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12433677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8C35C4338F for ; Thu, 12 Aug 2021 14:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72ED7610A5 for ; Thu, 12 Aug 2021 14:19:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236834AbhHLOUB (ORCPT ); Thu, 12 Aug 2021 10:20:01 -0400 Received: from gateway22.websitewelcome.com ([192.185.46.224]:38475 "EHLO gateway22.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231960AbhHLOUA (ORCPT ); Thu, 12 Aug 2021 10:20:00 -0400 Received: from cm10.websitewelcome.com (cm10.websitewelcome.com [100.42.49.4]) by gateway22.websitewelcome.com (Postfix) with ESMTP id BF0D0A90D for ; Thu, 12 Aug 2021 08:57:38 -0500 (CDT) Received: from gator4132.hostgator.com ([192.185.4.144]) by cmsmtp with SMTP id EBDGmziPiLjoREBDGmyxfY; Thu, 12 Aug 2021 08:57:38 -0500 X-Authority-Reason: nr=8 Received: from host-79-56-204-120.retail.telecomitalia.it ([79.56.204.120]:54910 helo=f34.bristot.me) by gator4132.hostgator.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mEBDF-004IPl-TA; Thu, 12 Aug 2021 08:57:38 -0500 From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , linux-trace-devel@vger.kernel.org Subject: [PATCH 2/5] trace/osnoise: Split workload start from the tracer start Date: Thu, 12 Aug 2021 15:57:21 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4132.hostgator.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kernel.org X-BWhitelist: no X-Source-IP: 79.56.204.120 X-Source-L: No X-Exim-ID: 1mEBDF-004IPl-TA X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: host-79-56-204-120.retail.telecomitalia.it (f34.bristot.me) [79.56.204.120]:54910 X-Source-Auth: kernel@bristot.me X-Email-Count: 15 X-Source-Cap: YnJpc3RvdG1lO2JyaXN0b3RtZTtnYXRvcjQxMzIuaG9zdGdhdG9yLmNvbQ== X-Local-Domain: no Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org In preparation from supporting multiple trace instances, create workload start/stop specific functions. No functional change. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Daniel Bristot de Oliveira Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira (Red Hat) --- kernel/trace/trace_osnoise.c | 53 ++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index aadb0d110155..9b283038fad2 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1545,7 +1545,7 @@ static int start_kthread(unsigned int cpu) * This starts the kernel thread that will look for osnoise on many * cpus. */ -static int start_per_cpu_kthreads(struct trace_array *tr) +static int start_per_cpu_kthreads(void) { struct cpumask *current_mask = &save_cpumask; int retval; @@ -1678,8 +1678,8 @@ osnoise_cpus_read(struct file *filp, char __user *ubuf, size_t count, return count; } -static void osnoise_tracer_start(struct trace_array *tr); -static void osnoise_tracer_stop(struct trace_array *tr); +static int osnoise_workload_start(void); +static void osnoise_workload_stop(void); /* * osnoise_cpus_write - Write function for "cpus" entry @@ -1701,7 +1701,6 @@ static ssize_t osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, loff_t *ppos) { - struct trace_array *tr = osnoise_trace; cpumask_var_t osnoise_cpumask_new; int running, err; char buf[256]; @@ -1726,7 +1725,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_lock(&trace_types_lock); running = osnoise_busy; if (running) - osnoise_tracer_stop(tr); + osnoise_workload_stop(); mutex_lock(&interface_lock); /* @@ -1740,7 +1739,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_unlock(&interface_lock); if (running) - osnoise_tracer_start(tr); + osnoise_workload_start(); mutex_unlock(&trace_types_lock); free_cpumask_var(osnoise_cpumask_new); @@ -1921,7 +1920,10 @@ static int osnoise_hook_events(void) return -EINVAL; } -static int __osnoise_tracer_start(struct trace_array *tr) +/* + * osnoise_workload_start - start the workload and hook to events + */ +static int osnoise_workload_start(void) { int retval; @@ -1936,7 +1938,7 @@ static int __osnoise_tracer_start(struct trace_array *tr) barrier(); trace_osnoise_callback_enabled = true; - retval = start_per_cpu_kthreads(tr); + retval = start_per_cpu_kthreads(); if (retval) { unhook_irq_events(); return retval; @@ -1947,6 +1949,26 @@ static int __osnoise_tracer_start(struct trace_array *tr) return 0; } +/* + * osnoise_workload_stop - stop the workload and unhook the events + */ +static void osnoise_workload_stop(void) +{ + if (!osnoise_busy) + return; + + trace_osnoise_callback_enabled = false; + barrier(); + + stop_per_cpu_kthreads(); + + unhook_irq_events(); + unhook_softirq_events(); + unhook_thread_events(); + + osnoise_busy = false; +} + static void osnoise_tracer_start(struct trace_array *tr) { int retval; @@ -1954,7 +1976,7 @@ static void osnoise_tracer_start(struct trace_array *tr) if (osnoise_busy) return; - retval = __osnoise_tracer_start(tr); + retval = osnoise_workload_start(); if (retval) pr_err(BANNER "Error starting osnoise tracer\n"); @@ -1965,16 +1987,7 @@ static void osnoise_tracer_stop(struct trace_array *tr) if (!osnoise_busy) return; - trace_osnoise_callback_enabled = false; - barrier(); - - stop_per_cpu_kthreads(); - - unhook_irq_events(); - unhook_softirq_events(); - unhook_thread_events(); - - osnoise_busy = false; + osnoise_workload_stop(); } static int osnoise_tracer_init(struct trace_array *tr) @@ -2017,7 +2030,7 @@ static void timerlat_tracer_start(struct trace_array *tr) osnoise_data.timerlat_tracer = 1; - retval = __osnoise_tracer_start(tr); + retval = osnoise_workload_start(); if (retval) goto out_err; From patchwork Thu Aug 12 13:57:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12433721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FF3C4338F for ; Thu, 12 Aug 2021 14:44:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0777610A5 for ; Thu, 12 Aug 2021 14:44:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236694AbhHLOo4 (ORCPT ); Thu, 12 Aug 2021 10:44:56 -0400 Received: from gateway21.websitewelcome.com ([192.185.45.159]:12996 "EHLO gateway21.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbhHLOo4 (ORCPT ); Thu, 12 Aug 2021 10:44:56 -0400 X-Greylist: delayed 1438 seconds by postgrey-1.27 at vger.kernel.org; Thu, 12 Aug 2021 10:44:55 EDT Received: from cm14.websitewelcome.com (cm14.websitewelcome.com [100.42.49.7]) by gateway21.websitewelcome.com (Postfix) with ESMTP id A244D400EE185 for ; Thu, 12 Aug 2021 08:57:40 -0500 (CDT) Received: from gator4132.hostgator.com ([192.185.4.144]) by cmsmtp with SMTP id EBDIm9M4o7sOiEBDImc9ML; Thu, 12 Aug 2021 08:57:40 -0500 X-Authority-Reason: nr=8 Received: from host-79-56-204-120.retail.telecomitalia.it ([79.56.204.120]:54910 helo=f34.bristot.me) by gator4132.hostgator.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mEBDH-004IPl-RU; Thu, 12 Aug 2021 08:57:40 -0500 From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , linux-trace-devel@vger.kernel.org Subject: [PATCH 3/5] trace/osnoise: Use start/stop_per_cpu_kthreads() on osnoise_cpus_write() Date: Thu, 12 Aug 2021 15:57:22 +0200 Message-Id: <95bb0ae3634f96f2e49b23f1509eb9c47a978266.1628775552.git.bristot@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4132.hostgator.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kernel.org X-BWhitelist: no X-Source-IP: 79.56.204.120 X-Source-L: No X-Exim-ID: 1mEBDH-004IPl-RU X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: host-79-56-204-120.retail.telecomitalia.it (f34.bristot.me) [79.56.204.120]:54910 X-Source-Auth: kernel@bristot.me X-Email-Count: 22 X-Source-Cap: YnJpc3RvdG1lO2JyaXN0b3RtZTtnYXRvcjQxMzIuaG9zdGdhdG9yLmNvbQ== X-Local-Domain: no Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org When writing a new CPU mask via osnoise/cpus, if the tracer is running, the workload is restarted to follow the new cpumask. The restart is currently done using osnoise_workload_start/stop(), which disables the workload *and* the instrumentation. However, disabling the instrumentation is not necessary. Calling start/stop_per_cpu_kthreads() is enough to apply the new osnoise/cpus config. No functional change. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Daniel Bristot de Oliveira Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira (Red Hat) --- kernel/trace/trace_osnoise.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 9b283038fad2..9f95dbd5b19b 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -1678,9 +1678,6 @@ osnoise_cpus_read(struct file *filp, char __user *ubuf, size_t count, return count; } -static int osnoise_workload_start(void); -static void osnoise_workload_stop(void); - /* * osnoise_cpus_write - Write function for "cpus" entry * @filp: The active open file structure @@ -1725,7 +1722,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_lock(&trace_types_lock); running = osnoise_busy; if (running) - osnoise_workload_stop(); + stop_per_cpu_kthreads(); mutex_lock(&interface_lock); /* @@ -1739,7 +1736,7 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, mutex_unlock(&interface_lock); if (running) - osnoise_workload_start(); + start_per_cpu_kthreads(); mutex_unlock(&trace_types_lock); free_cpumask_var(osnoise_cpumask_new); From patchwork Thu Aug 12 13:57:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12433685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA925C432BE for ; Thu, 12 Aug 2021 14:20:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B7737610A3 for ; Thu, 12 Aug 2021 14:20:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237427AbhHLOUm (ORCPT ); Thu, 12 Aug 2021 10:20:42 -0400 Received: from gateway30.websitewelcome.com ([192.185.144.21]:33063 "EHLO gateway30.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237341AbhHLOUm (ORCPT ); Thu, 12 Aug 2021 10:20:42 -0400 Received: from cm11.websitewelcome.com (cm11.websitewelcome.com [100.42.49.5]) by gateway30.websitewelcome.com (Postfix) with ESMTP id E6F8B11306 for ; Thu, 12 Aug 2021 08:57:42 -0500 (CDT) Received: from gator4132.hostgator.com ([192.185.4.144]) by cmsmtp with SMTP id EBDKmBXsqK61iEBDKm9bfS; Thu, 12 Aug 2021 08:57:42 -0500 X-Authority-Reason: nr=8 Received: from host-79-56-204-120.retail.telecomitalia.it ([79.56.204.120]:54910 helo=f34.bristot.me) by gator4132.hostgator.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mEBDJ-004IPl-OR; Thu, 12 Aug 2021 08:57:42 -0500 From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , linux-trace-devel@vger.kernel.org Subject: [PATCH 4/5] trace/osnoise: Support a list of trace_array *tr Date: Thu, 12 Aug 2021 15:57:23 +0200 Message-Id: <0f78cb78890f2bcc83376f309559798032a7fa46.1628775552.git.bristot@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4132.hostgator.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kernel.org X-BWhitelist: no X-Source-IP: 79.56.204.120 X-Source-L: No X-Exim-ID: 1mEBDJ-004IPl-OR X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: host-79-56-204-120.retail.telecomitalia.it (f34.bristot.me) [79.56.204.120]:54910 X-Source-Auth: kernel@bristot.me X-Email-Count: 29 X-Source-Cap: YnJpc3RvdG1lO2JyaXN0b3RtZTtnYXRvcjQxMzIuaG9zdGdhdG9yLmNvbQ== X-Local-Domain: no Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org osnoise/timerlat were built to run a single instance, and for this, a single variable is enough to store the current struct trace_array *tr with information about the tracing instance. This is done via the *osnoise_trace variable. A trace_array represents a trace instance. In preparation to support multiple instances, replace the *osnoise_trace variable with an RCU protected list of instances. The operations that refer to an instance now propagate to all elements of the list (all instances). Also, replace the osnoise_busy variable with a check if the list has elements (busy). No functional change is expected with this patch, i.e., only one instance is allowed yet. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Daniel Bristot de Oliveira Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira (Red Hat) --- kernel/trace/trace_osnoise.c | 253 +++++++++++++++++++++++++---------- 1 file changed, 184 insertions(+), 69 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 9f95dbd5b19b..9193c256377e 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -38,8 +38,6 @@ #define CREATE_TRACE_POINTS #include -static struct trace_array *osnoise_trace; - /* * Default values. */ @@ -50,6 +48,73 @@ static struct trace_array *osnoise_trace; #define DEFAULT_TIMERLAT_PERIOD 1000 /* 1ms */ #define DEFAULT_TIMERLAT_PRIO 95 /* FIFO 95 */ +/* + * trace_array of the enabled osnoise/timerlat instances. + */ +struct osnoise_instance { + struct list_head list; + struct trace_array *tr; +}; +struct list_head osnoise_instances; + +static bool osnoise_has_registered_instances(void) +{ + return !!list_first_or_null_rcu(&osnoise_instances, + struct osnoise_instance, + list); +} + +/* + * osnoise_register_instance - register a new trace instance + * + * Register a trace_array *tr in the list of instances running + * osnoise/timerlat tracers. + */ +static int osnoise_register_instance(struct trace_array *tr) +{ + struct osnoise_instance *inst; + + inst = kmalloc(sizeof(*inst), GFP_KERNEL); + if (!inst) + return -ENOMEM; + + INIT_LIST_HEAD_RCU(&inst->list); + inst->tr = tr; + list_add_tail_rcu(&inst->list, &osnoise_instances); + + return 0; +} + +/* + * osnoise_unregister_instance - unregister a registered trace instance + * + * Remove the trace_array *tr from the list of instances running + * osnoise/timerlat tracers. + */ +static void osnoise_unregister_instance(struct trace_array *tr) +{ + struct osnoise_instance *inst; + int found = 0; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + if (inst->tr == tr) { + list_del_rcu(&inst->list); + found = 1; + break; + } + } + rcu_read_unlock(); + + if (!found) { + WARN("osnoise unregister did not find tr %s\n", tr->name); + return; + } + + synchronize_rcu(); + kfree(inst); +} + /* * NMI runtime info. */ @@ -248,11 +313,6 @@ static struct osnoise_data { #endif }; -/* - * Boolean variable used to inform that the tracer is currently sampling. - */ -static bool osnoise_busy; - #ifdef CONFIG_PREEMPT_RT /* * Print the osnoise header info. @@ -315,19 +375,24 @@ static void print_osnoise_headers(struct seq_file *s) * osnoise_taint - report an osnoise error. */ #define osnoise_taint(msg) ({ \ - struct trace_array *tr = osnoise_trace; \ + struct osnoise_instance *inst; \ + struct trace_buffer *buffer; \ \ - trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \ + rcu_read_lock(); \ + list_for_each_entry_rcu(inst, &osnoise_instances, list) { \ + buffer = inst->tr->array_buffer.buffer; \ + trace_array_printk_buf(buffer, _THIS_IP_, msg); \ + } \ + rcu_read_unlock(); \ osnoise_data.tainted = true; \ }) /* * Record an osnoise_sample into the tracer buffer. */ -static void trace_osnoise_sample(struct osnoise_sample *sample) +static void +__trace_osnoise_sample(struct osnoise_sample *sample, struct trace_buffer *buffer) { - struct trace_array *tr = osnoise_trace; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct trace_event_call *call = &event_osnoise; struct ring_buffer_event *event; struct osnoise_entry *entry; @@ -350,6 +415,22 @@ static void trace_osnoise_sample(struct osnoise_sample *sample) trace_buffer_unlock_commit_nostack(buffer, event); } +/* + * Record an osnoise_sample on all osnoise instances. + */ +static void trace_osnoise_sample(struct osnoise_sample *sample) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __trace_osnoise_sample(sample, buffer); + } + rcu_read_unlock(); +} + #ifdef CONFIG_TIMERLAT_TRACER /* * Print the timerlat header info. @@ -387,14 +468,10 @@ static void print_timerlat_headers(struct seq_file *s) } #endif /* CONFIG_PREEMPT_RT */ -/* - * Record an timerlat_sample into the tracer buffer. - */ -static void trace_timerlat_sample(struct timerlat_sample *sample) +static void +__trace_timerlat_sample(struct timerlat_sample *sample, struct trace_buffer *buffer) { - struct trace_array *tr = osnoise_trace; struct trace_event_call *call = &event_osnoise; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct ring_buffer_event *event; struct timerlat_entry *entry; @@ -411,6 +488,22 @@ static void trace_timerlat_sample(struct timerlat_sample *sample) trace_buffer_unlock_commit_nostack(buffer, event); } +/* + * Record an timerlat_sample into the tracer buffer. + */ +static void trace_timerlat_sample(struct timerlat_sample *sample) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __trace_timerlat_sample(sample, buffer); + } + rcu_read_unlock(); +} + #ifdef CONFIG_STACKTRACE #define MAX_CALLS 256 @@ -450,29 +543,18 @@ static void timerlat_save_stack(int skip) return; } -/* - * timerlat_dump_stack - dump a stack trace previously saved - * - * Dump a saved stack trace into the trace buffer. - */ -static void timerlat_dump_stack(void) + +static void +__timerlat_dump_stack(struct trace_buffer *buffer, struct trace_stack *fstack, unsigned int size) { struct trace_event_call *call = &event_osnoise; - struct trace_array *tr = osnoise_trace; - struct trace_buffer *buffer = tr->array_buffer.buffer; struct ring_buffer_event *event; - struct trace_stack *fstack; struct stack_entry *entry; - unsigned int size; - - preempt_disable_notrace(); - fstack = this_cpu_ptr(&trace_stack); - size = fstack->stack_size; event = trace_buffer_lock_reserve(buffer, TRACE_STACK, sizeof(*entry) + size, tracing_gen_ctx()); if (!event) - goto out; + return; entry = ring_buffer_event_data(event); @@ -481,8 +563,29 @@ static void timerlat_dump_stack(void) if (!call_filter_check_discard(call, entry, buffer, event)) trace_buffer_unlock_commit_nostack(buffer, event); +} -out: +/* + * timerlat_dump_stack - dump a stack trace previously saved + */ +static void timerlat_dump_stack(void) +{ + struct osnoise_instance *inst; + struct trace_buffer *buffer; + struct trace_stack *fstack; + unsigned int size; + + preempt_disable_notrace(); + fstack = this_cpu_ptr(&trace_stack); + size = fstack->stack_size; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + buffer = inst->tr->array_buffer.buffer; + __timerlat_dump_stack(buffer, fstack, size); + + } + rcu_read_unlock(); preempt_enable_notrace(); } #else @@ -1077,12 +1180,37 @@ diff_osn_sample_stats(struct osnoise_variables *osn_var, struct osnoise_sample * */ static __always_inline void osnoise_stop_tracing(void) { - struct trace_array *tr = osnoise_trace; + struct osnoise_instance *inst; + struct trace_array *tr; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + tr = inst->tr; + trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, + "stop tracing hit on cpu %d\n", smp_processor_id()); + + tracer_tracing_off(tr); + } + rcu_read_unlock(); +} - trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, - "stop tracing hit on cpu %d\n", smp_processor_id()); +/* + * notify_new_max_latency - Notify a new max latency via fsnotify interface. + */ +void notify_new_max_latency(u64 latency) +{ + struct osnoise_instance *inst; + struct trace_array *tr; - tracer_tracing_off(tr); + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + tr = inst->tr; + if (tr->max_latency < latency) { + tr->max_latency = latency; + latency_fsnotify(tr); + } + } + rcu_read_unlock(); } /* @@ -1096,7 +1224,6 @@ static __always_inline void osnoise_stop_tracing(void) static int run_osnoise(void) { struct osnoise_variables *osn_var = this_cpu_osn_var(); - struct trace_array *tr = osnoise_trace; u64 start, sample, last_sample; u64 last_int_count, int_count; s64 noise = 0, max_noise = 0; @@ -1231,11 +1358,7 @@ static int run_osnoise(void) trace_osnoise_sample(&s); - /* Keep a running maximum ever recorded osnoise "latency" */ - if (max_noise > tr->max_latency) { - tr->max_latency = max_noise; - latency_fsnotify(tr); - } + notify_new_max_latency(max_noise); if (osnoise_data.stop_tracing_total) if (s.noise > osnoise_data.stop_tracing_total) @@ -1293,7 +1416,6 @@ static int osnoise_main(void *data) static enum hrtimer_restart timerlat_irq(struct hrtimer *timer) { struct osnoise_variables *osn_var = this_cpu_osn_var(); - struct trace_array *tr = osnoise_trace; struct timerlat_variables *tlat; struct timerlat_sample s; u64 now; @@ -1364,11 +1486,7 @@ static enum hrtimer_restart timerlat_irq(struct hrtimer *timer) trace_timerlat_sample(&s); - /* Keep a running maximum ever recorded os noise "latency" */ - if (diff > tr->max_latency) { - tr->max_latency = diff; - latency_fsnotify(tr); - } + notify_new_max_latency(diff); if (osnoise_data.stop_tracing) if (time_to_us(diff) >= osnoise_data.stop_tracing) @@ -1581,7 +1699,7 @@ static void osnoise_hotplug_workfn(struct work_struct *dummy) mutex_lock(&trace_types_lock); - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) goto out_unlock_trace; mutex_lock(&interface_lock); @@ -1716,11 +1834,10 @@ osnoise_cpus_write(struct file *filp, const char __user *ubuf, size_t count, goto err_free; /* - * trace_types_lock is taken to avoid concurrency on start/stop - * and osnoise_busy. + * trace_types_lock is taken to avoid concurrency on start/stop. */ mutex_lock(&trace_types_lock); - running = osnoise_busy; + running = osnoise_has_registered_instances(); if (running) stop_per_cpu_kthreads(); @@ -1941,8 +2058,6 @@ static int osnoise_workload_start(void) return retval; } - osnoise_busy = true; - return 0; } @@ -1951,7 +2066,7 @@ static int osnoise_workload_start(void) */ static void osnoise_workload_stop(void) { - if (!osnoise_busy) + if (osnoise_has_registered_instances()) return; trace_osnoise_callback_enabled = false; @@ -1962,28 +2077,28 @@ static void osnoise_workload_stop(void) unhook_irq_events(); unhook_softirq_events(); unhook_thread_events(); - - osnoise_busy = false; } static void osnoise_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_busy) + if (osnoise_has_registered_instances()) return; retval = osnoise_workload_start(); if (retval) pr_err(BANNER "Error starting osnoise tracer\n"); + osnoise_register_instance(tr); } static void osnoise_tracer_stop(struct trace_array *tr) { - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) return; + osnoise_unregister_instance(tr); osnoise_workload_stop(); } @@ -1991,14 +2106,12 @@ static int osnoise_tracer_init(struct trace_array *tr) { /* Only allow one instance to enable this */ - if (osnoise_busy) + if (osnoise_has_registered_instances()) return -EBUSY; - osnoise_trace = tr; tr->max_latency = 0; osnoise_tracer_start(tr); - return 0; } @@ -2022,7 +2135,7 @@ static void timerlat_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_busy) + if (osnoise_has_registered_instances()) return; osnoise_data.timerlat_tracer = 1; @@ -2031,6 +2144,8 @@ static void timerlat_tracer_start(struct trace_array *tr) if (retval) goto out_err; + osnoise_register_instance(tr); + return; out_err: pr_err(BANNER "Error starting timerlat tracer\n"); @@ -2040,7 +2155,7 @@ static void timerlat_tracer_stop(struct trace_array *tr) { int cpu; - if (!osnoise_busy) + if (!osnoise_has_registered_instances()) return; for_each_online_cpu(cpu) @@ -2054,11 +2169,9 @@ static void timerlat_tracer_stop(struct trace_array *tr) static int timerlat_tracer_init(struct trace_array *tr) { /* Only allow one instance to enable this */ - if (osnoise_busy) + if (osnoise_has_registered_instances()) return -EBUSY; - osnoise_trace = tr; - tr->max_latency = 0; timerlat_tracer_start(tr); @@ -2105,6 +2218,8 @@ __init static int init_osnoise_tracer(void) #endif osnoise_init_hotplug_support(); + INIT_LIST_HEAD_RCU(&osnoise_instances); + init_tracefs(); return 0; From patchwork Thu Aug 12 13:57:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Bristot de Oliveira X-Patchwork-Id: 12433681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 866B7C4320A for ; Thu, 12 Aug 2021 14:19:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 57A3D6023D for ; Thu, 12 Aug 2021 14:19:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230369AbhHLOUJ (ORCPT ); Thu, 12 Aug 2021 10:20:09 -0400 Received: from gateway31.websitewelcome.com ([192.185.144.219]:43731 "EHLO gateway31.websitewelcome.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237386AbhHLOUJ (ORCPT ); Thu, 12 Aug 2021 10:20:09 -0400 Received: from cm17.websitewelcome.com (cm17.websitewelcome.com [100.42.49.20]) by gateway31.websitewelcome.com (Postfix) with ESMTP id D04BE2693 for ; Thu, 12 Aug 2021 08:57:44 -0500 (CDT) Received: from gator4132.hostgator.com ([192.185.4.144]) by cmsmtp with SMTP id EBDMmVDxpMGeEEBDMmVXNO; Thu, 12 Aug 2021 08:57:44 -0500 X-Authority-Reason: nr=8 Received: from host-79-56-204-120.retail.telecomitalia.it ([79.56.204.120]:54910 helo=f34.bristot.me) by gator4132.hostgator.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mEBDM-004IPl-1O; Thu, 12 Aug 2021 08:57:44 -0500 From: Daniel Bristot de Oliveira To: linux-kernel@vger.kernel.org, Steven Rostedt Cc: Daniel Bristot de Oliveira , Ingo Molnar , Tom Zanussi , Masami Hiramatsu , linux-trace-devel@vger.kernel.org Subject: [PATCH 5/5] trace/osnoise: Allow multiple instances of the same tracer Date: Thu, 12 Aug 2021 15:57:24 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gator4132.hostgator.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kernel.org X-BWhitelist: no X-Source-IP: 79.56.204.120 X-Source-L: No X-Exim-ID: 1mEBDM-004IPl-1O X-Source: X-Source-Args: X-Source-Dir: X-Source-Sender: host-79-56-204-120.retail.telecomitalia.it (f34.bristot.me) [79.56.204.120]:54910 X-Source-Auth: kernel@bristot.me X-Email-Count: 36 X-Source-Cap: YnJpc3RvdG1lO2JyaXN0b3RtZTtnYXRvcjQxMzIuaG9zdGdhdG9yLmNvbQ== X-Local-Domain: no Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org Allow more than one instance of the same tracer. The workload will start when the first trace_array (instance) is registered and will stop when the last instance is unregistered. Osnoise and timerlat are still mutually exclusive because of the different behavior of the osnoise: tracepoints. Cc: Steven Rostedt Cc: Ingo Molnar Cc: Tom Zanussi Cc: Masami Hiramatsu Cc: Daniel Bristot de Oliveira Cc: linux-trace-devel@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Daniel Bristot de Oliveira (Red Hat) --- kernel/trace/trace_osnoise.c | 106 ++++++++++++++++++++++++++--------- 1 file changed, 80 insertions(+), 26 deletions(-) diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c index 9193c256377e..5f09fb55b359 100644 --- a/kernel/trace/trace_osnoise.c +++ b/kernel/trace/trace_osnoise.c @@ -64,6 +64,24 @@ static bool osnoise_has_registered_instances(void) list); } +/* + * osnoise_instance_registered - check if a tr is already registered + */ +static int osnoise_instance_registered(struct trace_array *tr) +{ + struct osnoise_instance *inst; + int found = 0; + + rcu_read_lock(); + list_for_each_entry_rcu(inst, &osnoise_instances, list) { + if (inst->tr == tr) + found = 1; + } + rcu_read_unlock(); + + return found; +} + /* * osnoise_register_instance - register a new trace instance * @@ -106,10 +124,8 @@ static void osnoise_unregister_instance(struct trace_array *tr) } rcu_read_unlock(); - if (!found) { - WARN("osnoise unregister did not find tr %s\n", tr->name); + if (!found) return; - } synchronize_rcu(); kfree(inst); @@ -1190,6 +1206,7 @@ static __always_inline void osnoise_stop_tracing(void) "stop tracing hit on cpu %d\n", smp_processor_id()); tracer_tracing_off(tr); + } rcu_read_unlock(); } @@ -2041,6 +2058,16 @@ static int osnoise_workload_start(void) { int retval; + /* + * Instances need to be registered after calling workload + * start. Hence, if there is already an instance, the + * workload was already registered. Otherwise, this + * code is on the way to register the first instance, + * and the workload will start. + */ + if (osnoise_has_registered_instances()) + return 0; + osn_var_reset_all(); retval = osnoise_hook_events(); @@ -2066,6 +2093,13 @@ static int osnoise_workload_start(void) */ static void osnoise_workload_stop(void) { + /* + * Instances need to be unregistered before calling + * stop. Hence, if there is a registered instance, more + * than one instance is running, and the workload will not + * yet stop. Otherwise, this code is on the way to disable + * the last instance, and the workload can stop. + */ if (osnoise_has_registered_instances()) return; @@ -2083,7 +2117,11 @@ static void osnoise_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_has_registered_instances()) + /* + * If the instance is already registered, there is no need to + * register it again. + */ + if (osnoise_instance_registered(tr)) return; retval = osnoise_workload_start(); @@ -2095,18 +2133,17 @@ static void osnoise_tracer_start(struct trace_array *tr) static void osnoise_tracer_stop(struct trace_array *tr) { - if (!osnoise_has_registered_instances()) - return; - osnoise_unregister_instance(tr); osnoise_workload_stop(); } static int osnoise_tracer_init(struct trace_array *tr) { - - /* Only allow one instance to enable this */ - if (osnoise_has_registered_instances()) + /* + * Only allow osnoise tracer if timerlat tracer is not running + * already. + */ + if (osnoise_data.timerlat_tracer) return -EBUSY; tr->max_latency = 0; @@ -2135,45 +2172,55 @@ static void timerlat_tracer_start(struct trace_array *tr) { int retval; - if (osnoise_has_registered_instances()) + /* + * If the instance is already registered, there is no need to + * register it again. + */ + if (osnoise_instance_registered(tr)) return; - osnoise_data.timerlat_tracer = 1; - retval = osnoise_workload_start(); if (retval) - goto out_err; + pr_err(BANNER "Error starting timerlat tracer\n"); osnoise_register_instance(tr); return; -out_err: - pr_err(BANNER "Error starting timerlat tracer\n"); } static void timerlat_tracer_stop(struct trace_array *tr) { int cpu; - if (!osnoise_has_registered_instances()) - return; - - for_each_online_cpu(cpu) - per_cpu(per_cpu_osnoise_var, cpu).sampling = 0; + osnoise_unregister_instance(tr); - osnoise_tracer_stop(tr); + /* + * Instruct the threads to stop only if this is the last instance. + */ + if (!osnoise_has_registered_instances()) { + for_each_online_cpu(cpu) + per_cpu(per_cpu_osnoise_var, cpu).sampling = 0; + } - osnoise_data.timerlat_tracer = 0; + osnoise_workload_stop(); } static int timerlat_tracer_init(struct trace_array *tr) { - /* Only allow one instance to enable this */ - if (osnoise_has_registered_instances()) + /* + * Only allow timerlat tracer if osnoise tracer is not running already. + */ + if (osnoise_has_registered_instances() && !osnoise_data.timerlat_tracer) return -EBUSY; - tr->max_latency = 0; + /* + * If this is the first instance, set timerlat_tracer to block + * osnoise tracer start. + */ + if (!osnoise_has_registered_instances()) + osnoise_data.timerlat_tracer = 1; + tr->max_latency = 0; timerlat_tracer_start(tr); return 0; @@ -2182,6 +2229,13 @@ static int timerlat_tracer_init(struct trace_array *tr) static void timerlat_tracer_reset(struct trace_array *tr) { timerlat_tracer_stop(tr); + + /* + * If this is the last instance, reset timerlat_tracer allowing + * osnoise to be started. + */ + if (!osnoise_has_registered_instances()) + osnoise_data.timerlat_tracer = 0; } static struct tracer timerlat_tracer __read_mostly = {