From patchwork Sat Aug 5 03:38:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Yejian X-Patchwork-Id: 13342499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D989C04E69 for ; Sat, 5 Aug 2023 03:38:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229673AbjHEDi6 (ORCPT ); Fri, 4 Aug 2023 23:38:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229652AbjHEDi4 (ORCPT ); Fri, 4 Aug 2023 23:38:56 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE1F24ED8; Fri, 4 Aug 2023 20:38:54 -0700 (PDT) Received: from dggpeml500012.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4RHpDH703GzVk4V; Sat, 5 Aug 2023 11:37:03 +0800 (CST) Received: from localhost.localdomain (10.67.175.61) by dggpeml500012.china.huawei.com (7.185.36.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Sat, 5 Aug 2023 11:38:52 +0800 From: Zheng Yejian To: , , , CC: , , Subject: [PATCH v2 1/2] tracing: Fix cpu buffers unavailable due to 'record_disabled' messed Date: Sat, 5 Aug 2023 11:38:15 +0800 Message-ID: <20230805033816.3284594-2-zhengyejian1@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230805033816.3284594-1-zhengyejian1@huawei.com> References: <20230805033816.3284594-1-zhengyejian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.175.61] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpeml500012.china.huawei.com (7.185.36.15) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org Trace ring buffer can no longer record anything after executing following commands at the shell prompt: # cd /sys/kernel/tracing # cat tracing_cpumask fff # echo 0 > tracing_cpumask # echo 1 > snapshot # echo fff > tracing_cpumask # echo 1 > tracing_on # echo "hello world" > trace_marker -bash: echo: write error: Bad file descriptor The root cause is that: 1. After `echo 0 > tracing_cpumask`, 'record_disabled' of cpu buffers in 'tr->array_buffer.buffer' became 1 (see tracing_set_cpumask()); 2. After `echo 1 > snapshot`, 'tr->array_buffer.buffer' is swapped with 'tr->max_buffer.buffer', then the 'record_disabled' became 0 (see update_max_tr()); 3. After `echo fff > tracing_cpumask`, the 'record_disabled' become -1; Then array_buffer and max_buffer are both unavailable due to value of 'record_disabled' is not 0. To fix it, enable or disable both array_buffer and max_buffer at the same time in tracing_set_cpumask(). Fixes: 71babb2705e2 ("tracing: change CPU ring buffer state from tracing_cpumask") Signed-off-by: Zheng Yejian --- kernel/trace/trace.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index b8870078ef58..b0e8eb6ea8ac 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -5277,11 +5277,17 @@ int tracing_set_cpumask(struct trace_array *tr, !cpumask_test_cpu(cpu, tracing_cpumask_new)) { atomic_inc(&per_cpu_ptr(tr->array_buffer.data, cpu)->disabled); ring_buffer_record_disable_cpu(tr->array_buffer.buffer, cpu); +#ifdef CONFIG_TRACER_MAX_TRACE + ring_buffer_record_disable_cpu(tr->max_buffer.buffer, cpu); +#endif } if (!cpumask_test_cpu(cpu, tr->tracing_cpumask) && cpumask_test_cpu(cpu, tracing_cpumask_new)) { atomic_dec(&per_cpu_ptr(tr->array_buffer.data, cpu)->disabled); ring_buffer_record_enable_cpu(tr->array_buffer.buffer, cpu); +#ifdef CONFIG_TRACER_MAX_TRACE + ring_buffer_record_enable_cpu(tr->max_buffer.buffer, cpu); +#endif } } arch_spin_unlock(&tr->max_lock); From patchwork Sat Aug 5 03:38:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zheng Yejian X-Patchwork-Id: 13342498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4A5EC41513 for ; Sat, 5 Aug 2023 03:38:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229668AbjHEDi5 (ORCPT ); Fri, 4 Aug 2023 23:38:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229379AbjHEDi4 (ORCPT ); Fri, 4 Aug 2023 23:38:56 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDB594ED7; Fri, 4 Aug 2023 20:38:54 -0700 (PDT) Received: from dggpeml500012.china.huawei.com (unknown [172.30.72.54]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4RHpF63703zrS5n; Sat, 5 Aug 2023 11:37:46 +0800 (CST) Received: from localhost.localdomain (10.67.175.61) by dggpeml500012.china.huawei.com (7.185.36.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Sat, 5 Aug 2023 11:38:52 +0800 From: Zheng Yejian To: , , , CC: , , Subject: [PATCH v2 2/2] selftests/ftrace: Add a basic testcase for snapshot Date: Sat, 5 Aug 2023 11:38:16 +0800 Message-ID: <20230805033816.3284594-3-zhengyejian1@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230805033816.3284594-1-zhengyejian1@huawei.com> References: <20230805033816.3284594-1-zhengyejian1@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.175.61] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpeml500012.china.huawei.com (7.185.36.15) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-trace-kernel@vger.kernel.org This testcase is constrived to reproduce a problem that the cpu buffers become unavailable which is due to 'record_disabled' of array_buffer and max_buffer being messed up. Local test result after bugfix: # ./ftracetest test.d/00basic/snapshot1.tc === Ftrace unit tests === [1] Snapshot and tracing_cpumask [PASS] [2] (instance) Snapshot and tracing_cpumask [PASS] # of passed: 2 # of failed: 0 # of unresolved: 0 # of untested: 0 # of unsupported: 0 # of xfailed: 0 # of undefined(test bug): 0 Signed-off-by: Zheng Yejian --- .../ftrace/test.d/00basic/snapshot1.tc | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/snapshot1.tc diff --git a/tools/testing/selftests/ftrace/test.d/00basic/snapshot1.tc b/tools/testing/selftests/ftrace/test.d/00basic/snapshot1.tc new file mode 100644 index 000000000000..63b76cf2a360 --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/00basic/snapshot1.tc @@ -0,0 +1,31 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: Snapshot and tracing_cpumask +# requires: trace_marker tracing_cpumask snapshot +# flags: instance + +# This testcase is constrived to reproduce a problem that the cpu buffers +# become unavailable which is due to 'record_disabled' of array_buffer and +# max_buffer being messed up. + +# Store origin cpumask +ORIG_CPUMASK=`cat tracing_cpumask` + +# Stop tracing all cpu +echo 0 > tracing_cpumask + +# Take a snapshot of the main buffer +echo 1 > snapshot + +# Restore origin cpumask, note that there should be some cpus being traced +echo ${ORIG_CPUMASK} > tracing_cpumask + +# Set tracing on +echo 1 > tracing_on + +# Write a log into buffer +echo "test input 1" > trace_marker + +# Ensure the log writed so that cpu buffers are still available +grep -q "test input 1" trace +exit 0