| Message ID | 171671825710.39694.6859036369216249956.stgit@devnote2 (mailing list archive) |
|---|---|
| Headers | show |
| Series | tracing: Fix some selftest issues | expand |
On Sun, 26 May 2024 19:10:57 +0900 "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > Hi, > > Here is a series of some fixes/improvements for the test modules and boot > time selftest of kprobe events. I found a WARNING message with some boot > time selftest configuration, which came from the combination of embedded > kprobe generate API tests module and ftrace boot-time selftest. So the main > problem is that the test module should not be built-in. But I also think > this WARNING message is useless (because there are warning messages already) > and the cleanup code is redundant. This series fixes those issues. Note, when I enable trace tests as builtin instead of modules, I just disable the bootup self tests when it detects this. This helps with doing tests via config options than having to add user space code that loads modules. Could you do something similar? -- Steve > > Thank you, > > --- > > Masami Hiramatsu (Google) (3): > tracing: Build event generation tests only as modules > tracing/kprobe: Remove unneeded WARN_ON_ONCE() in selftests > tracing/kprobe: Remove cleanup code unrelated to selftest >
On Mon, 27 May 2024 19:29:07 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Sun, 26 May 2024 19:10:57 +0900 > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > > > Hi, > > > > Here is a series of some fixes/improvements for the test modules and boot > > time selftest of kprobe events. I found a WARNING message with some boot > > time selftest configuration, which came from the combination of embedded > > kprobe generate API tests module and ftrace boot-time selftest. So the main > > problem is that the test module should not be built-in. But I also think > > this WARNING message is useless (because there are warning messages already) > > and the cleanup code is redundant. This series fixes those issues. > > Note, when I enable trace tests as builtin instead of modules, I just > disable the bootup self tests when it detects this. This helps with > doing tests via config options than having to add user space code that > loads modules. > > Could you do something similar? OK, in that case, I would like to move the test cleanup code in module_exit function into the end of module_init function. It looks there is no reason to split those into 2 parts. Thank you, > > -- Steve > > > > > > Thank you, > > > > --- > > > > Masami Hiramatsu (Google) (3): > > tracing: Build event generation tests only as modules > > tracing/kprobe: Remove unneeded WARN_ON_ONCE() in selftests > > tracing/kprobe: Remove cleanup code unrelated to selftest > >
On Wed, 29 May 2024 01:46:40 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > On Mon, 27 May 2024 19:29:07 -0400 > Steven Rostedt <rostedt@goodmis.org> wrote: > > > On Sun, 26 May 2024 19:10:57 +0900 > > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > > > > > Hi, > > > > > > Here is a series of some fixes/improvements for the test modules and boot > > > time selftest of kprobe events. I found a WARNING message with some boot > > > time selftest configuration, which came from the combination of embedded > > > kprobe generate API tests module and ftrace boot-time selftest. So the main > > > problem is that the test module should not be built-in. But I also think > > > this WARNING message is useless (because there are warning messages already) > > > and the cleanup code is redundant. This series fixes those issues. > > > > Note, when I enable trace tests as builtin instead of modules, I just > > disable the bootup self tests when it detects this. This helps with > > doing tests via config options than having to add user space code that > > loads modules. > > > > Could you do something similar? > > OK, in that case, I would like to move the test cleanup code in > module_exit function into the end of module_init function. > It looks there is no reason to split those into 2 parts. Wait, I would like to hear Tom's opinion. I found following usage comments in the code. * Following that are a few examples using the created events to test * various ways of tracing a synthetic event. * * To test, select CONFIG_SYNTH_EVENT_GEN_TEST and build the module. * Then: * * # insmod kernel/trace/synth_event_gen_test.ko * # cat /sys/kernel/tracing/trace * * You should see several events in the trace buffer - * "create_synth_test", "empty_synth_test", and several instances of * "gen_synth_test". * * To remove the events, remove the module: * * # rmmod synth_event_gen_test Tom, is that intended behavior ? and are you expected to reuse these events outside of the module? e.g. load the test module and run some test script in user space which uses those events? As far as I can see, those tests are not intended to be embedded in the kernel because those are expected to be removed. Thank you, > > Thank you, > > > > > -- Steve > > > > > > > > > > Thank you, > > > > > > --- > > > > > > Masami Hiramatsu (Google) (3): > > > tracing: Build event generation tests only as modules > > > tracing/kprobe: Remove unneeded WARN_ON_ONCE() in selftests > > > tracing/kprobe: Remove cleanup code unrelated to selftest > > > > > > -- > Masami Hiramatsu (Google) <mhiramat@kernel.org>
Hi Masami, On Wed, 2024-05-29 at 08:38 +0900, Masami Hiramatsu wrote: > On Wed, 29 May 2024 01:46:40 +0900 > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > On Mon, 27 May 2024 19:29:07 -0400 > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > On Sun, 26 May 2024 19:10:57 +0900 > > > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > > > > > > > Hi, > > > > > > > > Here is a series of some fixes/improvements for the test > > > > modules and boot > > > > time selftest of kprobe events. I found a WARNING message with > > > > some boot > > > > time selftest configuration, which came from the combination of > > > > embedded > > > > kprobe generate API tests module and ftrace boot-time selftest. > > > > So the main > > > > problem is that the test module should not be built-in. But I > > > > also think > > > > this WARNING message is useless (because there are warning > > > > messages already) > > > > and the cleanup code is redundant. This series fixes those > > > > issues. > > > > > > Note, when I enable trace tests as builtin instead of modules, I > > > just > > > disable the bootup self tests when it detects this. This helps > > > with > > > doing tests via config options than having to add user space code > > > that > > > loads modules. > > > > > > Could you do something similar? > > > > OK, in that case, I would like to move the test cleanup code in > > module_exit function into the end of module_init function. > > It looks there is no reason to split those into 2 parts. > > Wait, I would like to hear Tom's opinion. I found following usage > comments > in the code. > > * Following that are a few examples using the created events to test > * various ways of tracing a synthetic event. > * > * To test, select CONFIG_SYNTH_EVENT_GEN_TEST and build the module. > * Then: > * > * # insmod kernel/trace/synth_event_gen_test.ko > * # cat /sys/kernel/tracing/trace > * > * You should see several events in the trace buffer - > * "create_synth_test", "empty_synth_test", and several instances of > * "gen_synth_test". > * > * To remove the events, remove the module: > * > * # rmmod synth_event_gen_test > > Tom, is that intended behavior ? and are you expected to reuse these > events outside of the module? e.g. load the test module and run some > test script in user space which uses those events? > Yeah, this module was meant as a sample module showing how to create and generate synthetic events in-kernel. So the interested user insmods the module, looks at the trace stream and sees, ok the events are there as expected, so it does work, great, let's remove the module to get rid of them and go write our own. Having both the creation and cleanup in module_init() wouldn't allow the user the opportunity to do that i.e. verify the results by reading the trace file. Tom > As far as I can see, those tests are not intended to be embedded in > the > kernel because those are expected to be removed. > > Thank you, > > > > > Thank you, > > > > > > > > -- Steve > > > > > > > > > > > > > > Thank you, > > > > > > > > --- > > > > > > > > Masami Hiramatsu (Google) (3): > > > > tracing: Build event generation tests only as modules > > > > tracing/kprobe: Remove unneeded WARN_ON_ONCE() in > > > > selftests > > > > tracing/kprobe: Remove cleanup code unrelated to selftest > > > > > > > > > > -- > > Masami Hiramatsu (Google) <mhiramat@kernel.org> > >
On Wed, 29 May 2024 11:01:43 -0500 Tom Zanussi <tom.zanussi@linux.intel.com> wrote: > Hi Masami, > > On Wed, 2024-05-29 at 08:38 +0900, Masami Hiramatsu wrote: > > On Wed, 29 May 2024 01:46:40 +0900 > > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > > > On Mon, 27 May 2024 19:29:07 -0400 > > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > > On Sun, 26 May 2024 19:10:57 +0900 > > > > "Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote: > > > > > > > > > Hi, > > > > > > > > > > Here is a series of some fixes/improvements for the test > > > > > modules and boot > > > > > time selftest of kprobe events. I found a WARNING message with > > > > > some boot > > > > > time selftest configuration, which came from the combination of > > > > > embedded > > > > > kprobe generate API tests module and ftrace boot-time selftest. > > > > > So the main > > > > > problem is that the test module should not be built-in. But I > > > > > also think > > > > > this WARNING message is useless (because there are warning > > > > > messages already) > > > > > and the cleanup code is redundant. This series fixes those > > > > > issues. > > > > > > > > Note, when I enable trace tests as builtin instead of modules, I > > > > just > > > > disable the bootup self tests when it detects this. This helps > > > > with > > > > doing tests via config options than having to add user space code > > > > that > > > > loads modules. > > > > > > > > Could you do something similar? > > > > > > OK, in that case, I would like to move the test cleanup code in > > > module_exit function into the end of module_init function. > > > It looks there is no reason to split those into 2 parts. > > > > Wait, I would like to hear Tom's opinion. I found following usage > > comments > > in the code. > > > > * Following that are a few examples using the created events to test > > * various ways of tracing a synthetic event. > > * > > * To test, select CONFIG_SYNTH_EVENT_GEN_TEST and build the module. > > * Then: > > * > > * # insmod kernel/trace/synth_event_gen_test.ko > > * # cat /sys/kernel/tracing/trace > > * > > * You should see several events in the trace buffer - > > * "create_synth_test", "empty_synth_test", and several instances of > > * "gen_synth_test". > > * > > * To remove the events, remove the module: > > * > > * # rmmod synth_event_gen_test > > > > Tom, is that intended behavior ? and are you expected to reuse these > > events outside of the module? e.g. load the test module and run some > > test script in user space which uses those events? > > > > Yeah, this module was meant as a sample module showing how to create > and generate synthetic events in-kernel. > > So the interested user insmods the module, looks at the trace stream > and sees, ok the events are there as expected, so it does work, great, > let's remove the module to get rid of them and go write our own. > > Having both the creation and cleanup in module_init() wouldn't allow > the user the opportunity to do that i.e. verify the results by reading > the trace file. So, in summary, it is designed to be a module. Steve, I think these tests should be kept as modules. There are many reason to do so. - This test is designed to be used as module. - This can conflict with other boot time selftest if it is embedded. - We can make these tests and boot time selftest mutable exclusive but if we make these tests as modules, we can build and run both tests safely. - Embedding these tests leave new events when the kernel boot, which user must be cleaned up by manual. What would you think? Thank you, > > Tom > > > As far as I can see, those tests are not intended to be embedded in > > the > > kernel because those are expected to be removed. > > > > Thank you, > > > > > > > > Thank you, > > > > > > > > > > > -- Steve > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > > --- > > > > > > > > > > Masami Hiramatsu (Google) (3): > > > > > tracing: Build event generation tests only as modules > > > > > tracing/kprobe: Remove unneeded WARN_ON_ONCE() in > > > > > selftests > > > > > tracing/kprobe: Remove cleanup code unrelated to selftest > > > > > > > > > > > > > > -- > > > Masami Hiramatsu (Google) <mhiramat@kernel.org> > > > > > >
On Fri, 31 May 2024 11:37:21 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > So, in summary, it is designed to be a module. Steve, I think these tests > should be kept as modules. There are many reason to do so. > > - This test is designed to be used as module. > - This can conflict with other boot time selftest if it is embedded. > - We can make these tests and boot time selftest mutable exclusive but > if we make these tests as modules, we can build and run both tests > safely. > - Embedding these tests leave new events when the kernel boot, which > user must be cleaned up by manual. > > What would you think? I was mostly following what Ingo told me long ago, where having it built in is just one more way to test it ;-) But that said, from your first patch, you show the stack dump and mention: > Since the kprobes and synth event generation tests adds and enable > generated events in init_module() and delete it in exit_module(), > if we make it as built-in, those events are left in kernel and cause > kprobe event self-test failure. But you don't explain what exactly the conflict is. What about those events causes kprobe selftests to fail? -- Steve
On Fri, 31 May 2024 03:24:25 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 31 May 2024 11:37:21 +0900 > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > So, in summary, it is designed to be a module. Steve, I think these tests > > should be kept as modules. There are many reason to do so. > > > > - This test is designed to be used as module. > > - This can conflict with other boot time selftest if it is embedded. > > - We can make these tests and boot time selftest mutable exclusive but > > if we make these tests as modules, we can build and run both tests > > safely. > > - Embedding these tests leave new events when the kernel boot, which > > user must be cleaned up by manual. > > > > What would you think? > > I was mostly following what Ingo told me long ago, where having it > built in is just one more way to test it ;-) Ah, would you mean embedding the module is also a part of tests? > > But that said, from your first patch, you show the stack dump and > mention: > > > Since the kprobes and synth event generation tests adds and enable > > generated events in init_module() and delete it in exit_module(), > > if we make it as built-in, those events are left in kernel and cause > > kprobe event self-test failure. > > But you don't explain what exactly the conflict is. What about those > events causes kprobe selftests to fail? The major conflict happens when the boot-time test cleans up the kprobe events by dyn_events_release_all(&trace_kprobe_ops); And I removed it by [3/3] patch in this series :) because it does not needed and not confirmed there is no other kprobe events when the test starts. Also the warning message are redundant so I removed it by [2/3]. So without this [1/3], if we apply [2/3] and [3/3], the problem will be mitigated, but I think the root cause is that these modules are built-in. Thank you, > > > -- Steve
On Fri, 31 May 2024 23:20:47 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > The major conflict happens when the boot-time test cleans up the kprobe > events by > > dyn_events_release_all(&trace_kprobe_ops); > > And I removed it by [3/3] patch in this series :) because it does not > needed and not confirmed there is no other kprobe events when the test > starts. Also the warning message are redundant so I removed it by [2/3]. > > So without this [1/3], if we apply [2/3] and [3/3], the problem will be > mitigated, but I think the root cause is that these modules are built-in. I'm OK with making them module only, but I don't see any selftests for sythetic events. I think they should have a boot up test as well. If we remove them, let's add something to test them at boot up. Then the boot up code could clean it up. Or change the test module to be a boot up test that cleans itself up if it is compiled in as not a module? -- Steve
On Tue, 4 Jun 2024 09:57:46 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 31 May 2024 23:20:47 +0900 > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > The major conflict happens when the boot-time test cleans up the kprobe > > events by > > > > dyn_events_release_all(&trace_kprobe_ops); > > > > And I removed it by [3/3] patch in this series :) because it does not > > needed and not confirmed there is no other kprobe events when the test > > starts. Also the warning message are redundant so I removed it by [2/3]. > > > > So without this [1/3], if we apply [2/3] and [3/3], the problem will be > > mitigated, but I think the root cause is that these modules are built-in. > > I'm OK with making them module only, but I don't see any selftests for > sythetic events. I think they should have a boot up test as well. If we > remove them, let's add something to test them at boot up. Then the boot up > code could clean it up. > > Or change the test module to be a boot up test that cleans itself up if it > is compiled in as not a module? Yeah, I think we may need another test code for synthetic events, which also triggering the synthetic events. BTW, some these bootup tests can be ported on KUnit. Do you have a plan to use KUnit? Thank you, > > -- Steve >
On Tue, 4 Jun 2024 23:18:29 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > BTW, some these bootup tests can be ported on KUnit. Do you have a plan to > use KUnit? I haven't thought about that. If someone else wants to do it, I will happily review it ;-) -- Steve
On Fri, 31 May 2024 03:24:25 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 31 May 2024 11:37:21 +0900 > Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > > So, in summary, it is designed to be a module. Steve, I think these tests > > should be kept as modules. There are many reason to do so. > > > > - This test is designed to be used as module. > > - This can conflict with other boot time selftest if it is embedded. > > - We can make these tests and boot time selftest mutable exclusive but > > if we make these tests as modules, we can build and run both tests > > safely. > > - Embedding these tests leave new events when the kernel boot, which > > user must be cleaned up by manual. > > > > What would you think? > > I was mostly following what Ingo told me long ago, where having it > built in is just one more way to test it ;-) > > But that said, from your first patch, you show the stack dump and > mention: > > > Since the kprobes and synth event generation tests adds and enable > > generated events in init_module() and delete it in exit_module(), > > if we make it as built-in, those events are left in kernel and cause > > kprobe event self-test failure. > > But you don't explain what exactly the conflict is. What about those > events causes kprobe selftests to fail? I also found another problem on these modules. These modules get trace event file references to prevent removing probes. Therefore, if we embed these modules, we can not remove these events forever! /* * Now get the gen_kprobe_test event file. We need to prevent * the instance and event from disappearing from underneath * us, which trace_get_event_file() does (though in this case * we're using the top-level instance which never goes away). */ gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); if (IS_ERR(gen_kprobe_test)) { ret = PTR_ERR(gen_kprobe_test); goto delete; } This means most ftracetest fails because we can not clean up the tracing state by removing dynamic events, which are installed while booting. Note that these references (locks) will be removed when the module is unloaded. Thanks, > > > -- Steve
On Mon, 10 Jun 2024 11:10:01 +0900 Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote: > > But you don't explain what exactly the conflict is. What about those > > events causes kprobe selftests to fail? > > I also found another problem on these modules. These modules get trace > event file references to prevent removing probes. Therefore, if we > embed these modules, we can not remove these events forever! > > /* > * Now get the gen_kprobe_test event file. We need to prevent > * the instance and event from disappearing from underneath > * us, which trace_get_event_file() does (though in this case > * we're using the top-level instance which never goes away). > */ > gen_kprobe_test = trace_get_event_file(NULL, "kprobes", > "gen_kprobe_test"); > if (IS_ERR(gen_kprobe_test)) { > ret = PTR_ERR(gen_kprobe_test); > goto delete; > } > > This means most ftracetest fails because we can not clean up the > tracing state by removing dynamic events, which are installed while > booting. > Note that these references (locks) will be removed when the module > is unloaded. I'm fine if you want to force these as modules. Just make sure the change log lists all the issues of them not being modules. -- Steve
Hi, Here is a series of some fixes/improvements for the test modules and boot time selftest of kprobe events. I found a WARNING message with some boot time selftest configuration, which came from the combination of embedded kprobe generate API tests module and ftrace boot-time selftest. So the main problem is that the test module should not be built-in. But I also think this WARNING message is useless (because there are warning messages already) and the cleanup code is redundant. This series fixes those issues. Thank you, --- Masami Hiramatsu (Google) (3): tracing: Build event generation tests only as modules tracing/kprobe: Remove unneeded WARN_ON_ONCE() in selftests tracing/kprobe: Remove cleanup code unrelated to selftest kernel/trace/Kconfig | 4 ++-- kernel/trace/trace_kprobe.c | 29 ++++++++++++----------------- 2 files changed, 14 insertions(+), 19 deletions(-) -- Masami Hiramatsu (Google) <mhiramat@kernel.org>