| Message ID | 20230831125500.986862-1-bfoster@redhat.com (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | tracing: zero the pipe cpumask on alloc to avoid spurious -EBUSY | expand |
On 2023/8/31 20:55, Brian Foster wrote: > The pipe cpumask used to serialize opens between the main and percpu > trace pipes is not zeroed or initialized. This can result in > spurious -EBUSY returns if underlying memory is not fully zeroed. > This has been observed by immediate failure to read the main > trace_pipe file on an otherwise newly booted and idle system: > > # cat /sys/kernel/debug/tracing/trace_pipe > cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy > > Zero the allocation of pipe_cpumask to avoid the problem. > > Fixes: c2489bb7e6be ("tracing: Introduce pipe_cpumask to avoid race on trace_pipes") > Signed-off-by: Brian Foster <bfoster@redhat.com> > --- > > Hi, > > I ran into this problem just recently on one of my test VMs immediately > after updating to a v6.5 base. A revert of the aforementioned commit > addressed the problem. I'm not terribly familiar with the tracing code, > but on further inspection I noticed the cpumask doesn't appear to be > initialized anywhere. I suppose this could alternatively do a > cpumask_clear() or whatever after allocation, but either way this > addresses the problem for me. Yes, pipe_cpumask must be initialized. -- Thanks, Zheng Yejian > > Please CC on replies as I'm not subscribed to the list. Thanks. > > Brian > > kernel/trace/trace.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c > index 8e64aaad5361..2656ca3b9b39 100644 > --- a/kernel/trace/trace.c > +++ b/kernel/trace/trace.c > @@ -9486,7 +9486,7 @@ static struct trace_array *trace_array_create(const char *name) > if (!alloc_cpumask_var(&tr->tracing_cpumask, GFP_KERNEL)) > goto out_free_tr; > > - if (!alloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL)) > + if (!zalloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL)) > goto out_free_tr; > > tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS; > @@ -10431,7 +10431,7 @@ __init static int tracer_alloc_buffers(void) > if (trace_create_savedcmd() < 0) > goto out_free_temp_buffer; > > - if (!alloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL)) > + if (!zalloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL)) > goto out_free_savedcmd; > > /* TODO: make the number of buffers hot pluggable with CPUS */
On Thu, 31 Aug 2023 21:51:18 +0800 Zheng Yejian <zhengyejian1@huawei.com> wrote: > > Hi, > > > > I ran into this problem just recently on one of my test VMs immediately > > after updating to a v6.5 base. A revert of the aforementioned commit > > addressed the problem. I'm not terribly familiar with the tracing code, > > but on further inspection I noticed the cpumask doesn't appear to be > > initialized anywhere. I suppose this could alternatively do a > > cpumask_clear() or whatever after allocation, but either way this > > addresses the problem for me. > > Yes, pipe_cpumask must be initialized. Can I add a Reviewed-by tag from you? > > > > > Please CC on replies as I'm not subscribed to the list. Thanks. That's the default with Linux kernel lists. -- Steve
On 2023/9/1 04:33, Steven Rostedt wrote: > On Thu, 31 Aug 2023 21:51:18 +0800 > Zheng Yejian <zhengyejian1@huawei.com> wrote: > >>> Hi, >>> >>> I ran into this problem just recently on one of my test VMs immediately >>> after updating to a v6.5 base. A revert of the aforementioned commit >>> addressed the problem. I'm not terribly familiar with the tracing code, >>> but on further inspection I noticed the cpumask doesn't appear to be >>> initialized anywhere. I suppose this could alternatively do a >>> cpumask_clear() or whatever after allocation, but either way this >>> addresses the problem for me. >> >> Yes, pipe_cpumask must be initialized. > > Can I add a Reviewed-by tag from you? Of course :) Reviewed-by: Zheng Yejian <zhengyejian1@huawei.com> -- Thanks, Zheng Yejian > >> >>> >>> Please CC on replies as I'm not subscribed to the list. Thanks. > > That's the default with Linux kernel lists. > > -- Steve >
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 8e64aaad5361..2656ca3b9b39 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -9486,7 +9486,7 @@ static struct trace_array *trace_array_create(const char *name) if (!alloc_cpumask_var(&tr->tracing_cpumask, GFP_KERNEL)) goto out_free_tr; - if (!alloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL)) + if (!zalloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL)) goto out_free_tr; tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS; @@ -10431,7 +10431,7 @@ __init static int tracer_alloc_buffers(void) if (trace_create_savedcmd() < 0) goto out_free_temp_buffer; - if (!alloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL)) + if (!zalloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL)) goto out_free_savedcmd; /* TODO: make the number of buffers hot pluggable with CPUS */
The pipe cpumask used to serialize opens between the main and percpu trace pipes is not zeroed or initialized. This can result in spurious -EBUSY returns if underlying memory is not fully zeroed. This has been observed by immediate failure to read the main trace_pipe file on an otherwise newly booted and idle system: # cat /sys/kernel/debug/tracing/trace_pipe cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy Zero the allocation of pipe_cpumask to avoid the problem. Fixes: c2489bb7e6be ("tracing: Introduce pipe_cpumask to avoid race on trace_pipes") Signed-off-by: Brian Foster <bfoster@redhat.com> --- Hi, I ran into this problem just recently on one of my test VMs immediately after updating to a v6.5 base. A revert of the aforementioned commit addressed the problem. I'm not terribly familiar with the tracing code, but on further inspection I noticed the cpumask doesn't appear to be initialized anywhere. I suppose this could alternatively do a cpumask_clear() or whatever after allocation, but either way this addresses the problem for me. Please CC on replies as I'm not subscribed to the list. Thanks. Brian kernel/trace/trace.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)