| Message ID | 20211008111527.438276127@infradead.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | wchan: Fix wchan support | expand |
On Fri, Oct 08, 2021 at 01:15:27PM +0200, Peter Zijlstra wrote: > Hi, > > This fixes up wchan which is various degrees of broken across the > architectures. > > Patch 4 fixes wchan for x86, which has been returning 0 for the past many > releases. > > Patch 5 fixes the fundamental race against scheduling. > > Patch 6 deletes a lot and makes STACKTRACE unconditional > > patch 7 fixes up a few STACKTRACE arch oddities > > 0day says all builds are good, so it must be perfect :-) I'm planning on > queueing up at least the first 5 patches, but I'm hoping the last two patches > can be too. > > Also available here: > > git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/wchan These patches introduce a regression on ARM. Whereas before, I have /proc/*/wchan populated with non-zero values, with these patches they _all_ contain "0": root@clearfog21:~# cat /proc/*/wchan 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000root@clearfog21:~# I'll try to investigate what is going on later today.
On Thu, Oct 14, 2021 at 01:02:34PM +0100, Russell King (Oracle) wrote: > On Fri, Oct 08, 2021 at 01:15:27PM +0200, Peter Zijlstra wrote: > > Hi, > > > > This fixes up wchan which is various degrees of broken across the > > architectures. > > > > Patch 4 fixes wchan for x86, which has been returning 0 for the past many > > releases. > > > > Patch 5 fixes the fundamental race against scheduling. > > > > Patch 6 deletes a lot and makes STACKTRACE unconditional > > > > patch 7 fixes up a few STACKTRACE arch oddities > > > > 0day says all builds are good, so it must be perfect :-) I'm planning on > > queueing up at least the first 5 patches, but I'm hoping the last two patches > > can be too. > > > > Also available here: > > > > git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/wchan > > These patches introduce a regression on ARM. Whereas before, I have > /proc/*/wchan populated with non-zero values, with these patches they > _all_ contain "0": > > root@clearfog21:~# cat /proc/*/wchan > 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000root@clearfog21:~# > > I'll try to investigate what is going on later today. What is going on here is that the ARM stacktrace code refuses to trace non-current tasks in a SMP environment due to the racy nature of doing so if the non-current tasks are running. When walking the stack with frame pointers, we: - validate that the frame pointer is between the stack pointer and the top of stack defined by that stack pointer. - we then load the next stack pointer and next frame pointer from the stack. The reason this is unsafe when the task is not blocked is the stack can change at any moment, which can cause the value read as a stack pointer to be wildly different. If the read frame pointer value is roughly in agreement, we can end up reading any part of memory, which would be an information leak. The table based unwinding is much more complex being essentially a set of instructions to the unwinder code about which values to read from the stack into a set of pseudo-registers, corrections to the stack pointer, or transfers from the pseudo-registers. I haven't analysed this code enough to really know the implications of what could be possible if the values on the stack change while this code is running on another CPU (it's not my code!) There is an attempt to bounds-limit the virtual stack pointer after each unwind instruction is processed to catch the unwinder doing anything silly, so it may be safe in so far as it will fail should it encounter anything "stupid". However, get_wchan() is a different case; we know for certain that the task is blocked, so it won't be running on another CPU, and with your patch 4, we have this guarantee. However, that is not true of all callers to the stacktracing code, so I don't see how we can sanely switch to using the stacktracing code for this.
On Thu, Oct 14, 2021 at 02:38:19PM +0100, Russell King (Oracle) wrote: > What is going on here is that the ARM stacktrace code refuses to trace > non-current tasks in a SMP environment due to the racy nature of doing > so if the non-current tasks are running. > > When walking the stack with frame pointers, we: > > - validate that the frame pointer is between the stack pointer and the > top of stack defined by that stack pointer. > - we then load the next stack pointer and next frame pointer from the > stack. > > The reason this is unsafe when the task is not blocked is the stack can > change at any moment, which can cause the value read as a stack pointer > to be wildly different. If the read frame pointer value is roughly in > agreement, we can end up reading any part of memory, which would be an > information leak. It would be a good idea to add some guardrails to prevent that regardless. If there's stack corruption for any reason, the unwinder shouldn't make things worse. On x86 the unwinder relies on the caller to ensure the task is blocked (or current). If the caller doesn't do that, they might get garbage, and they get to keep the pieces. But an important part of that is that the unwinder has guardrails to ensure it handles stack corruption gracefully by never accessing out of bounds of the stack. When multiple stacks are involved in a kernel execution path (task, irq, exception, etc), the stacks link to each other (e.g., last word on the irq stack might point to the task stack). Also the irq/exception stack addresses are stored in percpu variables, and the task stack is in the task struct. So the unwinder can easily make sure it's in-bounds. See get_stack_info() in arch/x86/kernel/dumpstack_64.c.