| Message ID | 20241223093722.78570-1-laoar.shao@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state | expand |
On Mon, Dec 23, 2024 at 5:37 PM Yafang Shao <laoar.shao@gmail.com> wrote: > > We recently encountered an XFS deadlock issue, which is a known problem > resolved in the upstream kernel [0]. During the analysis of this issue, I > observed that a kernel thread in the TASK_WAKEKILL state could not be > detected as a hung task by the hung_task detector. The details are as > follows: > > Using the following command, I identified nine tasks stuck in the D state: > > $ ps -eLo state,comm,tid,wchan | grep ^D > D java 4177339 xfs_buf_lock > D kworker/93:3+xf 3025535 xfs_buf_lock > D kworker/87:0+xf 3426612 xfs_extent_busy_flush > D kworker/85:0+xf 3479378 xfs_buf_lock > D kworker/91:1+xf 3584478 xfs_buf_lock > D kworker/80:3+xf 3655680 xfs_buf_lock > D kworker/89:0+xf 3671691 xfs_buf_lock > D kworker/84:1+xf 3708397 xfs_buf_lock > D kworker/81:1+xf 4005763 xfs_buf_lock > > However, the hung_task detector only reported eight of these tasks: > > [3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds. > [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds. > [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds. > [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds. > [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds. > [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds. > [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds. > [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds. > > Task 3426612, although in the D state, was not reported as a hung task. > > I confirmed that task 3426612 remained in the D (disk sleep) state and > experienced no context switches over a long period: > > $ cat /proc/3426612/status | grep -E "State:|ctxt_switches:"; \ > sleep 60; echo "----"; \ > cat /proc/3426612/status | grep -E "State:|ctxt_switches:" > State: D (disk sleep) > voluntary_ctxt_switches: 7516 > nonvoluntary_ctxt_switches: 0 > ---- > State: D (disk sleep) > voluntary_ctxt_switches: 7516 > nonvoluntary_ctxt_switches: 0 > > The system's hung_task detector settings were configured as follows: > > kernel.hung_task_timeout_secs = 28 > kernel.hung_task_warnings = -1 > > The issue lies in the handling of task state in the XFS code. Specifically, > the thread in question (3426612) was set to the TASK_KILLABLE state in > xfs_extent_busy_flush(): > > xfs_extent_busy_flush > prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE); > > When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the > hung_task detector ignores it, as it assumes such tasks can be terminated. > However, in this case, the kernel thread cannot be killed, meaning it > effectively becomes a hung task. > > To address this issue, the hung_task detector should report the kthreads in > the TASK_WAKEKILL state. > > Link: https://lore.kernel.org/linux-xfs/20230620002021.1038067-5-david@fromorbit.com/ [0] > Signed-off-by: Yafang Shao <laoar.shao@gmail.com> > Cc: Dave Chinner <david@fromorbit.com> > --- > kernel/hung_task.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/kernel/hung_task.c b/kernel/hung_task.c > index c18717189f32..ed63fd84ce2e 100644 > --- a/kernel/hung_task.c > +++ b/kernel/hung_task.c > @@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > */ > state = READ_ONCE(t->__state); > if ((state & TASK_UNINTERRUPTIBLE) && > + (t->flags & PF_KTHREAD || > !(state & TASK_WAKEKILL) && > - !(state & TASK_NOLOAD)) > + !(state & TASK_NOLOAD))) > check_hung_task(t, timeout); > } > unlock: > -- > 2.43.5 > Please disregard this. There may be multiple hung tasks in the TASK_IDLE state. I will send a new one. -- Regards Yafang
diff --git a/kernel/hung_task.c b/kernel/hung_task.c index c18717189f32..ed63fd84ce2e 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) */ state = READ_ONCE(t->__state); if ((state & TASK_UNINTERRUPTIBLE) && + (t->flags & PF_KTHREAD || !(state & TASK_WAKEKILL) && - !(state & TASK_NOLOAD)) + !(state & TASK_NOLOAD))) check_hung_task(t, timeout); } unlock:
We recently encountered an XFS deadlock issue, which is a known problem resolved in the upstream kernel [0]. During the analysis of this issue, I observed that a kernel thread in the TASK_WAKEKILL state could not be detected as a hung task by the hung_task detector. The details are as follows: Using the following command, I identified nine tasks stuck in the D state: $ ps -eLo state,comm,tid,wchan | grep ^D D java 4177339 xfs_buf_lock D kworker/93:3+xf 3025535 xfs_buf_lock D kworker/87:0+xf 3426612 xfs_extent_busy_flush D kworker/85:0+xf 3479378 xfs_buf_lock D kworker/91:1+xf 3584478 xfs_buf_lock D kworker/80:3+xf 3655680 xfs_buf_lock D kworker/89:0+xf 3671691 xfs_buf_lock D kworker/84:1+xf 3708397 xfs_buf_lock D kworker/81:1+xf 4005763 xfs_buf_lock However, the hung_task detector only reported eight of these tasks: [3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds. [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds. [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds. [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds. [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds. [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds. [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds. [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds. Task 3426612, although in the D state, was not reported as a hung task. I confirmed that task 3426612 remained in the D (disk sleep) state and experienced no context switches over a long period: $ cat /proc/3426612/status | grep -E "State:|ctxt_switches:"; \ sleep 60; echo "----"; \ cat /proc/3426612/status | grep -E "State:|ctxt_switches:" State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0 ---- State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0 The system's hung_task detector settings were configured as follows: kernel.hung_task_timeout_secs = 28 kernel.hung_task_warnings = -1 The issue lies in the handling of task state in the XFS code. Specifically, the thread in question (3426612) was set to the TASK_KILLABLE state in xfs_extent_busy_flush(): xfs_extent_busy_flush prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE); When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the hung_task detector ignores it, as it assumes such tasks can be terminated. However, in this case, the kernel thread cannot be killed, meaning it effectively becomes a hung task. To address this issue, the hung_task detector should report the kthreads in the TASK_WAKEKILL state. Link: https://lore.kernel.org/linux-xfs/20230620002021.1038067-5-david@fromorbit.com/ [0] Signed-off-by: Yafang Shao <laoar.shao@gmail.com> Cc: Dave Chinner <david@fromorbit.com> --- kernel/hung_task.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)