| Message ID | 20220615162805.27962-1-pmladek@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | printk: Prevent printk kthreads from blocking direct console handling | expand |
On Wed, Jun 15, 2022 at 06:28:03PM +0200, Petr Mladek wrote: > There are reports that console kthreads prevented printing > messages during panic() or shutdown(), see > BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1 > BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com > > In both situations, printk() correctly tries to flush the consoles > directly but it fails to get the global console_lock(). Both > problems went away with these patches: > > 1st patch blocks console kthreads so that they do not start > handling new messages when the direct printing is preferred > by the system state. It is probably enough on its own. It > solves the problem when the kthreads actively did > the wrong thing. > > 2nd patch allows to wait for the console kthreads to release > the lock in any context. It should make it more reliable. > It would have been useful even for the legacy code. Thank you! For the series: Tested-by: Paul E. McKenney <paulmck@kernel.org> > More possible improvements: > > + the waiting might be necessary also in the suspend code paths > > + convert con->mutex to con->spinlock to avoid blocking > the global console_lock() when sleeping with con->lock > > + at least disable preemption around console_emit_next_record() > in console kthread to avoid sleeping in the console driver > code > > + somehow change the priority of the kthread so that it gets > scheduled immediately when the scheduler works > > Petr Mladek (2): > printk: Block console kthreads when direct printing will be required > printk: Wait for the global console lock when the system is going down > > include/linux/printk.h | 5 +++++ > kernel/panic.c | 2 ++ > kernel/printk/internal.h | 2 ++ > kernel/printk/printk.c | 8 +++++++- > kernel/printk/printk_safe.c | 32 ++++++++++++++++++++++++++++++++ > kernel/reboot.c | 2 ++ > 6 files changed, 50 insertions(+), 1 deletion(-) > > -- > 2.35.3 >
On Wed 2022-06-15 10:10:42, Paul E. McKenney wrote: > On Wed, Jun 15, 2022 at 06:28:03PM +0200, Petr Mladek wrote: > > There are reports that console kthreads prevented printing > > messages during panic() or shutdown(), see > > BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1 > > BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com > > > > In both situations, printk() correctly tries to flush the consoles > > directly but it fails to get the global console_lock(). Both > > problems went away with these patches: > > > > 1st patch blocks console kthreads so that they do not start > > handling new messages when the direct printing is preferred > > by the system state. It is probably enough on its own. It > > solves the problem when the kthreads actively did > > the wrong thing. > > > > 2nd patch allows to wait for the console kthreads to release > > the lock in any context. It should make it more reliable. > > It would have been useful even for the legacy code. > > Thank you! > > For the series: > > Tested-by: Paul E. McKenney <paulmck@kernel.org> Thanks a lot for testing. I have pushed it into printk/linux.git, branch rework/kthreads to give it a spin in linux-next. Best Regards, Petr
There are reports that console kthreads prevented printing messages during panic() or shutdown(), see BugLink: https://lore.kernel.org/r/20220610205038.GA3050413@paulmck-ThinkPad-P17-Gen-1 BugLink: https://lore.kernel.org/r/CAMdYzYpF4FNTBPZsEFeWRuEwSies36QM_As8osPWZSr2q-viEA@mail.gmail.com In both situations, printk() correctly tries to flush the consoles directly but it fails to get the global console_lock(). Both problems went away with these patches: 1st patch blocks console kthreads so that they do not start handling new messages when the direct printing is preferred by the system state. It is probably enough on its own. It solves the problem when the kthreads actively did the wrong thing. 2nd patch allows to wait for the console kthreads to release the lock in any context. It should make it more reliable. It would have been useful even for the legacy code. More possible improvements: + the waiting might be necessary also in the suspend code paths + convert con->mutex to con->spinlock to avoid blocking the global console_lock() when sleeping with con->lock + at least disable preemption around console_emit_next_record() in console kthread to avoid sleeping in the console driver code + somehow change the priority of the kthread so that it gets scheduled immediately when the scheduler works Petr Mladek (2): printk: Block console kthreads when direct printing will be required printk: Wait for the global console lock when the system is going down include/linux/printk.h | 5 +++++ kernel/panic.c | 2 ++ kernel/printk/internal.h | 2 ++ kernel/printk/printk.c | 8 +++++++- kernel/printk/printk_safe.c | 32 ++++++++++++++++++++++++++++++++ kernel/reboot.c | 2 ++ 6 files changed, 50 insertions(+), 1 deletion(-)