From patchwork Tue Apr 18 19:56:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Geert Uytterhoeven X-Patchwork-Id: 9686175 X-Patchwork-Delegate: geert@linux-m68k.org Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6D279602C9 for ; Tue, 18 Apr 2017 19:57:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 592F0223A6 for ; Tue, 18 Apr 2017 19:57:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D6E2283E8; Tue, 18 Apr 2017 19:57:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 28877223A6 for ; Tue, 18 Apr 2017 19:57:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757474AbdDRT5B (ORCPT ); Tue, 18 Apr 2017 15:57:01 -0400 Received: from mail-io0-f195.google.com ([209.85.223.195]:33742 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755272AbdDRT47 (ORCPT ); Tue, 18 Apr 2017 15:56:59 -0400 Received: by mail-io0-f195.google.com with SMTP id k87so1333395ioi.0; Tue, 18 Apr 2017 12:56:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to:cc; bh=Ga9IsQJR4r0WPclgh3gM0dDVKPJKXGp5ZpVfMYZtITo=; b=S2qN1OxEE1r8QU7a/cpBTFRF0QWEKDisGlw0C3QVKgfT7PN0jmrav3TFyhMC7bhK0R WqYbJRtEi2Hkb+WrK3XcTa40TRT4nsdp6fVs12IUYaGS8u1f6CbPoo9xCKewPQiZWuER kzCUUtSauIdOM3x2ALqSsyxKRr2Xs5nswJMPCOqcJ8LmGQ+u1zFs5+EyLxluAw6RdO7B n+cjo57ZhOj16VLQOmBtXTKW8SkdSDToytVK2d0UA/m5oK5HBFT8K1umOAvasJeXDs/1 AqHAKPmLQfQ2yw0pwRLACFBXA7ZNpFS0MQIGGKG9Vvch/I6buV9aOBWVpLJ89bsSGb8r km5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to:cc; bh=Ga9IsQJR4r0WPclgh3gM0dDVKPJKXGp5ZpVfMYZtITo=; b=dj0YsXxbAn1LZVGi05tcHsalVMXnkwdnh7fZCLCk18gxQQnUe13RUrcnOPzIXZmKeY osccO79lt8rfyBDQlGDPAFOgjet/4omW6PrYg7aeu5axqJh5EaW6U/Fyxk66aSBsa0+P WP4aAMhva5DPSId3JDsHI3bFzZPs+Z+Pms/G8Fx46UveE2qaCd9Vjlu664V5umRxUP87 Dtu8+f7rJOUCzXf07QFZML269apOOO+L07wYZ1GLAIcCgY/SdtK5tgh3xWBq4dslgbjY qpIOk1+vKer+8xCYYDEc4s9H4U7HGPjB1oC/XNWicD6UlL9UAASOFhurVTclUOqSKRp+ iquA== X-Gm-Message-State: AN3rC/46U79h1etNdm8XYmpxo+lwqLp1GImooukr6gxY6Hq37V4+ndlH Gdga91Vr6vWwx5hMB73c6LKGfsFUCA== X-Received: by 10.107.47.101 with SMTP id j98mr1075181ioo.203.1492545417927; Tue, 18 Apr 2017 12:56:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.39.212 with HTTP; Tue, 18 Apr 2017 12:56:56 -0700 (PDT) From: Geert Uytterhoeven Date: Tue, 18 Apr 2017 21:56:56 +0200 X-Google-Sender-Auth: vouJ0sclqGQ_FpRyecfDKq76Yxo Message-ID: Subject: "mm: move pcp and lru-pcp draining into single wq" broke resume from s2ram To: Michal Hocko , Andrew Morton , Linus Torvalds , Tetsuo Handa Cc: Linux Kernel Mailing List , Linux MM , Linux PM list , Vlastimil Babka , Mel Gorman , Linux-Renesas Sender: linux-renesas-soc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-renesas-soc@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi all, On Sat, Apr 8, 2017 at 10:48 AM, Linux Kernel Mailing List wrote: > Web: https://git.kernel.org/torvalds/c/ce612879ddc78ea7e4de4be80cba4ebf9caa07ee > Commit: ce612879ddc78ea7e4de4be80cba4ebf9caa07ee > Parent: cdcf4330d5660998d06fcd899b443693ab3d652f > Refname: refs/heads/master > Author: Michal Hocko > AuthorDate: Fri Apr 7 16:05:05 2017 -0700 > Committer: Linus Torvalds > CommitDate: Sat Apr 8 00:47:49 2017 -0700 > > mm: move pcp and lru-pcp draining into single wq > > We currently have 2 specific WQ_RECLAIM workqueues in the mm code. > vmstat_wq for updating pcp stats and lru_add_drain_wq dedicated to drain > per cpu lru caches. This seems more than necessary because both can run > on a single WQ. Both do not block on locks requiring a memory > allocation nor perform any allocations themselves. We will save one > rescuer thread this way. > > On the other hand drain_all_pages() queues work on the system wq which > doesn't have rescuer and so this depend on memory allocation (when all > workers are stuck allocating and new ones cannot be created). > > Initially we thought this would be more of a theoretical problem but > Hugh Dickins has reported: > > : 4.11-rc has been giving me hangs after hours of swapping load. At > : first they looked like memory leaks ("fork: Cannot allocate memory"); > : but for no good reason I happened to do "cat /proc/sys/vm/stat_refresh" > : before looking at /proc/meminfo one time, and the stat_refresh stuck > : in D state, waiting for completion of flush_work like many kworkers. > : kthreadd waiting for completion of flush_work in drain_all_pages(). > > This worker should be using WQ_RECLAIM as well in order to guarantee a > forward progress. We can reuse the same one as for lru draining and > vmstat. > > Link: http://lkml.kernel.org/r/20170307131751.24936-1-mhocko@kernel.org > Signed-off-by: Michal Hocko > Suggested-by: Tetsuo Handa > Acked-by: Vlastimil Babka > Acked-by: Mel Gorman > Tested-by: Yang Li > Tested-by: Hugh Dickins > Signed-off-by: Andrew Morton > Signed-off-by: Linus Torvalds This commit broke resume from s2ram on some of my Renesas ARM boards. On some boards the hang is 100% reproducible, on others it's intermittent (which was a PITA, as I had to bisect another independent s2ram bug as well). On r8a7791/koelsch: --- /tmp/good 2017-04-18 21:47:04.457156167 +0200 +++ /tmp/bad 2017-04-18 21:43:26.215240325 +0200 @@ -13,11 +13,178 @@ Enabling non-boot CPUs ... CPU1 is up PM: noirq resume of devices complete after N.N msecs PM: early resume of devices complete after N.N msecs -Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY driver [Micrel KSZ8041RNLI] (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=-1) -PM: resume of devices complete after N.N msecs -PM: resume devices took N.N seconds -PM: Finishing wakeup. -Restarting tasks ... done. -ata1: link resume succeeded after 1 retries -ata1: SATA link down (SStatus 0 SControl 300) -sh-eth ee700000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx +INFO: task kworker/u4:0:5 blocked for more than 120 seconds. + Not tainted 4.11.0-rc7-koelsch-00426-g70412b99f7936b37 #3470 +"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. +kworker/u4:0 D 0 5 2 0x00000000 +Workqueue: events_unbound async_run_entry_fn +[] (__schedule) from [] (schedule+0xb0/0xcc) +[] (schedule) from [] (schedule_timeout+0x18/0x1f4) +[] (schedule_timeout) from [] (wait_for_common+0x100/0x19c) +[] (wait_for_common) from [] (dpm_wait_for_superior+0x14/0x5c) +[] (dpm_wait_for_superior) from [] (device_resume+0x40/0x1a0) +[] (device_resume) from [] (async_resume+0x18/0x44) +[] (async_resume) from [] (async_run_entry_fn+0x44/0x114) +[] (async_run_entry_fn) from [] (process_one_work+0x1cc/0x31c) +[] (process_one_work) from [] (worker_thread+0x2b8/0x3f0) +[] (worker_thread) from [] (kthread+0x120/0x140) +[] (kthread) from [] (ret_from_fork+0x14/0x2c) +INFO: task kworker/u4:1:125 blocked for more than 120 seconds. + Not tainted 4.11.0-rc7-koelsch-00426-g70412b99f7936b37 #3470 +"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. +kworker/u4:1 D 0 125 2 0x00000000 +Workqueue: events_unbound async_run_entry_fn +[] (__schedule) from [] (schedule+0xb0/0xcc) +[] (schedule) from [] (schedule_timeout+0x18/0x1f4) +[] (schedule_timeout) from [] (wait_for_common+0x100/0x19c) +[] (wait_for_common) from [] (dpm_wait_for_superior+0x14/0x5c) +[] (dpm_wait_for_superior) from [] (device_resume+0x40/0x1a0) +[] (device_resume) from [] (async_resume+0x18/0x44) +[] (async_resume) from [] (async_run_entry_fn+0x44/0x114) +[] (async_run_entry_fn) from [] (process_one_work+0x1cc/0x31c) +[] (process_one_work) from [] (worker_thread+0x2b8/0x3f0) +[] (worker_thread) from [] (kthread+0x120/0x140) +[] (kthread) from [] (ret_from_fork+0x14/0x2c) ... On r8a7795/salvator-x, where I have working lockdep: PM: noirq resume of devices complete after 131.415 msecs PM: early resume of devices complete after 8.894 msecs INFO: task kworker/u16:2:276 blocked for more than 120 seconds. Not tainted 4.11.0-rc7-salvator-x-06706-g70412b99f7936b37 #1220 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u16:2 D 0 276 2 0x00000000 Workqueue: events_unbound async_run_entry_fn Call trace: [] __switch_to+0xa0/0xac [] __schedule+0x70c/0xb88 [] schedule+0x84/0xa4 [] schedule_timeout+0x30/0x400 [] wait_for_common+0x164/0x1a8 [] wait_for_completion+0x14/0x1c [] dpm_wait+0x30/0x38 [] dpm_wait_for_superior+0x28/0x7c [] device_resume+0x44/0x190 [] async_resume+0x24/0x54 [] async_run_entry_fn+0x4c/0x12c [] process_one_work+0x340/0x66c [] worker_thread+0x274/0x39c [] kthread+0x120/0x128 [] ret_from_fork+0x10/0x40 Showing all locks held in the system: 2 locks held by khungtaskd/52: #0: (rcu_read_lock){......}, at: [] watchdog+0xc0/0x618 #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x68/0x18c 2 locks held by kworker/u16:2/276: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:3/291: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 8 locks held by s2ram/1899: #0: (sb_writers#7){.+.+.+}, at: [] vfs_write+0xa8/0x15c #1: (&of->mutex){+.+.+.}, at: [] kernfs_fop_write+0xf0/0x194 #2: (s_active#48){.+.+.+}, at: [] kernfs_fop_write+0xf8/0x194 #3: (pm_mutex){+.+.+.}, at: [] pm_suspend+0x16c/0xabc #4: (&dev->mutex){......}, at: [] device_resume+0x58/0x190 #5: (cma_mutex){+.+...}, at: [] cma_alloc+0x150/0x374 #6: (lock){+.+...}, at: [] lru_add_drain_all+0x4c/0x1b4 #7: (cpu_hotplug.dep_map){++++++}, at: [] get_online_cpus+0x3c/0x9c 2 locks held by kworker/u16:1/1918: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:4/1919: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:5/1920: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:7/1922: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:9/1924: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:10/1925: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:11/1926: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:12/1927: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:13/1928: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:14/1929: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c 2 locks held by kworker/u16:16/1931: #0: ("events_unbound"){.+.+.+}, at: [] process_one_work+0x1c8/0x66c #1: ((&entry->work)){+.+...}, at: [] process_one_work+0x1c8/0x66c Thanks for your comments! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds