From patchwork Fri Dec 20 10:31:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Ridong X-Patchwork-Id: 13916507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E5CBE77188 for ; Fri, 20 Dec 2024 10:41:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DE256B0085; Fri, 20 Dec 2024 05:41:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 88E426B0088; Fri, 20 Dec 2024 05:41:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 755B46B0089; Fri, 20 Dec 2024 05:41:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 596D96B0085 for ; Fri, 20 Dec 2024 05:41:40 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 043104439E for ; Fri, 20 Dec 2024 10:41:39 +0000 (UTC) X-FDA: 82914995352.22.7168FAD Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf12.hostedemail.com (Postfix) with ESMTP id EAB254000A for ; Fri, 20 Dec 2024 10:41:20 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734691262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=KHYRzV/P4zP/iCxlJnJb4QnUihg/J5QF5RlZOnsYuZI=; b=IlSpQ/imFlTXEbE6ICX4pcoSzQ4BQ2W+KrbE3+yravf9I0ytnNKzO6JHyZTnTpFO0PeIWk rjV5Bh9rJhHX9if5OBNFlX620qGuLChKjmvTbCjkG46k3SO5iLJeLLINOvyOm/xYuRXIux xW9cmZ3Yg+1nBU1FJkIYXsclbHqyeCs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734691262; a=rsa-sha256; cv=none; b=ltyh4zaaK6CsPgOMGJXDtnqmJA8iFMygWC9lxUwKAfsdW0LYi8mBm6vnKYJjO2gK1s9dRW blsIsDKZxKH7Tc2Ul+sHHAEmZ6HhPafvlCURijYV8fMqPHrwD5LW5TWe+RXBZroajmRcMT sOlwUf779VvcWHF5GFsSJcNljLyiodQ= Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4YF3qb1Jjpz4f3jqF for ; Fri, 20 Dec 2024 18:41:15 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id AB74D1A0568 for ; Fri, 20 Dec 2024 18:41:29 +0800 (CST) Received: from hulk-vt.huawei.com (unknown [10.67.174.121]) by APP2 (Coremail) with SMTP id Syh0CgBXt+PKSWVn5CNcFA--.50187S2; Fri, 20 Dec 2024 18:41:29 +0800 (CST) From: Chen Ridong To: akpm@linux-foundation.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: [PATCH v2] memcg: fix soft lockup in the OOM process Date: Fri, 20 Dec 2024 10:31:23 +0000 Message-Id: <20241220103123.3677988-1-chenridong@huaweicloud.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CM-TRANSID: Syh0CgBXt+PKSWVn5CNcFA--.50187S2 X-Coremail-Antispam: 1UD129KBjvJXoWxXryxGryDKF18WF45XF4Utwb_yoWrCF4kpa srCa1UGw4rGr15Xr12yr1j9r1fAw48ua1UJrW2qr1UAr13Kwn8JrW7Jr4Utrn5WFWaqFya qrn0yw48Kr4qyaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUvFb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Cr0_Gr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I 0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40E x7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x 0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAa w2AFwI0_GFv_Wryl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxV Aqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a 6rW5MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6x kF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AK xVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvj xUIa0PDUUUU X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EAB254000A X-Rspam-User: X-Stat-Signature: kxekcf7mnu7qroon8gz9xnknanwocgsw X-HE-Tag: 1734691280-860641 X-HE-Meta: U2FsdGVkX18jcyhNlhdYC0y5w6H766f4p5DlUVlQAJgnm5ZvKaVDjUUWZWbA52IkRBhHkbZHwH9Quih+Gm3Wvkc99CERum3/CSAz5DZqbRysQiV2MSOgMfszdbC4NvOlElFhbI/8LYRoN6ckEjwx0blPpdAKj0aXExKx0nYEMNYoo+qqekfVv3S1kfxwGzBrH3IfXMDmkChR4TOu1AfySQRI6xOhzuznspqiyJn/uNmdPOzTgP8n269q+GhuP7Dhabv7WIy8miXU+m7N0fWMEjPBzjRuRylkf/U+3vh6pEJq5+yE/nf0gcpr9UJ2oAyz0Q62jdlo0tIKd5sVjsBA14R7R10ToVPV+2ADdZfKXgzCLuw/M3tL/929MJXe6WsElFfiklCZcOXoy37u5dalls6e+m6mgxy8jFNKnKEY3NrGRzUFfMCEGlsbFG1hFuRvw+EYIuy6WfZM+mYJEeYGcRht6L3hIniyEkswoPsbmKXiiRP39g3alt5ogfXRcf6FDHuf675zkKd11nj/bU9TD1rGYKUnOKDdoMGEe0UMPh0FRpxG/KY66ET2MO8ZFRxnUOaNW9yVSqFiB8ZRs6ZRXRdeIAs3dMGf7bJRbooIiEUZ1tmfGVdwNdunDvhAgtRply3p0Xik82Q/U9FAZHfTkP+QHYHaUnTNa5wwndRs6nM77H9DXDQKEQT9gPnI6vTbszkHd4fANc5mNzgl3WK6rRbGM+o9Um7wo+sDCVFyFi/MBHvZRXKXLQzsgmuipUcuHrwpX7V8dZ4my7eC63HRy+EyzFw7HK1FYBpNKabkUOaP4WsPrHR8BfH/vbXYUscbY8guW8wUEgJeBBEX3WIlbFielug3JKJTmlJ5PUxG9kblZX8DDU153wRjsgJ0nDDJQy5/eHMk/5NjzI5rfVe29j1CobDmTaa0YpsBgzXRB9HkT3+V9QATuFSGSTT2uXBaKi+m/YxLOlUtoDK5HBl kMoBda7I MNg1YfLZttT89TTM8GZh5cmBUiUJXMKnWA8Kjvyv+8qOL7zqAlZeKTOleZ1h+D6zuM15L/6SSocprOabT93NbDnSrQVH7gjQCjGw5x1ZS0p3+a5bcmaYdFeGzqZTmcmwQVl0AGPW1zLeApsM7PyeUPBtd4aXcynGmsTK47rYjZOZkkXk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chen Ridong A soft lockup issue was found in the product with about 56,000 tasks were in the OOM cgroup, it was traversing them when the soft lockup was triggered. watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [VM Thread:1503066] CPU: 2 PID: 1503066 Comm: VM Thread Kdump: loaded Tainted: G Hardware name: Huawei Cloud OpenStack Nova, BIOS RIP: 0010:console_unlock+0x343/0x540 RSP: 0000:ffffb751447db9a0 EFLAGS: 00000247 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00000000ffffffff RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000247 RBP: ffffffffafc71f90 R08: 0000000000000000 R09: 0000000000000040 R10: 0000000000000080 R11: 0000000000000000 R12: ffffffffafc74bd0 R13: ffffffffaf60a220 R14: 0000000000000247 R15: 0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2fe6ad91f0 CR3: 00000004b2076003 CR4: 0000000000360ee0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: vprintk_emit+0x193/0x280 printk+0x52/0x6e dump_task+0x114/0x130 mem_cgroup_scan_tasks+0x76/0x100 dump_header+0x1fe/0x210 oom_kill_process+0xd1/0x100 out_of_memory+0x125/0x570 mem_cgroup_out_of_memory+0xb5/0xd0 try_charge+0x720/0x770 mem_cgroup_try_charge+0x86/0x180 mem_cgroup_try_charge_delay+0x1c/0x40 do_anonymous_page+0xb5/0x390 handle_mm_fault+0xc4/0x1f0 This is because thousands of processes are in the OOM cgroup, it takes a long time to traverse all of them. As a result, this lead to soft lockup in the OOM process. To fix this issue, call 'cond_resched' in the 'mem_cgroup_scan_tasks' function per 1000 iterations. For global OOM, call 'touch_softlockup_watchdog' per 1000 iterations to avoid this issue. Fixes: 9cbb78bb3143 ("mm, memcg: introduce own oom handler to iterate only over its own threads") Signed-off-by: Chen Ridong --- include/linux/oom.h | 7 +++++++ mm/memcontrol.c | 6 +++++- mm/oom_kill.c | 7 ++++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/oom.h b/include/linux/oom.h index 1e0fc6931ce9..df2d336f653d 100644 --- a/include/linux/oom.h +++ b/include/linux/oom.h @@ -14,6 +14,13 @@ struct notifier_block; struct mem_cgroup; struct task_struct; +/* When it traverses for long time, to prevent softlockup, call + * cond_resched/touch_softlockup_watchdog very 1000 iterations. + * The 1000 value is not exactly right, it's used to mitigate the overhead + * of cond_resched/touch_softlockup_watchdog. + */ +#define SOFTLOCKUP_PREVENTION_LIMIT 1000 + enum oom_constraint { CONSTRAINT_NONE, CONSTRAINT_CPUSET, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5c373d275e7a..f4c12d6e7b37 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1161,6 +1161,7 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, { struct mem_cgroup *iter; int ret = 0; + int i = 0; BUG_ON(mem_cgroup_is_root(memcg)); @@ -1169,8 +1170,11 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, struct task_struct *task; css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it); - while (!ret && (task = css_task_iter_next(&it))) + while (!ret && (task = css_task_iter_next(&it))) { ret = fn(task, arg); + if (++i % SOFTLOCKUP_PREVENTION_LIMIT) + cond_resched(); + } css_task_iter_end(&it); if (ret) { mem_cgroup_iter_break(memcg, iter); diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 1c485beb0b93..05e613750e9b 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include "internal.h" @@ -430,10 +431,14 @@ static void dump_tasks(struct oom_control *oc) mem_cgroup_scan_tasks(oc->memcg, dump_task, oc); else { struct task_struct *p; + int i = 0; rcu_read_lock(); - for_each_process(p) + for_each_process(p) { + if (++i % SOFTLOCKUP_PREVENTION_LIMIT) + touch_softlockup_watchdog(); dump_task(p, oc); + } rcu_read_unlock(); } }