From patchwork Wed Jun 22 12:13:54 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Hocko X-Patchwork-Id: 9192593 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8EF8860756 for ; Wed, 22 Jun 2016 12:14:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7577E283F7 for ; Wed, 22 Jun 2016 12:14:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 69EDD283FE; Wed, 22 Jun 2016 12:14:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C72BA283F7 for ; Wed, 22 Jun 2016 12:14:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751926AbcFVMOI (ORCPT ); Wed, 22 Jun 2016 08:14:08 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:36008 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751755AbcFVMOH (ORCPT ); Wed, 22 Jun 2016 08:14:07 -0400 Received: by mail-wm0-f66.google.com with SMTP id c82so631873wme.3; Wed, 22 Jun 2016 05:14:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=NfWMEmyo+v3pdV12poqBfxHzTEXNgjrwesdDoXGn0AY=; b=M/Ak8xeEM5/XVc78QMoM2b0VoiJWMM2V4cvas2/jgLHjR2bs66K3opy/C7llx/TuaN ipsll5O6NWaG00EPX7gNBGy8lJ4RtyJkqdpgbbNyTkvYKYQ08j68X8J35rc8FpzExifB 0bs6cOOek6Tw5iLqsCS9sIHmFRosjeU8gtQPJBWBVD1vPyP2KtZDU+c6UMsYWw1U9abM lOYU2zSRJ9v0enzr8UJ9OvIL2vgDdrV6RwnQOoH09l9a6ZV62+BOjLTU3bLRfsZBm4G7 /bpfSGMRoS0BJz1rQv9v83WFIVJ3KtRU0v0TVgo2Mn6K9dHMV8h1QmlvCHdWUJtzcv35 FK1A== X-Gm-Message-State: ALyK8tITkwmU7NLdKxLRmPF57qhGSwC57J21KpyaHwNmxTO0ZA5Y+WwqSLtRoTc2zag8Og== X-Received: by 10.28.163.70 with SMTP id m67mr7941465wme.38.1466597645804; Wed, 22 Jun 2016 05:14:05 -0700 (PDT) Received: from tiehlicka.suse.cz ([80.188.202.66]) by smtp.gmail.com with ESMTPSA id e5sm73703496wjj.10.2016.06.22.05.14.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Jun 2016 05:14:05 -0700 (PDT) From: Michal Hocko To: "Rafael J. Wysocki" , Andrew Morton Cc: Tetsuo Handa , , LKML , linux-pm@vger.kernel.org, Michal Hocko Subject: [PATCH] oom, suspend: fix oom_reaper vs. oom_killer_disable race Date: Wed, 22 Jun 2016 14:13:54 +0200 Message-Id: <1466597634-16199-1-git-send-email-mhocko@kernel.org> X-Mailer: git-send-email 2.8.1 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Michal Hocko Tetsuo has reported the following potential oom_killer_disable vs. oom_reaper race: (1) freeze_processes() starts freezing user space threads. (2) Somebody (maybe a kenrel thread) calls out_of_memory(). (3) The OOM killer calls mark_oom_victim() on a user space thread P1 which is already in __refrigerator(). (4) oom_killer_disable() sets oom_killer_disabled = true. (5) P1 leaves __refrigerator() and enters do_exit(). (6) The OOM reaper calls exit_oom_victim(P1) before P1 can call exit_oom_victim(P1). (7) oom_killer_disable() returns while P1 not yet finished (8) P1 perform IO/interfere with the freezer. This situation is unfortunate. We cannot move oom_killer_disable after all the freezable kernel threads are frozen because the oom victim might depend on some of those kthreads to make a forward progress to exit so we could deadlock. It is also far from trivial to teach the oom_reaper to not call exit_oom_victim() because then we would lose a guarantee of the OOM killer and oom_killer_disable forward progress because exit_mm->mmput might block and never call exit_oom_victim. It seems the easiest way forward is to workaround this race by calling try_to_freeze_tasks again after oom_killer_disable. This will make sure that all the tasks are frozen or it bails out. Fixes: 449d777d7ad6 ("mm, oom_reaper: clear TIF_MEMDIE for all tasks queued for oom_reaper") Reported-by: Tetsuo Handa Signed-off-by: Michal Hocko --- Hi, this is a temporal fix that is easy enough for the current 4.7 cycle. Even though I suspect PM suspend race with the OOM killer is super unlikely it is better to not risk it considering the patch is quite trivial. I plan to come up with a proper solution later on after some other OOM killer/reaper changes settle down. It is really desirable that oom_killer_disable() is a full "barrier" rather than working around potential issues. So if this looks like a proper fix to you Rafael I would route this via Andrew. What do you think? kernel/power/process.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/power/process.c b/kernel/power/process.c index df058bed53ce..0c2ee9761d57 100644 --- a/kernel/power/process.c +++ b/kernel/power/process.c @@ -146,6 +146,18 @@ int freeze_processes(void) if (!error && !oom_killer_disable()) error = -EBUSY; + /* + * There is a hard to fix race between oom_reaper kernel thread + * and oom_killer_disable. oom_reaper calls exit_oom_victim + * before the victim reaches exit_mm so try to freeze all the tasks + * again and catch such a left over task. + */ + if (!error) { + pr_info("Double checking all user space processes after OOM killer disable... "); + error = try_to_freeze_tasks(true); + pr_cont("\n"); + } + if (error) thaw_processes(); return error;