From patchwork Mon Aug 26 20:14:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11115511 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCC2F16B1 for ; Mon, 26 Aug 2019 20:14:50 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A4B9021872 for ; Mon, 26 Aug 2019 20:14:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4B9021872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 59CC888F94; Mon, 26 Aug 2019 20:14:46 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6F8D788E33 for ; Mon, 26 Aug 2019 20:14:34 +0000 (UTC) Received: by mail-ed1-x543.google.com with SMTP id r12so28185923edo.5 for ; Mon, 26 Aug 2019 13:14:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o7ZUgQ2t4EFE1mwtpJ1URNzAUsS76Vtu8jaG9dFn1TA=; b=jN4etrmsbBaVmYjMuiPUYH0ybd6zPlp5wF5otAGb+YLZm/y0CiQ6BuwRrIeH2gS/3a e6h2X9Lu+tcsKbyXYOSPlo4vn9oAkm62HLCa+Fs8HRKbClXO4iQZyif2dWEL5bV5FiGI 21DonPP/YEsYEUB2xruvegqhUEYoyLvosook0Xcdglcn2IVgp77k985H+EbuOQSdEuAK spvNwkwcQRD+6iZXbhVds9G3FOAHK4YwA+D3SxnrtHzO5/Uh1m6ZQjjcSATWlh4Po4zr UaT06Z4Dr6HgqxOk1gPR0u+H/jtm1tLfh4QvELkg3dtar18TKoGMdRFzfIKLeJa2iQIh 7tuA== X-Gm-Message-State: APjAAAWR0FjV0agbq8fHRGLxC7KEBnDtd/34+tnB+ufLg24OKAoVYNAh MrGBzi8Cu54IBOvmD1e9IWTcFw== X-Google-Smtp-Source: APXvYqwI2mGanGH2sozUgL+eYbKM81N0+mUwsmoxd/Kj0A+ZkgeS/zd44/6ltxt/cAqXnGD7EXIVMA== X-Received: by 2002:a50:ab5d:: with SMTP id t29mr20833370edc.32.1566850472927; Mon, 26 Aug 2019 13:14:32 -0700 (PDT) Received: from phenom.ffwll.local (212-51-149-96.fiber7.init7.net. [212.51.149.96]) by smtp.gmail.com with ESMTPSA id j25sm3000780ejb.49.2019.08.26.13.14.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2019 13:14:32 -0700 (PDT) From: Daniel Vetter To: LKML Subject: [PATCH 1/5] mm, notifier: Add a lockdep map for invalidate_range_start/end Date: Mon, 26 Aug 2019 22:14:21 +0200 Message-Id: <20190826201425.17547-2-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190826201425.17547-1-daniel.vetter@ffwll.ch> References: <20190826201425.17547-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o7ZUgQ2t4EFE1mwtpJ1URNzAUsS76Vtu8jaG9dFn1TA=; b=He9JFSR4idYfY3cTrY5TrEj3IMz4MATcFkTbswKaFuTjJVLXrG8g5PlXrG5qVP0BAC vftisf9Y9ImOBlYxAlQuHQAvI72a+KbZ0fnsIWd3BIIgIdpM2sGiWSs2EGrPZfBpVrw1 hbx/vccXAWEoMwnC29A6T7olib4cw3Y+5UUpc= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Linux MM , Daniel Vetter , Jason Gunthorpe , Greg Kroah-Hartman , DRI Development , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Mike Rapoport , David Rientjes , Daniel Vetter , Andrew Morton , =?utf-8?q?Christian_K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This is a similar idea to the fs_reclaim fake lockdep lock. It's fairly easy to provoke a specific notifier to be run on a specific range: Just prep it, and then munmap() it. A bit harder, but still doable, is to provoke the mmu notifiers for all the various callchains that might lead to them. But both at the same time is really hard to reliable hit, especially when you want to exercise paths like direct reclaim or compaction, where it's not easy to control what exactly will be unmapped. By introducing a lockdep map to tie them all together we allow lockdep to see a lot more dependencies, without having to actually hit them in a single challchain while testing. On Jason's suggestion this is is rolled out for both invalidate_range_start and invalidate_range_end. They both have the same calling context, hence we can share the same lockdep map. Note that the annotation for invalidate_ranage_start is outside of the mm_has_notifiers(), to make sure lockdep is informed about all paths leading to this context irrespective of whether mmu notifiers are present for a given context. We don't do that on the invalidate_range_end side to avoid paying the overhead twice, there the lockdep annotation is pushed down behind the mm_has_notifiers() check. v2: Use lock_map_acquire/release() like fs_reclaim, to avoid confusion with this being a real mutex (Chris Wilson). v3: Rebase on top of Glisse's arg rework. v4: Also annotate invalidate_range_end (Jason Gunthorpe) Also annotate invalidate_range_start_nonblock, I somehow missed that one in the first version. Cc: Jason Gunthorpe Cc: Chris Wilson Cc: Andrew Morton Cc: David Rientjes Cc: "Jérôme Glisse" Cc: Michal Hocko Cc: "Christian König" Cc: Greg Kroah-Hartman Cc: Daniel Vetter Cc: Mike Rapoport Cc: linux-mm@kvack.org Reviewed-by: Jason Gunthorpe Signed-off-by: Daniel Vetter --- include/linux/mmu_notifier.h | 8 ++++++++ mm/mmu_notifier.c | 9 +++++++++ 2 files changed, 17 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 31aa971315a1..3f9829a1f32e 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -42,6 +42,10 @@ enum mmu_notifier_event { #ifdef CONFIG_MMU_NOTIFIER +#ifdef CONFIG_LOCKDEP +extern struct lockdep_map __mmu_notifier_invalidate_range_start_map; +#endif + /* * The mmu notifier_mm structure is allocated and installed in * mm->mmu_notifier_mm inside the mm_take_all_locks() protected @@ -341,19 +345,23 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm, static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { + lock_map_acquire(&__mmu_notifier_invalidate_range_start_map); if (mm_has_notifiers(range->mm)) { range->flags |= MMU_NOTIFIER_RANGE_BLOCKABLE; __mmu_notifier_invalidate_range_start(range); } + lock_map_release(&__mmu_notifier_invalidate_range_start_map); } static inline int mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range) { + lock_map_acquire(&__mmu_notifier_invalidate_range_start_map); if (mm_has_notifiers(range->mm)) { range->flags &= ~MMU_NOTIFIER_RANGE_BLOCKABLE; return __mmu_notifier_invalidate_range_start(range); } + lock_map_release(&__mmu_notifier_invalidate_range_start_map); return 0; } diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index d76ea27e2bbb..d48d3b2abd68 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -21,6 +21,13 @@ /* global SRCU for all MMs */ DEFINE_STATIC_SRCU(srcu); +#ifdef CONFIG_LOCKDEP +struct lockdep_map __mmu_notifier_invalidate_range_start_map = { + .name = "mmu_notifier_invalidate_range_start" +}; +EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_start_map); +#endif + /* * This function allows mmu_notifier::release callback to delay a call to * a function that will free appropriate resources. The function must be @@ -197,6 +204,7 @@ void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range, struct mmu_notifier *mn; int id; + lock_map_acquire(&__mmu_notifier_invalidate_range_start_map); id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(mn, &range->mm->mmu_notifier_mm->list, hlist) { /* @@ -220,6 +228,7 @@ void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range, mn->ops->invalidate_range_end(mn, range); } srcu_read_unlock(&srcu, id); + lock_map_release(&__mmu_notifier_invalidate_range_start_map); } EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_end); From patchwork Mon Aug 26 20:14:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11115495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C03E21399 for ; Mon, 26 Aug 2019 20:14:39 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A7FB7217F5 for ; Mon, 26 Aug 2019 20:14:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7FB7217F5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 198AD88E45; Mon, 26 Aug 2019 20:14:38 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by gabe.freedesktop.org (Postfix) with ESMTPS id DF01488E35 for ; Mon, 26 Aug 2019 20:14:35 +0000 (UTC) Received: by mail-ed1-x542.google.com with SMTP id t50so28194390edd.2 for ; Mon, 26 Aug 2019 13:14:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=43MMkU2nAs7jtnMnw+TAIk5AMkJWGIqIB3w6SHg2dOA=; b=OIXMaFsOifmuLQXTgeNiIKFlLFHgPJk9ubPV3CgPnJIWNW+W2xYF0ciBO8cGuCS/7i 8If5/uEGNay1T4b8a1tSZtBihodJaqnoTsRfc6C78P6ih19vn6Aq6bjPoV6poktPWDHi 1ggzh36oZ83xdeRAM/dNgcqjlh4Qe6U8KUEEFK0W3fZ1NLJZK6lMlQOrdLU6pz15/KIl qNcap7OrV7LxSNdWltxEz3vHOYdeqar2HiIvmBXc9Pj41b0Y6veRAz+fVopo/rS2p/ko qWM5E9vNx17O+Ujde0ctcMzEPnCupdj5AZfN9TGeMM3OXkqbwcJ5bJ8EKeRsQz9uy7Uk 7NQg== X-Gm-Message-State: APjAAAXFM6Z2Ribi9fBgnxyKfbAfLnmKgfSoYOFAJNt6ayTHJIIaNVgn PEyac7NaUdDIfe+gx0g1JOl5qg== X-Google-Smtp-Source: APXvYqz1alL5dlGwBBuOouoL2tngfggzrmMx20gafiZ6V+aQeNinylzwLOpzJO1HhOF4TDXl6u3nsA== X-Received: by 2002:a50:d0cc:: with SMTP id g12mr19859322edf.201.1566850474445; Mon, 26 Aug 2019 13:14:34 -0700 (PDT) Received: from phenom.ffwll.local (212-51-149-96.fiber7.init7.net. [212.51.149.96]) by smtp.gmail.com with ESMTPSA id j25sm3000780ejb.49.2019.08.26.13.14.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2019 13:14:33 -0700 (PDT) From: Daniel Vetter To: LKML Subject: [PATCH 2/5] mm, notifier: Prime lockdep Date: Mon, 26 Aug 2019 22:14:22 +0200 Message-Id: <20190826201425.17547-3-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190826201425.17547-1-daniel.vetter@ffwll.ch> References: <20190826201425.17547-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=43MMkU2nAs7jtnMnw+TAIk5AMkJWGIqIB3w6SHg2dOA=; b=jWs+/c0l0gQqLqnonUguMXmcEgRnSUc9uFPRnBjVNs1Nfg/BKnOJH3T6iKhSC9eb6q dpAfCEkDXZ40OJ02lIV0O9i7OYhqlEmJrKDApByW1yfFG3TBOGtD29nnDadAmOX0w/3a ohtG4NM02aYnLMkZ4fScN1kWA5UKzPn2mVZA0= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Linux MM , Daniel Vetter , Jason Gunthorpe , Greg Kroah-Hartman , DRI Development , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Mike Rapoport , David Rientjes , Daniel Vetter , Andrew Morton , =?utf-8?q?Christian_K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" We want to teach lockdep that mmu notifiers can be called from direct reclaim paths, since on many CI systems load might never reach that level (e.g. when just running fuzzer or small functional tests). Motivated by a discussion with Jason. I've put the annotation into mmu_notifier_register since only when we have mmu notifiers registered is there any point in teaching lockdep about them. Also, we already have a kmalloc(, GFP_KERNEL), so this is safe. Cc: Jason Gunthorpe Cc: Chris Wilson Cc: Andrew Morton Cc: David Rientjes Cc: "Jérôme Glisse" Cc: Michal Hocko Cc: "Christian König" Cc: Greg Kroah-Hartman Cc: Daniel Vetter Cc: Mike Rapoport Cc: linux-mm@kvack.org Reviewed-by: Jason Gunthorpe Signed-off-by: Daniel Vetter --- mm/mmu_notifier.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index d48d3b2abd68..0523555933c9 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -259,6 +259,13 @@ int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) lockdep_assert_held_write(&mm->mmap_sem); BUG_ON(atomic_read(&mm->mm_users) <= 0); + if (IS_ENABLED(CONFIG_LOCKDEP)) { + fs_reclaim_acquire(GFP_KERNEL); + lock_map_acquire(&__mmu_notifier_invalidate_range_start_map); + lock_map_release(&__mmu_notifier_invalidate_range_start_map); + fs_reclaim_release(GFP_KERNEL); + } + mn->mm = mm; mn->users = 1; From patchwork Mon Aug 26 20:14:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11115499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B74561399 for ; Mon, 26 Aug 2019 20:14:41 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9F994217F5 for ; Mon, 26 Aug 2019 20:14:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F994217F5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E3FE588E7E; Mon, 26 Aug 2019 20:14:38 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5852888E45 for ; Mon, 26 Aug 2019 20:14:37 +0000 (UTC) Received: by mail-ed1-x541.google.com with SMTP id s15so28186588edx.0 for ; Mon, 26 Aug 2019 13:14:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Hvc+94UsrSo0dvtxcfDnaVeEmPfhp0JVZ15+D+5kcxs=; b=c4n6CzMR0cz7hGhv/l8D2ZsyTwwwy0c7EYBZPAkJfV7HnW7cSbbKlDBmMUv9XhWJ2n jMegWyTy1wlSSWHiaV/lta1dLTqhVIyYWEhO66Hpy0Xef+4RqqAbw8X0mAmdO4Zm/3Vs 1YlaYKxualX0zPPKoOAFfAzxv2FgeN1IcowN4vhI+AkeyJnFLp73R5BAIz7QU+52tbxh AdNWYtjZp814KvYbwTBc7WCrjkMRdQK99UOgEuxxoJFMBqWiIxkJZEKS7BN4w+OtUpnb 0bRkETHxT6X4k3cnvzc9dNA1chK8cZilIbw7dysIPbT/xPy/FiM25x9SLG+c8ukPWCPk fVVw== X-Gm-Message-State: APjAAAUce2Bkb1D5SLqma2qMiXmgS/JJqllCJYLchuHVogTIY0hvE3Zy EM6/N/ppYgDR5Qq402MmPEUORg== X-Google-Smtp-Source: APXvYqwb474jg9zO2UYtdgTPRZkQGqmNA6F5tuCOyD7Luea082jyI2817UPjUHI4jh54moplcdAmZw== X-Received: by 2002:a17:906:3b09:: with SMTP id g9mr18654098ejf.297.1566850475866; Mon, 26 Aug 2019 13:14:35 -0700 (PDT) Received: from phenom.ffwll.local (212-51-149-96.fiber7.init7.net. [212.51.149.96]) by smtp.gmail.com with ESMTPSA id j25sm3000780ejb.49.2019.08.26.13.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2019 13:14:35 -0700 (PDT) From: Daniel Vetter To: LKML Subject: [PATCH 3/5] kernel.h: Add non_block_start/end() Date: Mon, 26 Aug 2019 22:14:23 +0200 Message-Id: <20190826201425.17547-4-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190826201425.17547-1-daniel.vetter@ffwll.ch> References: <20190826201425.17547-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Hvc+94UsrSo0dvtxcfDnaVeEmPfhp0JVZ15+D+5kcxs=; b=VFNW6E8Sf5IO/vio8FpctFWFmv5uJj0ggZOq4v0zfOgDSSsylE0MV0cDqxpUJ0O3+J jJbfISI7yr3sm+aoIHAjZ4ksh6Ts3a0rodKmpkYQuExF0IXSOGXHenwQjq/PFnh2xjem JCdCkwAuhHfBH1F/2PAMX7b12BePRCojTK8VM= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Feng Tang , Michal Hocko , Kees Cook , Linux MM , Peter Zijlstra , Daniel Vetter , Jann Horn , DRI Development , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Ingo Molnar , Thomas Gleixner , Randy Dunlap , David Rientjes , Wei Wang , Daniel Vetter , Andrew Morton , Andy Shevchenko , =?utf-8?q?Christian_K?= =?utf-8?q?=C3=B6nig?= , Masahiro Yamada Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In some special cases we must not block, but there's not a spinlock, preempt-off, irqs-off or similar critical section already that arms the might_sleep() debug checks. Add a non_block_start/end() pair to annotate these. This will be used in the oom paths of mmu-notifiers, where blocking is not allowed to make sure there's forward progress. Quoting Michal: "The notifier is called from quite a restricted context - oom_reaper - which shouldn't depend on any locks or sleepable conditionals. The code should be swift as well but we mostly do care about it to make a forward progress. Checking for sleepable context is the best thing we could come up with that would describe these demands at least partially." Peter also asked whether we want to catch spinlocks on top, but Michal said those are less of a problem because spinlocks can't have an indirect dependency upon the page allocator and hence close the loop with the oom reaper. Suggested by Michal Hocko. v2: - Improve commit message (Michal) - Also check in schedule, not just might_sleep (Peter) v3: It works better when I actually squash in the fixup I had lying around :-/ v4: Pick the suggestion from Andrew Morton to give non_block_start/end some good kerneldoc comments. I added that other blocking calls like wait_event pose similar issues, since that's the other example we discussed. Cc: Jason Gunthorpe Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Andrew Morton Cc: Michal Hocko Cc: David Rientjes Cc: "Christian König" Cc: Daniel Vetter Cc: "Jérôme Glisse" Cc: linux-mm@kvack.org Cc: Masahiro Yamada Cc: Wei Wang Cc: Andy Shevchenko Cc: Thomas Gleixner Cc: Jann Horn Cc: Feng Tang Cc: Kees Cook Cc: Randy Dunlap Cc: linux-kernel@vger.kernel.org Acked-by: Christian König (v1) Acked-by: Peter Zijlstra (Intel) Signed-off-by: Daniel Vetter Acked-by: Michal Hocko --- include/linux/kernel.h | 25 ++++++++++++++++++++++++- include/linux/sched.h | 4 ++++ kernel/sched/core.c | 19 ++++++++++++++----- 3 files changed, 42 insertions(+), 6 deletions(-) diff --git a/include/linux/kernel.h b/include/linux/kernel.h index 4fa360a13c1e..82f84cfe372f 100644 --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -217,7 +217,9 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset); * might_sleep - annotation for functions that can sleep * * this macro will print a stack trace if it is executed in an atomic - * context (spinlock, irq-handler, ...). + * context (spinlock, irq-handler, ...). Additional sections where blocking is + * not allowed can be annotated with non_block_start() and non_block_end() + * pairs. * * This is a useful debugging help to be able to catch problems early and not * be bitten later when the calling function happens to sleep when it is not @@ -233,6 +235,25 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset); # define cant_sleep() \ do { __cant_sleep(__FILE__, __LINE__, 0); } while (0) # define sched_annotate_sleep() (current->task_state_change = 0) +/** + * non_block_start - annotate the start of section where sleeping is prohibited + * + * This is on behalf of the oom reaper, specifically when it is calling the mmu + * notifiers. The problem is that if the notifier were to block on, for example, + * mutex_lock() and if the process which holds that mutex were to perform a + * sleeping memory allocation, the oom reaper is now blocked on completion of + * that memory allocation. Other blocking calls like wait_event() pose similar + * issues. + */ +# define non_block_start() \ + do { current->non_block_count++; } while (0) +/** + * non_block_end - annotate the end of section where sleeping is prohibited + * + * Closes a section opened by non_block_start(). + */ +# define non_block_end() \ + do { WARN_ON(current->non_block_count-- == 0); } while (0) #else static inline void ___might_sleep(const char *file, int line, int preempt_offset) { } @@ -241,6 +262,8 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset); # define might_sleep() do { might_resched(); } while (0) # define cant_sleep() do { } while (0) # define sched_annotate_sleep() do { } while (0) +# define non_block_start() do { } while (0) +# define non_block_end() do { } while (0) #endif #define might_sleep_if(cond) do { if (cond) might_sleep(); } while (0) diff --git a/include/linux/sched.h b/include/linux/sched.h index b6ec130dff9b..e8bb965f5019 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -980,6 +980,10 @@ struct task_struct { struct mutex_waiter *blocked_on; #endif +#ifdef CONFIG_DEBUG_ATOMIC_SLEEP + int non_block_count; +#endif + #ifdef CONFIG_TRACE_IRQFLAGS unsigned int irq_events; unsigned long hardirq_enable_ip; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 45dceec209f4..0d01c7994a9a 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3752,13 +3752,22 @@ static noinline void __schedule_bug(struct task_struct *prev) /* * Various schedule()-time debugging checks and statistics: */ -static inline void schedule_debug(struct task_struct *prev) +static inline void schedule_debug(struct task_struct *prev, bool preempt) { #ifdef CONFIG_SCHED_STACK_END_CHECK if (task_stack_end_corrupted(prev)) panic("corrupted stack end detected inside scheduler\n"); #endif +#ifdef CONFIG_DEBUG_ATOMIC_SLEEP + if (!preempt && prev->state && prev->non_block_count) { + printk(KERN_ERR "BUG: scheduling in a non-blocking section: %s/%d/%i\n", + prev->comm, prev->pid, prev->non_block_count); + dump_stack(); + add_taint(TAINT_WARN, LOCKDEP_STILL_OK); + } +#endif + if (unlikely(in_atomic_preempt_off())) { __schedule_bug(prev); preempt_count_set(PREEMPT_DISABLED); @@ -3870,7 +3879,7 @@ static void __sched notrace __schedule(bool preempt) rq = cpu_rq(cpu); prev = rq->curr; - schedule_debug(prev); + schedule_debug(prev, preempt); if (sched_feat(HRTICK)) hrtick_clear(rq); @@ -6641,7 +6650,7 @@ void ___might_sleep(const char *file, int line, int preempt_offset) rcu_sleep_check(); if ((preempt_count_equals(preempt_offset) && !irqs_disabled() && - !is_idle_task(current)) || + !is_idle_task(current) && !current->non_block_count) || system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING || oops_in_progress) return; @@ -6657,8 +6666,8 @@ void ___might_sleep(const char *file, int line, int preempt_offset) "BUG: sleeping function called from invalid context at %s:%d\n", file, line); printk(KERN_ERR - "in_atomic(): %d, irqs_disabled(): %d, pid: %d, name: %s\n", - in_atomic(), irqs_disabled(), + "in_atomic(): %d, irqs_disabled(): %d, non_block: %d, pid: %d, name: %s\n", + in_atomic(), irqs_disabled(), current->non_block_count, current->pid, current->comm); if (task_stack_end_corrupted(current)) From patchwork Mon Aug 26 20:14:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11115509 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B6D416B1 for ; Mon, 26 Aug 2019 20:14:49 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 433A021872 for ; Mon, 26 Aug 2019 20:14:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 433A021872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3803288F2D; Mon, 26 Aug 2019 20:14:46 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8579288E53 for ; Mon, 26 Aug 2019 20:14:38 +0000 (UTC) Received: by mail-ed1-x543.google.com with SMTP id h13so28117570edq.10 for ; Mon, 26 Aug 2019 13:14:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bew81wZtU1Y0ks/RIjoWK2FqRSw2UObINLTLlREzuuI=; b=Ygy9KWvDg4ka3/hAXj76tmYzDwgzvmqABp5KaMHAra0em644bm4SMUjUuKrPi9TaLv J49b3xWz3bmL/XBlQSyI0smqbeqbnn/t/e8YIUZqMnDgz/BGXPHxfkC2yXkWMvyAmn1V qJPxsUFULEhRlyGAanOcYbRnOikX6zK+ASNqD+yAzNJBjqJzecN924vkABzIvmNkTqXv PC0x/QheZKrZc5QtSWG6w4y0JPe93PDFdj/EKsjawQ5zP/U1l0UQX8NdYLyVwTMhEm9Z /+qRFLZZ0xya+mm3L/cGz7H+r1x6df/sRnR5RBlDa+jVhhqH76auaWyUOcaL+FZmLYYa 2oYw== X-Gm-Message-State: APjAAAWzjQAMavVUcCWsmCqtqNc3TF8p0DhCJEVXZPF1J1oeA215ACxu e9zo21u4NXPuDBw+Pl/MvxiTfw== X-Google-Smtp-Source: APXvYqw8z9afSvhgONVa7l2XkwoZjbx6t7Y2tVjSzj6EiE5l88NiQj8YRIrJNGWJ+8ua4BQqqD4zVA== X-Received: by 2002:a17:907:207a:: with SMTP id qp26mr18160870ejb.12.1566850477119; Mon, 26 Aug 2019 13:14:37 -0700 (PDT) Received: from phenom.ffwll.local (212-51-149-96.fiber7.init7.net. [212.51.149.96]) by smtp.gmail.com with ESMTPSA id j25sm3000780ejb.49.2019.08.26.13.14.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2019 13:14:36 -0700 (PDT) From: Daniel Vetter To: LKML Subject: [PATCH 4/5] mm, notifier: Catch sleeping/blocking for !blockable Date: Mon, 26 Aug 2019 22:14:24 +0200 Message-Id: <20190826201425.17547-5-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190826201425.17547-1-daniel.vetter@ffwll.ch> References: <20190826201425.17547-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bew81wZtU1Y0ks/RIjoWK2FqRSw2UObINLTLlREzuuI=; b=eqf/zxHzmkxFzdusChJdtOxtD1T//2StyL+3IeJkwUOueNAgekF2+ClgnH/jmo0xK7 qr4GW3Irit/IupsG1Na0ULH20pR7piGumm9IQOHaYtrHyUARGCU3fZukTvNoWL+1YqKl pT4WlAHyRMSIzeLbFjs0YtZDoFJZPwP/EgQ5o= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Linux MM , Daniel Vetter , DRI Development , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , David Rientjes , Daniel Vetter , Andrew Morton , =?utf-8?q?Christian_K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" We need to make sure implementations don't cheat and don't have a possible schedule/blocking point deeply burried where review can't catch it. I'm not sure whether this is the best way to make sure all the might_sleep() callsites trigger, and it's a bit ugly in the code flow. But it gets the job done. Inspired by an i915 patch series which did exactly that, because the rules haven't been entirely clear to us. v2: Use the shiny new non_block_start/end annotations instead of abusing preempt_disable/enable. v3: Rebase on top of Glisse's arg rework. v4: Rebase on top of more Glisse rework. v5: Also annotate invalidate_range_end in the same style. I hope I got Jason's request for this right. Cc: Jason Gunthorpe Cc: Andrew Morton Cc: Michal Hocko Cc: David Rientjes Cc: "Christian König" Cc: Daniel Vetter Cc: "Jérôme Glisse" Cc: linux-mm@kvack.org Reviewed-by: Christian König (v1) Reviewed-by: Jérôme Glisse (v4) Signed-off-by: Daniel Vetter --- mm/mmu_notifier.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 0523555933c9..b17f3fd3779b 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -181,7 +181,13 @@ int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(mn, &range->mm->mmu_notifier_mm->list, hlist) { if (mn->ops->invalidate_range_start) { - int _ret = mn->ops->invalidate_range_start(mn, range); + int _ret; + + if (!mmu_notifier_range_blockable(range)) + non_block_start(); + _ret = mn->ops->invalidate_range_start(mn, range); + if (!mmu_notifier_range_blockable(range)) + non_block_end(); if (_ret) { pr_info("%pS callback failed with %d in %sblockable context.\n", mn->ops->invalidate_range_start, _ret, @@ -224,8 +230,13 @@ void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range, mn->ops->invalidate_range(mn, range->mm, range->start, range->end); - if (mn->ops->invalidate_range_end) + if (mn->ops->invalidate_range_end) { + if (!mmu_notifier_range_blockable(range)) + non_block_start(); mn->ops->invalidate_range_end(mn, range); + if (!mmu_notifier_range_blockable(range)) + non_block_end(); + } } srcu_read_unlock(&srcu, id); lock_map_release(&__mmu_notifier_invalidate_range_start_map); From patchwork Mon Aug 26 20:14:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Daniel Vetter X-Patchwork-Id: 11115505 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A44B16B1 for ; Mon, 26 Aug 2019 20:14:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 41D6221872 for ; Mon, 26 Aug 2019 20:14:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 41D6221872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 20ADC88F1B; Mon, 26 Aug 2019 20:14:46 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4339C88EA7 for ; Mon, 26 Aug 2019 20:14:40 +0000 (UTC) Received: by mail-ed1-x541.google.com with SMTP id g8so28158523edm.6 for ; Mon, 26 Aug 2019 13:14:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bUhdeRpbg8gRloy4w33VnXTSeooO5sqpvDT0wx5DMzY=; b=m/i5r8EvQb1WnP/3pUJ4RMLMMuxHHHYQ36TXmIwX1vyD23NjoOIbQeOr8OYwyG+kBE WLjTj3lEusyCBIsvUP8P/R5H+ljM+/hhdqIg3ucGTgRGCGgG/k82muilNyJ4/+EiIUWL KC7m+PdDwPIstytBvOXrxxnS1vdQh8l3yl8UvG/tQ6gA386VKRGID70yXO0GOMtePBcR ZQ6cSz/5CiPTm4dhrl7H8QqmBDMtXo3XQQlnnMUtpmXgkaDF5rtE6WpWThxDPRVOAm2Q xmlzGT6GOlnTfoNfx3sgsylsMbNJy5TJylRVHTfzXC9/iIGNaBO2MNv3tcUQftDhRc8T 8lXg== X-Gm-Message-State: APjAAAWodRtYcb7wvgc227g8Bnn3rlLrVsi2uVPxvsu2bS2Qg8EnBxdB k/NF0yAjI9/ye8FIf8H6BmATcg== X-Google-Smtp-Source: APXvYqwQX5rJn4xiT0PzmoOjk8RceDZHB/u+ViHnJA99I4EPasO1Ql6cGV43P/XDRPUyqHKPLLvlzQ== X-Received: by 2002:aa7:c552:: with SMTP id s18mr21104247edr.0.1566850478492; Mon, 26 Aug 2019 13:14:38 -0700 (PDT) Received: from phenom.ffwll.local (212-51-149-96.fiber7.init7.net. [212.51.149.96]) by smtp.gmail.com with ESMTPSA id j25sm3000780ejb.49.2019.08.26.13.14.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2019 13:14:37 -0700 (PDT) From: Daniel Vetter To: LKML Subject: [PATCH 5/5] mm, notifier: annotate with might_sleep() Date: Mon, 26 Aug 2019 22:14:25 +0200 Message-Id: <20190826201425.17547-6-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190826201425.17547-1-daniel.vetter@ffwll.ch> References: <20190826201425.17547-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bUhdeRpbg8gRloy4w33VnXTSeooO5sqpvDT0wx5DMzY=; b=QViZPMOCt4TCBDSlKAVCdrC+U37mohUaGpt7t54fKUAJnII/M30PJe3HVgsMvO32I/ JHmsQisAEo+uwv1GVmUHudMfC1x01ExLcWbO28gdRMZiZ/4jYiOIUrGZ6L9+5vRcVnku F9YxwgBgX7xnEA70Wp+hl/fz9/FJFtPM7Z4dQ= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michal Hocko , Linux MM , Daniel Vetter , DRI Development , Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , David Rientjes , Daniel Vetter , Andrew Morton , =?utf-8?q?Christian_K=C3=B6nig?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since mmu notifiers don't exist for more processes, but could block in interesting places, add some annotations. This should help make sure core mm keeps up its end of the mmu notifier contract. The checks here are outside of all notifier checks because of that. They compile away without CONFIG_DEBUG_ATOMIC_SLEEP. Suggested by Jason. Cc: Jason Gunthorpe Cc: Andrew Morton Cc: Michal Hocko Cc: David Rientjes Cc: "Christian König" Cc: Daniel Vetter Cc: "Jérôme Glisse" Cc: linux-mm@kvack.org Signed-off-by: Daniel Vetter --- include/linux/mmu_notifier.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 3f9829a1f32e..8b71813417e7 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -345,6 +345,8 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm, static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { + might_sleep(); + lock_map_acquire(&__mmu_notifier_invalidate_range_start_map); if (mm_has_notifiers(range->mm)) { range->flags |= MMU_NOTIFIER_RANGE_BLOCKABLE; @@ -368,6 +370,9 @@ mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range) static inline void mmu_notifier_invalidate_range_end(struct mmu_notifier_range *range) { + if (mmu_notifier_range_blockable(range)) + might_sleep(); + if (mm_has_notifiers(range->mm)) __mmu_notifier_invalidate_range_end(range, false); }