From patchwork Fri Jan 28 13:09:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12728514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F05AC433EF for ; Fri, 28 Jan 2022 13:10:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E7DB6B0088; Fri, 28 Jan 2022 08:10:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 298156B0087; Fri, 28 Jan 2022 08:10:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D95316B0088; Fri, 28 Jan 2022 08:10:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay031.a.hostedemail.com [64.99.140.31]) by kanga.kvack.org (Postfix) with ESMTP id 646366B0082 for ; Fri, 28 Jan 2022 08:10:12 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2F899229B6 for ; Fri, 28 Jan 2022 13:10:12 +0000 (UTC) X-FDA: 79079729064.03.2D96307 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf19.hostedemail.com (Postfix) with ESMTP id 8D3911A0005 for ; Fri, 28 Jan 2022 13:10:11 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-ed; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=P67mhsIt1ppjFTFW+x1YN2vEk/hxfE1OzpykzhPxla8=; b=j05HraC270uf9eSPytFr4+XRdIB2UMC4nP+hXv9KOHmP5Tpxd1I3xZVKhGwxrIWQxFito 6SX6oF6SPaYp8DyAA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-rsa; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=P67mhsIt1ppjFTFW+x1YN2vEk/hxfE1OzpykzhPxla8=; b=ATrqRJnge0cYDvYRn6k7PBbijeM0h2dmb6XlkF6kgZBYAzZTNWQhpOEyYdfd3ig3EAxlm XRJY7kymtFXBbDuHn20R/ysQ3y+T/UjXQH9ehvC7rWhtJ6mKKjSO4iDbz2j+PZjNMZ5oONj rt8Y0w7HuZxI6oRK0/a1DdPwp8U6Wpk2z6peNkcODaj8xWGYHJnozGnTJmWHzkNbzc9DTQB FGnIuAqR5K+Q++6P4Uz9di1TOldV0BQCB1qqF4PEYmbt1WY5n7VTjqrQr0i+VvEIhcG4heR 9ZzYLWjDurPxXV+wh3Zz/ADRzNspdvk3dD/t6Wz24QmuwistOpeXw6u6QHuA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 29F9D160AAC; Fri, 28 Jan 2022 05:10:07 -0800 (PST) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 0BE3F2055C; Fri, 28 Jan 2022 05:10:07 -0800 (PST) From: Michel Lespinasse To: Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton Cc: kernel-team@fb.com, Laurent Dufour , Jerome Glisse , Peter Zijlstra , Michal Hocko , Vlastimil Babka , Davidlohr Bueso , Matthew Wilcox , Liam Howlett , Rik van Riel , Paul McKenney , Song Liu , Suren Baghdasaryan , Minchan Kim , Joel Fernandes , David Rientjes , Axel Rasmussen , Andy Lutomirski , Michel Lespinasse Subject: [PATCH v2 23/35] mm: add mmu_notifier_lock Date: Fri, 28 Jan 2022 05:09:54 -0800 Message-Id: <20220128131006.67712-24-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220128131006.67712-1-michel@lespinasse.org> References: <20220128131006.67712-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8D3911A0005 X-Stat-Signature: 8ww94gbsebnstmzs45yqjuti8tcpzkbq Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-52-ed header.b=j05HraC2; dkim=pass header.d=lespinasse.org header.s=srv-52-rsa header.b=ATrqRJng; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf19.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspam-User: nil X-HE-Tag: 1643375411-212742 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce mmu_notifier_lock as a per-mm percpu_rw_semaphore, as well as the code to initialize and destroy it together with the mm. This lock will be used to prevent races between mmu_notifier_register() and speculative fault handlers that need to fire MMU notifications without holding any of the mmap or rmap locks. Signed-off-by: Michel Lespinasse --- include/linux/mm_types.h | 6 +++++- include/linux/mmu_notifier.h | 27 +++++++++++++++++++++++++-- kernel/fork.c | 3 ++- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 305f05d2a4bc..f77e2dec038d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -462,6 +462,7 @@ struct vm_area_struct { } __randomize_layout; struct kioctx_table; +struct percpu_rw_semaphore; struct mm_struct { struct { struct vm_area_struct *mmap; /* list of VMAs */ @@ -608,7 +609,10 @@ struct mm_struct { struct file __rcu *exe_file; #ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_subscriptions *notifier_subscriptions; -#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct percpu_rw_semaphore *mmu_notifier_lock; +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ +#endif /* CONFIG_MMU_NOTIFIER */ #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS pgtable_t pmd_huge_pte; /* protected by page_table_lock */ #endif diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 45fc2c81e370..ace76fe91c0c 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -6,6 +6,8 @@ #include #include #include +#include +#include #include #include @@ -499,15 +501,35 @@ static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, __mmu_notifier_invalidate_range(mm, start, end); } -static inline void mmu_notifier_subscriptions_init(struct mm_struct *mm) +static inline bool mmu_notifier_subscriptions_init(struct mm_struct *mm) { +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + mm->mmu_notifier_lock = kzalloc(sizeof(struct percpu_rw_semaphore), GFP_KERNEL); + if (!mm->mmu_notifier_lock) + return false; + if (percpu_init_rwsem(mm->mmu_notifier_lock)) { + kfree(mm->mmu_notifier_lock); + return false; + } +#endif + mm->notifier_subscriptions = NULL; + return true; } static inline void mmu_notifier_subscriptions_destroy(struct mm_struct *mm) { if (mm_has_notifiers(mm)) __mmu_notifier_subscriptions_destroy(mm); + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + if (!in_atomic()) { + percpu_free_rwsem(mm->mmu_notifier_lock); + kfree(mm->mmu_notifier_lock); + } else { + percpu_rwsem_async_destroy(mm->mmu_notifier_lock); + } +#endif } @@ -724,8 +746,9 @@ static inline void mmu_notifier_invalidate_range(struct mm_struct *mm, { } -static inline void mmu_notifier_subscriptions_init(struct mm_struct *mm) +static inline bool mmu_notifier_subscriptions_init(struct mm_struct *mm) { + return true; } static inline void mmu_notifier_subscriptions_destroy(struct mm_struct *mm) diff --git a/kernel/fork.c b/kernel/fork.c index 2e5f2e8de31a..db92e42d0087 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1069,7 +1069,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm_init_owner(mm, p); mm_init_pasid(mm); RCU_INIT_POINTER(mm->exe_file, NULL); - mmu_notifier_subscriptions_init(mm); + if (!mmu_notifier_subscriptions_init(mm)) + goto fail_nopgd; init_tlb_flush_pending(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte = NULL;