[061/146] mm: protect free_pgtables with mmap_lock write lock in exit_mmap

Commit Message

Andrew Morton Jan. 14, 2022, 10:06 p.m. UTC

From: Suren Baghdasaryan <[e-mail]>
Subject: mm: protect free_pgtables with mmap_lock write lock in exit_mmap

oom-reaper and process_mrelease system call should protect against races
with exit_mmap which can destroy page tables while they walk the VMA tree.
oom-reaper protects from that race by setting MMF_OOM_VICTIM and by
relying on exit_mmap to set MMF_OOM_SKIP before taking and releasing
mmap_write_lock.  process_mrelease has to elevate mm->mm_users to prevent
such race.

Both oom-reaper and process_mrelease hold mmap_read_lock when walking the
VMA tree.  The locking rules and mechanisms could be simpler if exit_mmap
takes mmap_write_lock while executing destructive operations such as
free_pgtables.

Change exit_mmap to hold the mmap_write_lock when calling unlock_range,
free_pgtables and remove_vma.  Note also that because oom-reaper checks
VM_LOCKED flag, unlock_range() should not be allowed to race with it.

Before this patch, remove_vma used to be called with no locks held,
however with fput being executed asynchronously and vm_ops->close not
being allowed to hold mmap_lock (it is called from __split_vma with
mmap_sem held for write), changing that should be fine.

In most cases this lock should be uncontended.  Previously, Kirill
reported ~4% regression caused by a similar change [1].  We reran the same
test and although the individual results are quite noisy, the percentiles
show lower regression with 1.6% being the worst case [2].  The change
allows oom-reaper and process_mrelease to execute safely under
mmap_read_lock without worries that exit_mmap might destroy page tables
from under them.

[1] https://lore.kernel.org/all/[e-mail]/
[2] https://lore.kernel.org/all/CAJuCfpGC9-c9P40x7oy=[e-mail]/

Link: https://lkml.kernel.org/r/[e-mail]
Signed-off-by: Suren Baghdasaryan <[e-mail]>
Acked-by: Michal Hocko <[e-mail]>
Cc: David Rientjes <[e-mail]>
Cc: Matthew Wilcox <[e-mail]>
Cc: Johannes Weiner <[e-mail]>
Cc: Roman Gushchin <[e-mail]>
Cc: Rik van Riel <[e-mail]>
Cc: Minchan Kim <[e-mail]>
Cc: Kirill A. Shutemov <[e-mail]>
Cc: Andrea Arcangeli <[e-mail]>
Cc: Christian Brauner <[e-mail]>
Cc: Christoph Hellwig <[e-mail]>
Cc: Oleg Nesterov <[e-mail]>
Cc: David Hildenbrand <[e-mail]>
Cc: Jann Horn <[e-mail]>
Cc: Shakeel Butt <[e-mail]>
Cc: Andy Lutomirski <[e-mail]>
Cc: Christian Brauner <[e-mail]>
Cc: Florian Weimer <[e-mail]>
Cc: Jan Engelhardt <[e-mail]>
Cc: Tim Murray <[e-mail]>
Cc: Jason Gunthorpe <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 mm/mmap.c |   16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

Patch

--- a/mm/mmap.c~mm-protect-free_pgtables-with-mmap_lock-write-lock-in-exit_mmap
+++ a/mm/mmap.c
@@ -3149,25 +3149,27 @@  void exit_mmap(struct mm_struct *mm)
 		 * to mmu_notifier_release(mm) ensures mmu notifier callbacks in
 		 * __oom_reap_task_mm() will not block.
 		 *
-		 * This needs to be done before calling munlock_vma_pages_all(),
+		 * This needs to be done before calling unlock_range(),
 		 * which clears VM_LOCKED, otherwise the oom reaper cannot
 		 * reliably test it.
 		 */
 		(void)__oom_reap_task_mm(mm);
 
 		set_bit(MMF_OOM_SKIP, &mm->flags);
-		mmap_write_lock(mm);
-		mmap_write_unlock(mm);
 	}
 
+	mmap_write_lock(mm);
 	if (mm->locked_vm)
 		unlock_range(mm->mmap, ULONG_MAX);
 
 	arch_exit_mmap(mm);
 
 	vma = mm->mmap;
-	if (!vma)	/* Can happen if dup_mmap() received an OOM */
+	if (!vma) {
+		/* Can happen if dup_mmap() received an OOM */
+		mmap_write_unlock(mm);
 		return;
+	}
 
 	lru_add_drain();
 	flush_cache_mm(mm);
@@ -3178,16 +3180,14 @@  void exit_mmap(struct mm_struct *mm)
 	free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
 	tlb_finish_mmu(&tlb);
 
-	/*
-	 * Walk the list again, actually closing and freeing it,
-	 * with preemption enabled, without holding any MM locks.
-	 */
+	/* Walk the list again, actually closing and freeing it. */
 	while (vma) {
 		if (vma->vm_flags & VM_ACCOUNT)
 			nr_accounted += vma_pages(vma);
 		vma = remove_vma(vma);
 		cond_resched();
 	}
+	mmap_write_unlock(mm);
 	vm_unacct_memory(nr_accounted);
 }