[1/3] userfaultfd: UFFDIO_REMAP: rmap preparation

Commit Message

Suren Baghdasaryan Sept. 14, 2023, 3:26 p.m. UTC

From: Andrea Arcangeli <[e-mail]>

As far as the rmap code is concerned, UFFDIO_REMAP only alters the
page->mapping and page->index. It does it while holding the page
lock. However folio_referenced() is doing rmap walks without taking the
folio lock first, so folio_lock_anon_vma_read() must be updated to
re-check that the folio->mapping didn't change after we obtained the
anon_vma read lock.

UFFDIO_REMAP takes the anon_vma lock for writing before altering the
folio->mapping, so if the folio->mapping is still the same after
obtaining the anon_vma read lock (without the folio lock), the rmap
walks can go ahead safely (and UFFDIO_REMAP will wait the rmap walk to
complete before proceeding).

UFFDIO_REMAP serializes against itself with the folio lock.

All other places taking the anon_vma lock while holding the mmap_lock
for writing, don't need to check if the folio->mapping has changed
after taking the anon_vma lock, regardless of the folio lock, because
UFFDIO_REMAP holds the mmap_lock for reading.

There's one constraint enforced to allow this simplification: the
source pages passed to UFFDIO_REMAP must be mapped only in one vma,
but this constraint is an acceptable tradeoff for UFFDIO_REMAP
users.

The source addresses passed to UFFDIO_REMAP can be set as
VM_DONTCOPY with MADV_DONTFORK to avoid any risk of the mapcount of
the pages increasing if some thread of the process forks() before
UFFDIO_REMAP run.

Signed-off-by: Andrea Arcangeli <[e-mail]>
Signed-off-by: Suren Baghdasaryan <[e-mail]>
---
 mm/rmap.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

Comments

Matthew Wilcox Sept. 14, 2023, 5:56 p.m. UTC | #1

On Thu, Sep 14, 2023 at 08:26:11AM -0700, Suren Baghdasaryan wrote:
> From: Andrea Arcangeli <[e-mail]>
> 
> As far as the rmap code is concerned, UFFDIO_REMAP only alters the
> page->mapping and page->index. It does it while holding the page

folio->mapping and folio->index.  It does it while holding the folio

No other objections ...

Suren Baghdasaryan Sept. 14, 2023, 6:34 p.m. UTC | #2

On Thu, Sep 14, 2023 at 10:56 AM Matthew Wilcox <[e-mail]> wrote:
>
> On Thu, Sep 14, 2023 at 08:26:11AM -0700, Suren Baghdasaryan wrote:
> > From: Andrea Arcangeli <[e-mail]>
> >
> > As far as the rmap code is concerned, UFFDIO_REMAP only alters the
> > page->mapping and page->index. It does it while holding the page
>
> folio->mapping and folio->index.  It does it while holding the folio

Ack.

>
> No other objections ...

Thanks for the review!

Patch

diff --git a/mm/rmap.c b/mm/rmap.c
index ec7f8e6c9e48..c1ebbd23fa61 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -542,6 +542,7 @@  struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 	struct anon_vma *root_anon_vma;
 	unsigned long anon_mapping;
 
+repeat:
 	rcu_read_lock();
 	anon_mapping = (unsigned long)READ_ONCE(folio->mapping);
 	if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
@@ -586,6 +587,18 @@  struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 	rcu_read_unlock();
 	anon_vma_lock_read(anon_vma);
 
+	/*
+	 * Check if UFFDIO_REMAP changed the anon_vma. This is needed
+	 * because we don't assume the folio was locked.
+	 */
+	if (unlikely((unsigned long) READ_ONCE(folio->mapping) !=
+		     anon_mapping)) {
+		anon_vma_unlock_read(anon_vma);
+		put_anon_vma(anon_vma);
+		anon_vma = NULL;
+		goto repeat;
+	}
+
 	if (atomic_dec_and_test(&anon_vma->refcount)) {
 		/*
 		 * Oops, we held the last refcount, release the lock