mm: Fix MAP_POPULATE and mlock() for DAX
diff mbox

Message ID 1434493710-11138-1-git-send-email-toshi.kani@hp.com
State Not Applicable
Headers show

Commit Message

Toshi Kani June 16, 2015, 10:28 p.m. UTC
DAX has the following issues in a shared or read-only private
mmap'd file.
 - mmap(MAP_POPULATE) does not pre-fault
 - mlock() fails with -ENOMEM

DAX uses VM_MIXEDMAP for mmap'd files, which do not have struct
page associated with the ranges.  Both MAP_POPULATE and mlock()
call __mm_populate(), which in turn calls __get_user_pages().
Because __get_user_pages() requires a valid page returned from
follow_page_mask(), MAP_POPULATE and mlock(), i.e. FOLL_POPULATE,
fail in the first page.

Change __get_user_pages() to proceed FOLL_POPULATE when the
translation is set but its page does not exist (-EFAULT), and
@pages is not requested.  With that, MAP_POPULATE and mlock()
set translations to the requested range and complete successfully.

MAP_POPULATE still provides a major performance improvement to
DAX as it will avoid page faults during initial access to the
pages.

mlock() continues to set VM_LOCKED to vma and populate the range.
Since there is no struct page, the range is pinned without marking
pages mlocked.

Note, MAP_POPULATE and mlock() already work for a write-able
private mmap'd file on DAX since populate_vma_page_range() breaks
COW, which allocates page caches.

Signed-off-by: Toshi Kani <toshi.kani@hp.com>
---
 mm/gup.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

Comments

Kirill A. Shutemov June 20, 2015, 7:46 p.m. UTC | #1
On Tue, Jun 16, 2015 at 04:28:30PM -0600, Toshi Kani wrote:
> DAX has the following issues in a shared or read-only private
> mmap'd file.
>  - mmap(MAP_POPULATE) does not pre-fault
>  - mlock() fails with -ENOMEM
> 
> DAX uses VM_MIXEDMAP for mmap'd files, which do not have struct
> page associated with the ranges.  Both MAP_POPULATE and mlock()
> call __mm_populate(), which in turn calls __get_user_pages().
> Because __get_user_pages() requires a valid page returned from
> follow_page_mask(), MAP_POPULATE and mlock(), i.e. FOLL_POPULATE,
> fail in the first page.
> 
> Change __get_user_pages() to proceed FOLL_POPULATE when the
> translation is set but its page does not exist (-EFAULT), and
> @pages is not requested.  With that, MAP_POPULATE and mlock()
> set translations to the requested range and complete successfully.
> 
> MAP_POPULATE still provides a major performance improvement to
> DAX as it will avoid page faults during initial access to the
> pages.
> 
> mlock() continues to set VM_LOCKED to vma and populate the range.
> Since there is no struct page, the range is pinned without marking
> pages mlocked.
> 
> Note, MAP_POPULATE and mlock() already work for a write-able
> private mmap'd file on DAX since populate_vma_page_range() breaks
> COW, which allocates page caches.

I don't think that's true in all cases.

We would fail to break COW for mlock() if the mapping is populated with
read-only entries by the mlock() time. In this case follow_page_mask()
would fail with -EFAULT and faultin_page() will never executed.
Toshi Kani June 22, 2015, 8:55 p.m. UTC | #2
On Sat, 2015-06-20 at 22:46 +0300, Kirill A. Shutemov wrote:
> On Tue, Jun 16, 2015 at 04:28:30PM -0600, Toshi Kani wrote:
> > DAX has the following issues in a shared or read-only private
> > mmap'd file.
> >  - mmap(MAP_POPULATE) does not pre-fault
> >  - mlock() fails with -ENOMEM
> > 
> > DAX uses VM_MIXEDMAP for mmap'd files, which do not have struct
> > page associated with the ranges.  Both MAP_POPULATE and mlock()
> > call __mm_populate(), which in turn calls __get_user_pages().
> > Because __get_user_pages() requires a valid page returned from
> > follow_page_mask(), MAP_POPULATE and mlock(), i.e. FOLL_POPULATE,
> > fail in the first page.
> > 
> > Change __get_user_pages() to proceed FOLL_POPULATE when the
> > translation is set but its page does not exist (-EFAULT), and
> > @pages is not requested.  With that, MAP_POPULATE and mlock()
> > set translations to the requested range and complete successfully.
> > 
> > MAP_POPULATE still provides a major performance improvement to
> > DAX as it will avoid page faults during initial access to the
> > pages.
> > 
> > mlock() continues to set VM_LOCKED to vma and populate the range.
> > Since there is no struct page, the range is pinned without marking
> > pages mlocked.
> > 
> > Note, MAP_POPULATE and mlock() already work for a write-able
> > private mmap'd file on DAX since populate_vma_page_range() breaks
> > COW, which allocates page caches.
> 
> I don't think that's true in all cases.
> 
> We would fail to break COW for mlock() if the mapping is populated with
> read-only entries by the mlock() time. In this case follow_page_mask()
> would fail with -EFAULT and faultin_page() will never executed.

No, mlock() always breaks COW as populate_vma_page_range() sets
FOLL_WRITE in case of write-able private mmap.

  /*
   * We want to touch writable mappings with a write fault in order
   * to break COW, except for shared mappings because these don't COW
   * and we would not want to dirty them for nothing.
   */
  if ((vma->vm_flags & (VM_WRITE | VM_SHARED)) == VM_WRITE)
           gup_flags |= FOLL_WRITE;

Thanks,
-Toshi

Patch
diff mbox

diff --git a/mm/gup.c b/mm/gup.c
index 6297f6b..16d536f 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -490,8 +490,20 @@  retry:
 			}
 			BUG();
 		}
-		if (IS_ERR(page))
+		if (IS_ERR(page)) {
+			/*
+			 * No page may be associated with VM_MIXEDMAP. Proceed
+			 * FOLL_POPULATE when the translation is set but its
+			 * page does not exist (-EFAULT), and @pages is not
+			 * requested by the caller.
+			 */
+			if ((PTR_ERR(page) == -EFAULT) && (!pages) &&
+			    (gup_flags & FOLL_POPULATE) &&
+			    (vma->vm_flags & VM_MIXEDMAP))
+				goto next_page;
+
 			return i ? i : PTR_ERR(page);
+		}
 		if (pages) {
 			pages[i] = page;
 			flush_anon_page(vma, page, start);