| Message ID | 1431624680-20153-23-git-send-email-aarcange@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, 14 May 2015 19:31:19 +0200 Andrea Arcangeli <aarcange@redhat.com> wrote: > If the rwsem starves writers it wasn't strictly a bug but lockdep > doesn't like it and this avoids depending on lowlevel implementation > details of the lock. > > ... > > @@ -229,13 +246,33 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, > > if (!zeropage) > err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, > - dst_addr, src_addr); > + dst_addr, src_addr, &page); > else > err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, > dst_addr); > > cond_resched(); > > + if (unlikely(err == -EFAULT)) { > + void *page_kaddr; > + > + BUILD_BUG_ON(zeropage); I'm not sure what this is trying to do. BUILD_BUG_ON(local_variable)? It goes bang in my build. I'll just delete it. > + up_read(&dst_mm->mmap_sem); > + BUG_ON(!page); > + > + page_kaddr = kmap(page); > + err = copy_from_user(page_kaddr, > + (const void __user *) src_addr, > + PAGE_SIZE); > + kunmap(page); > + if (unlikely(err)) { > + err = -EFAULT; > + goto out; > + } > + goto retry; > + } else > + BUG_ON(page); > + -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, May 22, 2015 at 01:18:22PM -0700, Andrew Morton wrote: > On Thu, 14 May 2015 19:31:19 +0200 Andrea Arcangeli <aarcange@redhat.com> wrote: > > > If the rwsem starves writers it wasn't strictly a bug but lockdep > > doesn't like it and this avoids depending on lowlevel implementation > > details of the lock. > > > > ... > > > > @@ -229,13 +246,33 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, > > > > if (!zeropage) > > err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, > > - dst_addr, src_addr); > > + dst_addr, src_addr, &page); > > else > > err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, > > dst_addr); > > > > cond_resched(); > > > > + if (unlikely(err == -EFAULT)) { > > + void *page_kaddr; > > + > > + BUILD_BUG_ON(zeropage); > > I'm not sure what this is trying to do. BUILD_BUG_ON(local_variable)? > > It goes bang in my build. I'll just delete it. Yes, it has to be a false positive failure, so it's fine to drop it. My gcc 4.8.4 can go inside the static called function and see that only mcopy_atomic_pte can return -EFAULT. RHEL7 (4.8.3) gcc didn't complain either. Perhaps to make the BUILD_BUG_ON work with older gcc, it requrires a local variable set explicitly in the callee, but it's not worth it. It would be bad if we end up in the -EFAULT path in the zeropage case (if somebody later adds an apparently innocent -EFAULT retval and unexpectedly ends up in the mcopy_atomic_pte retry logic), but it's not important, the caller should be reviewed before improvising new retvals anyway. The retry loop addition and the BUILD_BUG_ON is all about the copy_from_user run while we already hold the mmap_sem (potentially of a different process in the non-cooperative case but it's a problem if it's the current task mmap_sem in case the rwlock implementation changes to avoid write starvation and becomes non-reentrant). lockdep definitely complains (even if I think in practice it'd be safe to read-lock recurse, we just got lockdep complains never deadlocks in fact). I didn't want to call gup_fast as copy_from_user is faster and I got an usable user mapping with likely TLB entry hot too. The lockdep warnings we hit I think were associated with NUMA hinting faults or something infrequent like that, the fast path doesn't need to retry. Thanks, Andrea -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
There's a more serious failure with i386 allmodconfig: fs/userfaultfd.c:145:2: note: in expansion of macro 'BUILD_BUG_ON' BUILD_BUG_ON(sizeof(struct uffd_msg) != 32); I'm surprised the feature is even reachable on i386 builds? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index c54c761..f8fe494 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -21,26 +21,39 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, - unsigned long src_addr) + unsigned long src_addr, + struct page **pagep) { struct mem_cgroup *memcg; pte_t _dst_pte, *dst_pte; spinlock_t *ptl; - struct page *page; void *page_kaddr; int ret; + struct page *page; - ret = -ENOMEM; - page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, dst_vma, dst_addr); - if (!page) - goto out; - - page_kaddr = kmap(page); - ret = -EFAULT; - if (copy_from_user(page_kaddr, (const void __user *) src_addr, - PAGE_SIZE)) - goto out_kunmap_release; - kunmap(page); + if (!*pagep) { + ret = -ENOMEM; + page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, dst_vma, dst_addr); + if (!page) + goto out; + + page_kaddr = kmap_atomic(page); + ret = copy_from_user(page_kaddr, + (const void __user *) src_addr, + PAGE_SIZE); + kunmap_atomic(page_kaddr); + + /* fallback to copy_from_user outside mmap_sem */ + if (unlikely(ret)) { + ret = -EFAULT; + *pagep = page; + /* don't free the page */ + goto out; + } + } else { + page = *pagep; + *pagep = NULL; + } /* * The memory barrier inside __SetPageUptodate makes sure that @@ -82,9 +95,6 @@ out_release_uncharge_unlock: out_release: page_cache_release(page); goto out; -out_kunmap_release: - kunmap(page); - goto out_release; } static int mfill_zeropage_pte(struct mm_struct *dst_mm, @@ -139,7 +149,8 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, ssize_t err; pmd_t *dst_pmd; unsigned long src_addr, dst_addr; - long copied = 0; + long copied; + struct page *page; /* * Sanitize the command parameters: @@ -151,6 +162,11 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, BUG_ON(src_start + len <= src_start); BUG_ON(dst_start + len <= dst_start); + src_addr = src_start; + dst_addr = dst_start; + copied = 0; + page = NULL; +retry: down_read(&dst_mm->mmap_sem); /* @@ -160,10 +176,10 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, err = -EINVAL; dst_vma = find_vma(dst_mm, dst_start); if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) - goto out; + goto out_unlock; if (dst_start < dst_vma->vm_start || dst_start + len > dst_vma->vm_end) - goto out; + goto out_unlock; /* * Be strict and only allow __mcopy_atomic on userfaultfd @@ -175,14 +191,14 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, * belonging to the userfaultfd and not syscalls. */ if (!dst_vma->vm_userfaultfd_ctx.ctx) - goto out; + goto out_unlock; /* * FIXME: only allow copying on anonymous vmas, tmpfs should * be added. */ if (dst_vma->vm_ops) - goto out; + goto out_unlock; /* * Ensure the dst_vma has a anon_vma or this page @@ -191,12 +207,13 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, */ err = -ENOMEM; if (unlikely(anon_vma_prepare(dst_vma))) - goto out; + goto out_unlock; - for (src_addr = src_start, dst_addr = dst_start; - src_addr < src_start + len; ) { + while (src_addr < src_start + len) { pmd_t dst_pmdval; + BUG_ON(dst_addr >= dst_start + len); + dst_pmd = mm_alloc_pmd(dst_mm, dst_addr); if (unlikely(!dst_pmd)) { err = -ENOMEM; @@ -229,13 +246,33 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, if (!zeropage) err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, - dst_addr, src_addr); + dst_addr, src_addr, &page); else err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, dst_addr); cond_resched(); + if (unlikely(err == -EFAULT)) { + void *page_kaddr; + + BUILD_BUG_ON(zeropage); + up_read(&dst_mm->mmap_sem); + BUG_ON(!page); + + page_kaddr = kmap(page); + err = copy_from_user(page_kaddr, + (const void __user *) src_addr, + PAGE_SIZE); + kunmap(page); + if (unlikely(err)) { + err = -EFAULT; + goto out; + } + goto retry; + } else + BUG_ON(page); + if (!err) { dst_addr += PAGE_SIZE; src_addr += PAGE_SIZE; @@ -248,8 +285,11 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, break; } -out: +out_unlock: up_read(&dst_mm->mmap_sem); +out: + if (page) + page_cache_release(page); BUG_ON(copied < 0); BUG_ON(err > 0); BUG_ON(!copied && !err);
If the rwsem starves writers it wasn't strictly a bug but lockdep doesn't like it and this avoids depending on lowlevel implementation details of the lock. Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> --- mm/userfaultfd.c | 92 ++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 66 insertions(+), 26 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html