| Message ID | 20181218223557.5202-2-mike.kravetz@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | hugetlbfs: use i_mmap_rwsem for better synchronization | expand |
Hi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag,
fixing commit: 39dde65c9940 [PATCH] shared page table for hugetlb page.
The bot has tested the following trees: v4.19.10, v4.14.89, v4.9.146, v4.4.168, v3.18.130,
v4.19.10: Build OK!
v4.14.89: Failed to apply! Possible dependencies:
285b8dcaacfc ("mm, hugetlbfs: pass fault address to no page handler")
v4.9.146: Failed to apply! Possible dependencies:
1a1aad8a9b7b ("userfaultfd: hugetlbfs: add userfaultfd hugetlb hook")
369cd2121be4 ("userfaultfd: hugetlbfs: userfaultfd_huge_must_wait for hugepmd ranges")
7868a2087ec1 ("mm/hugetlb: add size parameter to huge_pte_offset()")
82b0f8c39a38 ("mm: join struct fault_env and vm_fault")
8fb5debc5fcd ("userfaultfd: hugetlbfs: add hugetlb_mcopy_atomic_pte for userfaultfd support")
953c66c2b22a ("mm: THP page cache support for ppc64")
ace71a19cec5 ("mm: introduce page_vma_mapped_walk()")
fd60775aea80 ("mm, thp: avoid unlikely branches for split_huge_pmd")
v4.4.168: Failed to apply! Possible dependencies:
01c8f1c44b83 ("mm, dax, gpu: convert vm_insert_mixed to pfn_t")
0e749e54244e ("dax: increase granularity of dax_clear_blocks() operations")
34c0fd540e79 ("mm, dax, pmem: introduce pfn_t")
369cd2121be4 ("userfaultfd: hugetlbfs: userfaultfd_huge_must_wait for hugepmd ranges")
52db400fcd50 ("pmem, dax: clean up clear_pmem()")
66b3923a1a0f ("arm64: hugetlb: add support for PTE contiguous bit")
7868a2087ec1 ("mm/hugetlb: add size parameter to huge_pte_offset()")
82b0f8c39a38 ("mm: join struct fault_env and vm_fault")
9973c98ecfda ("dax: add support for fsync/sync")
ac401cc78242 ("dax: New fault locking")
b2e0d1625e19 ("dax: fix lifetime of in-kernel dax mappings with dax_map_atomic()")
bae473a423f6 ("mm: introduce fault_env")
bc2466e42573 ("dax: Use radix tree entry lock to protect cow faults")
e4b274915863 ("DAX: move RADIX_DAX_ definitions to dax.c")
f9fe48bece3a ("dax: support dirty DAX entries in radix tree")
v3.18.130: Failed to apply! Possible dependencies:
1038628d80e9 ("userfaultfd: uAPI")
15b726ef048b ("userfaultfd: optimize read() and poll() to be O(1)")
25edd8bffd0f ("userfaultfd: linux/Documentation/vm/userfaultfd.txt")
2f4b829c625e ("arm64: Add support for hardware updates of the access and dirty pte bits")
369cd2121be4 ("userfaultfd: hugetlbfs: userfaultfd_huge_must_wait for hugepmd ranges")
3f602d2724b1 ("userfaultfd: Rename uffd_api.bits into .features")
66b3923a1a0f ("arm64: hugetlb: add support for PTE contiguous bit")
6910fa16dbe1 ("arm64: enable PTE type bit in the mask for pte_modify")
736d2169338a ("parisc: Add Huge Page and HUGETLBFS support")
7868a2087ec1 ("mm/hugetlb: add size parameter to huge_pte_offset()")
82b0f8c39a38 ("mm: join struct fault_env and vm_fault")
83cde9e8ba95 ("mm: use new helper functions around the i_mmap_mutex")
86039bd3b4e6 ("userfaultfd: add new syscall to provide memory externalization")
874bfcaf79e3 ("mm/xip: share the i_mmap_rwsem")
8d2afd96c203 ("userfaultfd: solve the race between UFFDIO_COPY|ZEROPAGE and read")
93ef666a094f ("arm64: Macros to check/set/unset the contiguous bit")
a9b85f9415fd ("userfaultfd: change the read API to return a uffd_msg")
ac401cc78242 ("dax: New fault locking")
ba85c702e4b2 ("userfaultfd: wake pending userfaults")
bae473a423f6 ("mm: introduce fault_env")
bc2466e42573 ("dax: Use radix tree entry lock to protect cow faults")
d475c6346a38 ("dax,ext2: replace XIP read and write with DAX I/O")
de1414a654e6 ("fs: export inode_to_bdi and use it in favor of mapping->backing_dev_info")
dfa37dc3fc1f ("userfaultfd: allow signals to interrupt a userfault")
e4b274915863 ("DAX: move RADIX_DAX_ definitions to dax.c")
ecf35a237a85 ("arm64: PTE/PMD contiguous bit definition")
f24ffde43237 ("parisc: expose number of page table levels on Kconfig level")
How should we proceed with this patch?
--
Thanks,
Sasha
On Tue, Dec 18, 2018 at 02:35:56PM -0800, Mike Kravetz wrote: > While looking at BUGs associated with invalid huge page map counts, > it was discovered and observed that a huge pte pointer could become > 'invalid' and point to another task's page table. Consider the > following: > > A task takes a page fault on a shared hugetlbfs file and calls > huge_pte_alloc to get a ptep. Suppose the returned ptep points to a > shared pmd. > > Now, another task truncates the hugetlbfs file. As part of truncation, > it unmaps everyone who has the file mapped. If the range being > truncated is covered by a shared pmd, huge_pmd_unshare will be called. > For all but the last user of the shared pmd, huge_pmd_unshare will > clear the pud pointing to the pmd. If the task in the middle of the > page fault is not the last user, the ptep returned by huge_pte_alloc > now points to another task's page table or worse. This leads to bad > things such as incorrect page map/reference counts or invalid memory > references. > > To fix, expand the use of i_mmap_rwsem as follows: > - i_mmap_rwsem is held in read mode whenever huge_pmd_share is called. > huge_pmd_share is only called via huge_pte_alloc, so callers of > huge_pte_alloc take i_mmap_rwsem before calling. In addition, callers > of huge_pte_alloc continue to hold the semaphore until finished with > the ptep. > - i_mmap_rwsem is held in write mode whenever huge_pmd_unshare is called. > > Cc: <stable@vger.kernel.org> > Fixes: 39dde65c9940 ("shared page table for hugetlb page") > Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> Other the few questions below. The patch looks reasonable to me. > @@ -3252,11 +3253,23 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, > > for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) { > spinlock_t *src_ptl, *dst_ptl; > + > src_pte = huge_pte_offset(src, addr, sz); > if (!src_pte) > continue; > + > + /* > + * i_mmap_rwsem must be held to call huge_pte_alloc. > + * Continue to hold until finished with dst_pte, otherwise > + * it could go away if part of a shared pmd. > + * > + * Technically, i_mmap_rwsem is only needed in the non-cow > + * case as cow mappings are not shared. > + */ > + i_mmap_lock_read(mapping); Any reason you do lock/unlock on each iteration rather than around whole loop? > dst_pte = huge_pte_alloc(dst, addr, sz); > if (!dst_pte) { > + i_mmap_unlock_read(mapping); > ret = -ENOMEM; > break; > } ... > @@ -3772,14 +3789,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, > }; > > /* > - * hugetlb_fault_mutex must be dropped before > - * handling userfault. Reacquire after handling > - * fault to make calling code simpler. > + * hugetlb_fault_mutex and i_mmap_rwsem must be > + * dropped before handling userfault. Reacquire > + * after handling fault to make calling code simpler. > */ > hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, > idx, haddr); > mutex_unlock(&hugetlb_fault_mutex_table[hash]); > + i_mmap_unlock_read(mapping); > + Do we have order of hugetlb_fault_mutex vs. i_mmap_lock documented? I *looks* correct to me, but it's better to write it down somewhere. Mayby add to the header of mm/rmap.c? > ret = handle_userfault(&vmf, VM_UFFD_MISSING); > + > + i_mmap_lock_read(mapping); > mutex_lock(&hugetlb_fault_mutex_table[hash]); > goto out; > }
On 12/21/18 2:05 AM, Kirill A. Shutemov wrote: > On Tue, Dec 18, 2018 at 02:35:56PM -0800, Mike Kravetz wrote: >> While looking at BUGs associated with invalid huge page map counts, >> it was discovered and observed that a huge pte pointer could become >> 'invalid' and point to another task's page table. Consider the >> following: >> >> A task takes a page fault on a shared hugetlbfs file and calls >> huge_pte_alloc to get a ptep. Suppose the returned ptep points to a >> shared pmd. >> >> Now, another task truncates the hugetlbfs file. As part of truncation, >> it unmaps everyone who has the file mapped. If the range being >> truncated is covered by a shared pmd, huge_pmd_unshare will be called. >> For all but the last user of the shared pmd, huge_pmd_unshare will >> clear the pud pointing to the pmd. If the task in the middle of the >> page fault is not the last user, the ptep returned by huge_pte_alloc >> now points to another task's page table or worse. This leads to bad >> things such as incorrect page map/reference counts or invalid memory >> references. >> >> To fix, expand the use of i_mmap_rwsem as follows: >> - i_mmap_rwsem is held in read mode whenever huge_pmd_share is called. >> huge_pmd_share is only called via huge_pte_alloc, so callers of >> huge_pte_alloc take i_mmap_rwsem before calling. In addition, callers >> of huge_pte_alloc continue to hold the semaphore until finished with >> the ptep. >> - i_mmap_rwsem is held in write mode whenever huge_pmd_unshare is called. >> >> Cc: <stable@vger.kernel.org> >> Fixes: 39dde65c9940 ("shared page table for hugetlb page") >> Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> > > Other the few questions below. The patch looks reasonable to me. Thanks for taking a look. > >> @@ -3252,11 +3253,23 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, >> >> for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) { >> spinlock_t *src_ptl, *dst_ptl; >> + >> src_pte = huge_pte_offset(src, addr, sz); >> if (!src_pte) >> continue; >> + >> + /* >> + * i_mmap_rwsem must be held to call huge_pte_alloc. >> + * Continue to hold until finished with dst_pte, otherwise >> + * it could go away if part of a shared pmd. >> + * >> + * Technically, i_mmap_rwsem is only needed in the non-cow >> + * case as cow mappings are not shared. >> + */ >> + i_mmap_lock_read(mapping); > > Any reason you do lock/unlock on each iteration rather than around whole > loop? I am simply mirroring the page table locking. This is not necessary. The page table lock can change while processing the range, but the i_mmap_rwsem can not. Therefore, we can hold around the whole loop. I will modify, test and put out an updated patch later today. >> dst_pte = huge_pte_alloc(dst, addr, sz); >> if (!dst_pte) { >> + i_mmap_unlock_read(mapping); >> ret = -ENOMEM; >> break; >> } > > ... > >> @@ -3772,14 +3789,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, >> }; >> >> /* >> - * hugetlb_fault_mutex must be dropped before >> - * handling userfault. Reacquire after handling >> - * fault to make calling code simpler. >> + * hugetlb_fault_mutex and i_mmap_rwsem must be >> + * dropped before handling userfault. Reacquire >> + * after handling fault to make calling code simpler. >> */ >> hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, >> idx, haddr); >> mutex_unlock(&hugetlb_fault_mutex_table[hash]); >> + i_mmap_unlock_read(mapping); >> + > > Do we have order of hugetlb_fault_mutex vs. i_mmap_lock documented? > I *looks* correct to me, but it's better to write it down somewhere. > Mayby add to the header of mm/rmap.c? No it is not documented. I don't think there is much/any documentation for hugetlb_fault_mutex at all. I will add it to the lock documentation in mm/rmap.c as you suggest.
diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 309fb8c969af..ab4c77b8c72c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3239,6 +3239,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, int cow; struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); + struct address_space *mapping = vma->vm_file->f_mapping; unsigned long mmun_start; /* For mmu_notifiers */ unsigned long mmun_end; /* For mmu_notifiers */ int ret = 0; @@ -3252,11 +3253,23 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; + src_pte = huge_pte_offset(src, addr, sz); if (!src_pte) continue; + + /* + * i_mmap_rwsem must be held to call huge_pte_alloc. + * Continue to hold until finished with dst_pte, otherwise + * it could go away if part of a shared pmd. + * + * Technically, i_mmap_rwsem is only needed in the non-cow + * case as cow mappings are not shared. + */ + i_mmap_lock_read(mapping); dst_pte = huge_pte_alloc(dst, addr, sz); if (!dst_pte) { + i_mmap_unlock_read(mapping); ret = -ENOMEM; break; } @@ -3271,8 +3284,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * after taking the lock below. */ dst_entry = huge_ptep_get(dst_pte); - if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) + if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) { + i_mmap_unlock_read(mapping); continue; + } dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); @@ -3321,6 +3336,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } spin_unlock(src_ptl); spin_unlock(dst_ptl); + + i_mmap_unlock_read(mapping); } if (cow) @@ -3772,14 +3789,18 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, }; /* - * hugetlb_fault_mutex must be dropped before - * handling userfault. Reacquire after handling - * fault to make calling code simpler. + * hugetlb_fault_mutex and i_mmap_rwsem must be + * dropped before handling userfault. Reacquire + * after handling fault to make calling code simpler. */ hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, idx, haddr); mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); + ret = handle_userfault(&vmf, VM_UFFD_MISSING); + + i_mmap_lock_read(mapping); mutex_lock(&hugetlb_fault_mutex_table[hash]); goto out; } @@ -3927,6 +3948,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (ptep) { + /* + * Since we hold no locks, ptep could be stale. That is + * OK as we are only making decisions based on content and + * not actually modifying content here. + */ entry = huge_ptep_get(ptep); if (unlikely(is_hugetlb_entry_migration(entry))) { migration_entry_wait_huge(vma, mm, ptep); @@ -3934,20 +3960,31 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) return VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); - } else { - ptep = huge_pte_alloc(mm, haddr, huge_page_size(h)); - if (!ptep) - return VM_FAULT_OOM; } + /* + * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold + * until finished with ptep. This prevents huge_pmd_unshare from + * being called elsewhere and making the ptep no longer valid. + * + * ptep could have already be assigned via huge_pte_offset. That + * is OK, as huge_pte_alloc will return the same value unless + * something changed. + */ mapping = vma->vm_file->f_mapping; - idx = vma_hugecache_offset(h, vma, haddr); + i_mmap_lock_read(mapping); + ptep = huge_pte_alloc(mm, haddr, huge_page_size(h)); + if (!ptep) { + i_mmap_unlock_read(mapping); + return VM_FAULT_OOM; + } /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate * the same page in the page cache. */ + idx = vma_hugecache_offset(h, vma, haddr); hash = hugetlb_fault_mutex_hash(h, mm, vma, mapping, idx, haddr); mutex_lock(&hugetlb_fault_mutex_table[hash]); @@ -4035,6 +4072,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } out_mutex: mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); /* * Generally it's safe to hold refcount during waiting page lock. But * here we just wait to defer the next page fault to avoid busy loop and @@ -4639,10 +4677,12 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, * Search for a shareable pmd page for hugetlb. In any case calls pmd_alloc() * and returns the corresponding pte. While this is not necessary for the * !shared pmd case because we can allocate the pmd later as well, it makes the - * code much cleaner. pmd allocation is essential for the shared case because - * pud has to be populated inside the same i_mmap_rwsem section - otherwise - * racing tasks could either miss the sharing (see huge_pte_offset) or select a - * bad pmd for sharing. + * code much cleaner. + * + * This routine must be called with i_mmap_rwsem held in at least read mode. + * For hugetlbfs, this prevents removal of any page table entries associated + * with the address space. This is important as we are setting up sharing + * based on existing page table entries (mappings). */ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) { @@ -4659,7 +4699,6 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) if (!vma_shareable(vma, addr)) return (pte_t *)pmd_alloc(mm, pud, addr); - i_mmap_lock_write(mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { if (svma == vma) continue; @@ -4689,7 +4728,6 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) spin_unlock(ptl); out: pte = (pte_t *)pmd_alloc(mm, pud, addr); - i_mmap_unlock_write(mapping); return pte; } @@ -4700,7 +4738,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) * indicated by page_count > 1, unmap is achieved by clearing pud and * decrementing the ref count. If count == 1, the pte page is not shared. * - * called with page table lock held. + * Called with page table lock held and i_mmap_rwsem held in write mode. * * returns: 1 successfully unmapped a shared pte page * 0 the underlying pte page is not shared, or it is the last user diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 0cd3de3550f0..b992d1295578 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1028,7 +1028,19 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn, if (kill) collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED); - unmap_success = try_to_unmap(hpage, ttu); + if (!PageHuge(hpage)) { + unmap_success = try_to_unmap(hpage, ttu); + } else { + /* + * For hugetlb pages, try_to_unmap could potentially call + * huge_pmd_unshare. Because of this, take semaphore in + * write mode here and set TTU_RMAP_LOCKED to indicate we + * have taken the lock at this higer level. + */ + i_mmap_lock_write(mapping); + unmap_success = try_to_unmap(hpage, ttu|TTU_RMAP_LOCKED); + i_mmap_unlock_write(mapping); + } if (!unmap_success) pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n", pfn, page_mapcount(hpage)); diff --git a/mm/migrate.c b/mm/migrate.c index 84381b55b2bd..725edaef238a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1307,8 +1307,19 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, goto put_anon; if (page_mapped(hpage)) { + struct address_space *mapping = page_mapping(hpage); + + /* + * try_to_unmap could potentially call huge_pmd_unshare. + * Because of this, take semaphore in write mode here and + * set TTU_RMAP_LOCKED to let lower levels know we have + * taken the lock. + */ + i_mmap_lock_write(mapping); try_to_unmap(hpage, - TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); + TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS| + TTU_RMAP_LOCKED); + i_mmap_unlock_write(mapping); page_was_mapped = 1; } diff --git a/mm/rmap.c b/mm/rmap.c index 85b7f9423352..322e656d0225 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1374,6 +1374,9 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, /* * If sharing is possible, start and end will be adjusted * accordingly. + * + * If called for a huge page, caller must hold i_mmap_rwsem + * in write mode as it is possible to call huge_pmd_unshare. */ adjust_range_if_pmd_sharing_possible(vma, &start, &end); } diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 458acda96f20..48368589f519 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -267,10 +267,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, VM_BUG_ON(dst_addr & ~huge_page_mask(h)); /* - * Serialize via hugetlb_fault_mutex + * Serialize via i_mmap_rwsem and hugetlb_fault_mutex. + * i_mmap_rwsem ensures the dst_pte remains valid even + * in the case of shared pmds. fault mutex prevents + * races with other faulting threads. */ - idx = linear_page_index(dst_vma, dst_addr); mapping = dst_vma->vm_file->f_mapping; + i_mmap_lock_read(mapping); + idx = linear_page_index(dst_vma, dst_addr); hash = hugetlb_fault_mutex_hash(h, dst_mm, dst_vma, mapping, idx, dst_addr); mutex_lock(&hugetlb_fault_mutex_table[hash]); @@ -279,6 +283,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pte = huge_pte_alloc(dst_mm, dst_addr, huge_page_size(h)); if (!dst_pte) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); goto out_unlock; } @@ -286,6 +291,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pteval = huge_ptep_get(dst_pte); if (!huge_pte_none(dst_pteval)) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); goto out_unlock; } @@ -293,6 +299,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_addr, src_addr, &page); mutex_unlock(&hugetlb_fault_mutex_table[hash]); + i_mmap_unlock_read(mapping); vm_alloc_shared = vm_shared; cond_resched();
While looking at BUGs associated with invalid huge page map counts, it was discovered and observed that a huge pte pointer could become 'invalid' and point to another task's page table. Consider the following: A task takes a page fault on a shared hugetlbfs file and calls huge_pte_alloc to get a ptep. Suppose the returned ptep points to a shared pmd. Now, another task truncates the hugetlbfs file. As part of truncation, it unmaps everyone who has the file mapped. If the range being truncated is covered by a shared pmd, huge_pmd_unshare will be called. For all but the last user of the shared pmd, huge_pmd_unshare will clear the pud pointing to the pmd. If the task in the middle of the page fault is not the last user, the ptep returned by huge_pte_alloc now points to another task's page table or worse. This leads to bad things such as incorrect page map/reference counts or invalid memory references. To fix, expand the use of i_mmap_rwsem as follows: - i_mmap_rwsem is held in read mode whenever huge_pmd_share is called. huge_pmd_share is only called via huge_pte_alloc, so callers of huge_pte_alloc take i_mmap_rwsem before calling. In addition, callers of huge_pte_alloc continue to hold the semaphore until finished with the ptep. - i_mmap_rwsem is held in write mode whenever huge_pmd_unshare is called. Cc: <stable@vger.kernel.org> Fixes: 39dde65c9940 ("shared page table for hugetlb page") Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> --- mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++----------- mm/memory-failure.c | 14 ++++++++- mm/migrate.c | 13 ++++++++- mm/rmap.c | 3 ++ mm/userfaultfd.c | 11 +++++-- 5 files changed, 91 insertions(+), 20 deletions(-)