| Message ID | 20181218223557.5202-3-mike.kravetz@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | hugetlbfs: use i_mmap_rwsem for better synchronization | expand |
Hi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag,
fixing commit: ebed4bfc8da8 [PATCH] hugetlb: fix absurd HugePages_Rsvd.
The bot has tested the following trees: v4.19.10, v4.14.89, v4.9.146, v4.4.168, v3.18.130,
v4.19.10: : Build OK!
v4.14.89: Failed to apply! Possible dependencies:
285b8dcaacfc ("mm, hugetlbfs: pass fault address to no page handler")
v4.9.146: Failed to apply! Possible dependencies:
1a1aad8a9b7b ("userfaultfd: hugetlbfs: add userfaultfd hugetlb hook")
29f3ad7d8380 ("fs: Provide function to unmap metadata for a range of blocks")
334fd34d76f2 ("vfs: Add page_cache_seek_hole_data helper")
369cd2121be4 ("userfaultfd: hugetlbfs: userfaultfd_huge_must_wait for hugepmd ranges")
7868a2087ec1 ("mm/hugetlb: add size parameter to huge_pte_offset()")
7fc9e4722435 ("fs: Introduce filemap_range_has_page()")
82b0f8c39a38 ("mm: join struct fault_env and vm_fault")
8bea80520750 ("mm/hugetlb.c: use huge_pte_lock instead of opencoding the lock")
953c66c2b22a ("mm: THP page cache support for ppc64")
d72dc8a25afc ("mm: make pagevec_lookup() update index")
v4.4.168: Failed to apply! Possible dependencies:
0070e28d97e7 ("radix_tree: loop based on shift count, not height")
00f47b581105 ("radix-tree: rewrite radix_tree_tag_clear")
0e749e54244e ("dax: increase granularity of dax_clear_blocks() operations")
1366c37ed84b ("radix tree test harness")
29f3ad7d8380 ("fs: Provide function to unmap metadata for a range of blocks")
334fd34d76f2 ("vfs: Add page_cache_seek_hole_data helper")
339e6353046d ("radix_tree: tag all internal tree nodes as indirect pointers")
4aae8d1c051e ("mm/hugetlbfs: unmap pages if page fault raced with hole punch")
52db400fcd50 ("pmem, dax: clean up clear_pmem()")
72e2936c04f7 ("mm: remove unnecessary condition in remove_inode_hugepages")
7fc9e4722435 ("fs: Introduce filemap_range_has_page()")
83929372f629 ("filemap: prepare find and delete operations for huge pages")
ac401cc78242 ("dax: New fault locking")
b2e0d1625e19 ("dax: fix lifetime of in-kernel dax mappings with dax_map_atomic()")
d604c324524b ("radix-tree: introduce radix_tree_replace_clear_tags()")
d72dc8a25afc ("mm: make pagevec_lookup() update index")
e4b274915863 ("DAX: move RADIX_DAX_ definitions to dax.c")
e61452365372 ("radix_tree: add support for multi-order entries")
f9fe48bece3a ("dax: support dirty DAX entries in radix tree")
v3.18.130: Failed to apply! Possible dependencies:
1817889e3b2c ("mm/hugetlbfs: fix bugs in fallocate hole punch of areas with holes")
1c5ecae3a93f ("hugetlbfs: add minimum size accounting to subpools")
1dd308a7b49d ("mm/hugetlb: document the reserve map/region tracking routines")
5e9113731a3c ("mm/hugetlb: add cache of descriptors to resv_map for region_add")
83cde9e8ba95 ("mm: use new helper functions around the i_mmap_mutex")
b5cec28d36f5 ("hugetlbfs: truncate_hugepages() takes a range of pages")
c672c7f29f2f ("mm/hugetlb: expose hugetlb fault mutex for use by fallocate")
cf3ad20bfead ("mm/hugetlb: compute/return the number of regions added by region_add()")
feba16e25a57 ("mm/hugetlb: add region_del() to delete a specific range of entries")
How should we proceed with this patch?
--
Thanks,
Sasha
On Tue, Dec 18, 2018 at 02:35:57PM -0800, Mike Kravetz wrote: > Instead of writing the required complicated code for this rare > occurrence, just eliminate the race. i_mmap_rwsem is now held in read > mode for the duration of page fault processing. Hold i_mmap_rwsem > longer in truncation and hold punch code to cover the call to > remove_inode_hugepages. One of remove_inode_hugepages() callers is noticeably missing -- hugetlbfs_evict_inode(). Why? It at least deserves a comment on why the lock rule doesn't apply to it.
On 12/21/18 2:28 AM, Kirill A. Shutemov wrote: > On Tue, Dec 18, 2018 at 02:35:57PM -0800, Mike Kravetz wrote: >> Instead of writing the required complicated code for this rare >> occurrence, just eliminate the race. i_mmap_rwsem is now held in read >> mode for the duration of page fault processing. Hold i_mmap_rwsem >> longer in truncation and hold punch code to cover the call to >> remove_inode_hugepages. > > One of remove_inode_hugepages() callers is noticeably missing -- > hugetlbfs_evict_inode(). Why? > > It at least deserves a comment on why the lock rule doesn't apply to it. In the case of hugetlbfs_evict_inode, the vfs layer guarantees there are no more users of the inode/file. Therefore, it is safe to call without holding the mutex. But, I did add this comment to remove_inode_hugepages. * Callers of this routine must hold the i_mmap_rwsem in write mode to prevent * races with page faults. So, I violated the rule that I documented. Thanks for catching. I will update the comments to note this excpetion to the rule. Another option is to simply take the semaphore and still note why it is technically not needed. Since there are no users there will be no contention of the semaphore and the overhead should be negligible.
On Fri, Dec 21, 2018 at 10:28:25AM -0800, Mike Kravetz wrote: > On 12/21/18 2:28 AM, Kirill A. Shutemov wrote: > > On Tue, Dec 18, 2018 at 02:35:57PM -0800, Mike Kravetz wrote: > >> Instead of writing the required complicated code for this rare > >> occurrence, just eliminate the race. i_mmap_rwsem is now held in read > >> mode for the duration of page fault processing. Hold i_mmap_rwsem > >> longer in truncation and hold punch code to cover the call to > >> remove_inode_hugepages. > > > > One of remove_inode_hugepages() callers is noticeably missing -- > > hugetlbfs_evict_inode(). Why? > > > > It at least deserves a comment on why the lock rule doesn't apply to it. > > In the case of hugetlbfs_evict_inode, the vfs layer guarantees there are > no more users of the inode/file. I'm not convinced that it is true. See documentation for ->evict_inode() in Documentation/filesystems/porting: Caller does *not* evict the pagecache or inode-associated metadata buffers; the method has to use truncate_inode_pages_final() to get rid of those. Is hugetlbfs special here?
On 12/21/18 12:21 PM, Kirill A. Shutemov wrote: > On Fri, Dec 21, 2018 at 10:28:25AM -0800, Mike Kravetz wrote: >> On 12/21/18 2:28 AM, Kirill A. Shutemov wrote: >>> On Tue, Dec 18, 2018 at 02:35:57PM -0800, Mike Kravetz wrote: >>>> Instead of writing the required complicated code for this rare >>>> occurrence, just eliminate the race. i_mmap_rwsem is now held in read >>>> mode for the duration of page fault processing. Hold i_mmap_rwsem >>>> longer in truncation and hold punch code to cover the call to >>>> remove_inode_hugepages. >>> >>> One of remove_inode_hugepages() callers is noticeably missing -- >>> hugetlbfs_evict_inode(). Why? >>> >>> It at least deserves a comment on why the lock rule doesn't apply to it. >> >> In the case of hugetlbfs_evict_inode, the vfs layer guarantees there are >> no more users of the inode/file. > > I'm not convinced that it is true. See documentation for ->evict_inode() > in Documentation/filesystems/porting: > > Caller does *not* evict the pagecache or inode-associated > metadata buffers; the method has to use truncate_inode_pages_final() to get rid > of those. > We may be talking about different things. When I say there are no more users, I am talking about users via user space. We get to the hugetlbfs evict inode code via iput->iput_final->evict. In this path the count on the inode is zero, and is marked (I_FREEING) so that nobody will start using it. As a result, there can be no additional page faults against the file. This is what we are using i_mmap_rwsem to prevent. The Documentation above says that the ->evict_inode() method must evict from pagecache and get rid of metadatta buffers. hugetlbfs_evict_inode does this remove_inode_hugepages evicts pages from page cache (and frees them) as well as cleaning up the hugetlbfs specific reserve map metadata. Am I misunderstanding your question/concern? I have decided to add the locking (although unnecessary) with something like this in hugetlbfs_evict_inode. /* * The vfs layer guarantees that there are no other users of this * inode. Therefore, it would be safe to call remove_inode_hugepages * without holding i_mmap_rwsem. We acquire and hold here to be * consistent with other callers. Since there will be no contention * on the semaphore, overhead is negligible. */ i_mmap_lock_write(mapping); remove_inode_hugepages(inode, 0, LLONG_MAX); i_mmap_unlock_write(mapping);
On Fri, Dec 21, 2018 at 02:17:32PM -0800, Mike Kravetz wrote: > Am I misunderstanding your question/concern? No. Thanks for the clarification. > > I have decided to add the locking (although unnecessary) with something like > this in hugetlbfs_evict_inode. > > /* > * The vfs layer guarantees that there are no other users of this > * inode. Therefore, it would be safe to call remove_inode_hugepages > * without holding i_mmap_rwsem. We acquire and hold here to be > * consistent with other callers. Since there will be no contention > * on the semaphore, overhead is negligible. > */ > i_mmap_lock_write(mapping); > remove_inode_hugepages(inode, 0, LLONG_MAX); > i_mmap_unlock_write(mapping); LGTM.
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 32920a10100e..a9c00c6ef80d 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -383,17 +383,16 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) * truncation is indicated by end of range being LLONG_MAX * In this case, we first scan the range and release found pages. * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv - * maps and global counts. Page faults can not race with truncation - * in this routine. hugetlb_no_page() prevents page faults in the - * truncated range. It checks i_size before allocation, and again after - * with the page table lock for the page held. The same lock must be - * acquired to unmap a page. + * maps and global counts. * hole punch is indicated if end is not LLONG_MAX * In the hole punch case we scan the range and release found pages. * Only when releasing a page is the associated region/reserv map * deleted. The region/reserv map for ranges without associated - * pages are not modified. Page faults can race with hole punch. - * This is indicated if we find a mapped page. + * pages are not modified. + * + * Callers of this routine must hold the i_mmap_rwsem in write mode to prevent + * races with page faults. + * * Note: If the passed end of range value is beyond the end of file, but * not LLONG_MAX this routine still performs a hole punch operation. */ @@ -423,32 +422,14 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, for (i = 0; i < pagevec_count(&pvec); ++i) { struct page *page = pvec.pages[i]; - u32 hash; index = page->index; - hash = hugetlb_fault_mutex_hash(h, current->mm, - &pseudo_vma, - mapping, index, 0); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - /* - * If page is mapped, it was faulted in after being - * unmapped in caller. Unmap (again) now after taking - * the fault mutex. The mutex will prevent faults - * until we finish removing the page. - * - * This race can only happen in the hole punch case. - * Getting here in a truncate operation is a bug. + * A mapped page is impossible as callers should unmap + * all references before calling. And, i_mmap_rwsem + * prevents the creation of additional mappings. */ - if (unlikely(page_mapped(page))) { - BUG_ON(truncate_op); - - i_mmap_lock_write(mapping); - hugetlb_vmdelete_list(&mapping->i_mmap, - index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); - i_mmap_unlock_write(mapping); - } + VM_BUG_ON(page_mapped(page)); lock_page(page); /* @@ -470,7 +451,6 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, } unlock_page(page); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); } huge_pagevec_release(&pvec); cond_resched(); @@ -505,8 +485,8 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); + i_mmap_unlock_write(mapping); return 0; } @@ -540,8 +520,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len) hugetlb_vmdelete_list(&mapping->i_mmap, hole_start >> PAGE_SHIFT, hole_end >> PAGE_SHIFT); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); + i_mmap_unlock_write(mapping); inode_unlock(inode); } @@ -624,7 +604,11 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, /* addr is the offset within the file (zero based) */ addr = index * hpage_size; - /* mutex taken here, fault path and hole punch */ + /* + * fault mutex taken here, protects against fault path + * and hole punch. inode_lock previously taken protects + * against truncation. + */ hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping, index, addr); mutex_lock(&hugetlb_fault_mutex_table[hash]); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ab4c77b8c72c..25a0cd2f8b39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3760,16 +3760,16 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } /* - * Use page lock to guard against racing truncation - * before we get page_table_lock. + * We can not race with truncation due to holding i_mmap_rwsem. + * Check once here for faults beyond end of file. */ + size = i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >= size) + goto out; + retry: page = find_lock_page(mapping, idx); if (!page) { - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto out; - /* * Check for page in userfault range */ @@ -3859,9 +3859,6 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } ptl = huge_pte_lock(h, mm, ptep); - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto backout; ret = 0; if (!huge_pte_none(huge_ptep_get(ptep))) @@ -3964,8 +3961,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with ptep. This serves two purposes: + * 1) It prevents huge_pmd_unshare from being called elsewhere + * and making the ptep no longer valid. + * 2) It synchronizes us with file truncation. * * ptep could have already be assigned via huge_pte_offset. That * is OK, as huge_pte_alloc will return the same value unless
hugetlbfs page faults can race with truncate and hole punch operations. Current code in the page fault path attempts to handle this by 'backing out' operations if we encounter the race. One obvious omission in the current code is removing a page newly added to the page cache. This is pretty straight forward to address, but there is a more subtle and difficult issue of backing out hugetlb reservations. To handle this correctly, the 'reservation state' before page allocation needs to be noted so that it can be properly backed out. There are four distinct possibilities for reservation state: shared/reserved, shared/no-resv, private/reserved and private/no-resv. Backing out a reservation may require memory allocation which could fail so that needs to be taken into account as well. Instead of writing the required complicated code for this rare occurrence, just eliminate the race. i_mmap_rwsem is now held in read mode for the duration of page fault processing. Hold i_mmap_rwsem longer in truncation and hold punch code to cover the call to remove_inode_hugepages. With this modification, code in remove_inode_hugepages checking for races becomes 'dead' as it can not longer happen. Remove the dead code and expand comments to explain reasoning. Similarly, checks for races with truncation in the page fault path can be simplified and removed. Cc: <stable@vger.kernel.org> Fixes: ebed4bfc8da8 ("hugetlb: fix absurd HugePages_Rsvd") Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> --- fs/hugetlbfs/inode.c | 50 +++++++++++++++----------------------------- mm/hugetlb.c | 21 +++++++++---------- 2 files changed, 27 insertions(+), 44 deletions(-)