| Message ID | 20210116003105.182918-3-mike.kravetz@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | create hugetlb flags to consolidate state | expand |
On Fri, Jan 15, 2021 at 04:31:02PM -0800, Mike Kravetz wrote: > +++ b/fs/hugetlbfs/inode.c > @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, > > mutex_unlock(&hugetlb_fault_mutex_table[hash]); > > - set_page_huge_active(page); > + hugetlb_set_page_flag(page, HP_Migratable); I had understood the request to be more like ... SetHPageMigratable(page); > +++ b/include/linux/hugetlb.h > @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, > * HP_Restore_Reserve - Set when a hugetlb page consumes a reservation at > * allocation time. Cleared when page is fully instantiated. Free > * routine checks flag to restore a reservation on error paths. > + * HP_Migratable - Set after a newly allocated page is added to the page > + * cache and/or page tables. Indicates the page is a candidate for > + * migration. > */ > enum hugetlb_page_flags { > HP_Restore_Reserve = 0, > + HP_Migratable, > }; and name these HPG_restore_reserve and HPG_migratable and generate the calls to hugetlb_set_page_flag etc from macros, eg: #define TESTHPAGEFLAG(uname, lname) \ static __always_inline bool HPage##uname(struct page *page) \ { return test_bit(HPG_##lname, &page->private); } ... #define HPAGEFLAG(uname, lname) \ TESTHPAGEFLAG(uname, lname) \ SETHPAGEFLAG(uname, lname) \ CLEARHPAGEFLAG(uname, lname) HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) just to mirror page-flags.h more closely.
On Sat, Jan 16, 2021 at 12:26 PM Matthew Wilcox <willy@infradead.org> wrote: > > On Fri, Jan 15, 2021 at 04:31:02PM -0800, Mike Kravetz wrote: > > +++ b/fs/hugetlbfs/inode.c > > @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, > > > > mutex_unlock(&hugetlb_fault_mutex_table[hash]); > > > > - set_page_huge_active(page); > > + hugetlb_set_page_flag(page, HP_Migratable); > > I had understood the request to be more like ... > > SetHPageMigratable(page); > > > +++ b/include/linux/hugetlb.h > > @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, > > * HP_Restore_Reserve - Set when a hugetlb page consumes a reservation at > > * allocation time. Cleared when page is fully instantiated. Free > > * routine checks flag to restore a reservation on error paths. > > + * HP_Migratable - Set after a newly allocated page is added to the page > > + * cache and/or page tables. Indicates the page is a candidate for > > + * migration. > > */ > > enum hugetlb_page_flags { > > HP_Restore_Reserve = 0, > > + HP_Migratable, > > }; > > and name these HPG_restore_reserve and HPG_migratable > > and generate the calls to hugetlb_set_page_flag etc from macros, eg: > > #define TESTHPAGEFLAG(uname, lname) \ > static __always_inline bool HPage##uname(struct page *page) \ > { return test_bit(HPG_##lname, &page->private); } > ... > #define HPAGEFLAG(uname, lname) \ > TESTHPAGEFLAG(uname, lname) \ > SETHPAGEFLAG(uname, lname) \ > CLEARHPAGEFLAG(uname, lname) > > HPAGEFLAG(RestoreReserve, restore_reserve) > HPAGEFLAG(Migratable, migratable) > > just to mirror page-flags.h more closely. I prefer this suggestion. I also made the same suggestion in the previous RFC version.
On Sat, Jan 16, 2021 at 04:24:16AM +0000, Matthew Wilcox wrote: > and name these HPG_restore_reserve and HPG_migratable > > and generate the calls to hugetlb_set_page_flag etc from macros, eg: > > #define TESTHPAGEFLAG(uname, lname) \ > static __always_inline bool HPage##uname(struct page *page) \ > { return test_bit(HPG_##lname, &page->private); } > ... > #define HPAGEFLAG(uname, lname) \ > TESTHPAGEFLAG(uname, lname) \ > SETHPAGEFLAG(uname, lname) \ > CLEARHPAGEFLAG(uname, lname) > > HPAGEFLAG(RestoreReserve, restore_reserve) > HPAGEFLAG(Migratable, migratable) > > just to mirror page-flags.h more closely. That is on me. I thought that given the low number of flags, we coud get away with: hugetlb_{set,test,clear}_page_flag(page, flag) and call it from the code. But some of the flags need to be set/tested outside hugetlb code, so it indeed looks nicer and more consistent to follow page-flags.h convention. Sorry for the noise.
On 1/16/21 4:06 AM, Oscar Salvador wrote: > On Sat, Jan 16, 2021 at 04:24:16AM +0000, Matthew Wilcox wrote: >> and name these HPG_restore_reserve and HPG_migratable >> >> and generate the calls to hugetlb_set_page_flag etc from macros, eg: >> >> #define TESTHPAGEFLAG(uname, lname) \ >> static __always_inline bool HPage##uname(struct page *page) \ >> { return test_bit(HPG_##lname, &page->private); } >> ... >> #define HPAGEFLAG(uname, lname) \ >> TESTHPAGEFLAG(uname, lname) \ >> SETHPAGEFLAG(uname, lname) \ >> CLEARHPAGEFLAG(uname, lname) >> >> HPAGEFLAG(RestoreReserve, restore_reserve) >> HPAGEFLAG(Migratable, migratable) >> >> just to mirror page-flags.h more closely. > > That is on me. > I thought that given the low number of flags, we coud get away with: > > hugetlb_{set,test,clear}_page_flag(page, flag) > > and call it from the code. > But some of the flags need to be set/tested outside hugetlb code, so > it indeed looks nicer and more consistent to follow page-flags.h convention. > > Sorry for the noise. Thanks everyone! I was unsure about the best way to go for this. Will send out a new version in a few days using the page-flag style macros.
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b8a661780c4a..89bc9062b4f6 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 64f8c7a64186..353d81913cc7 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HP_Restore_Reserve - Set when a hugetlb page consumes a reservation at * allocation time. Cleared when page is fully instantiated. Free * routine checks flag to restore a reservation on error paths. + * HP_Migratable - Set after a newly allocated page is added to the page + * cache and/or page tables. Indicates the page is a candidate for + * migration. */ enum hugetlb_page_flags { HP_Restore_Reserve = 0, + HP_Migratable, }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index bc6fd1ee7dd6..04a34c08e0a6 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -592,15 +592,9 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); -bool page_huge_active(struct page *page); #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) - -static inline bool page_huge_active(struct page *page) -{ - return 0; -} #endif diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b01002d8fc2b..c43cebf2f278 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,30 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Test to determine whether the hugepage is "active/in-use" (i.e. being linked - * to hstate->hugepage_activelist.) - * - * This function can be called for tail pages, but never returns true for them. - */ -bool page_huge_active(struct page *page) -{ - return PageHeadHuge(page) && PagePrivate(&page[1]); -} - -/* never called for tail page */ -void set_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - SetPagePrivate(&page[1]); -} - -static void clear_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - ClearPagePrivate(&page[1]); -} - /* * Internal hugetlb specific page flag. Do not use outside of the hugetlb * code @@ -1438,7 +1414,7 @@ static void __free_huge_page(struct page *page) } spin_lock(&hugetlb_lock); - clear_page_huge_active(page); + hugetlb_clear_page_flag(page, HP_Migratable); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); hugetlb_cgroup_uncharge_page_rsvd(hstate_index(h), @@ -4221,7 +4197,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - set_page_huge_active(new_page); + hugetlb_set_page_flag(new_page, HP_Migratable); /* Make the old page be freed below */ new_page = old_page; } @@ -4458,12 +4434,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spin_unlock(ptl); /* - * Only make newly allocated pages active. Existing pages found - * in the pagecache could be !page_huge_active() if they have been - * isolated for migration. + * Only set HP_Migratable in newly allocated pages. Existing pages + * found in the pagecache may not have HP_Migratable set if they have + * been isolated for migration. */ if (new_page) - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); unlock_page(page); out: @@ -4774,7 +4750,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); if (vm_shared) unlock_page(page); ret = 0; @@ -5592,12 +5568,13 @@ bool isolate_huge_page(struct page *page, struct list_head *list) bool ret = true; spin_lock(&hugetlb_lock); - if (!PageHeadHuge(page) || !page_huge_active(page) || + if (!PageHeadHuge(page) || + !hugetlb_test_page_flag(page, HP_Migratable) || !get_page_unless_zero(page)) { ret = false; goto unlock; } - clear_page_huge_active(page); + hugetlb_clear_page_flag(page, HP_Migratable); list_move_tail(&page->lru, list); unlock: spin_unlock(&hugetlb_lock); @@ -5608,7 +5585,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f9d57b9be8c7..10cdd281dd29 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1260,7 +1260,13 @@ static int scan_movable_pages(unsigned long start, unsigned long end, if (!PageHuge(page)) continue; head = compound_head(page); - if (page_huge_active(head)) + /* + * This test is racy as we hold no reference or lock. The + * hugetlb page could have been free'ed and head is no longer + * a hugetlb page before the following check. In such unlikely + * cases false positives and negatives are possible. + */ + if (hugetlb_test_page_flag(head, HP_Migratable)) goto found; skip = compound_nr(head) - (page - head); pfn += skip - 1;
Use the new hugetlb page specific flag HP_Migratable to replace the page_huge_active interfaces. By it's name, page_huge_active implied that a huge page was on the active list. However, that is not really what code checking the flag wanted to know. It really wanted to determine if the huge page could be migrated. This happens when the page is actually added the page cache and/or task page table. This is the reasoning behind the name change. The VM_BUG_ON_PAGE() calls in the *_huge_active() interfaces are not really necessary as we KNOW the page is a hugetlb page. Therefore, they are removed. The routine page_huge_active checked for PageHeadHuge before testing the active bit. This is unnecessary in the case where we hold a reference or lock and know it is a hugetlb head page. page_huge_active is also called without holding a reference or lock (scan_movable_pages), and can race with code freeing the page. The extra check in page_huge_active shortened the race window, but did not prevent the race. Offline code calling scan_movable_pages already deals with these races, so removing the check is acceptable. Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 4 ++++ include/linux/page-flags.h | 6 ----- mm/hugetlb.c | 45 ++++++++++---------------------------- mm/memory_hotplug.c | 8 ++++++- 5 files changed, 23 insertions(+), 42 deletions(-)