diff mbox series

[1/3] mm: Allow anon exclusive check over hugetlb tail pages

Message ID 20240403013249.1418299-2-peterx@redhat.com (mailing list archive)
State New
Headers show
Series mm/gup: fixups for hugetlb gup rework series | expand

Commit Message

Peter Xu April 3, 2024, 1:32 a.m. UTC 
From: Peter Xu <peterx@redhat.com>

PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used
to be called mostly in hugetlb specific paths and the head page was
guaranteed.

As we move forward towards merging hugetlb paths into generic mm, we may
start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge
pages) for such check.  Allow it to properly fetch the head, in which case
the anon-exclusiveness of the head will always represents the tail page.

There's already a sign of it when we look at the fast-gup which already
contain the hugetlb processing altogether: we used to have a specific
commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in
gup_must_unshare() via GUP-fast") covering that area.  Now with this more
generic change, that can also go away.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/linux/page-flags.h |  8 +++++++-
 mm/internal.h              | 10 ----------
 2 files changed, 7 insertions(+), 11 deletions(-)

Comments

David Hildenbrand April 3, 2024, 7:37 a.m. UTC | #1 
On 03.04.24 03:32, peterx@redhat.com wrote:
> From: Peter Xu <peterx@redhat.com>
> 
> PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used
> to be called mostly in hugetlb specific paths and the head page was
> guaranteed.
> 
> As we move forward towards merging hugetlb paths into generic mm, we may
> start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge
> pages) for such check.  Allow it to properly fetch the head, in which case
> the anon-exclusiveness of the head will always represents the tail page.
> 
> There's already a sign of it when we look at the fast-gup which already

"GUP-fast" ;)

> contain the hugetlb processing altogether: we used to have a specific
> commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in
> gup_must_unshare() via GUP-fast") covering that area.  Now with this more
> generic change, that can also go away.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   include/linux/page-flags.h |  8 +++++++-
>   mm/internal.h              | 10 ----------
>   2 files changed, 7 insertions(+), 11 deletions(-)
> 
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 888353c209c0..225357f48a79 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
>   static __always_inline int PageAnonExclusive(const struct page *page)
>   {
>   	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> -	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> +	/*
> +	 * Allow the anon-exclusive check to work on hugetlb tail pages.
> +	 * Here hugetlb pages will always guarantee the anon-exclusiveness
> +	 * of the head page represents the tail pages.
> +	 */
> +	if (PageHuge(page) && !PageHead(page))
> +		page = compound_head(page);
>   	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
>   }
>   
> diff --git a/mm/internal.h b/mm/internal.h
> index 9512de7398d5..87f6e4fd56a5 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1259,16 +1259,6 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
>   	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
>   		smp_rmb();
>   
> -	/*
> -	 * During GUP-fast we might not get called on the head page for a
> -	 * hugetlb page that is mapped using cont-PTE, because GUP-fast does
> -	 * not work with the abstracted hugetlb PTEs that always point at the
> -	 * head page. For hugetlb, PageAnonExclusive only applies on the head
> -	 * page (as it cannot be partially COW-shared), so lookup the head page.
> -	 */
> -	if (unlikely(!PageHead(page) && PageHuge(page)))
> -		page = compound_head(page);
> -
>   	/*
>   	 * Note that PageKsm() pages cannot be exclusive, and consequently,
>   	 * cannot get pinned.


Acked-by: David Hildenbrand <david@redhat.com>
Matthew Wilcox April 4, 2024, 12:05 a.m. UTC | #2 
On Tue, Apr 02, 2024 at 09:32:47PM -0400, peterx@redhat.com wrote:
> +++ b/include/linux/page-flags.h
> @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
>  static __always_inline int PageAnonExclusive(const struct page *page)
>  {
>  	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> -	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> +	/*
> +	 * Allow the anon-exclusive check to work on hugetlb tail pages.
> +	 * Here hugetlb pages will always guarantee the anon-exclusiveness
> +	 * of the head page represents the tail pages.
> +	 */
> +	if (PageHuge(page) && !PageHead(page))
> +		page = compound_head(page);

I think this should be written as:

	/*
	 * HugeTLB stores this information on the head page; THP keeps
	 * it per page
	 */
	if (PageHuge(page))
		page = compound_head(page);

>  	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
>  }
Peter Xu April 4, 2024, 1:45 p.m. UTC | #3 
On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> On Tue, Apr 02, 2024 at 09:32:47PM -0400, peterx@redhat.com wrote:
> > +++ b/include/linux/page-flags.h
> > @@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
> >  static __always_inline int PageAnonExclusive(const struct page *page)
> >  {
> >  	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
> > -	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
> > +	/*
> > +	 * Allow the anon-exclusive check to work on hugetlb tail pages.
> > +	 * Here hugetlb pages will always guarantee the anon-exclusiveness
> > +	 * of the head page represents the tail pages.
> > +	 */
> > +	if (PageHuge(page) && !PageHead(page))
> > +		page = compound_head(page);
> 
> I think this should be written as:
> 
> 	/*
> 	 * HugeTLB stores this information on the head page; THP keeps
> 	 * it per page
> 	 */

This comment does look clean and cleaner indeed.  And yes, mention THP can
be helpful too.

> 	if (PageHuge(page))
> 		page = compound_head(page);

I would think PageHead() check would help us to avoid compound_head() on
heads, which should still be the majority cases iiuc (assuming page->flags
is already around in the cache anyway).  I've no strong opinion though, as
I can hardly tell a difference in reality.

> 
> >  	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
> >  }
> 

Thanks,
Matthew Wilcox April 4, 2024, 2:06 p.m. UTC | #4 
On Thu, Apr 04, 2024 at 09:45:58AM -0400, Peter Xu wrote:
> On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> > 	if (PageHuge(page))
> > 		page = compound_head(page);
> 
> I would think PageHead() check would help us to avoid compound_head() on
> heads, which should still be the majority cases iiuc (assuming page->flags
> is already around in the cache anyway).  I've no strong opinion though, as
> I can hardly tell a difference in reality.

compound_head() includes a check for PageHead().  Adding the check just
makes things slower.
Peter Xu April 4, 2024, 2:21 p.m. UTC | #5 
On Thu, Apr 04, 2024 at 03:06:25PM +0100, Matthew Wilcox wrote:
> On Thu, Apr 04, 2024 at 09:45:58AM -0400, Peter Xu wrote:
> > On Thu, Apr 04, 2024 at 01:05:57AM +0100, Matthew Wilcox wrote:
> > > 	if (PageHuge(page))
> > > 		page = compound_head(page);
> > 
> > I would think PageHead() check would help us to avoid compound_head() on
> > heads, which should still be the majority cases iiuc (assuming page->flags
> > is already around in the cache anyway).  I've no strong opinion though, as
> > I can hardly tell a difference in reality.
> 
> compound_head() includes a check for PageHead().  Adding the check just
> makes things slower.

They check different fields (compound_head, offset 0x8 for the former).
Again I'm ok with either way to go and I don't expect measurable
difference.. so if you prefer this I'm happy with it too.

Andrew, could you help update with Matthew's fixup?  The comment is
definitely better than what I wrote in all cases.

Thanks,
Andrew Morton April 4, 2024, 8:31 p.m. UTC | #6 
On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <peterx@redhat.com> wrote:

> Andrew, could you help update with Matthew's fixup?  The comment is
> definitely better than what I wrote in all cases.


From: Andrew Morton <akpm@linux-foundation.org>
Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
Date: Thu Apr  4 01:27:47 PM PDT 2024

simplify PageAnonExclusive(), per Matthew

Link: https://lkml.kernel.org/r/Zg3u5Sh9EbbYPhaI@casper.infradead.org
Cc: David Hildenbrand <david@redhat.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Peter Xu <peterx@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 include/linux/page-flags.h |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

--- a/include/linux/page-flags.h~mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
+++ a/include/linux/page-flags.h
@@ -1096,11 +1096,10 @@ static __always_inline int PageAnonExclu
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
 	/*
-	 * Allow the anon-exclusive check to work on hugetlb tail pages.
-	 * Here hugetlb pages will always guarantee the anon-exclusiveness
-	 * of the head page represents the tail pages.
+	 * HugeTLB stores this information on the head page; THP keeps it per
+	 * page
 	 */
-	if (PageHuge(page) && !PageHead(page))
+	if (PageHuge(page))
 		page = compound_head(page);
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }
Peter Xu April 4, 2024, 8:36 p.m. UTC | #7 
On Thu, Apr 04, 2024 at 01:31:37PM -0700, Andrew Morton wrote:
> On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <peterx@redhat.com> wrote:
> 
> > Andrew, could you help update with Matthew's fixup?  The comment is
> > definitely better than what I wrote in all cases.
> 
> From: Andrew Morton <akpm@linux-foundation.org>
> Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
> Date: Thu Apr  4 01:27:47 PM PDT 2024
> 
> simplify PageAnonExclusive(), per Matthew

Thank you!
diff mbox series

Patch

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 888353c209c0..225357f48a79 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -1095,7 +1095,13 @@  PAGEFLAG(Isolated, isolated, PF_ANY);
 static __always_inline int PageAnonExclusive(const struct page *page)
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+	/*
+	 * Allow the anon-exclusive check to work on hugetlb tail pages.
+	 * Here hugetlb pages will always guarantee the anon-exclusiveness
+	 * of the head page represents the tail pages.
+	 */
+	if (PageHuge(page) && !PageHead(page))
+		page = compound_head(page);
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }
 
diff --git a/mm/internal.h b/mm/internal.h
index 9512de7398d5..87f6e4fd56a5 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1259,16 +1259,6 @@  static inline bool gup_must_unshare(struct vm_area_struct *vma,
 	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
 		smp_rmb();
 
-	/*
-	 * During GUP-fast we might not get called on the head page for a
-	 * hugetlb page that is mapped using cont-PTE, because GUP-fast does
-	 * not work with the abstracted hugetlb PTEs that always point at the
-	 * head page. For hugetlb, PageAnonExclusive only applies on the head
-	 * page (as it cannot be partially COW-shared), so lookup the head page.
-	 */
-	if (unlikely(!PageHead(page) && PageHuge(page)))
-		page = compound_head(page);
-
 	/*
 	 * Note that PageKsm() pages cannot be exclusive, and consequently,
 	 * cannot get pinned.