[v7,1/3] mm/mmap: separate writenotify and dirty tracking logic

Commit Message

Lorenzo Stoakes May 2, 2023, 4:34 p.m. UTC

vma_wants_writenotify() is specifically intended for setting PTE page table
flags, accounting for existing PTE flag state and whether that might
already be read-only while mixing this check with a check whether the
filesystem performs dirty tracking.

Separate out the notions of dirty tracking and a PTE write notify checking
in order that we can invoke the dirty tracking check from elsewhere.

Note that this change introduces a very small duplicate check of the
separated out vm_ops_needs_writenotify(). This is necessary to avoid making
vma_needs_dirty_tracking() needlessly complicated (e.g. passing a
check_writenotify flag or having it assume this check was already
performed). This is such a small check that it doesn't seem too egregious
to do this.

Signed-off-by: Lorenzo Stoakes <[e-mail]>
Reviewed-by: John Hubbard <[e-mail]>
Reviewed-by: Mika Penttilä <[e-mail]>
Reviewed-by: Jan Kara <[e-mail]>
Reviewed-by: Jason Gunthorpe <[e-mail]>
---
 include/linux/mm.h |  1 +
 mm/mmap.c          | 36 +++++++++++++++++++++++++++---------
 2 files changed, 28 insertions(+), 9 deletions(-)

Comments

David Hildenbrand May 2, 2023, 4:38 p.m. UTC | #1

On 02.05.23 18:34, Lorenzo Stoakes wrote:
> vma_wants_writenotify() is specifically intended for setting PTE page table
> flags, accounting for existing PTE flag state and whether that might
> already be read-only while mixing this check with a check whether the
> filesystem performs dirty tracking.
> 
> Separate out the notions of dirty tracking and a PTE write notify checking
> in order that we can invoke the dirty tracking check from elsewhere.
> 
> Note that this change introduces a very small duplicate check of the
> separated out vm_ops_needs_writenotify(). This is necessary to avoid making
> vma_needs_dirty_tracking() needlessly complicated (e.g. passing a
> check_writenotify flag or having it assume this check was already
> performed). This is such a small check that it doesn't seem too egregious
> to do this.
> 
> Signed-off-by: Lorenzo Stoakes <[e-mail]>
> Reviewed-by: John Hubbard <[e-mail]>
> Reviewed-by: Mika Penttilä <[e-mail]>
> Reviewed-by: Jan Kara <[e-mail]>
> Reviewed-by: Jason Gunthorpe <[e-mail]>
> ---
>   include/linux/mm.h |  1 +
>   mm/mmap.c          | 36 +++++++++++++++++++++++++++---------
>   2 files changed, 28 insertions(+), 9 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 27ce77080c79..7b1d4e7393ef 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2422,6 +2422,7 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,
>   #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
>   					    MM_CP_UFFD_WP_RESOLVE)
>   
> +bool vma_needs_dirty_tracking(struct vm_area_struct *vma);
>   int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
>   static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
>   {
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 5522130ae606..295c5f2e9bd9 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1475,6 +1475,31 @@ SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
>   }
>   #endif /* __ARCH_WANT_SYS_OLD_MMAP */
>   
> +/* Do VMA operations imply write notify is required? */
> +static bool vm_ops_needs_writenotify(const struct vm_operations_struct *vm_ops)
> +{
> +	return vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite);
> +}
> +
> +/*
> + * Does this VMA require the underlying folios to have their dirty state
> + * tracked?
> + */
> +bool vma_needs_dirty_tracking(struct vm_area_struct *vma)
> +{

Sorry for not noticing this earlier, but ...

what about MAP_PRIVATE mappings? When we write, we populate an anon 
page, which will work as expected ... because we don't have to notify 
the fs?

I think you really also want the "If it was private or non-writable, the 
write bit is already clear */" part as well and remove "false" in that case.

Or was there a good reason to disallow private mappings as well?

Lorenzo Stoakes May 2, 2023, 4:53 p.m. UTC | #2

On Tue, May 02, 2023 at 06:38:53PM +0200, David Hildenbrand wrote:
> On 02.05.23 18:34, Lorenzo Stoakes wrote:
> > vma_wants_writenotify() is specifically intended for setting PTE page table
> > flags, accounting for existing PTE flag state and whether that might
> > already be read-only while mixing this check with a check whether the
> > filesystem performs dirty tracking.
> >
> > Separate out the notions of dirty tracking and a PTE write notify checking
> > in order that we can invoke the dirty tracking check from elsewhere.
> >
> > Note that this change introduces a very small duplicate check of the
> > separated out vm_ops_needs_writenotify(). This is necessary to avoid making
> > vma_needs_dirty_tracking() needlessly complicated (e.g. passing a
> > check_writenotify flag or having it assume this check was already
> > performed). This is such a small check that it doesn't seem too egregious
> > to do this.
> >
> > Signed-off-by: Lorenzo Stoakes <[e-mail]>
> > Reviewed-by: John Hubbard <[e-mail]>
> > Reviewed-by: Mika Penttilä <[e-mail]>
> > Reviewed-by: Jan Kara <[e-mail]>
> > Reviewed-by: Jason Gunthorpe <[e-mail]>
> > ---
> >   include/linux/mm.h |  1 +
> >   mm/mmap.c          | 36 +++++++++++++++++++++++++++---------
> >   2 files changed, 28 insertions(+), 9 deletions(-)
> >
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index 27ce77080c79..7b1d4e7393ef 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -2422,6 +2422,7 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,
> >   #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
> >   					    MM_CP_UFFD_WP_RESOLVE)
> > +bool vma_needs_dirty_tracking(struct vm_area_struct *vma);
> >   int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
> >   static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
> >   {
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index 5522130ae606..295c5f2e9bd9 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -1475,6 +1475,31 @@ SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
> >   }
> >   #endif /* __ARCH_WANT_SYS_OLD_MMAP */
> > +/* Do VMA operations imply write notify is required? */
> > +static bool vm_ops_needs_writenotify(const struct vm_operations_struct *vm_ops)
> > +{
> > +	return vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite);
> > +}
> > +
> > +/*
> > + * Does this VMA require the underlying folios to have their dirty state
> > + * tracked?
> > + */
> > +bool vma_needs_dirty_tracking(struct vm_area_struct *vma)
> > +{
>
> Sorry for not noticing this earlier, but ...

pints_owed++

>
> what about MAP_PRIVATE mappings? When we write, we populate an anon page,
> which will work as expected ... because we don't have to notify the fs?
>
> I think you really also want the "If it was private or non-writable, the
> write bit is already clear */" part as well and remove "false" in that case.
>

Not sure a 'write bit is already clear' case is relevant to checking
whether a filesystem dirty tracks? That seems specific entirely to the page
table bits.

That's why I didn't include it,

A !VM_WRITE shouldn't be GUP-writable except for FOLL_FORCE, and that
surely could be problematic if VM_MAYWRITE later?

Thinking about it though a !VM_SHARE should probably can be safely assumed
to not be dirty-trackable, so we probably do need to add a check for
!VM_SHARED -> !vma_needs_dirty_tracking

> Or was there a good reason to disallow private mappings as well?
>

Until the page is CoW'd walking the page tables will get you to the page
cache page right? This was the reason I (perhaps rather too quickly) felt
MAP_PRIVATE should be excluded.

However a FOLL_WRITE would trigger CoW... and then we'd be trivially OK.

So yeah, ok perhaps I dismissed that a little too soon. I was concerned
about some sort of egregious FOLL_FORCE case where somehow we'd end up with
the page cache folio. But actually, that probably can't happen...

> --
> Thanks,
>
> David / dhildenb
>

Lorenzo Stoakes May 2, 2023, 5:09 p.m. UTC | #3

On Tue, May 02, 2023 at 05:53:46PM +0100, Lorenzo Stoakes wrote:
> On Tue, May 02, 2023 at 06:38:53PM +0200, David Hildenbrand wrote:
> > On 02.05.23 18:34, Lorenzo Stoakes wrote:
> > > vma_wants_writenotify() is specifically intended for setting PTE page table
> > > flags, accounting for existing PTE flag state and whether that might
> > > already be read-only while mixing this check with a check whether the
> > > filesystem performs dirty tracking.
> > >
> > > Separate out the notions of dirty tracking and a PTE write notify checking
> > > in order that we can invoke the dirty tracking check from elsewhere.
> > >
> > > Note that this change introduces a very small duplicate check of the
> > > separated out vm_ops_needs_writenotify(). This is necessary to avoid making
> > > vma_needs_dirty_tracking() needlessly complicated (e.g. passing a
> > > check_writenotify flag or having it assume this check was already
> > > performed). This is such a small check that it doesn't seem too egregious
> > > to do this.
> > >
> > > Signed-off-by: Lorenzo Stoakes <[e-mail]>
> > > Reviewed-by: John Hubbard <[e-mail]>
> > > Reviewed-by: Mika Penttilä <[e-mail]>
> > > Reviewed-by: Jan Kara <[e-mail]>
> > > Reviewed-by: Jason Gunthorpe <[e-mail]>
> > > ---
> > >   include/linux/mm.h |  1 +
> > >   mm/mmap.c          | 36 +++++++++++++++++++++++++++---------
> > >   2 files changed, 28 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > > index 27ce77080c79..7b1d4e7393ef 100644
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -2422,6 +2422,7 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,
> > >   #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
> > >   					    MM_CP_UFFD_WP_RESOLVE)
> > > +bool vma_needs_dirty_tracking(struct vm_area_struct *vma);
> > >   int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
> > >   static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
> > >   {
> > > diff --git a/mm/mmap.c b/mm/mmap.c
> > > index 5522130ae606..295c5f2e9bd9 100644
> > > --- a/mm/mmap.c
> > > +++ b/mm/mmap.c
> > > @@ -1475,6 +1475,31 @@ SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
> > >   }
> > >   #endif /* __ARCH_WANT_SYS_OLD_MMAP */
> > > +/* Do VMA operations imply write notify is required? */
> > > +static bool vm_ops_needs_writenotify(const struct vm_operations_struct *vm_ops)
> > > +{
> > > +	return vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite);
> > > +}
> > > +
> > > +/*
> > > + * Does this VMA require the underlying folios to have their dirty state
> > > + * tracked?
> > > + */
> > > +bool vma_needs_dirty_tracking(struct vm_area_struct *vma)
> > > +{
> >
> > Sorry for not noticing this earlier, but ...
>
> pints_owed++
>
> >
> > what about MAP_PRIVATE mappings? When we write, we populate an anon page,
> > which will work as expected ... because we don't have to notify the fs?
> >
> > I think you really also want the "If it was private or non-writable, the
> > write bit is already clear */" part as well and remove "false" in that case.
> >
>
> Not sure a 'write bit is already clear' case is relevant to checking
> whether a filesystem dirty tracks? That seems specific entirely to the page
> table bits.
>
> That's why I didn't include it,
>
> A !VM_WRITE shouldn't be GUP-writable except for FOLL_FORCE, and that
> surely could be problematic if VM_MAYWRITE later?
>
> Thinking about it though a !VM_SHARE should probably can be safely assumed
> to not be dirty-trackable, so we probably do need to add a check for
> !VM_SHARED -> !vma_needs_dirty_tracking
>

On second thoughts, we explicitly check FOLL_FORCE && !is_cow_mapping() in
check_vma_flags() so that case cannot occur.

So actually yes we should probably include this on the basis of that and
the fact that a FOLL_WRITE operation will CoW the MAP_PRIVATE mapping.

This was an (over)abundance of caution.

Will fix on respin.

> > Or was there a good reason to disallow private mappings as well?
> >
>
> Until the page is CoW'd walking the page tables will get you to the page
> cache page right? This was the reason I (perhaps rather too quickly) felt
> MAP_PRIVATE should be excluded.
>
> However a FOLL_WRITE would trigger CoW... and then we'd be trivially OK.
>
> So yeah, ok perhaps I dismissed that a little too soon. I was concerned
> about some sort of egregious FOLL_FORCE case where somehow we'd end up with
> the page cache folio. But actually, that probably can't happen...
>
> > --
> > Thanks,
> >
> > David / dhildenb
> >

David Hildenbrand May 2, 2023, 5:16 p.m. UTC | #4

On 02.05.23 19:09, Lorenzo Stoakes wrote:
> On Tue, May 02, 2023 at 05:53:46PM +0100, Lorenzo Stoakes wrote:
>> On Tue, May 02, 2023 at 06:38:53PM +0200, David Hildenbrand wrote:
>>> On 02.05.23 18:34, Lorenzo Stoakes wrote:
>>>> vma_wants_writenotify() is specifically intended for setting PTE page table
>>>> flags, accounting for existing PTE flag state and whether that might
>>>> already be read-only while mixing this check with a check whether the
>>>> filesystem performs dirty tracking.
>>>>
>>>> Separate out the notions of dirty tracking and a PTE write notify checking
>>>> in order that we can invoke the dirty tracking check from elsewhere.
>>>>
>>>> Note that this change introduces a very small duplicate check of the
>>>> separated out vm_ops_needs_writenotify(). This is necessary to avoid making
>>>> vma_needs_dirty_tracking() needlessly complicated (e.g. passing a
>>>> check_writenotify flag or having it assume this check was already
>>>> performed). This is such a small check that it doesn't seem too egregious
>>>> to do this.
>>>>
>>>> Signed-off-by: Lorenzo Stoakes <[e-mail]>
>>>> Reviewed-by: John Hubbard <[e-mail]>
>>>> Reviewed-by: Mika Penttilä <[e-mail]>
>>>> Reviewed-by: Jan Kara <[e-mail]>
>>>> Reviewed-by: Jason Gunthorpe <[e-mail]>
>>>> ---
>>>>    include/linux/mm.h |  1 +
>>>>    mm/mmap.c          | 36 +++++++++++++++++++++++++++---------
>>>>    2 files changed, 28 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/include/linux/mm.h b/include/linux/mm.h
>>>> index 27ce77080c79..7b1d4e7393ef 100644
>>>> --- a/include/linux/mm.h
>>>> +++ b/include/linux/mm.h
>>>> @@ -2422,6 +2422,7 @@ extern unsigned long move_page_tables(struct vm_area_struct *vma,
>>>>    #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
>>>>    					    MM_CP_UFFD_WP_RESOLVE)
>>>> +bool vma_needs_dirty_tracking(struct vm_area_struct *vma);
>>>>    int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
>>>>    static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
>>>>    {
>>>> diff --git a/mm/mmap.c b/mm/mmap.c
>>>> index 5522130ae606..295c5f2e9bd9 100644
>>>> --- a/mm/mmap.c
>>>> +++ b/mm/mmap.c
>>>> @@ -1475,6 +1475,31 @@ SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
>>>>    }
>>>>    #endif /* __ARCH_WANT_SYS_OLD_MMAP */
>>>> +/* Do VMA operations imply write notify is required? */
>>>> +static bool vm_ops_needs_writenotify(const struct vm_operations_struct *vm_ops)
>>>> +{
>>>> +	return vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite);
>>>> +}
>>>> +
>>>> +/*
>>>> + * Does this VMA require the underlying folios to have their dirty state
>>>> + * tracked?
>>>> + */
>>>> +bool vma_needs_dirty_tracking(struct vm_area_struct *vma)
>>>> +{
>>>
>>> Sorry for not noticing this earlier, but ...
>>
>> pints_owed++

Having tired eyes and jumping back and forth between tasks really seems 
to start getting expensive ;)

>>
>>>
>>> what about MAP_PRIVATE mappings? When we write, we populate an anon page,
>>> which will work as expected ... because we don't have to notify the fs?
>>>
>>> I think you really also want the "If it was private or non-writable, the
>>> write bit is already clear */" part as well and remove "false" in that case.
>>>
>>
>> Not sure a 'write bit is already clear' case is relevant to checking
>> whether a filesystem dirty tracks? That seems specific entirely to the page
>> table bits.
>>
>> That's why I didn't include it,
>>
>> A !VM_WRITE shouldn't be GUP-writable except for FOLL_FORCE, and that
>> surely could be problematic if VM_MAYWRITE later?
>>
>> Thinking about it though a !VM_SHARE should probably can be safely assumed
>> to not be dirty-trackable, so we probably do need to add a check for
>> !VM_SHARED -> !vma_needs_dirty_tracking
>>
> 
> On second thoughts, we explicitly check FOLL_FORCE && !is_cow_mapping() in
> check_vma_flags() so that case cannot occur.
> 
> So actually yes we should probably include this on the basis of that and
> the fact that a FOLL_WRITE operation will CoW the MAP_PRIVATE mapping.
> 

Yes, we only allow to FOLL_FORCE write to (exclusive) anonymous pages 
that are mapped read-only. If it's not that, we trigger a (fake) write 
fault.

Patch

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 27ce77080c79..7b1d4e7393ef 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2422,6 +2422,7 @@  extern unsigned long move_page_tables(struct vm_area_struct *vma,
 #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
 					    MM_CP_UFFD_WP_RESOLVE)
 
+bool vma_needs_dirty_tracking(struct vm_area_struct *vma);
 int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
 static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
 {
diff --git a/mm/mmap.c b/mm/mmap.c
index 5522130ae606..295c5f2e9bd9 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1475,6 +1475,31 @@  SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
 }
 #endif /* __ARCH_WANT_SYS_OLD_MMAP */
 
+/* Do VMA operations imply write notify is required? */
+static bool vm_ops_needs_writenotify(const struct vm_operations_struct *vm_ops)
+{
+	return vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite);
+}
+
+/*
+ * Does this VMA require the underlying folios to have their dirty state
+ * tracked?
+ */
+bool vma_needs_dirty_tracking(struct vm_area_struct *vma)
+{
+	/* Does the filesystem need to be notified? */
+	if (vm_ops_needs_writenotify(vma->vm_ops))
+		return true;
+
+	/* Specialty mapping? */
+	if (vma->vm_flags & VM_PFNMAP)
+		return false;
+
+	/* Can the mapping track the dirty pages? */
+	return vma->vm_file && vma->vm_file->f_mapping &&
+		mapping_can_writeback(vma->vm_file->f_mapping);
+}
+
 /*
  * Some shared mappings will want the pages marked read-only
  * to track write events. If so, we'll downgrade vm_page_prot
@@ -1484,14 +1509,13 @@  SYSCALL_DEFINE1(old_mmap, struct mmap_arg_struct __user *, arg)
 int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot)
 {
 	vm_flags_t vm_flags = vma->vm_flags;
-	const struct vm_operations_struct *vm_ops = vma->vm_ops;
 
 	/* If it was private or non-writable, the write bit is already clear */
 	if ((vm_flags & (VM_WRITE|VM_SHARED)) != ((VM_WRITE|VM_SHARED)))
 		return 0;
 
 	/* The backer wishes to know when pages are first written to? */
-	if (vm_ops && (vm_ops->page_mkwrite || vm_ops->pfn_mkwrite))
+	if (vm_ops_needs_writenotify(vma->vm_ops))
 		return 1;
 
 	/* The open routine did something to the protections that pgprot_modify
@@ -1511,13 +1535,7 @@  int vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot)
 	if (userfaultfd_wp(vma))
 		return 1;
 
-	/* Specialty mapping? */
-	if (vm_flags & VM_PFNMAP)
-		return 0;
-
-	/* Can the mapping track the dirty pages? */
-	return vma->vm_file && vma->vm_file->f_mapping &&
-		mapping_can_writeback(vma->vm_file->f_mapping);
+	return vma_needs_dirty_tracking(vma);
 }
 
 /*