| Message ID | 20240416033457.32154-2-ioworker0@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | mm/madvise: enhance lazyfreeing with mTHP in madvise_free | expand |
On 16/04/2024 04:34, Lance Yang wrote: > This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). > By doing so, we can use the same function for both use cases > (madvise_pageout and madvise_free), and it also provides the flexibility > to only clear the dirty flag in the future if needed. > > Suggested-by: Ryan Roberts <ryan.roberts@arm.com> > Signed-off-by: Lance Yang <ioworker0@gmail.com> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com> > --- > include/linux/mm_types.h | 9 +++++ > include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- > mm/madvise.c | 3 +- > 3 files changed, 55 insertions(+), 31 deletions(-) > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index c432add95913..28822cd65d2a 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -1367,6 +1367,15 @@ enum fault_flag { > > typedef unsigned int __bitwise zap_flags_t; > > +/* Flags for clear_young_dirty_ptes(). */ > +typedef int __bitwise cydp_t; > + > +/* Clear the access bit */ > +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) > + > +/* Clear the dirty bit */ > +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) > + > /* > * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each > * other. Here is what they mean, and how to use them: > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index e2f45e22a6d1..18019f037bae 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, > } > #endif > > -#ifndef mkold_ptes > -/** > - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. > - * @vma: VMA the pages are mapped into. > - * @addr: Address the first page is mapped at. > - * @ptep: Page table pointer for the first entry. > - * @nr: Number of entries to mark old. > - * > - * May be overridden by the architecture; otherwise, implemented as a simple > - * loop over ptep_test_and_clear_young(). > - * > - * Note that PTE bits in the PTE range besides the PFN can differ. For example, > - * some PTEs might be write-protected. > - * > - * Context: The caller holds the page table lock. The PTEs map consecutive > - * pages that belong to the same folio. The PTEs are all in the same PMD. > - */ > -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, > - pte_t *ptep, unsigned int nr) > -{ > - for (;;) { > - ptep_test_and_clear_young(vma, addr, ptep); > - if (--nr == 0) > - break; > - ptep++; > - addr += PAGE_SIZE; > - } > -} > -#endif > - > #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG > #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) > static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, > @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, > } > #endif > > +#ifndef clear_young_dirty_ptes > +/** > + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the > + * same folio as old/clean. > + * @mm: Address space the pages are mapped into. > + * @addr: Address the first page is mapped at. > + * @ptep: Page table pointer for the first entry. > + * @nr: Number of entries to mark old/clean. > + * @flags: Flags to modify the PTE batch semantics. > + * > + * May be overridden by the architecture; otherwise, implemented by > + * get_and_clear/modify/set for each pte in the range. > + * > + * Note that PTE bits in the PTE range besides the PFN can differ. For example, > + * some PTEs might be write-protected. > + * > + * Context: The caller holds the page table lock. The PTEs map consecutive > + * pages that belong to the same folio. The PTEs are all in the same PMD. > + */ > +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, > + unsigned long addr, pte_t *ptep, > + unsigned int nr, cydp_t flags) > +{ > + pte_t pte; > + > + for (;;) { > + if (flags == CYDP_CLEAR_YOUNG) > + ptep_test_and_clear_young(vma, addr, ptep); > + else { > + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); > + if (flags & CYDP_CLEAR_YOUNG) > + pte = pte_mkold(pte); > + if (flags & CYDP_CLEAR_DIRTY) > + pte = pte_mkclean(pte); > + set_pte_at(vma->vm_mm, addr, ptep, pte); > + } > + if (--nr == 0) > + break; > + ptep++; > + addr += PAGE_SIZE; > + } > +} > +#endif > + > static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, > pte_t *ptep) > { > diff --git a/mm/madvise.c b/mm/madvise.c > index f59169888b8e..edb592adb749 100644 > --- a/mm/madvise.c > +++ b/mm/madvise.c > @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, > continue; > > if (!pageout && pte_young(ptent)) { > - mkold_ptes(vma, addr, pte, nr); > + clear_young_dirty_ptes(vma, addr, pte, nr, > + CYDP_CLEAR_YOUNG); > tlb_remove_tlb_entries(tlb, pte, nr, addr); > } >
On Tue, Apr 16, 2024 at 11:03 PM David Hildenbrand <david@redhat.com> wrote: > > On 16.04.24 05:34, Lance Yang wrote: > > This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). > > By doing so, we can use the same function for both use cases > > (madvise_pageout and madvise_free), and it also provides the flexibility > > to only clear the dirty flag in the future if needed. > > > > Suggested-by: Ryan Roberts <ryan.roberts@arm.com> > > Signed-off-by: Lance Yang <ioworker0@gmail.com> > > --- > > include/linux/mm_types.h | 9 +++++ > > include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- > > mm/madvise.c | 3 +- > > 3 files changed, 55 insertions(+), 31 deletions(-) > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index c432add95913..28822cd65d2a 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -1367,6 +1367,15 @@ enum fault_flag { > > > > typedef unsigned int __bitwise zap_flags_t; > > > > +/* Flags for clear_young_dirty_ptes(). */ > > +typedef int __bitwise cydp_t; > > + > > +/* Clear the access bit */ > > +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) > > + > > +/* Clear the dirty bit */ > > +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) > > + > > /* > > * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each > > * other. Here is what they mean, and how to use them: > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > > index e2f45e22a6d1..18019f037bae 100644 > > --- a/include/linux/pgtable.h > > +++ b/include/linux/pgtable.h > > @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, > > } > > #endif > > > > -#ifndef mkold_ptes > > -/** > > - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. > > - * @vma: VMA the pages are mapped into. > > - * @addr: Address the first page is mapped at. > > - * @ptep: Page table pointer for the first entry. > > - * @nr: Number of entries to mark old. > > - * > > - * May be overridden by the architecture; otherwise, implemented as a simple > > - * loop over ptep_test_and_clear_young(). > > - * > > - * Note that PTE bits in the PTE range besides the PFN can differ. For example, > > - * some PTEs might be write-protected. > > - * > > - * Context: The caller holds the page table lock. The PTEs map consecutive > > - * pages that belong to the same folio. The PTEs are all in the same PMD. > > - */ > > -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, > > - pte_t *ptep, unsigned int nr) > > -{ > > - for (;;) { > > - ptep_test_and_clear_young(vma, addr, ptep); > > - if (--nr == 0) > > - break; > > - ptep++; > > - addr += PAGE_SIZE; > > - } > > -} > > -#endif > > - > > #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG > > #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) > > static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, > > @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, > > } > > #endif > > > > +#ifndef clear_young_dirty_ptes > > +/** > > + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the > > + * same folio as old/clean. > > + * @mm: Address space the pages are mapped into. > > + * @addr: Address the first page is mapped at. > > + * @ptep: Page table pointer for the first entry. > > + * @nr: Number of entries to mark old/clean. > > + * @flags: Flags to modify the PTE batch semantics. > > + * > > + * May be overridden by the architecture; otherwise, implemented by > > + * get_and_clear/modify/set for each pte in the range. > > + * > > + * Note that PTE bits in the PTE range besides the PFN can differ. For example, > > + * some PTEs might be write-protected. > > + * > > + * Context: The caller holds the page table lock. The PTEs map consecutive > > + * pages that belong to the same folio. The PTEs are all in the same PMD. > > + */ > > +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, > > + unsigned long addr, pte_t *ptep, > > + unsigned int nr, cydp_t flags) > > +{ > > + pte_t pte; > > + > > + for (;;) { > > + if (flags == CYDP_CLEAR_YOUNG) > > + ptep_test_and_clear_young(vma, addr, ptep); > > + else { > > + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); > > + if (flags & CYDP_CLEAR_YOUNG) > > + pte = pte_mkold(pte); > > + if (flags & CYDP_CLEAR_DIRTY) > > + pte = pte_mkclean(pte); > > + set_pte_at(vma->vm_mm, addr, ptep, pte); > > + } > > + if (--nr == 0) > > + break; > > + ptep++; > > + addr += PAGE_SIZE; > > + } > > +} > Hey David, Thanks for taking time to review! > The complier *might* generate a bit faster code if you check for > CYDP_CLEAR_YOUNG outside of the loop, so you don't have to recheck on > each loop iteration. Nice! I think moving the CYDP_CLEAR_YOUNG check outside of the loop could speed up the code. I'll make this change in the next version. > > For now, nothing to lose sleep about > > Acked-by: David Hildenbrand <david@redhat.com> Thanks again for the review! Best, Lance > > -- > Cheers, > > David / dhildenb >
On Wed, Apr 17, 2024 at 12:25 AM Ryan Roberts <ryan.roberts@arm.com> wrote: > > On 16/04/2024 04:34, Lance Yang wrote: > > This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). > > By doing so, we can use the same function for both use cases > > (madvise_pageout and madvise_free), and it also provides the flexibility > > to only clear the dirty flag in the future if needed. > > > > Suggested-by: Ryan Roberts <ryan.roberts@arm.com> > > Signed-off-by: Lance Yang <ioworker0@gmail.com> > > Reviewed-by: Ryan Roberts <ryan.roberts@arm.com> Hey Ryan, Thanks for taking time to review! Best, Lance > > > --- > > include/linux/mm_types.h | 9 +++++ > > include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- > > mm/madvise.c | 3 +- > > 3 files changed, 55 insertions(+), 31 deletions(-) > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index c432add95913..28822cd65d2a 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -1367,6 +1367,15 @@ enum fault_flag { > > > > typedef unsigned int __bitwise zap_flags_t; > > > > +/* Flags for clear_young_dirty_ptes(). */ > > +typedef int __bitwise cydp_t; > > + > > +/* Clear the access bit */ > > +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) > > + > > +/* Clear the dirty bit */ > > +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) > > + > > /* > > * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each > > * other. Here is what they mean, and how to use them: > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > > index e2f45e22a6d1..18019f037bae 100644 > > --- a/include/linux/pgtable.h > > +++ b/include/linux/pgtable.h > > @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, > > } > > #endif > > > > -#ifndef mkold_ptes > > -/** > > - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. > > - * @vma: VMA the pages are mapped into. > > - * @addr: Address the first page is mapped at. > > - * @ptep: Page table pointer for the first entry. > > - * @nr: Number of entries to mark old. > > - * > > - * May be overridden by the architecture; otherwise, implemented as a simple > > - * loop over ptep_test_and_clear_young(). > > - * > > - * Note that PTE bits in the PTE range besides the PFN can differ. For example, > > - * some PTEs might be write-protected. > > - * > > - * Context: The caller holds the page table lock. The PTEs map consecutive > > - * pages that belong to the same folio. The PTEs are all in the same PMD. > > - */ > > -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, > > - pte_t *ptep, unsigned int nr) > > -{ > > - for (;;) { > > - ptep_test_and_clear_young(vma, addr, ptep); > > - if (--nr == 0) > > - break; > > - ptep++; > > - addr += PAGE_SIZE; > > - } > > -} > > -#endif > > - > > #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG > > #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) > > static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, > > @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, > > } > > #endif > > > > +#ifndef clear_young_dirty_ptes > > +/** > > + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the > > + * same folio as old/clean. > > + * @mm: Address space the pages are mapped into. > > + * @addr: Address the first page is mapped at. > > + * @ptep: Page table pointer for the first entry. > > + * @nr: Number of entries to mark old/clean. > > + * @flags: Flags to modify the PTE batch semantics. > > + * > > + * May be overridden by the architecture; otherwise, implemented by > > + * get_and_clear/modify/set for each pte in the range. > > + * > > + * Note that PTE bits in the PTE range besides the PFN can differ. For example, > > + * some PTEs might be write-protected. > > + * > > + * Context: The caller holds the page table lock. The PTEs map consecutive > > + * pages that belong to the same folio. The PTEs are all in the same PMD. > > + */ > > +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, > > + unsigned long addr, pte_t *ptep, > > + unsigned int nr, cydp_t flags) > > +{ > > + pte_t pte; > > + > > + for (;;) { > > + if (flags == CYDP_CLEAR_YOUNG) > > + ptep_test_and_clear_young(vma, addr, ptep); > > + else { > > + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); > > + if (flags & CYDP_CLEAR_YOUNG) > > + pte = pte_mkold(pte); > > + if (flags & CYDP_CLEAR_DIRTY) > > + pte = pte_mkclean(pte); > > + set_pte_at(vma->vm_mm, addr, ptep, pte); > > + } > > + if (--nr == 0) > > + break; > > + ptep++; > > + addr += PAGE_SIZE; > > + } > > +} > > +#endif > > + > > static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, > > pte_t *ptep) > > { > > diff --git a/mm/madvise.c b/mm/madvise.c > > index f59169888b8e..edb592adb749 100644 > > --- a/mm/madvise.c > > +++ b/mm/madvise.c > > @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, > > continue; > > > > if (!pageout && pte_young(ptent)) { > > - mkold_ptes(vma, addr, pte, nr); > > + clear_young_dirty_ptes(vma, addr, pte, nr, > > + CYDP_CLEAR_YOUNG); > > tlb_remove_tlb_entries(tlb, pte, nr, addr); > > } > > >
Hey David, Ryan,
How about this change?
static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
unsigned int nr, cydp_t flags)
{
if (flags == CYDP_CLEAR_YOUNG) {
for (;;) {
ptep_test_and_clear_young(vma, addr, ptep);
if (--nr == 0)
break;
ptep++;
addr += PAGE_SIZE;
}
return;
}
pte_t pte;
for (;;) {
pte = ptep_get_and_clear(vma->vm_mm, addr, ptep);
if (flags & CYDP_CLEAR_YOUNG)
pte = pte_mkold(pte);
if (flags & CYDP_CLEAR_DIRTY)
pte = pte_mkclean(pte);
if (--nr == 0)
break;
ptep++;
addr += PAGE_SIZE;
}
}
Thanks,
Lance
On 17.04.24 07:04, Lance Yang wrote: > Hey David, Ryan, > > How about this change? > > static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, > unsigned long addr, pte_t *ptep, > unsigned int nr, cydp_t flags) > { > if (flags == CYDP_CLEAR_YOUNG) { > for (;;) { > ptep_test_and_clear_young(vma, addr, ptep); > if (--nr == 0) > break; > ptep++; > addr += PAGE_SIZE; > } > return; > } > > pte_t pte; > > for (;;) { > pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); > > if (flags & CYDP_CLEAR_YOUNG) > pte = pte_mkold(pte); > if (flags & CYDP_CLEAR_DIRTY) > pte = pte_mkclean(pte); > > if (--nr == 0) > break; > ptep++; > addr += PAGE_SIZE; > } > } Likely it might be best to just KIS for now and leave it as is. The compiler should optimize out based on flags already, that's what I ignored.
On Wed, Apr 17, 2024 at 4:19 PM David Hildenbrand <david@redhat.com> wrote: > > On 17.04.24 07:04, Lance Yang wrote: > > Hey David, Ryan, > > > > How about this change? > > > > static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, > > unsigned long addr, pte_t *ptep, > > unsigned int nr, cydp_t flags) > > { > > if (flags == CYDP_CLEAR_YOUNG) { > > for (;;) { > > ptep_test_and_clear_young(vma, addr, ptep); > > if (--nr == 0) > > break; > > ptep++; > > addr += PAGE_SIZE; > > } > > return; > > } > > > > pte_t pte; > > > > for (;;) { > > pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); > > > > if (flags & CYDP_CLEAR_YOUNG) > > pte = pte_mkold(pte); > > if (flags & CYDP_CLEAR_DIRTY) > > pte = pte_mkclean(pte); > > > > if (--nr == 0) > > break; > > ptep++; > > addr += PAGE_SIZE; > > } > > } > > Likely it might be best to just KIS for now and leave it as is. The > compiler should optimize out based on flags already, that's what I ignored. Got it. Let's keep it as is for now :) Thanks, Lance > > -- > Cheers, > > David / dhildenb >
On 17/04/2024 10:01, Lance Yang wrote: > On Wed, Apr 17, 2024 at 4:19 PM David Hildenbrand <david@redhat.com> wrote: >> >> On 17.04.24 07:04, Lance Yang wrote: >>> Hey David, Ryan, >>> >>> How about this change? >>> >>> static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, >>> unsigned long addr, pte_t *ptep, >>> unsigned int nr, cydp_t flags) >>> { >>> if (flags == CYDP_CLEAR_YOUNG) { >>> for (;;) { >>> ptep_test_and_clear_young(vma, addr, ptep); >>> if (--nr == 0) >>> break; >>> ptep++; >>> addr += PAGE_SIZE; >>> } >>> return; >>> } >>> >>> pte_t pte; >>> >>> for (;;) { >>> pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); >>> >>> if (flags & CYDP_CLEAR_YOUNG) >>> pte = pte_mkold(pte); >>> if (flags & CYDP_CLEAR_DIRTY) >>> pte = pte_mkclean(pte); >>> >>> if (--nr == 0) >>> break; >>> ptep++; >>> addr += PAGE_SIZE; >>> } >>> } >> >> Likely it might be best to just KIS for now and leave it as is. The >> compiler should optimize out based on flags already, that's what I ignored. > > Got it. Let's keep it as is for now :) Yep agreed; you're passing the flags as constants so the compiler should be completely removing one half of the conditional. And if the input isn't a constant, I'd still expect the compiler to hoist the conditional out of the loop. > > Thanks, > Lance > >> >> -- >> Cheers, >> >> David / dhildenb >>
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c432add95913..28822cd65d2a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1367,6 +1367,15 @@ enum fault_flag { typedef unsigned int __bitwise zap_flags_t; +/* Flags for clear_young_dirty_ptes(). */ +typedef int __bitwise cydp_t; + +/* Clear the access bit */ +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) + +/* Clear the dirty bit */ +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) + /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each * other. Here is what they mean, and how to use them: diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2f45e22a6d1..18019f037bae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, } #endif -#ifndef mkold_ptes -/** - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as old. - * @vma: VMA the pages are mapped into. - * @addr: Address the first page is mapped at. - * @ptep: Page table pointer for the first entry. - * @nr: Number of entries to mark old. - * - * May be overridden by the architecture; otherwise, implemented as a simple - * loop over ptep_test_and_clear_young(). - * - * Note that PTE bits in the PTE range besides the PFN can differ. For example, - * some PTEs might be write-protected. - * - * Context: The caller holds the page table lock. The PTEs map consecutive - * pages that belong to the same folio. The PTEs are all in the same PMD. - */ -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - for (;;) { - ptep_test_and_clear_young(vma, addr, ptep); - if (--nr == 0) - break; - ptep++; - addr += PAGE_SIZE; - } -} -#endif - #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, } #endif +#ifndef clear_young_dirty_ptes +/** + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the + * same folio as old/clean. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to mark old/clean. + * @flags: Flags to modify the PTE batch semantics. + * + * May be overridden by the architecture; otherwise, implemented by + * get_and_clear/modify/set for each pte in the range. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + if (flags == CYDP_CLEAR_YOUNG) + ptep_test_and_clear_young(vma, addr, ptep); + else { + pte = ptep_get_and_clear(vma->vm_mm, addr, ptep); + if (flags & CYDP_CLEAR_YOUNG) + pte = pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte = pte_mkclean(pte); + set_pte_at(vma->vm_mm, addr, ptep, pte); + } + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} +#endif + static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/mm/madvise.c b/mm/madvise.c index f59169888b8e..edb592adb749 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; if (!pageout && pte_young(ptent)) { - mkold_ptes(vma, addr, pte, nr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG); tlb_remove_tlb_entries(tlb, pte, nr, addr); }
This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). By doing so, we can use the same function for both use cases (madvise_pageout and madvise_free), and it also provides the flexibility to only clear the dirty flag in the future if needed. Suggested-by: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Lance Yang <ioworker0@gmail.com> --- include/linux/mm_types.h | 9 +++++ include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- mm/madvise.c | 3 +- 3 files changed, 55 insertions(+), 31 deletions(-)