[v10,0/4] mm/madvise: enhance lazyfreeing with mTHP in madvise_free

Message

Lance Yang April 18, 2024, 1:44 p.m. UTC

Hi All,

This patchset adds support for lazyfreeing multi-size THP (mTHP) without
needing to first split the large folio via split_folio(). However, we
still need to split a large folio that is not fully mapped within the
target range.

If a large folio is locked or shared, or if we fail to split it, we just
leave it in place and advance to the next PTE in the range. But note that
the behavior is changed; previously, any failure of this sort would cause
the entire operation to give up. As large folios become more common,
sticking to the old way could result in wasted opportunities.

Performance Testing
===================

On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of
the same size results in the following runtimes for madvise(MADV_FREE)
in seconds (shorter is better):

Folio Size |   Old    |   New    | Change
------------------------------------------
      4KiB | 0.590251 | 0.590259 |    0%
     16KiB | 2.990447 | 0.185655 |  -94%
     32KiB | 2.547831 | 0.104870 |  -95%
     64KiB | 2.457796 | 0.052812 |  -97%
    128KiB | 2.281034 | 0.032777 |  -99%
    256KiB | 2.230387 | 0.017496 |  -99%
    512KiB | 2.189106 | 0.010781 |  -99%
   1024KiB | 2.183949 | 0.007753 |  -99%
   2048KiB | 0.002799 | 0.002804 |    0%

---
This patchset applies against mm-unstable (f52bcd4a9f60). 

The performance numbers are from v2. I did a quick benchmark run of v10 and
nothing significantly changed.

Changes since v9 [9]
====================
 - mm/madvise: optimize lazyfreeing with mTHP in madvise_free
    - Pass any_dirty to madvise_folio_pte_batch() (Thanks to David Hildenbrand)
 - Pick up AB's, Thanks to David!

Changes since v8 [8]
====================
 - mm/madvise: optimize lazyfreeing with mTHP in madvise_free
    - Leave the split folio code as is in the caller (per David Hildenbrand)
    - Use cydp_flags here that will make this easier to read (per
      David Hildenbrand)
 - Pick up RB's, Thanks to Ryan!

Changes since v7 [7]
====================
 - mm/madvise: optimize lazyfreeing with mTHP in madvise_free
    - Remove the duplicated check for the mapcount (per Ryan Roberts,
      David Hildenbrand)
 - Pick up AB's and RB's. Thanks to Ryan and David!

Changes since v6 [6]
====================
 - Fix a bug with incorrect bitwise operations (Thanks to Ryan Roberts)
 - Use a cmpxchg loop to only clear one of the flags to prevent race with
   the HW (per Ryan Roberts)

Changes since v5 [5]
====================
 - Convert mkold_ptes() to clear_young_dirty_ptes() (per Ryan Roberts)
 - Use the __bitwise flags as the input for clear_young_dirty_ptes()
   (per David Hildenbrand)
 - Follow the pattern already established by the original code
   (per Ryan Roberts)

Changes since v4 [4]
====================
 - The first patch implements the MADV_FREE change and introduces
   mkold_clean_ptes() with a generic implementation. The second patch
   specializes mkold_clean_ptes() for arm64, providing a performance boost
   specific to arm64 (per Ryan Roberts)
 - Drop the full parameter and call ptep_get_and_clear() in mkold_clean_ptes()
   (per Ryan Roberts)
 - Keep the previous behavior that avoids locking the folio if it wasn't in the
   swapcache or if it wasn't dirty (per Ryan Roberts)

Changes since v3 [3]
====================
 - Rename refresh_full_ptes -> mkold_clean_ptes (per Ryan Roberts)
 - Override mkold_clean_ptes() for arm64 to make it faster (per Ryan Roberts)
 - Update the changelog

Changes since v2 [2]
====================
 - Only skip all the PTEs for nr_pages when the number of batched PTEs matches
   nr_pages (per Barry Song)
 - Change folio_pte_batch() to consume an optional *any_dirty and *any_young
   function (per David Hildenbrand)
 - Move the ptep_get_and_clear_full() loop into refresh_full_ptes() (per
   David Hildenbrand)
 - Follow a similar pattern for madvise_free_pte_range() (per Ryan Roberts)

Changes since v1 [1]
====================
 - Update the performance numbers
 - Update the changelog (per Ryan Roberts)
 - Check the COW folio (per Yin Fengwei)
 - Check if we are mapping all subpages (per Barry Song, David Hildenbrand,
   Ryan Roberts)

[1] https://lore.kernel.org/linux-mm/[e-mail]
[2] https://lore.kernel.org/linux-mm/[e-mail]
[3] https://lore.kernel.org/linux-mm/[e-mail]
[4] https://lore.kernel.org/linux-mm/[e-mail]
[5] https://lore.kernel.org/linux-mm/[e-mail]
[6] https://lore.kernel.org/linux-mm/[e-mail]
[7] https://lore.kernel.org/linux-mm/[e-mail] 
[8] https://lore.kernel.org/linux-mm/[e-mail]
[9] https://lore.kernel.org/linux-mm/[e-mail]

Thanks,
Lance

Lance Yang (4):
 mm/madvise: introduce clear_young_dirty_ptes() batch helper
 mm/arm64: override clear_young_dirty_ptes() batch helper
 mm/memory: add any_dirty optional pointer to folio_pte_batch()
 mm/madvise: optimize lazyfreeing with mTHP in madvise_free

 arch/arm64/include/asm/pgtable.h |  55 +++++++++++++++++++++++++++++++++++++++
 arch/arm64/mm/contpte.c          |  29 +++++++++++++++++++++++
 include/linux/mm_types.h         |   9 ++++++++
 include/linux/pgtable.h          |  74 +++++++++++++++++++++------------------
 mm/internal.h                    |  12 ++++++++--
 mm/madvise.c                     | 107 +++++++++++++++++++--------------------
 mm/memory.c                      |   4 ++--
 7 files changed, 209 insertions(+), 81 deletions(-)