[v5,0/9] mm: multi-gen LRU: Walk secondary MMU page tables while aging

Message

James Houghton June 11, 2024, 12:21 a.m. UTC

This patchset makes it possible for MGLRU to consult secondary MMUs
while doing aging, not just during eviction. This allows for more
accurate reclaim decisions, which is especially important for proactive
reclaim.

This series does the following:
 1. Improve locking for the existing test/clear_young notifiers for x86
    and arm64.
 2. Add a new notifier test_clear_young_fast_only(), implemented only by
    KVM/x86.
 3. Incorporate test_clear_young_fast_only() into MGLRU aging.

To make aging work for more than just x86, the
test_clear_young_fast_only() notifier must be implemented by those other
architectures.

access_tracking_perf_test now has a mode (-p) to check performance of
MGLRU aging while the VM is faulting memory in. See the v4 cover
letter[1] for performance data collected with this test.

Previous versions of this series included logic in MGLRU and KVM to
support batching the updates to secondary page tables. This version
removes this logic, as it was complex and not necessary to enable
proactive reclaim. This optimization, as well as the additional
optimizations for arm64 and powerpc, can be done in a later series.

=== Previous Versions ===

This v5 re-adds a lot of logic that was present in v3 and earlier
versions of the series. There is an important difference I want to call
out:

 - should_look_around() can sometimes require two notifiers instead of
   one. This is necessary if I forbid myself from modifying
   mmu_notifier_clear_young().

   It may simply be better to do what v2/v3 did have and not have a
   fast-only notifier, and merge them all. This makes the API slightly
   more complex. I'm not sure which is better.

Change log:

Since v4[1]:
 - Removed Kconfig that controlled when aging was enabled. Aging will
   be done whenever the architecture supports it (thanks Yu).
 - Added a new MMU notifier, test_clear_young_fast_only(), specifically
   for MGLRU to use.
 - Add kvm_fast_{test_,}age_gfn, implemented by x86.
 - Fix locking for clear_flush_young().
 - Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes
   (thanks Sean).
 - Fix WARN_ON and other cleanup for the arm64 locking changes
   (thanks Oliver).

Since v3[2]:
 - Vastly simplified the series (thanks David). Removed mmu notifier
   batching logic entirely.
 - Cleaned up how locking is done for mmu_notifier_test/clear_young
   (thanks David).
 - Look-around is now only done when there are no secondary MMUs
   subscribed to MMU notifiers.
 - CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added.
 - Fixed the lockless implementation of kvm_{test,}age_gfn for x86
   (thanks David).
 - Added MGLRU functional and performance tests to
   access_tracking_perf_test (thanks Axel).
 - In v3, an mm would be completely ignored (for aging) if there was a
   secondary MMU but support for secondary MMU walking was missing. Now,
   missing secondary MMU walking support simply skips the notifier
   calls (except for eviction).
 - Added a sanity check for that range->lockless and range->on_lock are
   never both provided for the memslot walk.

For the changes since v2[3], see v3.

Based on 6.10-rc3.

[1]: https://lore.kernel.org/linux-mm/[e-mail]/
[2]: https://lore.kernel.org/linux-mm/[e-mail]/
[3]: https://lore.kernel.org/kvmarm/[e-mail]/

James Houghton (8):
  KVM: Add lockless memslot walk to KVM
  KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn
  KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn
  mm: Add test_clear_young_fast_only MMU notifier
  KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn
  KVM: x86: Implement kvm_fast_test_age_gfn and kvm_fast_age_gfn
  mm: multi-gen LRU: Have secondary MMUs participate in aging
  KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test

Yu Zhao (1):
  KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask

 Documentation/admin-guide/mm/multigen_lru.rst |   6 +-
 arch/arm64/kvm/Kconfig                        |   1 +
 arch/arm64/kvm/hyp/pgtable.c                  |  15 +-
 arch/arm64/kvm/mmu.c                          |  26 +-
 arch/x86/include/asm/kvm_host.h               |  14 +
 arch/x86/kvm/Kconfig                          |   2 +
 arch/x86/kvm/mmu.h                            |   6 -
 arch/x86/kvm/mmu/mmu.c                        |  60 ++-
 arch/x86/kvm/mmu/spte.h                       |   1 -
 arch/x86/kvm/mmu/tdp_iter.h                   |  27 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |  67 ++-
 include/linux/kvm_host.h                      |   8 +
 include/linux/mmu_notifier.h                  |  50 +++
 include/linux/mmzone.h                        |   6 +-
 include/trace/events/kvm.h                    |  22 +
 mm/mmu_notifier.c                             |  26 ++
 mm/rmap.c                                     |   9 +-
 mm/vmscan.c                                   | 185 +++++++--
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++--
 .../selftests/kvm/include/lru_gen_util.h      |  55 +++
 .../testing/selftests/kvm/lib/lru_gen_util.c  | 391 ++++++++++++++++++
 virt/kvm/Kconfig                              |   7 +
 virt/kvm/kvm_main.c                           |  73 +++-
 24 files changed, 1283 insertions(+), 140 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h
 create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c


base-commit: 83a7eefedc9b56fe7bfeff13b6c7356688ffa670