[v2,0/6] mm/arm64: re-enable HVO

Message

Yu Zhao Nov. 7, 2024, 8:20 p.m. UTC

HVO was disabled by commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable
HUGETLB_PAGE_OPTIMIZE_VMEMMAP") due to the following reason:

  This is deemed UNPREDICTABLE by the Arm architecture without a
  break-before-make sequence (make the PTE invalid, TLBI, write the
  new valid PTE). However, such sequence is not possible since the
  vmemmap may be concurrently accessed by the kernel.

This series presents one of the previously discussed approaches to
re-enable HugeTLB Vmemmap Optimization (HVO) on arm64. Other
approaches that have been discussed include:
  A. Handle kernel PF while doing BBM [1],
  B. Use stop_machine() while doing BBM [2], and,
  C. Enable FEAT_BBM level 2 and keep the memory contents at the old
     and new output addresses unchanged to avoid BBM (D8.16.1-2) [3].

A quick comparison between this approach (D) and the above approaches:
  --+------------------------------+-----------------------------+
    |              Pros            |             Cons            |
  --+------------------------------+-----------------------------+
  A | Low latency, h/w independent | Predictability concerns [4] |
  B | Predictable, h/w independent | High latency                |
  C | Predictable, low latency     | H/w dependent, complex      |
  D | Predictable, h/w independent | Medium latency              |
  --+------------------------------+-----------------------------+

This approach is being tested for Google's production systems, which
generally find the "cons" above acceptable, making it the preferred
trade-off for our use cases:
  +------------------------------+------------+----------+--------+
  | HugeTLB operations           | Before [0] + After    | Change |
  +------------------------------+------------+----------+--------+
  | Alloc 600 1GB                | 0m3.526s   | 0m3.649s | +4%    |
  | Free 600 1GB                 | 0m0.880s   | 0m0.917s | +4%    |
  | Demote 600 1GB to 307200 2MB | 0m1.575s   | 0m3.640s | +231%  |
  | Free 307200 2MB              | 0m0.946s   | 0m2.921s | +309%  |
  +------------------------------+------------+----------+--------+

[0] For comparison purposes, this only includes the last patch in the
    series, i.e., CONFIG_ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP=y.
[1] https://lore.kernel.org/[e-mail]/
[2] https://lore.kernel.org/[e-mail]/
[3] https://lore.kernel.org/[e-mail]/
[4] https://lore.kernel.org/20240326125409.GA9552@willie-the-truck/

Major changes from v1, based on Marc Zyngier's help:
  1. Switched from CPU masks to a counter when pausing remote CPUs.
  2. Removed unnecessary memory barriers.

Yu Zhao (6):
  mm/hugetlb_vmemmap: batch-update PTEs
  mm/hugetlb_vmemmap: add arch-independent helpers
  irqchip/gic-v3: support SGI broadcast
  arm64: broadcast IPIs to pause remote CPUs
  arm64: pause remote CPUs to update vmemmap
  arm64: select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP

 arch/arm64/Kconfig               |   1 +
 arch/arm64/include/asm/pgalloc.h |  69 ++++++++
 arch/arm64/include/asm/smp.h     |   3 +
 arch/arm64/kernel/smp.c          |  85 +++++++++-
 drivers/irqchip/irq-gic-v3.c     |  31 +++-
 include/linux/mm_types.h         |   7 +
 mm/hugetlb_vmemmap.c             | 262 +++++++++++++++++++++----------
 7 files changed, 362 insertions(+), 96 deletions(-)


base-commit: 80fb25341631b75f57b84f99cc35b95ca2aad329