[00/27] hugetlb/CMA improvements for large systems

Message

Frank van der Linden Jan. 27, 2025, 11:21 p.m. UTC

On large systems, we observed some issues with hugetlb and CMA:

1) When specifying a large number of hugetlb boot pages (hugepages=
   on the commandline), the kernel may run out of memory before it
   even gets to HVO. For example, if you have a 3072G system, and
   want to use 3024 1G hugetlb pages for VMs, that should leave
   you plenty of space for the hypervisor, provided you have the
   hugetlb vmemmap optimization (HVO) enabled. However, since
   the vmemmap pages are always allocated first, and then later
   in boot freed, you will actually run yourself out of memory
   before you can do HVO. This means not getting all the hugetlb
   pages you want, and worse, failure to boot if there is an
   allocation failure in the system from which it can't recover.

2) There is a system setup where you might want to use hugetlb_cma
   with a large value (say, again, 3024 out of 3072G like above),
   and then lower that if system usage allows it, to make room
   for non-hugetlb processes. For this, a variation of the problem
   above applies: the kernel runs out of unmovable space to allocate
   from before you finish boot, since your CMA area takes up all
   the space.

3) CMA wants to use one big contiguous area for allocations. Which
   fails if you have the aforementioned 3T system with a gap in the
   middle of physical memory (like the < 40bits BIOS DMA area seen on
   some AMD systems). You then won't be able to set up a CMA area for
   one of the NUMA nodes, leading to loss of half of your hugetlb
   CMA area.

4) Under the scenario mentioned in 2), when trying to grow the
   number of hugetlb pages after dropping it for a while, new
   CMA allocations may fail occasionally. This is not unexpected,
   some transient references on pages may prevent cma_alloc
   from succeeding under memory pressure. However, the hugetlb
   code then falls back to a normal contiguous alloc, which may
   end up succeeding. This is not always desired behavior. If
   you have a large CMA area, then the kernel has a restricted
   amount of memory it can do unmovable allocations from (a well
   known issue). A normal contiguous alloc may eat further in to
   this space.


To resolve these issues, do the following:

* Add hooks to the section init code to do custom initialization
  of memmap pages. Hugetlb bootmem (memblock) allocated pages can
  then be pre-HVOed. This avoids allocating a large number of
  vmemmap pages early in boot, only to have them be freed again
  later, and also avoids running out of memory as described under 1).
  Using these hooks for hugetlb is optional. It requires moving
  hugetlb bootmem allocation to an earlier spot by the architecture.
  This has been enabled on x86.
* hugetlb_cma doesn't care about the CMA area it uses being one
  large contiguous range. Multiple smaller ranges are fine. The
  only requirements are that the areas should be on one NUMA node,
  and individual gigantic pages should be allocatable from them. So,
  implement multi-range support for CMA, avoiding issue 3).
* Introduce a hugetlb_cma_only option on the commandline. This only
  allows allocations from CMA for gigantic pages, if hugetlb_cma=
  is also specified.
* With hugetlb_cma_only active, it also makes sense to be able to
  pre-allocate gigantic hugetlb pages at boot time from the CMA
  area(s). Add a rudimentary early CMA allocation interface, that
  just grabs a piece of memblock-allocated space from the CMA
  area, which gets marked as allocated in the CMA bitmap
  when the CMA area is initialized. With this, hugepages= can
  be supported with hugetlb_cma=, making scenario 2) work.

Additionally, fix some minor bugs, with one worth mentioning:
since hugetlb gigantic bootmem pages are allocated by memblock,
they may span multiple zones, as memblock doesn't (and mostly
can't) know about zones. This can cause problems. A hugetlb page
spanning multiple zones is bad, and it's worse with HVO, when
the de-HVO step effectively sneakily re-assigns pages to a
different zone than originally configured, since the tail pages
all inherit the zone from the first 60 tail pages. This condition
is not common, but can be easily reproduced using ZONE_MOVABLE.
To fix this, add checks to see if gigantic bootmem pages intersect
with multiple zones, and do not use them if they do, giving them
back to the page allocator instead.

The first patch is kind of along for the ride, except that
maintaining an available_count for a CMA area is convenient
for the multiple range support.

Frank van der Linden (27):
  mm/cma: export total and free number of pages for CMA areas
  mm, cma: support multiple contiguous ranges, if requested
  mm/cma: introduce cma_intersects function
  mm, hugetlb: use cma_declare_contiguous_multi
  mm/hugetlb: fix round-robin bootmem allocation
  mm/hugetlb: remove redundant __ClearPageReserved
  mm/hugetlb: use online nodes for bootmem allocation
  mm/hugetlb: convert cmdline parameters from setup to early
  x86/mm: make register_page_bootmem_memmap handle PTE mappings
  mm/bootmem_info: export register_page_bootmem_memmap
  mm/sparse: allow for alternate vmemmap section init at boot
  mm/hugetlb: set migratetype for bootmem folios
  mm: define __init_reserved_page_zone function
  mm/hugetlb: check bootmem pages for zone intersections
  mm/sparse: add vmemmap_*_hvo functions
  mm/hugetlb: deal with multiple calls to hugetlb_bootmem_alloc
  mm/hugetlb: move huge_boot_pages list init to hugetlb_bootmem_alloc
  mm/hugetlb: add pre-HVO framework
  mm/hugetlb_vmemmap: fix hugetlb_vmemmap_restore_folios definition
  mm/hugetlb: do pre-HVO for bootmem allocated pages
  x86/setup: call hugetlb_bootmem_alloc early
  x86/mm: set ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT
  mm/cma: simplify zone intersection check
  mm/cma: introduce a cma validate function
  mm/cma: introduce interface for early reservations
  mm/hugetlb: add hugetlb_cma_only cmdline option
  mm/hugetlb: enable bootmem allocation from CMA areas

 Documentation/ABI/testing/sysfs-kernel-mm-cma |  13 +
 .../admin-guide/kernel-parameters.txt         |   7 +
 arch/powerpc/mm/hugetlbpage.c                 |   6 +
 arch/powerpc/mm/init_64.c                     |   1 +
 arch/s390/mm/init.c                           |  13 +-
 arch/x86/Kconfig                              |   1 +
 arch/x86/kernel/setup.c                       |   4 +-
 arch/x86/mm/init_64.c                         |  16 +-
 include/linux/bootmem_info.h                  |   7 +
 include/linux/cma.h                           |   9 +
 include/linux/hugetlb.h                       |  25 +
 include/linux/mm.h                            |  13 +-
 include/linux/mmzone.h                        |  35 +
 mm/Kconfig                                    |   8 +
 mm/bootmem_info.c                             |   4 +-
 mm/cma.c                                      | 749 +++++++++++++++---
 mm/cma.h                                      |  42 +-
 mm/cma_debug.c                                |   5 +-
 mm/cma_sysfs.c                                |  20 +
 mm/hugetlb.c                                  | 403 ++++++++--
 mm/hugetlb_vmemmap.c                          | 199 ++++-
 mm/hugetlb_vmemmap.h                          |  23 +-
 mm/internal.h                                 |  19 +
 mm/mm_init.c                                  |  78 +-
 mm/sparse-vmemmap.c                           | 168 +++-
 mm/sparse.c                                   |  87 +-
 26 files changed, 1677 insertions(+), 278 deletions(-)