From patchwork Mon Jan 27 23:21:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13951841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94591C02188 for ; Mon, 27 Jan 2025 23:22:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEB0E2801BE; Mon, 27 Jan 2025 18:22:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E9A832801BC; Mon, 27 Jan 2025 18:22:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D892C2801BE; Mon, 27 Jan 2025 18:22:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B861F2801BC for ; Mon, 27 Jan 2025 18:22:23 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6822412056D for ; Mon, 27 Jan 2025 23:22:23 +0000 (UTC) X-FDA: 83054807766.15.D6AE828 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf04.hostedemail.com (Postfix) with ESMTP id B25FE40011 for ; Mon, 27 Jan 2025 23:22:21 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QxUNh2BT; spf=pass (imf04.hostedemail.com: domain of 3LBWYZwQKCPkgwemhpphmf.dpnmjovy-nnlwbdl.psh@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3LBWYZwQKCPkgwemhpphmf.dpnmjovy-nnlwbdl.psh@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738020141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=bt42mxdeTSwsweXjQuaToh/AJV98hNyz7ODp4fuHPgA=; b=M9rHq0cVvDo0WKXkdYDxTEDVy2tccwdRQF9pCoOshdcRqIwmTEuWyoLRhpBYj4yso7T7H8 IPc0axod5XTUExOTXzXvBtcDHqJsUgaAZ5Woubk8+T0Y0FpFuuTnJBL0D8rqgV9USmBvI0 qWssEfChly3SyqawydsuCQXLQpRZ5OQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738020141; a=rsa-sha256; cv=none; b=XDsn5YV/OOC5mPqKHR91+wg29OakDB0Ep+dzZxXR9Q6/VdW/M0Q7l+7wtmAn2J2cJWJgV0 vYEd/0+zITRrWpJ+Ctcd+/5POgJ60WTD3KTCzhw8SsW+CyNyaaRAhBcQ8q3Tcwqodu3Feb OpwwBGbbQpc6Da0Nh14Rc6xATiLQ+ac= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QxUNh2BT; spf=pass (imf04.hostedemail.com: domain of 3LBWYZwQKCPkgwemhpphmf.dpnmjovy-nnlwbdl.psh@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3LBWYZwQKCPkgwemhpphmf.dpnmjovy-nnlwbdl.psh@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21681a2c0d5so88408505ad.2 for ; Mon, 27 Jan 2025 15:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738020140; x=1738624940; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=bt42mxdeTSwsweXjQuaToh/AJV98hNyz7ODp4fuHPgA=; b=QxUNh2BTk5BieDQbxKKFEBUKIoQS6O4kk4Hl8vcbh/6VlRhgq96WCnzoL5AmY0pRyH BQI2dAGyv3P29/JVE/chv6+pIHHShHjfBmZjy+h91EnWwqHJhQZjGkqHplZxGLqv1qLB GCWW3MrhXkSr4Rrp7Wwunre5dBAK/hEvhoivcYZhL0EfXhsr1/ydtr7BfQOaXg1Yp2Ia vmBidzFlF3doDblpTs8uPEr+x5c6w5IVNRJ7QEc2y2ZMXnzvisUh5XE34lo6ZJjWYuTg 3ZKCRcxwfM5qTBjaZPQAy8GDB0/gAgVvJD4TY1BaKCv6lEAj4OVThKW3KvVpjEGCUObp 54Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738020140; x=1738624940; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bt42mxdeTSwsweXjQuaToh/AJV98hNyz7ODp4fuHPgA=; b=L6aOWy7hOh+ROanIhnUCxEH8CNuafj43z16NLer8Gx8GeDk/LdD6aE1XlAHzIO6cqr bg41Vkfa27558zkeRB+StxXeUbk1vNvZ4s0eccISfvqumivEWM1lZroaeCACa+Q4vcD7 6Ocl8xHeEU++jrmJAjtHNLTPP9rzu7lm4pxbOJVqohuEs3F4Oves51vbNm5fga12Af5+ rjPqmcfwUmtlM/dRH2AU0IjJWVZuHvuU9QNp3ZPcfgcRFhCwTdAoeKjvH+T4omdlSKDw JsWzg8TW7+ZIX2zXjfEZsHbiW4e6j2qXb+TTVherw6vosFIt55Not3tRb5cY56PZSeqa Vlcw== X-Forwarded-Encrypted: i=1; AJvYcCU7NxScBWxJKaTrJGf4osJKNIUF/seDNcEt/AvI8xiRXOpznZSZD7HtP/LVmzdNHMGWhm91gRUXRQ==@kvack.org X-Gm-Message-State: AOJu0YxoDKJF8ejN72DVCQ8C/A5AO5yjRU6p/XuEXdPBOcaOzKI+AxAb vD8vXYNduDpSfIAdkIa4xWlCxPqvV1dpNabU7T/4F/pFToNXD//C1f327OLPIQBZmkj4hQ== X-Google-Smtp-Source: AGHT+IENp6Hf6X07ZGRP/ZOiM/Qhql0exnKb7+yA54A1OtWRZyyUWBIjXdha2MoOL+llYNae/4R/5Gdj X-Received: from pfbdc11.prod.google.com ([2002:a05:6a00:35cb:b0:728:e3af:6bb0]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:734e:b0:1e8:bd15:6819 with SMTP id adf61e73a8af0-1eb214ea482mr67763394637.22.1738020140411; Mon, 27 Jan 2025 15:22:20 -0800 (PST) Date: Mon, 27 Jan 2025 23:21:40 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250127232207.3888640-1-fvdl@google.com> Subject: [PATCH 00/27] hugetlb/CMA improvements for large systems From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usama.arif@bytedance.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Stat-Signature: bgd71q8xrd4fw9oxh3zn7p8jpqs8oikf X-Rspam-User: X-Rspamd-Queue-Id: B25FE40011 X-Rspamd-Server: rspam03 X-HE-Tag: 1738020141-299393 X-HE-Meta: U2FsdGVkX1+TuymOdHEIgs2XH7x/2O6arM34ALCDrgqyoGoX8Jzj/Nbk9uRCDqujdl8e3vfsEuCP64VmhXEVGjzo+3Ga2btOjjI8KoXNXi6Ns+mWJhrbw4gyFweLSl8+GN45oNKDTxq6CXg2tj8f9ZSTeqL/sP8Wdr06FpMHVHomjrZhqtrQoAQweiaxFxmB4IPdS4PxDUGFJJJJWkd+ZeFMf1ZfxIWOEx6Ubqpm78vjsNMQFwr5uyMk77fWLnUDik1V5RgnMo1J0wxqpFsAHOrdpxe55QM42YJoCar6/jK0dgkPHZTpktvgqvGaCzSWBjGXp+L2O9lDtJxMxXLdUa8ecL/RvmEqjlpzMAOgm/AZV8RdX/VqZjuKcjZH8LhYxzjDd3tDWcIkV9BhhXmB6+sTGl852DsZPh5UiLoabPYdIALARlTua132M4mzgkVcrbYNjk4BJI7zsyXrswhJ9esGjMyAEKrOyK0tAVmBGhVzoNvlRcICIi2CIG7wV+tcnf+e6HxYnMl2Oy1lqKOVAcV3vvhhxY+8wNdKlIBRpGWDZObMtS1AxvKEE4cQP+xskNAB1gdouDtjKfIN4ngXMjqFInu4u89SH98tkPLRpKsJ2LN2nDEKzgwi6kXnhGh5omXLa2iJe/Tkir2G4BE6Q7keXW8UR/8fdXkQ5EEonF20LnwuotHRilQHz/gtW0X18koU9F5eJTHqiUWLy+tfaOjqoUcqXl7QURe99qjAOGdjVLKMgi/WLgDbTILB9nFNPlzJeanclv0zutWhvAUazVwEnJLO5q4YDH/6LEg4q0uVhGYbK9JG7AfQaR0K260X3+OQ7kurrcp2naYjxK+a2+87zCuWO4mrTLgd1xkrH1mak4BMniGrtxSpBoywopwkQrMfRVBJj8rGHsGT8gltUxrBxrY2lsb8pUl0MVg1Lh2MMLK+f7hEC0YPC+aQ2M1QiHO4CVCC5pOp6kBqff/ rG3K67J4 nFuqGSY8x3FAYxz8i+m95l6cGv58eERfyifNbVK24CXHHMPEsWzTkYttZTxuO1fvGp9e37YLFVzqwGwp8JAfNcsh8nUeTHAw3pRFPIwlv9Bj1jmNg0sgO2uquMkYrYTe4mgzU476mOtGqWdY0Ew22AfhGwsL+PryErzNofLMlWolmK/8f+dTj5RpnVPjExC+Ym+xVVUEixyZx1RsTfQaJQuH/hSpb8h9wnYa958Hn36uDqNGXk++8B8nZuy3wjN26pg8KTiOcazCZiF/DachtEqkg2c+AcDSXhoKgsSRNi+f6jsrYVn1Akbs8dkkU8mAG69kf4bUiRg7p9sctXClLfD/7FvUsK+Pb/H11XbA0srFhhxe1EcmlTrueK+51NjMBpHm7QfKRmP/eK1b9m8COANdzACmuRuJ00Tn9eN8BvltmLkQDOTlMAFABC4ERGTaF2OXS6fepfocPd4uFfKoQX3bJoUOM/rWJg2qOXWOa3aPNECLPlONp+yB+0sf8+lB8KsC5F163ePfOvXhc0hnezDbIZ9WAs46goSkd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On large systems, we observed some issues with hugetlb and CMA: 1) When specifying a large number of hugetlb boot pages (hugepages= on the commandline), the kernel may run out of memory before it even gets to HVO. For example, if you have a 3072G system, and want to use 3024 1G hugetlb pages for VMs, that should leave you plenty of space for the hypervisor, provided you have the hugetlb vmemmap optimization (HVO) enabled. However, since the vmemmap pages are always allocated first, and then later in boot freed, you will actually run yourself out of memory before you can do HVO. This means not getting all the hugetlb pages you want, and worse, failure to boot if there is an allocation failure in the system from which it can't recover. 2) There is a system setup where you might want to use hugetlb_cma with a large value (say, again, 3024 out of 3072G like above), and then lower that if system usage allows it, to make room for non-hugetlb processes. For this, a variation of the problem above applies: the kernel runs out of unmovable space to allocate from before you finish boot, since your CMA area takes up all the space. 3) CMA wants to use one big contiguous area for allocations. Which fails if you have the aforementioned 3T system with a gap in the middle of physical memory (like the < 40bits BIOS DMA area seen on some AMD systems). You then won't be able to set up a CMA area for one of the NUMA nodes, leading to loss of half of your hugetlb CMA area. 4) Under the scenario mentioned in 2), when trying to grow the number of hugetlb pages after dropping it for a while, new CMA allocations may fail occasionally. This is not unexpected, some transient references on pages may prevent cma_alloc from succeeding under memory pressure. However, the hugetlb code then falls back to a normal contiguous alloc, which may end up succeeding. This is not always desired behavior. If you have a large CMA area, then the kernel has a restricted amount of memory it can do unmovable allocations from (a well known issue). A normal contiguous alloc may eat further in to this space. To resolve these issues, do the following: * Add hooks to the section init code to do custom initialization of memmap pages. Hugetlb bootmem (memblock) allocated pages can then be pre-HVOed. This avoids allocating a large number of vmemmap pages early in boot, only to have them be freed again later, and also avoids running out of memory as described under 1). Using these hooks for hugetlb is optional. It requires moving hugetlb bootmem allocation to an earlier spot by the architecture. This has been enabled on x86. * hugetlb_cma doesn't care about the CMA area it uses being one large contiguous range. Multiple smaller ranges are fine. The only requirements are that the areas should be on one NUMA node, and individual gigantic pages should be allocatable from them. So, implement multi-range support for CMA, avoiding issue 3). * Introduce a hugetlb_cma_only option on the commandline. This only allows allocations from CMA for gigantic pages, if hugetlb_cma= is also specified. * With hugetlb_cma_only active, it also makes sense to be able to pre-allocate gigantic hugetlb pages at boot time from the CMA area(s). Add a rudimentary early CMA allocation interface, that just grabs a piece of memblock-allocated space from the CMA area, which gets marked as allocated in the CMA bitmap when the CMA area is initialized. With this, hugepages= can be supported with hugetlb_cma=, making scenario 2) work. Additionally, fix some minor bugs, with one worth mentioning: since hugetlb gigantic bootmem pages are allocated by memblock, they may span multiple zones, as memblock doesn't (and mostly can't) know about zones. This can cause problems. A hugetlb page spanning multiple zones is bad, and it's worse with HVO, when the de-HVO step effectively sneakily re-assigns pages to a different zone than originally configured, since the tail pages all inherit the zone from the first 60 tail pages. This condition is not common, but can be easily reproduced using ZONE_MOVABLE. To fix this, add checks to see if gigantic bootmem pages intersect with multiple zones, and do not use them if they do, giving them back to the page allocator instead. The first patch is kind of along for the ride, except that maintaining an available_count for a CMA area is convenient for the multiple range support. Frank van der Linden (27): mm/cma: export total and free number of pages for CMA areas mm, cma: support multiple contiguous ranges, if requested mm/cma: introduce cma_intersects function mm, hugetlb: use cma_declare_contiguous_multi mm/hugetlb: fix round-robin bootmem allocation mm/hugetlb: remove redundant __ClearPageReserved mm/hugetlb: use online nodes for bootmem allocation mm/hugetlb: convert cmdline parameters from setup to early x86/mm: make register_page_bootmem_memmap handle PTE mappings mm/bootmem_info: export register_page_bootmem_memmap mm/sparse: allow for alternate vmemmap section init at boot mm/hugetlb: set migratetype for bootmem folios mm: define __init_reserved_page_zone function mm/hugetlb: check bootmem pages for zone intersections mm/sparse: add vmemmap_*_hvo functions mm/hugetlb: deal with multiple calls to hugetlb_bootmem_alloc mm/hugetlb: move huge_boot_pages list init to hugetlb_bootmem_alloc mm/hugetlb: add pre-HVO framework mm/hugetlb_vmemmap: fix hugetlb_vmemmap_restore_folios definition mm/hugetlb: do pre-HVO for bootmem allocated pages x86/setup: call hugetlb_bootmem_alloc early x86/mm: set ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT mm/cma: simplify zone intersection check mm/cma: introduce a cma validate function mm/cma: introduce interface for early reservations mm/hugetlb: add hugetlb_cma_only cmdline option mm/hugetlb: enable bootmem allocation from CMA areas Documentation/ABI/testing/sysfs-kernel-mm-cma | 13 + .../admin-guide/kernel-parameters.txt | 7 + arch/powerpc/mm/hugetlbpage.c | 6 + arch/powerpc/mm/init_64.c | 1 + arch/s390/mm/init.c | 13 +- arch/x86/Kconfig | 1 + arch/x86/kernel/setup.c | 4 +- arch/x86/mm/init_64.c | 16 +- include/linux/bootmem_info.h | 7 + include/linux/cma.h | 9 + include/linux/hugetlb.h | 25 + include/linux/mm.h | 13 +- include/linux/mmzone.h | 35 + mm/Kconfig | 8 + mm/bootmem_info.c | 4 +- mm/cma.c | 749 +++++++++++++++--- mm/cma.h | 42 +- mm/cma_debug.c | 5 +- mm/cma_sysfs.c | 20 + mm/hugetlb.c | 403 ++++++++-- mm/hugetlb_vmemmap.c | 199 ++++- mm/hugetlb_vmemmap.h | 23 +- mm/internal.h | 19 + mm/mm_init.c | 78 +- mm/sparse-vmemmap.c | 168 +++- mm/sparse.c | 87 +- 26 files changed, 1677 insertions(+), 278 deletions(-)