From patchwork Mon Jan 27 23:22:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13951868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 688F6C0218A for ; Mon, 27 Jan 2025 23:24:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59E0B2801DD; Mon, 27 Jan 2025 18:23:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 529462801D9; Mon, 27 Jan 2025 18:23:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A1D42801DD; Mon, 27 Jan 2025 18:23:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0FC1B2801D9 for ; Mon, 27 Jan 2025 18:23:10 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A166F1C7519 for ; Mon, 27 Jan 2025 23:23:09 +0000 (UTC) X-FDA: 83054809698.26.9A698FA Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf01.hostedemail.com (Postfix) with ESMTP id BF71E40008 for ; Mon, 27 Jan 2025 23:23:07 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rQEZ0nww; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3WhWYZwQKCCkKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3WhWYZwQKCCkKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738020187; a=rsa-sha256; cv=none; b=RSfG54Iq+8Faz9jjeNEP77OA3kpMZqWrBJpKDtQqoCk5ygdITpbAVCjNsb4bhY3Vnu399e kj/TLUs9i3/GKBpxrckdPaj78oM5DFx9fsDW8OzdeJ0vTsKTeAdVI+kkp90fothHfwkmNG A7iZdDgVrshIo2MEiVivwvHm7BcvzJo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rQEZ0nww; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3WhWYZwQKCCkKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3WhWYZwQKCCkKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738020187; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Gm3hQSpmhBarlZRgBgM0zsWbfOG8zObqZGHewKZCOZk=; b=5DsyDc0n8X76keKjEkLG4YAZ7BaCS/hOm5QUGnj3TPFcOMDvXHoJl9m7YhiGwZQGoIECCw 4QPmKKs0VORhXvakKZjozj3oZTEa5c9lYDU+eJa8vxbWiYOr5Zs6d2r/6bl4Gc0PprEoTB AtiAR2Dy8H0x5P0nEPutNJEwbuiz7hE= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f780a3d6e5so10078072a91.0 for ; Mon, 27 Jan 2025 15:23:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738020186; x=1738624986; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Gm3hQSpmhBarlZRgBgM0zsWbfOG8zObqZGHewKZCOZk=; b=rQEZ0nwwTphehx9wu48RgkDDPvD5SFZ9ciLIDpbPHGZc6Bj+YvOOrzYAnt2V/aVfYD 2+Rg3ffo1Vw4knjQXX6yCKDkBwZ2WXOyLRo9/yyz0pGst48KJ4pzDcYZCDYcCpWbOQRk i7m9XfH0vAFxX7XrXYLuM5ly6PZX0/Xg1iMGns9ab6Yk4b3QijDKFQ9Rh2bpT4g/Uju3 Pb+FMpq0xTOACVHY/sT9yChsGqVmZlbtY89ehqlvdvFXCO7SwSw40FcXLPbtrBuWB+vG d1aw8bv/FZKl/6GVnEf6CoaKloBwdS6Xpv0iwkuQ1bVqKVHjDGzRw8D+wtJ5bSf+emPM LNcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738020186; x=1738624986; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Gm3hQSpmhBarlZRgBgM0zsWbfOG8zObqZGHewKZCOZk=; b=b528bLEfwreYi382VO/mpI+s0GL1Qf3bB47JCxII47SkuYzEjKbX/BIYuB07XgKbDB k5OsFy9n580wsispah5VGBVBsnmUTraHiAe13YmJZLlQdnuY2eyv2GnWqQZgDzblwUsy sWXb/9h3ZBIjPeSgClFBbWQPawTvhhgv81oc+Psjv272wFUCHe3Z+r4L2pz2tDTzVWmH 8mrcc0wElIwPN7RqH9BScbw1PHOe+lbsf3YYtQXq3GBLVg8sRAOVDzjeNLd4gfmQNqER Xtzrtp3XvNE7Nx0PLVGxMZjLcih4t7tBv2MNG5aZ3jAHoKdgQiGkNhJOHyOqMJIaoZiH vAyQ== X-Forwarded-Encrypted: i=1; AJvYcCXxc+eQEnV0/E0w/esSJRKPRZZsFUMIBkA87uPnoBnxCX9JdXvQjGgLf6vD2Vz/pGroNjyUdFaM+w==@kvack.org X-Gm-Message-State: AOJu0YxVlYnNY3MYfJZAk1P+K6JaP/3HLtLBeKxY9yxE6bGF5BPEJ6kq 0TEB3PQG9ZGSoU7FMyplV5rKK21ZxJAH/Ly6wzP+7Vs0ne2io/PJ4wM55Qo/3Wb0he+vmQ== X-Google-Smtp-Source: AGHT+IH19j30fP8A5nC7LbdjLhXfAefG5JdyL9TVfOVI+iunswaOnTqgk5DlFd7Zc5gd2cUKDhngrEFu X-Received: from pfbjo40.prod.google.com ([2002:a05:6a00:90a8:b0:728:e508:8a48]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2917:b0:725:b201:2362 with SMTP id d2e1a72fcca58-72dafa409b5mr61127310b3a.11.1738020186643; Mon, 27 Jan 2025 15:23:06 -0800 (PST) Date: Mon, 27 Jan 2025 23:22:07 +0000 In-Reply-To: <20250127232207.3888640-1-fvdl@google.com> Mime-Version: 1.0 References: <20250127232207.3888640-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250127232207.3888640-28-fvdl@google.com> Subject: [PATCH 27/27] mm/hugetlb: enable bootmem allocation from CMA areas From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usama.arif@bytedance.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Madhavan Srinivasan , Michael Ellerman , linuxppc-dev@lists.ozlabs.org X-Rspam-User: X-Rspamd-Queue-Id: BF71E40008 X-Rspamd-Server: rspam10 X-Stat-Signature: jrkxw5cbyf4uukcam5bt1wxzt3emfops X-HE-Tag: 1738020187-534090 X-HE-Meta: U2FsdGVkX1/+db7Sc+XFNgRof7i1r2/ma+J44792dksOXLCJqdD+prx631VI/oSciU576Q4cWxJFmtwdalJxLhqji7FexaD6WPAB+bAxI/AA/cM0Nx0dsBPKKU2uMRHaFJLxyDN/wXyJWTAq2T6x6Zrp1Lu4UqLC6gaRjTUtMTztIuWgUkbySpZ0kTtULSUUtJ3S/fAHL3GurSSQ8n2hXO/tK7cSCXb3Kl6fEc6nY6XwQZlZPL72HgY05OOWsetljCqGmD41PmHk+c9m1PPGBGBJFIp0d8WNbFzCbJS2RpHMEUE6dUwoL2yiwed+rDjjXr7cvs7e7aKTLfx4B0fhYXUU5thETbs4fmU659TSIoJ5mGnAf4XS1WN1r9dyZLQLqG0T7iHaF/bh4tYRVM3/8bnCuhb3QEKVqzDHzKi2NAUqFs0bJDsCPzaRoWGC3YZ0jfxEF0vGy31FpMjFcxmYMXsoTI55/GF02SXM9YFigsdsDRr9YN6JgJRtJaR90yz8oPA/8RIgXWv2WmG+1rK8gf3VXk0KC1g4Kjhk98u/VqyH55W7r9ro9hRi1KIN79fQcPGlel8D5qZdN5jGwT6z4f6R4B1y9lCHlTd6ODB7Oy9YiEYOI9kBhzkwUMmTG2EDrtWgUe6O5IJIFFBCpdFEweu+lEzFNNkAR1olJtk7to8uqqUWb7ASLB9pGPjRkllM5n8T2VryT8YCJvd5CTmREZsl+w79EttxbVeERpzhoW+HTtH1lykD5y1+wdMLKCsN+QGZJV7vo91GCN93Fb9O0iPYSSkVqrpbc4+UrATZtFbJbFIlghAsRstlxm2IgmMyxsEr/UFOfx9K0YOlXWyS947zn5CuG6f3nWFuJLB5OyViZ7BMxFzXhIMIjlJvoc++Jhy21IOWOLftSVh/gAxL23XXmsjuorkhOtYDAlOeeC0a5tTagRD+EgjfKWs0U1pkPPF2wEn8aqs76rxQ7sO E70CwwpQ q0XTI9pCMvPl5dUDcA/GCLvbb/JR7cxK51BNvW1KJCA7ZaV58vQpWDaewZ6V9fNK9ZkQ9xinq6OlvbE/As9dd97yMbKRNHY18aJWhtOvnpWt/Qzo9u/r5dZKjdLkip/cr3OPwihmWMGKhLhqWr3jTr+DVTXfgABnQCzgEHSE2xFC08E0F7hy/12sGvRmPNYUpvdp92FkjFkBGU/5Ba689FXl7Ipifunzycqp53Af00MNNVFUhaaLTP4/NL4fEAp5yJXsb4iei8nyWOdcNhnUpelKc6yciAlkPBj+igkF+M47dTqTBic6zwHg5+NSsQ4ykse0L/jbHoN5GMGH9WgW4GhJ0LFrQ04zcBjx3VKPDYpIIUVzfDxqUapysxVebs9r9YhYug1Tv8iQCCKdC13s1vYXfd9gU70Il+A31ycVLc6x2as0MEuC3rRcYJ0nziRlT0soGgrN8g6UMGnSG7hsNIMGuggKXxDFkiwiXCMxu/J2BkTPZ6TU/AUbanjR3BKYv2sA7HKVrG8dSOZK37qytE1kEDTAAcYIaB6q142DtiC0RWkJYdh5JfLcrr2Fo0q5vbfXB953DK28U9RXFqrPDxWDKrqy8uihM6vuLr0LICGF/yWM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If hugetlb_cma_only is enabled, we know that hugetlb pages can only be allocated from CMA. Now that there is an interface to do early reservations from a CMA area (returning memblock memory), it can be used to allocate hugetlb pages from CMA. This also allows for doing pre-HVO on these pages (if enabled). Make sure to initialize the page structures and associated data correctly. Create a flag to signal that a hugetlb page has been allocated from CMA to make things a little easier. Some configurations of powerpc have a special hugetlb bootmem allocator, so introduce a boolean arch_specific_huge_bootmem_alloc that returns true if such an allocator is present. In that case, CMA bootmem allocations can't be used, so check that function before trying. Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Frank van der Linden --- arch/powerpc/mm/hugetlbpage.c | 5 ++ include/linux/hugetlb.h | 7 ++ mm/hugetlb.c | 135 +++++++++++++++++++++++++--------- 3 files changed, 114 insertions(+), 33 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index d3c1b749dcfc..e53e4b4c8ef6 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -121,6 +121,11 @@ bool __init hugetlb_node_alloc_supported(void) { return false; } + +bool __init arch_specific_huge_bootmem_alloc(struct hstate *h) +{ + return (firmware_has_feature(FW_FEATURE_LPAR) && !radix_enabled()); +} #endif diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2512463bca49..bca3052fb175 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -591,6 +591,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -650,6 +651,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE @@ -678,14 +680,18 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; +struct cma; + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; unsigned long flags; + struct cma *cma; }; #define HUGE_BOOTMEM_HVO 0x0001 #define HUGE_BOOTMEM_ZONES_VALID 0x0002 +#define HUGE_BOOTMEM_CMA 0x0004 bool hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m); @@ -711,6 +717,7 @@ bool __init hugetlb_node_alloc_supported(void); void __init hugetlb_add_hstate(unsigned order); bool __init arch_hugetlb_valid_size(unsigned long size); +bool __init arch_specific_huge_bootmem_alloc(struct hstate *h); struct hstate *size_to_hstate(unsigned long size); #ifndef HUGE_MAX_HSTATE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 32ebde9039e2..183e8d0c2fb4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -61,7 +61,7 @@ static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; #endif static bool hugetlb_cma_only; -static unsigned long hugetlb_cma_size __initdata; +static unsigned long hugetlb_cma_size; __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; __initdata unsigned long hstate_boot_nrinvalid[HUGE_MAX_HSTATE]; @@ -132,8 +132,10 @@ static void hugetlb_free_folio(struct folio *folio) #ifdef CONFIG_CMA int nid = folio_nid(folio); - if (cma_free_folio(hugetlb_cma[nid], folio)) + if (folio_test_hugetlb_cma(folio)) { + WARN_ON(!cma_free_folio(hugetlb_cma[nid], folio)); return; + } #endif folio_put(folio); } @@ -1509,6 +1511,9 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, break; } } + + if (folio) + folio_set_hugetlb_cma(folio); } #endif if (!folio) { @@ -3175,6 +3180,63 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, return ERR_PTR(-ENOSPC); } +/* + * Some architectures do their own bootmem allocation, so they can't use + * early CMA allocation. So, allow for this function to be redefined. + */ +bool __init __attribute((weak)) +arch_specific_huge_bootmem_alloc(struct hstate *h) +{ + return false; +} + +static bool __init hugetlb_early_cma(struct hstate *h) +{ + if (arch_specific_huge_bootmem_alloc(h)) + return false; + + return (hstate_is_gigantic(h) && hugetlb_cma_size && hugetlb_cma_only); +} + +static __init void *alloc_bootmem(struct hstate *h, int nid) +{ + struct huge_bootmem_page *m; + unsigned long flags; + struct cma *cma; + +#ifdef CONFIG_CMA + if (hugetlb_early_cma(h)) { + flags = HUGE_BOOTMEM_CMA; + cma = hugetlb_cma[nid]; + m = cma_reserve_early(cma, huge_page_size(h)); + } else +#endif + { + flags = 0; + cma = NULL; + m = memblock_alloc_try_nid_raw(huge_page_size(h), + huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + } + + if (m) { + /* + * Use the beginning of the huge page to store the + * huge_bootmem_page struct (until gather_bootmem + * puts them into the mem_map). + * + * Put them into a private list first because mem_map + * is not up yet. + */ + INIT_LIST_HEAD(&m->list); + list_add(&m->list, &huge_boot_pages[nid]); + m->hstate = h; + m->flags = flags; + m->cma = cma; + } + + return m; +} + int alloc_bootmem_huge_page(struct hstate *h, int nid) __attribute__ ((weak, alias("__alloc_bootmem_huge_page"))); int __alloc_bootmem_huge_page(struct hstate *h, int nid) @@ -3184,17 +3246,14 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) /* do node specific alloc */ if (nid != NUMA_NO_NODE) { - m = memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + m = alloc_bootmem(h, node); if (!m) return 0; goto found; } /* allocate from next node when distributing huge pages */ for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_ONLINE]) { - m = memblock_alloc_try_nid_raw( - huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); + m = alloc_bootmem(h, node); if (m) break; } @@ -3203,7 +3262,6 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) return 0; found: - /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3213,18 +3271,6 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), huge_page_size(h) - PAGE_SIZE); - /* - * Use the beginning of the huge page to store the - * huge_bootmem_page struct (until gather_bootmem - * puts them into the mem_map). - * - * Put them into a private list first because mem_map - * is not up yet. - */ - INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[node]); - m->hstate = h; - m->flags = 0; return 1; } @@ -3265,13 +3311,25 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) +{ + return (m->flags & HUGE_BOOTMEM_HVO); +} + +static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) +{ + return (m->flags & HUGE_BOOTMEM_CMA); +} + /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) - * here. + * here, or MIGRATE_CMA if this was a page allocated through an early CMA + * reservation. * - * Note that this will not write the page struct, it is ok (and necessary) - * to do this on vmemmap optimized folios. + * In case of vmemmap optimized folios, the tail vmemmap pages are mapped + * read-only, but that's ok - for sparse vmemmap this does not write to + * the page structure. */ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, struct hstate *h) @@ -3280,9 +3338,13 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); - for (i = 0; i < nr_pages; i += pageblock_nr_pages) - set_pageblock_migratetype(folio_page(folio, i), + for (i = 0; i < nr_pages; i += pageblock_nr_pages) { + if (folio_test_hugetlb_cma(folio)) + init_cma_pageblock(folio_page(folio, i)); + else + set_pageblock_migratetype(folio_page(folio, i), MIGRATE_MOVABLE); + } } static void __init prep_and_add_bootmem_folios(struct hstate *h, @@ -3319,7 +3381,7 @@ bool __init hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m) { unsigned long start_pfn; - bool valid; + bool valid = false; if (m->flags & HUGE_BOOTMEM_ZONES_VALID) { /* @@ -3328,10 +3390,16 @@ bool __init hugetlb_bootmem_page_zones_valid(int nid, return true; } + if (hugetlb_bootmem_page_earlycma(m)) { + valid = cma_validate_zones(m->cma); + goto out; + } + start_pfn = virt_to_phys(m) >> PAGE_SHIFT; valid = !pfn_range_intersects_zones(nid, start_pfn, pages_per_huge_page(m->hstate)); +out: if (!valid) hstate_boot_nrinvalid[hstate_index(m->hstate)]++; @@ -3360,11 +3428,6 @@ static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, } } -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return (m->flags & HUGE_BOOTMEM_HVO); -} - /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3414,6 +3477,9 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) */ folio_set_hugetlb_vmemmap_optimized(folio); + if (hugetlb_bootmem_page_earlycma(m)) + folio_set_hugetlb_cma(folio); + list_add(&folio->lru, &folio_list); /* @@ -3606,8 +3672,11 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; - /* skip gigantic hugepages allocation if hugetlb_cma enabled */ - if (hstate_is_gigantic(h) && hugetlb_cma_size) { + /* + * Skip gigantic hugepages allocation if early CMA + * reservations are not available. + */ + if (hstate_is_gigantic(h) && hugetlb_cma_size && !hugetlb_early_cma(h)) { pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); return; }