From patchwork Mon Jul 24 13:46:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13324802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0EC6C001DF for ; Mon, 24 Jul 2023 13:46:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06D448E0005; Mon, 24 Jul 2023 09:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F373B8E0001; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8EA1900002; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9FD608E0001 for ; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 67047B2152 for ; Mon, 24 Jul 2023 13:46:53 +0000 (UTC) X-FDA: 81046631106.18.9B44655 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by imf26.hostedemail.com (Postfix) with ESMTP id 6B89F14000E for ; Mon, 24 Jul 2023 13:46:50 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ikYTxmvN; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690206411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2+5f3AgE150Mh6cYaW30LczJjyIbsUmp9ItlqNwo9Gs=; b=uBkOMlNxzGUKNtavUbdI8A3O69er6gwmclehSAYHl2kcNgLEBFx88eXc/Tko+RGMmYgjWZ sUSrRjpTUX2UlgvUBa/BN4PZd0gk7aVfSH6wAFtm4Q6nXCauhBg1kmPdwRbCcY6lnUWAfU IPXNf6mWssq0zCCulOKXuvCgISfIL3I= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ikYTxmvN; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690206411; a=rsa-sha256; cv=none; b=REPP1SmjkSvRv/CpgilIcQjFg7EWQItfesryrDQPTjLP4YXfqC+6cWixhCwgTbabPiqypg Uq+JWvBKkbEHg8AyaN7oEBUO6T7faWDi9GOGk474BVC8QXIHXHeQT5ljBSF6m7gnjZwrvw pBz1xHFt+Sjzwl19c/XWr6V3sa3lGo8= Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-3fbc244d307so41942675e9.1 for ; Mon, 24 Jul 2023 06:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690206408; x=1690811208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2+5f3AgE150Mh6cYaW30LczJjyIbsUmp9ItlqNwo9Gs=; b=ikYTxmvNKkByuJlR0NutAuZwBS+HffntLHXBTOm8ABSvwCfWL2fKqLXT+iOY+era+y 0f+kQ2586uXX6d82nfLBSpOQr1gqnMFjKQ47G8ZLbVTI8NSLejjGc+kDPEvl1HVxnBUF jCpEFBNwQAA+XsldO7xDMXzVVhi5WukMODB/6kuUFR9VRoS+9pioBp657mkjAf4PNZ/i F4wa0/ndwm2yPUB7BijO2XndG4yNOsOBIbw8Z0pGdeoakBDEl8/5c+TqIcujNE6fR8ZQ FpFK7b0NO5bBOQMrm+8GlX2hKWNa2zqyvycuRIQfoA56RG/Cn9fc7U79QcopUi7lSSGW lQSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690206408; x=1690811208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2+5f3AgE150Mh6cYaW30LczJjyIbsUmp9ItlqNwo9Gs=; b=VOvuPcU1Y/nNZTncPh/tyXzaLBWp0rDK/W70sBSV58L84E+RJ/BHhzBInJ1BAZkbKx bgopmINIwtyjTuiZndIokh0ZP70V15uAcxt4aywtSYx0hhy+zxXQVh7eJxhxVxVDIP/y sqJxOx4vfoznP3vPZ7zJ5zTrBzTcmN/aWW1ZQF8IuGfSOzuxXeNFp6QtUfwvo9Q7Nyxa Aq0Ig5OaeJSkWuFRYQp4oI5B11i80uYX9U7enN93VwKw5IFH7e2ON6cOvHX6qpsO5Dzi /scxpLe+xnhE7FHuNbalfcuCYKYwwWi1Wf94nZLnRjKWvus8rQrwoLowpC64z7TSWXh6 m2bw== X-Gm-Message-State: ABy/qLbm6gynJFhsoBPrUT5fXRpO6RtX4xI8qLd2QHnVvMFP5coRzBH+ aOsCO2SNEtPGjoaK/IzZywgJDNLXktLbGqKG5dI= X-Google-Smtp-Source: APBJJlG2oRr8ivYhrN6NtzN8nuU0+P+lfubBc/YogasGIn8tgMOZpf/GJE4V1pfPjhr20IzDbaEKIw== X-Received: by 2002:a1c:750a:0:b0:3fd:ad65:ea8b with SMTP id o10-20020a1c750a000000b003fdad65ea8bmr3233762wmc.12.1690206408483; Mon, 24 Jul 2023 06:46:48 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:d7c4:7f46:8fed:f874]) by smtp.gmail.com with ESMTPSA id e19-20020a05600c219300b003fbe791a0e8sm10209354wme.0.2023.07.24.06.46.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jul 2023 06:46:48 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [RFC 1/4] mm/hugetlb: Skip prep of tail pages when HVO is enabled Date: Mon, 24 Jul 2023 14:46:41 +0100 Message-Id: <20230724134644.1299963-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230724134644.1299963-1-usama.arif@bytedance.com> References: <20230724134644.1299963-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6B89F14000E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6y6owp3tgp4qtr4am3tkx4cgwnoewsez X-HE-Tag: 1690206410-339825 X-HE-Meta: U2FsdGVkX18vagZ4BG87dQqzJVTLWa5Glk75489/CKRnkZ2s0kZeoUqYI9A1rHbucSp/pVVEMwVbWZ4x6+JTuQCWuc4ltsQ04t2eExrFzrjY6ZhPUf7NRRzE9N+GIuIWxyD0r9H0ZeeBcC0jJaQBrWOA7GpntCoLVQWDVI5hqMQa0/LboVDInDcn0fiXKkfqm8KcKc8KeOhnIJr0ekWCiWs28THcbQa3hjeRs2m6lWk6llydzxD8CyxIPcgRoYcAQG115kcHTEy9Ky5AuOFxb37Yv6IT2WeZzkXRcIFhgVEqv9jYe8LpKqZG7dCpYIOBOQKBtOpnrrQFb8slUp4mbs6htzs8u3YfXvvR6+GcApgpfaf3jNI8KJ3qGRtPmtRrMOrLqGMjirHT79+kQM9X19f1SK0/10XobmEoFd5dFi+XcsLo/jjm9dMPAKuoKsOjWF+hz0Fdo5sYKwn0eM9f/CvEYSZsM5SF3eOdINDABt2O6LwatdDPc7gk80zUvunZNL5fxfWvehp9JINP5Xs0zEdzpgWjVAfDDG9IEKLWZ8kOu8tyHdoumWeu4BzUMnOUvryMr8EX5bkC4Azok5MYj95boCWCPgOaKlFZhvJe+pRIMyhenSpSE7F69h4YHOvgKm5/Gk49ZTM7rEt4EdaZhTX7FOxPY3ivsAkQHv1sXVLbvaLLjJqPGoNMS2/CM+24ZKgg5uku5BRmI3vzgXpSZqBEwazjYIma/Z3mC5h5hkHUUFsRYk/QoQBfTEeoMVizF3Uadq/0JgqiOo3qov/AM2Q90dSeEQoEvp8nrdEQiXYv/nS1xWT13LZ692Yxb/m2P8LvHrKJ2G3PFT8e2ffPl21xyHO9rwniPEv08hocJwwJGY3rX8D+sSQDTvi3WrBsI5Y8fyUcTrirXU1i3evmsSSV3EtRS+RkjSfK7hXQ1VmfY+AqSWaPgxOynDr+/cbSOgTNqD1/OIbvcew/OPB FuOZYFx/ eWIWLcjPKORLuON4u1i/bBTFAn16MUR3FU1hTMF+2RITS9q5yG4EgYiOUvXgQi9xtrdwWK3G6RMLUa/CQoJRopfoX8Ue0C31q4DU1sz/wxLt4vz1xmhKES5JZPnWAFp94So1aPA/8a4ZsTllVzsSxKnrHFSzp9xomzI7ghmeOF31N53kUN7Ea7OLGtwo9xNMsI4p4hTzK1LuQRyeWvDnIcpRi2QXLI74kZml57o3koySyE+wHGUTk306G2q2Jmtal6QgnKB0xYw+F/cE3QBCOujgvEwX8N9M5KLvSQmV7/f6g2aMJOK2L3xmtF+ZRQ1cjphg/1iH9WaPl01GJSKDqd5bSNAEtYO33A6Ou2jBkrTL5NwutQjVZD63vjI/EugOvcySuj6MLNLQHKXSktZ7GWfr4v08njV6sPyd4FQMqA94+ZD4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 = 262080 struct pages per hugepage. Signed-off-by: Usama Arif --- mm/hugetlb.c | 30 +++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 1 + 3 files changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..24352abbb9e5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1943,13 +1943,22 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni } static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) + unsigned int order, bool demote, + bool hugetlb_vmemmap_optimizable) { int i, j; int nr_pages = 1 << order; struct page *p; __folio_clear_reserved(folio); + + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_optimize + * in prep_new_huge_page. Hence, reduce nr_pages to the pages that will be kept. + */ + if (hugetlb_vmemmap_optimizable) + nr_pages = HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); + for (i = 0; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2020,15 +2029,15 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, } static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) + unsigned int order, bool hugetlb_vmemmap_optimizable) { - return __prep_compound_gigantic_folio(folio, order, false); + return __prep_compound_gigantic_folio(folio, order, false, hugetlb_vmemmap_optimizable); } static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) + unsigned int order, bool hugetlb_vmemmap_optimizable) { - return __prep_compound_gigantic_folio(folio, order, true); + return __prep_compound_gigantic_folio(folio, order, true, hugetlb_vmemmap_optimizable); } /* @@ -2185,7 +2194,8 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, &folio->page))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3211,8 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, page))) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3635,9 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) subpage = folio_page(folio, i); inner_folio = page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio_for_demote(folio, + target_hstate->order, + vmemmap_should_optimize(target_hstate, subpage)); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..3525c514c061 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -57,4 +57,5 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); #endif /* _LINUX_HUGETLB_VMEMMAP_H */ From patchwork Mon Jul 24 13:46:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13324801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72E5AC0015E for ; Mon, 24 Jul 2023 13:46:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9B04900003; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B18938E0002; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 992DA900002; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8211C8E0001 for ; Mon, 24 Jul 2023 09:46:53 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 52F32160AAA for ; Mon, 24 Jul 2023 13:46:53 +0000 (UTC) X-FDA: 81046631106.17.DFF465D Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by imf16.hostedemail.com (Postfix) with ESMTP id 6037B180023 for ; Mon, 24 Jul 2023 13:46:50 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hpDj6ajL; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.49 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690206411; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TrBvXJXG5pOuPGnFnmmWgOCSm+Hu58ziTnCPKJdrxFI=; b=BmtoU9+/NrEYo1YQS2aRindCt15OokDfP2pw9ew3IlrhqWLCV87klk11d9H4EDC2e5U4x2 yrFCZ8jeMFhdaaTq7aMXHdDcepO7KqmJiNvg/b6kk7H90IhwkxXypcDEyjeX8mZDv1gFPr UPsIdWbjN9PIHGImP6tmMk1cA2pMHIA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=hpDj6ajL; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf16.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.49 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690206411; a=rsa-sha256; cv=none; b=cnzqgdmGtOC7cKft39Bz2MHZ5l0yIGm2zNJhfW7kA55RmQ464/fg8pZdVNnNHe6ysMHzb7 hzQv41lRWidzhmnvkPgx34Dun640M9+0LN/BnEIWm4MEvYts8rXtXMwTIS9n2lhpAyfkXC MJCqetBBKg8Ad1H15q90jOYpv0I3+PQ= Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-3fb4146e8fcso27503795e9.0 for ; Mon, 24 Jul 2023 06:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690206409; x=1690811209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TrBvXJXG5pOuPGnFnmmWgOCSm+Hu58ziTnCPKJdrxFI=; b=hpDj6ajLvbAW9mGF/M6fPkHM3QLBe4nIO3sZ+5rXgwwUYggGi/WBHwUVl1j7lubKb8 LIXvVDqKO3ib1Q5crCeVVLcCwz6zNW1WL7yBf3YfRSnBoROpth2S/AcSHCUOGFSvOamJ WwynED0rZsIpTLVIUvw0WZlcGPrJtSbolW7vEwQvV4fqE/ypPbFP/TVjajy/h0SLgyJd DEkKUg1jUWxEp6QBOkRtSMiC3xBufpqtPJyRabtQ/i1vuBg6r5wxRkqLOsuG4q264q1h bnSfu1davANntTXapNjREHqzZwcU0yVx4hSTqtUWrFB7eFjVaKtY5FR0QwLONqCmO8lV fFVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690206409; x=1690811209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TrBvXJXG5pOuPGnFnmmWgOCSm+Hu58ziTnCPKJdrxFI=; b=VzOFVB40zFBxVwCRwNrFFUKboaYE8YlDts2yhDkF7Q9EMY7uxbAtrFCxf1U5SttUey xA25rOC7+9yuz2f/r89+QmZYTSfhtZbS7aHqmTT+lh9IV6RY2A4TPQIpM9XDrff6GMPr Znn1ZvsUS6JrqwP9lUKthcmPrgJM+vVr28V9jNPxClckGc6pWxZplHqvLzrU8tMdV1vI 93d+As1GPktfNjyeJPSHwFZkytlkY4IZYhOuOF461u9opnmXV0Yxf2MQJWAAeI4gM3i+ 2F1XkvMsrlCA7Ms1DZPCwa4lGXDmQg5X1BJPC7GKdZhzFWY4/kVB9SFb8oB30mnk07sk dtmA== X-Gm-Message-State: ABy/qLapJ5waAkTPxCPdS65FUuUZLNmeFsQQV8GVtLpcTUkrRi4TIgLb iCM33oqTPxHaHUr5/MfHnQByq49WdliqLQZV2Pw= X-Google-Smtp-Source: APBJJlHk8KJWAC/OoxngikxSJIK1cZRDdcxZvpPXhgKaCZyVzjAtt+mOGG+/DXri1zptdNJ2s+Z0zg== X-Received: by 2002:a7b:c8d7:0:b0:3fc:180:6463 with SMTP id f23-20020a7bc8d7000000b003fc01806463mr8081280wml.7.1690206409462; Mon, 24 Jul 2023 06:46:49 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:d7c4:7f46:8fed:f874]) by smtp.gmail.com with ESMTPSA id e19-20020a05600c219300b003fbe791a0e8sm10209354wme.0.2023.07.24.06.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jul 2023 06:46:48 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [RFC 2/4] mm/memblock: Add hugepage_size member to struct memblock_region Date: Mon, 24 Jul 2023 14:46:42 +0100 Message-Id: <20230724134644.1299963-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230724134644.1299963-1-usama.arif@bytedance.com> References: <20230724134644.1299963-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6037B180023 X-Stat-Signature: sjsjckcz7cnzu9qgyj8dpigiopic3i69 X-HE-Tag: 1690206410-348364 X-HE-Meta: U2FsdGVkX1+qhMpjDzUpA4Xv1euc9N5HkDwySHSARLYexM5cJTIaLyGei5oAKXiYxpLTM7mNes04jUiRNkThZKIDN/RdA17Ai8TyuF/6TNl8CoiGrYsofEXWEvfgdoFldBAEWP4XHHh4jXecAX+PfvL1Y4nXsxatcc/YCzR0qRq9yAs70NlDBJRNlX7wYIW+JgDnhOeDyCrtEcJKsGvynEmKppYWOQ2FuX6aBDZGXbF618pAfevRK8bjlQqoa2EInRQs8YRkbnSWtLGd0p/McEjbyw67n7GOV+aqVoFztnA1CB49EpSUzcUMwMRne5Jyts7uG5+xvZ2yvk2TT3FDXiDdfRl0beeqQBuEzCSXdJy2a0VRD5g0VPIvv2vLH6Y4pnMayCHUQ22PJHuUpvwqki98uxrE80u0m8q4+6VInlGYw/5pg5ABbSJf4lnQvS2OWTQo5USkaCJRZkFbIf0hVVArLOrREfF+XqWIClmBkJW5aLsIXbtfRR1VKsCVuGfsl6ZT+g4qKp4iedSGhmzagmmmFPlLkQ13YS46MRbrMZEZWl3xNVm8sji9Bfc9Zgbqlmw9XCOZ/bLeDyu6M04wafOohJb7w0QzFCgl367i7PslA/JT+12dYsW4jh4Rlx5i+6fdQajGYxjhCmE5VbnP7wDXTLteiyYTgYDOSAgZAR0xRJJ8J3Rlueov4KVbpz/bVVEFtVDaiRFhHzkpTfz9zfobWKLimJLKcDkOPiegxxBN42RQSqBDKYhemdTTfyVftmG9dJ+EicWOQkfF1pBsIHI+tqwA7ltuk57NSFe2+eZphn0SrMtb+twMwNsMiQmmQTZTZ0TpxNZ463EtFYQ09d4wgvJIHt9W/C5HBVaHibfkyq6H0kb8pIwkjL8eQbXAd1fGYzF4L+AaCxk3mH4Ed/aSO/mr8xvSsDcs7w1Dgb6tKcdL78oW5WFRDkhOXC/S6rJqbJtq3sqR5Riwezh rB9UEat7 p5UeADKruUryQUM2+A/RXYzlWCO6n1DIcSIzXdAusNGGQaGB32+F+OEtKoOXdZ+B7oXtuxU3a8v3vQlgECURNWw7k30mooVIb1U9JYBwkZhWQZ1aeLUuHXdOidI57eiT/NWCMJDhrQ7XpGSZHzI/OtmUyCu14f1gA2CSst9RmDue+Z5yUXsqc9RbTkfraNc5E9sl7A0i8lgozrBWdQTcVRrypv4CJlGGdGzAsO9pIHAxobDo2a2VIrd7RcNlMrQBt1yEOEnn9VwuZNyGlblZk9BTxJbnl65IRCkfJ2Q/DDjlYsm9KD2l8TqhCDXwrNO+Zf1L2r8NUwPl5kAnG5NBBC6LtJivUp7cOoL1LEMk0X5ZTO4/Rk/WrNMlwB58mSDQhx29xOeHL72bVysWKqGdX53cxIkSiz0zbyLKl8VwcgAXiaQIASj2b9vzrl/eYPiy5kYuxnVMe/cO5sKOR8UPuPawudA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This propagates the hugepage size from the memblock APIs (memblock_alloc_try_nid_raw and memblock_alloc_range_nid) so that it can be stored in struct memblock region. This does not introduce any functional change and hugepage_size is not used in this commit. It is just a setup for the next commit where huge_pagesize is used to skip initialization of struct pages that will be freed later when HVO is enabled. Signed-off-by: Usama Arif --- arch/arm64/mm/kasan_init.c | 2 +- arch/powerpc/platforms/pasemi/iommu.c | 2 +- arch/powerpc/platforms/pseries/setup.c | 4 +- arch/powerpc/sysdev/dart_iommu.c | 2 +- include/linux/memblock.h | 8 ++- mm/cma.c | 4 +- mm/hugetlb.c | 6 +- mm/memblock.c | 60 ++++++++++++-------- mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 2 +- tools/testing/memblock/tests/alloc_nid_api.c | 2 +- 11 files changed, 56 insertions(+), 38 deletions(-) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index f17d066e85eb..39992a418891 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -50,7 +50,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node) void *p = memblock_alloc_try_nid_raw(PAGE_SIZE, PAGE_SIZE, __pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_NOLEAKTRACE, - node); + node, 0); if (!p) panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%llx\n", __func__, PAGE_SIZE, PAGE_SIZE, node, diff --git a/arch/powerpc/platforms/pasemi/iommu.c b/arch/powerpc/platforms/pasemi/iommu.c index 375487cba874..6963cdf76bce 100644 --- a/arch/powerpc/platforms/pasemi/iommu.c +++ b/arch/powerpc/platforms/pasemi/iommu.c @@ -201,7 +201,7 @@ static int __init iob_init(struct device_node *dn) /* For 2G space, 8x64 pages (2^21 bytes) is max total l2 size */ iob_l2_base = memblock_alloc_try_nid_raw(1UL << 21, 1UL << 21, MEMBLOCK_LOW_LIMIT, 0x80000000, - NUMA_NO_NODE); + NUMA_NO_NODE, 0); if (!iob_l2_base) panic("%s: Failed to allocate %lu bytes align=0x%lx max_addr=%x\n", __func__, 1UL << 21, 1UL << 21, 0x80000000); diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index e2a57cfa6c83..cec7198b59d2 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -160,7 +160,7 @@ static void __init fwnmi_init(void) */ mce_data_buf = memblock_alloc_try_nid_raw(RTAS_ERROR_LOG_MAX * nr_cpus, RTAS_ERROR_LOG_MAX, MEMBLOCK_LOW_LIMIT, - ppc64_rma_size, NUMA_NO_NODE); + ppc64_rma_size, NUMA_NO_NODE, 0); if (!mce_data_buf) panic("Failed to allocate %d bytes below %pa for MCE buffer\n", RTAS_ERROR_LOG_MAX * nr_cpus, &ppc64_rma_size); @@ -176,7 +176,7 @@ static void __init fwnmi_init(void) size = sizeof(struct slb_entry) * mmu_slb_size * nr_cpus; slb_ptr = memblock_alloc_try_nid_raw(size, sizeof(struct slb_entry), MEMBLOCK_LOW_LIMIT, - ppc64_rma_size, NUMA_NO_NODE); + ppc64_rma_size, NUMA_NO_NODE, 0); if (!slb_ptr) panic("Failed to allocate %zu bytes below %pa for slb area\n", size, &ppc64_rma_size); diff --git a/arch/powerpc/sysdev/dart_iommu.c b/arch/powerpc/sysdev/dart_iommu.c index 98096bbfd62e..86c676b61899 100644 --- a/arch/powerpc/sysdev/dart_iommu.c +++ b/arch/powerpc/sysdev/dart_iommu.c @@ -239,7 +239,7 @@ static void __init allocate_dart(void) */ dart_tablebase = memblock_alloc_try_nid_raw(SZ_16M, SZ_16M, MEMBLOCK_LOW_LIMIT, SZ_2G, - NUMA_NO_NODE); + NUMA_NO_NODE, 0); if (!dart_tablebase) panic("Failed to allocate 16MB below 2GB for DART table\n"); diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..bb8019540d73 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -63,6 +63,7 @@ struct memblock_region { #ifdef CONFIG_NUMA int nid; #endif + phys_addr_t hugepage_size; }; /** @@ -400,7 +401,8 @@ phys_addr_t memblock_phys_alloc_range(phys_addr_t size, phys_addr_t align, phys_addr_t start, phys_addr_t end); phys_addr_t memblock_alloc_range_nid(phys_addr_t size, phys_addr_t align, phys_addr_t start, - phys_addr_t end, int nid, bool exact_nid); + phys_addr_t end, int nid, bool exact_nid, + phys_addr_t hugepage_size); phys_addr_t memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid); static __always_inline phys_addr_t memblock_phys_alloc(phys_addr_t size, @@ -415,7 +417,7 @@ void *memblock_alloc_exact_nid_raw(phys_addr_t size, phys_addr_t align, int nid); void *memblock_alloc_try_nid_raw(phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, phys_addr_t max_addr, - int nid); + int nid, phys_addr_t hugepage_size); void *memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, phys_addr_t max_addr, int nid); @@ -431,7 +433,7 @@ static inline void *memblock_alloc_raw(phys_addr_t size, { return memblock_alloc_try_nid_raw(size, align, MEMBLOCK_LOW_LIMIT, MEMBLOCK_ALLOC_ACCESSIBLE, - NUMA_NO_NODE); + NUMA_NO_NODE, 0); } static inline void *memblock_alloc_from(phys_addr_t size, diff --git a/mm/cma.c b/mm/cma.c index a4cfe995e11e..a270905aa7f2 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -334,7 +334,7 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, if (!memblock_bottom_up() && memblock_end >= SZ_4G + size) { memblock_set_bottom_up(true); addr = memblock_alloc_range_nid(size, alignment, SZ_4G, - limit, nid, true); + limit, nid, true, 0); memblock_set_bottom_up(false); } #endif @@ -353,7 +353,7 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, if (!addr) { addr = memblock_alloc_range_nid(size, alignment, base, - limit, nid, true); + limit, nid, true, 0); if (!addr) { ret = -ENOMEM; goto err; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 24352abbb9e5..5ba7fd702458 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3168,7 +3168,8 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) /* do node specific alloc */ if (nid != NUMA_NO_NODE) { m = memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid, + hugetlb_vmemmap_optimizable(h) ? huge_page_size(h) : 0); if (!m) return 0; goto found; @@ -3177,7 +3178,8 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) { m = memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); + 0, MEMBLOCK_ALLOC_ACCESSIBLE, node, + hugetlb_vmemmap_optimizable(h) ? huge_page_size(h) : 0); /* * Use the beginning of the huge page to store the * huge_bootmem_page struct (until gather_bootmem diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..e92d437bcb51 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -549,7 +549,8 @@ static void __init_memblock memblock_insert_region(struct memblock_type *type, int idx, phys_addr_t base, phys_addr_t size, int nid, - enum memblock_flags flags) + enum memblock_flags flags, + phys_addr_t hugepage_size) { struct memblock_region *rgn = &type->regions[idx]; @@ -558,6 +559,7 @@ static void __init_memblock memblock_insert_region(struct memblock_type *type, rgn->base = base; rgn->size = size; rgn->flags = flags; + rgn->hugepage_size = hugepage_size; memblock_set_region_node(rgn, nid); type->cnt++; type->total_size += size; @@ -581,7 +583,7 @@ static void __init_memblock memblock_insert_region(struct memblock_type *type, */ static int __init_memblock memblock_add_range(struct memblock_type *type, phys_addr_t base, phys_addr_t size, - int nid, enum memblock_flags flags) + int nid, enum memblock_flags flags, phys_addr_t hugepage_size) { bool insert = false; phys_addr_t obase = base; @@ -598,6 +600,7 @@ static int __init_memblock memblock_add_range(struct memblock_type *type, type->regions[0].base = base; type->regions[0].size = size; type->regions[0].flags = flags; + type->regions[0].hugepage_size = hugepage_size; memblock_set_region_node(&type->regions[0], nid); type->total_size = size; return 0; @@ -646,7 +649,7 @@ static int __init_memblock memblock_add_range(struct memblock_type *type, end_rgn = idx + 1; memblock_insert_region(type, idx++, base, rbase - base, nid, - flags); + flags, hugepage_size); } } /* area below @rend is dealt with, forget about it */ @@ -661,7 +664,7 @@ static int __init_memblock memblock_add_range(struct memblock_type *type, start_rgn = idx; end_rgn = idx + 1; memblock_insert_region(type, idx, base, end - base, - nid, flags); + nid, flags, hugepage_size); } } @@ -705,7 +708,7 @@ int __init_memblock memblock_add_node(phys_addr_t base, phys_addr_t size, memblock_dbg("%s: [%pa-%pa] nid=%d flags=%x %pS\n", __func__, &base, &end, nid, flags, (void *)_RET_IP_); - return memblock_add_range(&memblock.memory, base, size, nid, flags); + return memblock_add_range(&memblock.memory, base, size, nid, flags, 0); } /** @@ -726,7 +729,7 @@ int __init_memblock memblock_add(phys_addr_t base, phys_addr_t size) memblock_dbg("%s: [%pa-%pa] %pS\n", __func__, &base, &end, (void *)_RET_IP_); - return memblock_add_range(&memblock.memory, base, size, MAX_NUMNODES, 0); + return memblock_add_range(&memblock.memory, base, size, MAX_NUMNODES, 0, 0); } /** @@ -782,7 +785,7 @@ static int __init_memblock memblock_isolate_range(struct memblock_type *type, type->total_size -= base - rbase; memblock_insert_region(type, idx, rbase, base - rbase, memblock_get_region_node(rgn), - rgn->flags); + rgn->flags, 0); } else if (rend > end) { /* * @rgn intersects from above. Split and redo the @@ -793,7 +796,7 @@ static int __init_memblock memblock_isolate_range(struct memblock_type *type, type->total_size -= end - rbase; memblock_insert_region(type, idx--, rbase, end - rbase, memblock_get_region_node(rgn), - rgn->flags); + rgn->flags, 0); } else { /* @rgn is fully contained, record it */ if (!*end_rgn) @@ -863,14 +866,20 @@ int __init_memblock memblock_phys_free(phys_addr_t base, phys_addr_t size) return memblock_remove_range(&memblock.reserved, base, size); } -int __init_memblock memblock_reserve(phys_addr_t base, phys_addr_t size) +int __init_memblock memblock_reserve_huge(phys_addr_t base, phys_addr_t size, + phys_addr_t hugepage_size) { phys_addr_t end = base + size - 1; memblock_dbg("%s: [%pa-%pa] %pS\n", __func__, &base, &end, (void *)_RET_IP_); - return memblock_add_range(&memblock.reserved, base, size, MAX_NUMNODES, 0); + return memblock_add_range(&memblock.reserved, base, size, MAX_NUMNODES, 0, hugepage_size); +} + +int __init_memblock memblock_reserve(phys_addr_t base, phys_addr_t size) +{ + return memblock_reserve_huge(base, size, 0); } #ifdef CONFIG_HAVE_MEMBLOCK_PHYS_MAP @@ -881,7 +890,7 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) memblock_dbg("%s: [%pa-%pa] %pS\n", __func__, &base, &end, (void *)_RET_IP_); - return memblock_add_range(&physmem, base, size, MAX_NUMNODES, 0); + return memblock_add_range(&physmem, base, size, MAX_NUMNODES, 0, 0); } #endif @@ -1365,6 +1374,7 @@ __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone, * @end: the upper bound of the memory region to allocate (phys address) * @nid: nid of the free area to find, %NUMA_NO_NODE for any node * @exact_nid: control the allocation fall back to other nodes + * @hugepage_size: size of the hugepages in bytes * * The allocation is performed from memory region limited by * memblock.current_limit if @end == %MEMBLOCK_ALLOC_ACCESSIBLE. @@ -1385,7 +1395,7 @@ __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone, phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, phys_addr_t align, phys_addr_t start, phys_addr_t end, int nid, - bool exact_nid) + bool exact_nid, phys_addr_t hugepage_size) { enum memblock_flags flags = choose_memblock_flags(); phys_addr_t found; @@ -1402,14 +1412,14 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, again: found = memblock_find_in_range_node(size, align, start, end, nid, flags); - if (found && !memblock_reserve(found, size)) + if (found && !memblock_reserve_huge(found, size, hugepage_size)) goto done; if (nid != NUMA_NO_NODE && !exact_nid) { found = memblock_find_in_range_node(size, align, start, end, NUMA_NO_NODE, flags); - if (found && !memblock_reserve(found, size)) + if (found && !memblock_reserve_huge(found, size, hugepage_size)) goto done; } @@ -1469,7 +1479,7 @@ phys_addr_t __init memblock_phys_alloc_range(phys_addr_t size, __func__, (u64)size, (u64)align, &start, &end, (void *)_RET_IP_); return memblock_alloc_range_nid(size, align, start, end, NUMA_NO_NODE, - false); + false, 0); } /** @@ -1488,7 +1498,7 @@ phys_addr_t __init memblock_phys_alloc_range(phys_addr_t size, phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid) { return memblock_alloc_range_nid(size, align, 0, - MEMBLOCK_ALLOC_ACCESSIBLE, nid, false); + MEMBLOCK_ALLOC_ACCESSIBLE, nid, false, 0); } /** @@ -1514,7 +1524,7 @@ phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t ali static void * __init memblock_alloc_internal( phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, phys_addr_t max_addr, - int nid, bool exact_nid) + int nid, bool exact_nid, phys_addr_t hugepage_size) { phys_addr_t alloc; @@ -1530,12 +1540,12 @@ static void * __init memblock_alloc_internal( max_addr = memblock.current_limit; alloc = memblock_alloc_range_nid(size, align, min_addr, max_addr, nid, - exact_nid); + exact_nid, hugepage_size); /* retry allocation without lower limit */ if (!alloc && min_addr) alloc = memblock_alloc_range_nid(size, align, 0, max_addr, nid, - exact_nid); + exact_nid, hugepage_size); if (!alloc) return NULL; @@ -1571,7 +1581,7 @@ void * __init memblock_alloc_exact_nid_raw( &max_addr, (void *)_RET_IP_); return memblock_alloc_internal(size, align, min_addr, max_addr, nid, - true); + true, 0); } /** @@ -1585,25 +1595,29 @@ void * __init memblock_alloc_exact_nid_raw( * is preferred (phys address), or %MEMBLOCK_ALLOC_ACCESSIBLE to * allocate only from memory limited by memblock.current_limit value * @nid: nid of the free area to find, %NUMA_NO_NODE for any node + * @hugepage_size: size of the hugepages in bytes * * Public function, provides additional debug information (including caller * info), if enabled. Does not zero allocated memory, does not panic if request * cannot be satisfied. * + * If hugepage_size is not 0 and HVO is enabled, then only the struct pages + * that are not freed by HVO are initialized using the hugepage_size parameter. + * * Return: * Virtual address of allocated memory block on success, NULL on failure. */ void * __init memblock_alloc_try_nid_raw( phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, phys_addr_t max_addr, - int nid) + int nid, phys_addr_t hugepage_size) { memblock_dbg("%s: %llu bytes align=0x%llx nid=%d from=%pa max_addr=%pa %pS\n", __func__, (u64)size, (u64)align, nid, &min_addr, &max_addr, (void *)_RET_IP_); return memblock_alloc_internal(size, align, min_addr, max_addr, nid, - false); + false, hugepage_size); } /** @@ -1634,7 +1648,7 @@ void * __init memblock_alloc_try_nid( __func__, (u64)size, (u64)align, nid, &min_addr, &max_addr, (void *)_RET_IP_); ptr = memblock_alloc_internal(size, align, - min_addr, max_addr, nid, false); + min_addr, max_addr, nid, false, 0); if (ptr) memset(ptr, 0, size); diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..c36d768bb671 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1615,7 +1615,7 @@ void __init *memmap_alloc(phys_addr_t size, phys_addr_t align, else ptr = memblock_alloc_try_nid_raw(size, align, min_addr, MEMBLOCK_ALLOC_ACCESSIBLE, - nid); + nid, 0); if (ptr && size > 0) page_init_poison(ptr, size); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a044a130405b..56b8b8e684df 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -43,7 +43,7 @@ static void * __ref __earlyonly_bootmem_alloc(int node, unsigned long goal) { return memblock_alloc_try_nid_raw(size, align, goal, - MEMBLOCK_ALLOC_ACCESSIBLE, node); + MEMBLOCK_ALLOC_ACCESSIBLE, node, 0); } void * __meminit vmemmap_alloc_block(unsigned long size, int node) diff --git a/tools/testing/memblock/tests/alloc_nid_api.c b/tools/testing/memblock/tests/alloc_nid_api.c index 49bb416d34ff..225044366fbb 100644 --- a/tools/testing/memblock/tests/alloc_nid_api.c +++ b/tools/testing/memblock/tests/alloc_nid_api.c @@ -43,7 +43,7 @@ static inline void *run_memblock_alloc_nid(phys_addr_t size, max_addr, nid); if (alloc_nid_test_flags & TEST_F_RAW) return memblock_alloc_try_nid_raw(size, align, min_addr, - max_addr, nid); + max_addr, nid, 0); return memblock_alloc_try_nid(size, align, min_addr, max_addr, nid); } From patchwork Mon Jul 24 13:46:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13324804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D67EC001DF for ; Mon, 24 Jul 2023 13:46:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BCB1900004; Mon, 24 Jul 2023 09:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96B90900002; Mon, 24 Jul 2023 09:46:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80B7A900004; Mon, 24 Jul 2023 09:46:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6C672900002 for ; Mon, 24 Jul 2023 09:46:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2A59340B43 for ; Mon, 24 Jul 2023 13:46:54 +0000 (UTC) X-FDA: 81046631148.13.82D538F Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf30.hostedemail.com (Postfix) with ESMTP id 28BDB80010 for ; Mon, 24 Jul 2023 13:46:51 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MgcuJRY5; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690206412; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=gKqdeZQEfMGPFgS/mH5DSHT6HOKKBpjLeysU9NI8qIv1q1k6JXgkntLj02KB+6euyadcsL /wKMNlIGj69Va2vxNW3uowP43gI3Gyxn3Bq3F+Wm2woWcsSvpRVVIqpFD3iYZig5aMKF/X h3L79p+KtxPsJJYG0WVZzQF3qGDj4gY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MgcuJRY5; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690206412; a=rsa-sha256; cv=none; b=Vh0P7sVNnMlB38LtAxJC1b3Ni8oq7XH+kqAjKkVQdA8tkALk8NtRMhKEm06i8Ro/iZF4gI eshGBvOZkA6RocrnSbi9s4o6SOu3TqKAMr5ZsnNxtgCsbL8NR/bvtQNCtOinpe+D0b0F14 V7kPbdO9V5kKYWZU+cye6DzXwVJOUyo= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-3fbc1218262so41886375e9.3 for ; Mon, 24 Jul 2023 06:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690206410; x=1690811210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=MgcuJRY5CZqiaM6oN+2ZgOrZfx75ny19uB226sSasic7Mnx54MWMiPH2vVvPOLfApJ prGjNWXiRaSzV4+PdVcYOKWYfpjwq8s/M8+afykb+Nc06bZYe/+gbxa1+Be4Rm3jeWI0 8cCsZq8cfyydU59ItSQYEJs5yxH0yKdyeTfFZmwx6fK1dDSSKyPHBvlQzTgukYT3kPUX EQcr7s+o30vHTb1ipJOy2fJMjidD+R8BdV4yReZet1FTj9rlNgXk8HK6ZpTxwwfR3+Y5 MrrxBs3KhFwb0TnnMWnCPZSjTb//F/UJ43wX8Vj0/TqJXnIo2EGyycAVn4dvvn7wm2nd L09Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690206410; x=1690811210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=Qzu2+/G2UstzHrMbRNlGWnJMo8N6bT77T9ppAG5mrKqv1PKkHvtC3YQ/Jd9vy8FNH0 ZnvJNiGAB8iavUVKnD94P8CBGjGFblBGzXCnkJw5dwD8JqqcNgjGupgPlF2pxTzn7jYT CYlTKiorYVNMN1Vbf44E7TSjyKQeXmRksLB8fWIfbjxzSNPOub6adVPUW57/YEVvkGDb CFwBKT9GKHEKbV6mFpoi/I4uX7C4YImhh83jl9TapS965M3AT78cJVCXwDKlIauk/jmu K1uX47gqX2mL9pZHlchxTy5zhRgTptN6+l7fjFPSRaPhfRCrfPh2mlp/m86t7e2hAKgB qY/w== X-Gm-Message-State: ABy/qLaK3ZvQBqLjE5As8fHNunSZNdJOZTxd01uo6EUrbqtS7RGJwILN qe5HYWQJYa+5wU3YTyGWAbu2DofB6otV5KYb/4M= X-Google-Smtp-Source: APBJJlGKyOnJzA0snmPQ5aknmyxWInB37EjZPNgnEdY747vTPxkIvkOio6Xp4jLPbIkb3rasSdJxGQ== X-Received: by 2002:a7b:c349:0:b0:3fc:5bcc:a909 with SMTP id l9-20020a7bc349000000b003fc5bcca909mr7478357wmj.2.1690206410523; Mon, 24 Jul 2023 06:46:50 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:d7c4:7f46:8fed:f874]) by smtp.gmail.com with ESMTPSA id e19-20020a05600c219300b003fbe791a0e8sm10209354wme.0.2023.07.24.06.46.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jul 2023 06:46:49 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [RFC 3/4] mm/hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Mon, 24 Jul 2023 14:46:43 +0100 Message-Id: <20230724134644.1299963-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230724134644.1299963-1-usama.arif@bytedance.com> References: <20230724134644.1299963-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 28BDB80010 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ped8r9kxfuefauf1scgksg5c9ozu71bd X-HE-Tag: 1690206411-896135 X-HE-Meta: U2FsdGVkX18A4tLTEebjIObexQ4nc7K5p1C809UOApf2hCr7Ta60DOi0Zx7fucUeJ+3Yd4XGbJpmzbshlhZv4jvughmKB2us5I1ZB+EHMo9UicanwUhIXDbbZwUE2QMaQg+y1i91L5yk24qynBOLWap7ZmQ1J0jEY9JstWgJv0bk/Y019ccxYH93WZnZ/fM3NwbdrJyH8LB8fbvwHoR4knejOa9PYIP6dQsyN7GboDsmmRyPBRrkfvz7ygpJ+/4r1Y4xauIxnrM1CF9nUUORNxEXjEIv7tPE+Gm21znC5pwPkXgg5W130mX4nQwIOYPyp2Vsv3OBqPToKOTeQER5uIv8KfD77jEFZGzdY/0P+HGdUWIzzxY88y63wPdl4us6av1ycFDSVpEOZzKI5im3svdDFBoJIhWb7MqZCm7DyrsZLb/9uq60e1386KgSZgtsMGdGoi1rR7mUkXrP9UKYvQwVraS4gNMk1x2wG1zdOzm0RFWKNHi4GD5wMy1fRCn4uB+o9/dGRpDaLfFP4QBMzy1MbrVhzr8nnpkgiqvCg5cYR44Jspti8Lty6m57XoLCfjfKlTGVZcgbrG6zpr4hAjZjjwPfnq0TP7/Nw3qy+1BIcKlN6QclKZ5wOBtTKtLfd1mfCE7d8yweq8ECUsXxdWpO1fq/UMN1q0AZpVUvlh58WZB8u9DadjMrdOB/Yb/fbeSYHrFlv//Lc0Pa1uhNSytCQsuvGlHc+UDATdJgLPFsXukpro41Y148ljBN37zB2jQkxDtzcS1AUi22traKQs/fMuqGnl5Iyt1RH7XRvmbX82pvD+mwWmtLZ5Zev02LqXd893HDSqljKb3bCqd56AnJOG/8SkOMO8TEbkVuSEnz/ZbRHljubQsfxYRNk9BPNVBB1QegD0ZqyH68W7PFqynRb8QTrgbnQPMl0Z//7DM0YKmRxb8mgYA7RVmL+NF7BoC1iMvy5a4bnEzfUlG LZtCFmGX Nx0zw4WC372EY7izuvkKisSamL3oFtZyeX/TFyQbfYCN3QCHc4lME4BL9Rj2ZxOa75S8DKvuIP3I6iE17pmqMysatT5vuP9/d8Z09nI5ucvli3dDO7yql82h+R2MMRH00LLn5cGSVPe329Nb3Fmw39q7SwNuvFG61/4YYyK2cIwSy/Qjl++Daq5+Lj3DrrMqJGuV9yAZLikLTu5sOLZEubO/tdqNLDyubHrfEf2QeQe1f+PUvfaoVxtUVHq7ZyCIZQX/z7UcwuVnyWXa1u+QCudNVC1v95SrCfk7RY1pDFSUNsSE+MxnlmmtKfKClPQbLXcQCSiZW552hWnsFXkwOz42u3RJxZus7y7S3O9SRzXedfi9KXYIqDNP+xnKuDGw3Dv7xqFChVTJLp6aQ6+yoJDj5QRTS4Jukxs4dSPOlBmKoKiiwpEIx4zg39DCy6OPXIgbFQ7b39UMkF40G0N+y3U5XoQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000095, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Mon Jul 24 13:46:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13324805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12D6AC0015E for ; Mon, 24 Jul 2023 13:47:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB42D900006; Mon, 24 Jul 2023 09:46:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4966900002; Mon, 24 Jul 2023 09:46:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B41E1900006; Mon, 24 Jul 2023 09:46:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 99BBE900002 for ; Mon, 24 Jul 2023 09:46:55 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 37331C0A97 for ; Mon, 24 Jul 2023 13:46:55 +0000 (UTC) X-FDA: 81046631190.05.4D31669 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf12.hostedemail.com (Postfix) with ESMTP id 4833740015 for ; Mon, 24 Jul 2023 13:46:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=U9TJoPZM; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690206413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l4pGELgjvEXBJuRM+kyS6zBxvRB1Y0OOMkMsDX033Vw=; b=GdJNUJBXCl0c04WQ6Y2vjrKvkg8DjL3+mjoRTyCJJ7vst2I8gGC35byx7O6Qw9Iot6ZhIg 1/pIkfGhAEdE66X8HRgsiX/fbktuj4yNrMcspXpskLgHaFGItVKLKEZa7Rbbf6iW3tWu5V nv2GFuytybM1E3XizShWzkaKzJUO03Y= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=U9TJoPZM; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf12.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690206413; a=rsa-sha256; cv=none; b=SBManTHS64aAAiOb3trPOmqYRtRtIQRu+ENB0SVXUHnQlEVX+2U+e/onH15U8l4XnN7NHD 2sZ/YmU5aNwe/eKkuuNgcvBLj8OG5a9QXLQrD11UNPX3UuvI83mXYPLTjkfLBfviXdKP0K pCkbQlJcCM/Pqu52TN13GDaSDSKO/Xc= Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-3fbf1b82d9cso34181315e9.2 for ; Mon, 24 Jul 2023 06:46:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690206411; x=1690811211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l4pGELgjvEXBJuRM+kyS6zBxvRB1Y0OOMkMsDX033Vw=; b=U9TJoPZM8Vb+e/+TDxZOJ/G64+DOvXK8duWaCY98EJ9LR85s7Ir6jydu8BZa479KxZ jRykc91Lr65/WdO14wqIbrmESX+QVJSfPKmbkz08W6PegHJYRKhZsHYX5/ZeohmFO5I6 ROm+xlqvOa/+KqvRaMncQyPQY2le2+8HqRa+EVD/36i0+8/kad2mKe6z940vhrF7rJAO MA+rdL862CQeWZy+wv9ZnuKLEUaUrm6Ho1ie1q6HfyD0bJpWH4Cwb1xB+e3/cLjPXN9C v8pFji+l0xoy/MHWdK9ePxYNKHToDuhq5eys6H6aN5bmin9SIzYcVj3G2fhYuOBSlPSk kY4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690206411; x=1690811211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l4pGELgjvEXBJuRM+kyS6zBxvRB1Y0OOMkMsDX033Vw=; b=IGDQgjhJAET0XbZ1913+Esm7XLF6iElNe5bTuZeGjr6Qd8BUKRwl87MQ6Ju/xM/S+z K9ZrxByJQI8+pJopUEftIjw9YqXlZxubl2si24oHQdt2DLO+UGacBqf7qn8M/AKdM/WE R2YMTJOE/zlI9jVl3MTH7wjHdrgwC2T35Vov3OZnj1N09FKBBR5PIn1XdlSQ3pArXrfW XVm45XwpBpw+zkdmuPExpyi8n39hjJP0dEW4w12TEFJPBVaDuLdg6tLiUJp0NQnmbshr 6sgtp3CA9N/uBI/cBbIkKYB0Z7I6pj9X7kYSVyK0qeDgdUdjgxcP9ES3DNNWesmKP7nb O1Jw== X-Gm-Message-State: ABy/qLY2YtLgskMBjVMGN/+rlwC7541cat6qgFmMDUfGllhayccKjfuF lWfTnupfiqHK7do6T3+i5m8Fn4tzXZNS7oUu+yg= X-Google-Smtp-Source: APBJJlGC9ngrxmvPhoLL2+ZVdDO/kVFluie5EE/bs+tXQXKrVPr0EB5rctNwlrUxp2EATgtzWXfzgg== X-Received: by 2002:a1c:7703:0:b0:3fa:d160:fc6d with SMTP id t3-20020a1c7703000000b003fad160fc6dmr6581149wmi.30.1690206411484; Mon, 24 Jul 2023 06:46:51 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:d7c4:7f46:8fed:f874]) by smtp.gmail.com with ESMTPSA id e19-20020a05600c219300b003fbe791a0e8sm10209354wme.0.2023.07.24.06.46.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 Jul 2023 06:46:50 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [RFC 4/4] mm/memblock: Skip initialization of struct pages freed later by HVO Date: Mon, 24 Jul 2023 14:46:44 +0100 Message-Id: <20230724134644.1299963-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230724134644.1299963-1-usama.arif@bytedance.com> References: <20230724134644.1299963-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4833740015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: aicmu8pc4r6kg1f1dsohkybzjg3xpde3 X-HE-Tag: 1690206412-314989 X-HE-Meta: U2FsdGVkX1/06QP8+BCm/X+yUlpfFQKTVYk1fXoaMVuPPCXGK3kEmKG623CbZ/piiKfOa6GkAaoim+emGi/gsUH4HFwS4f/V3UprSRY6hfa15n9cWG8IBxopo+keGF6g/htm++tWsc1+++nwfKH+huY6hKyp93IIak53SzumPul6n4JevzikHmD0vO1GYWTonBM16t5cNyL14ZQlju3bRTKgQsIW5VJ69wtjIbqfvw7zIvXo+iz9zfF5RGYWRl5CEVUOc/X6XFsXe3nwTGNWQfu5BY5o8X0TspcBaWHTLRaWfNRoC9YvpRnyPDjB/5HYckASe+SMa47VLxW+h2IwK1xd7L2CIK4YuQJ64rVS/b1sSX9+AwNzZhUPNPEqxZiIO3iyNm7XYZcsFvwDHR3rSduczP4YFBO0Uf8MBPJmb6ik/DzAdChHIsONO4MRE01I7oJ1Kj9HRsMe65uUeMkrr6f6rLQq7agU6wd2orM4d8d8Xp4U1ry9KAJ5pgbdIB2B/jwmM96pvqvSL1HUF2sfmdnaJ1EHWovbMLP1Q4L4VCIvBgqyQN60Maf/P/Ov/p0/QiUZFAD93iQsbqYjaqsDMM3GjspSHWmyPUn1W6r7YgJHUxaSiKkFEWINyW8AO9y8hGClMTuNdhWMQ/1hAbgZN6iqwfMx80q8Is5w2GZT6VJdnNMgAt7YmIepfw0V7qK+YIfJh4iz2BOrpQgAjZFmxWFevmZik8BxCAATX0iC03PcWsPF8XAH6U/GebjPZpzR94eAqlhBOG2FICUW2dOfT5Fxu1KGH/t1PTTv0CqMZDeeccyNUsz5wgP6GZbHRFOkmV0N/yvsYrVev9vG386Z/jdml/brTW/OEuis32WLph8ZGADVxxkH8skW/7euaf/rLgFyASYVHbnP+f6Apef1rFXkqF8dy2FMl9Wg8PI3iTEM1DwjRzno00XaTaskwVDyZi5bVlr8RjjVYFoB2Jo OqwmMBTp QxHgpM6XAiygixLRkv5M7XWO95zglCGobY6TNkSU36jp5ylnNyJsxdHPuPLbpV7GQECOn8YNXkIVbu38iRqvcrwR1LNcp3UltRxT+58oH23t5AleAfGP2fJp3g3ZmKYe3Rn81NUfih4+CmPq94KzKieaDwLhGujrNYJqUmZZGEctonfcEt5lM5teW6n9lB9hY8BvK4U+SX3AnbkOZ6e7UN1pAPLZnNFULMcR3ofCQuOighp8eHnK5CLV4qSk4QyQYmQ4nCVssJXj4T0x8+2Uck6BVEkKzY4xU7YQv7sFFMDSuzPXkdahZR8Nw5uLDI1GVxpaPm7jMC/CiClRJD5zWs/xjIfs/Vue3KBNOtcDpb45arhEdCa5S/iWANt6JerfPk/rEW/j3tygbNoriwPzgVNWSLg/EMAxPCifXPA/nBJuDOmQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. As memmap_init_reserved_pages is only called at boot time, we don't need to worry about memory hotplug. Hugepage regions are kept separate from non hugepage regions in memblock_merge_regions so that initialization for unused struct pages can be skipped for the entire region. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ mm/memblock.c | 27 ++++++++++++++++++++++----- 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3525c514c061..8b9a1563f7b9 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -58,4 +58,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) return hugetlb_vmemmap_optimizable_size(h) != 0; } bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */ diff --git a/mm/memblock.c b/mm/memblock.c index e92d437bcb51..62072a0226de 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -21,6 +21,7 @@ #include #include "internal.h" +#include "hugetlb_vmemmap.h" #define INIT_MEMBLOCK_REGIONS 128 #define INIT_PHYSMEM_REGIONS 4 @@ -519,7 +520,8 @@ static void __init_memblock memblock_merge_regions(struct memblock_type *type, if (this->base + this->size != next->base || memblock_get_region_node(this) != memblock_get_region_node(next) || - this->flags != next->flags) { + this->flags != next->flags || + this->hugepage_size != next->hugepage_size) { BUG_ON(this->base + this->size > next->base); i++; continue; @@ -2125,10 +2127,25 @@ static void __init memmap_init_reserved_pages(void) /* initialize struct pages for the reserved regions */ for_each_reserved_mem_region(region) { nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; - - reserve_bootmem_region(start, end, nid); + /* + * If the region is for hugepages and if HVO is enabled, then those + * struct pages which will be freed later don't need to be initialized. + * This can save significant time when a large number of hugepages are + * allocated at boot time. As this is at boot time, we don't need to + * worry about memory hotplug. + */ + if (region->hugepage_size && vmemmap_optimize_enabled) { + for (start = region->base; + start < region->base + region->size; + start += region->hugepage_size) { + end = start + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + reserve_bootmem_region(start, end, nid); + } + } else { + start = region->base; + end = start + region->size; + reserve_bootmem_region(start, end, nid); + } } }