From patchwork Thu Jul 27 20:46:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A14BC00528 for ; Thu, 27 Jul 2023 20:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CC826B0074; Thu, 27 Jul 2023 16:46:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77B726B0075; Thu, 27 Jul 2023 16:46:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64A0B6B0078; Thu, 27 Jul 2023 16:46:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5701F6B0074 for ; Thu, 27 Jul 2023 16:46:36 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1270D1602D5 for ; Thu, 27 Jul 2023 20:46:36 +0000 (UTC) X-FDA: 81058575192.19.0A9CF81 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 35FD61C0003 for ; Thu, 27 Jul 2023 20:46:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Qh4G5pkJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=Ox9e6A5J06C7TlBz2SQgn7AP7BbOoVN0Ctxe+KXEw8sN5j3KQFRyPoxD7pKYB8LxPeGiK/ 5YM1B5fkv/BwEL/7WvwsHw7OMLY8hSIUmEszLsU1jbqvpXlSIWCZkKGPy3XAyHGsNOI3QU ODibnmywK4AxC3wvjDZN71+nms79dwY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Qh4G5pkJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490794; a=rsa-sha256; cv=none; b=5c8soK3l0YjkPPGQXhPt3y6IJN+hhDZIZYA3rBAGYga16CXs+1AA6qKEG+0ox6gFMnNg04 n0a0ussFGHhObzRpIYa4XemMIMcHyCWtNoaQ4A9Uln7usKWnCF9Kcx+yEJ0jsVF07p+kKE abJjv01BZtje/F7LfZAttvnTMF/2axg= Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2b9b9f0387dso22177481fa.0 for ; Thu, 27 Jul 2023 13:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=Qh4G5pkJ2CklC1dn5cBz4RJXwctIF25YGEWE2oQ3ZrOzxnKalFKae/8pznmb31lV7C WTzPyronTTWL8g0r5I8lSbz/AdU7gvJCGq8K2Qd4vWzRvFiQ/FfCwXl6dsQ9CmNizTh+ m0AH/lVmLiMfUecg6qsdvv9XMrT/D7wOGOTWcG9Zt50oRSe1Bavo9Ko7yaB4wjJ+Qnnr BNWwV0vvSuj9U4WuwZrl1x2sIhyQwELI33CtL1QqIzp+Rtzz205zsveFrvcH9XBhLeZX G7lPHvXhJp/f1c2Jmw7kl1BPg6BGcbtzSnM+LE3IJIaKxCmCB8OfQyprqC6Rh3SHBpsg Wpxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490792; x=1691095592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oaqNw2Hm1guEkvjK0qvKkO1TrGxzTe44nwo0h0OS4xY=; b=IJMKCnhPKMUa1HMl36PBLA/DfUH5P6RDns4V3Fvoc8fDRGwzkgxqvl3MNenZh9Mf5n AyxTpuFdlD+ep0aZ+5tohLt/4NFoEpjbMcF1ELTr2bN0nssnNbafuEc5QunSxlNd5VMh Ij9DNS/7H10hpRztU7xYlLvqk78Hj8E3FqBjZQFTcCRzLKUq4IWzqXOPOOV3NZ5tvuy1 OFbEN4t4admf31HW8JZSkSGT7gaER3cXI2FmQGc1xFVRf5SlDZTXnE10sDg/V5ajj0l9 r0XXegbXg3ouSpYQ9vNQNKvkFY0wbb7xlmj5Z/IM33Xe3JMFlIY+SxI2fJcXdRReHJx+ OzMg== X-Gm-Message-State: ABy/qLZLcqAo2vzBl1L9jQN3FjpsQKYGMIMpz5q8AnuxG9mRe1U15OGZ duA2VwWEOR1eFsvVNg2JurlDbNjR4R1/akfBB9s= X-Google-Smtp-Source: APBJJlEpvm0P4NAPU6V060zctU8cT38iuWapH30/cDkVR/gUSQA0rrC4oV9wP9Dr8ocbwFl306QznA== X-Received: by 2002:a05:651c:1042:b0:2b6:da64:321 with SMTP id x2-20020a05651c104200b002b6da640321mr95486ljm.45.1690490792019; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:31 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Thu, 27 Jul 2023 21:46:19 +0100 Message-Id: <20230727204624.1942372-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: rx7h4ba1xbfmjzn4thjnc7w93k1qcm7p X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 35FD61C0003 X-HE-Tag: 1690490793-689274 X-HE-Meta: U2FsdGVkX1+3Sl+zD73F/yaP2VcGcn24C12cZRnvgZLz2XeWIGnTDjBrlJ6OvOXFQN1TZ7PhJfuu1434MAHfs2gqvendxZG869hkmwlwVod3Ql7POOwzOCcQarOAhZXnjwHOyvXRS9Awcj6Y3ftkCV4lXuku+9mieHvqn6j00ja1NG1pzV64dx4HvKUOsU+ryx7UpQfRcQc4TJsndghxCX/HAaUZDijV2cgR5QURUPgeP2osy9c7kO+wSVi4rJEr9zbu+UGrh5d7OXhsXKgPk8h1fTVTXUsVv0azujjIE6jQ4tpJ5ugpbFs7TET9HfjiA+LrJdJ1T9PxoYF6QpqVdYaCFAKIkAngrdkmDDPWh9SEqx3G5k/V8/NH0nPSuWzEUSsWdbWbzKWd3XpVanU8WkAs17LMyvKamO/n+X35OedyjTKk26PRcTIftq1UwGrOS5tqc3z0pdsfDX+I90r3bc91Yifs8TAM+jUkD8QlFhTE9YCTnWM6Lpco2G7vQXzNIrcJN5tsmRh8m1HC8NEJQtciBxj+RHJBZnGiHpQQWh2anKV+ZaiQQwcyPceoCrfztbf2KGdyX1C6X7jkFEJWpHZadf5uJZKwblF1FQrbG9tk2/PJ409V/GLmEvsrKSP1zUxWDXb6tISGP3cjdm3QB5lTXMspax2wCwqPO6Tfqa0iQZZDXIN5zSqL+Qd32m/5fLhExKGALU0KXEVJLoWYXeu8xNxubCE3j5YH0FB25mE3opydWYDgdzDcbsbaTs4gknfl2K4kMNepUg27JEZIZXBgJrb2Y8WFDYboJTB3vDi3h+1y935IOWCB9wV0U9kcvfyxx8fZNtqGPbwdhKhLNvUZlxoNmI7Z/VpigFdSQwJaC6/Xg5/U/4FINVXNjxY1g/5nW7QDdvKlg6qGeyQjX9r3LGGTls2wbnvHI9sM/xl/taLX87DQguoTe9s8MfD4rjrDUOs5Y5riBMm9lHk qHwqjfZW XmGVPeiizaLO5TJnlV06WSvzlVcYBqdHYFGedknHzyUsZKHpoM277FfnADqtmgfvEjhe5CpWUfxJhE0IqtTJnxpw0ndd1Roex4rQP/rprIXdEJDjrBlLReBV0WCQBHogTvR8JzmsK4n/dtJRkJ1iOId0yXYiC9uw5i8QRLleaea+JTen6ywTE52dNWS7qKq8H1tQDIaLPpqZthIf84FzupxfjUpH15wf0EI0cxJ7s8UzW1glQszcNIgleFnjliF3y7i+UTi76xpK7uuAmvFbcTIaNOFwjl/R6MSIvkl30GTbzOYB5q5t8/WkXBnlLfvOu4lIf+KicbgJGKYug3X+8a5Ji9/raiY0LGOJzbVKW7WqxGRw7WqHWor+y4ovc8MhAzmxETI71sHVHBQ2JdDx0BbZQ0iAUz2KVswUu2zUzez3kT4M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 = 262080 struct pages per hugepage. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 +++++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 7 +++++++ 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..58cf5978bee1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1943,13 +1943,24 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni } static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) + unsigned int order, bool demote, + bool hvo) { int i, j; int nr_pages = 1 << order; struct page *p; __folio_clear_reserved(folio); + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_optimize + * in prep_new_huge_page. Hence, reduce nr_pages to the pages that will be kept. + */ + if (hvo) + nr_pages = HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); +#endif + for (i = 0; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2020,15 +2031,15 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, } static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, false); + return __prep_compound_gigantic_folio(folio, order, false, hvo); } static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) + unsigned int order, bool hvo) { - return __prep_compound_gigantic_folio(folio, order, true); + return __prep_compound_gigantic_folio(folio, order, true, hvo); } /* @@ -2185,7 +2196,8 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, &folio->page))) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3213,8 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, huge_page_order(h), + vmemmap_should_optimize(h, page))) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3637,9 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) subpage = folio_page(folio, i); inner_folio = page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio_for_demote(folio, + target_hstate->order, + vmemmap_should_optimize(target_hstate, subpage)); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..07555d2dc0cb 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate { return 0; } + +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) From patchwork Thu Jul 27 20:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E0E1C04A94 for ; Thu, 27 Jul 2023 20:46:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 646A96B0075; Thu, 27 Jul 2023 16:46:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A93D6B0078; Thu, 27 Jul 2023 16:46:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ABD66B007B; Thu, 27 Jul 2023 16:46:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2DD116B0075 for ; Thu, 27 Jul 2023 16:46:37 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EF54C1A0413 for ; Thu, 27 Jul 2023 20:46:36 +0000 (UTC) X-FDA: 81058575192.11.774200B Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf26.hostedemail.com (Postfix) with ESMTP id 1BA2F140005 for ; Thu, 27 Jul 2023 20:46:34 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZZUlyW4Y; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490795; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=vypgaY3fUee11okUBYVJS4zMPfkDIHXhPcba1NC4Tx9e0BPHzkOXKAwqTfWrfugNhJalLG 1jwvMmIhIfZs2NoTDovSJhHs1vSEZ7xKNvXPWXlLUI2mcuX5OQlBua229c9upwj4bdHmw0 vFw4JdmpzQ37IBx/ZfXMCqm9if4zN34= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490795; a=rsa-sha256; cv=none; b=XaLA9DdWGmiRyvzbFp+6Yc3lt7bkQQ9me/F+84YJFORKU4tPnK8wqzEJ+lFf4vZuIUD3ui mgbMMxVT35oNRJ5nN2u1tDfhguaYDVhThAmZQsCq+re9/o9WEm3WH3UIiSWh9K6gumOwuF m0C109jM0i+/hxH6I/W4wYyY9onIJD0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZZUlyW4Y; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-316eabffaa6so1397691f8f.2 for ; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=ZZUlyW4YzysoCHAYMp2/sA6YrKmTeac3t3azsTg4jDgno5140SsxzboZ2DX+sYyTar QBt4NCcbIwVqU6ECOkAxzfNVYTJSA4UoAO9vNAzRgOzSciNduCZzg6hsTy30yZygE/FX Y3IhdC9ei8B4xOpa33YxJ+rAYbRMQmfBhHWbkpR2MzbCVvhFxC5cl2XdP9qY+d1bHhgd lqfn5I8dfHTphHuAk/OGTexXituKKGFsXZtEGV8Pe+xS2H+P9mt/hFkl6RI2Q/3BRFmT 8QbfWy8NIL5JwByXBCIWXDjISwNspvj4RcypyTUIR9HRGShbKOG5H+xfqxUKSelybJ2e i7JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490793; x=1691095593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=Hp0AM7ktdswWSqeqeH63zTT4ofNQ7nVn65MPuEvebdLt64L1PaFNXHY+a4fwOOTwe3 vQmvsI6po/0tAejL2FIG8HQ9ugl4+zEZeqeFTqWlmTtZjWuVetlnHCLaM2tc/aLndILM pl3Btu8QK9+IV6HoVSArrwE5bKrPAo8tMYLM47azpTiLCIkjLPYdzclmDFzwR98tcMoE KGwehpn1UebWaWBDhlDA1ZOYO7fw9vjawAvg6p6QLpaK6+nMhsO6ZqPc4xnDJGca0pWv P8AOxy5LqM0zF7ScA0fWfi323HXMozQQjRJCCWwOSxpFP+TySANtxHdI/fP90gFTm4Ip jsFw== X-Gm-Message-State: ABy/qLbDyMQCUqSgebkx96Ep6L4FbXDyC8f2/IA8QuKTeUNC45Cjt4dP m0mbn/pUy/I7bSwaYoxXaL8lHkWKmYE5DdkhkKg= X-Google-Smtp-Source: APBJJlG66jp7ldTsiVfnEBvuqVAQDlHt8IOfvNpmcOCKQllr9otAKlvj+wF8PQJddxyevra73/5ciQ== X-Received: by 2002:adf:ef4f:0:b0:314:32b6:af3 with SMTP id c15-20020adfef4f000000b0031432b60af3mr204498wrp.5.1690490792997; Thu, 27 Jul 2023 13:46:32 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:32 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Thu, 27 Jul 2023 21:46:20 +0100 Message-Id: <20230727204624.1942372-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: xozfffutcqipnp6koz68dqd7ysii5chj X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1BA2F140005 X-Rspam-User: X-HE-Tag: 1690490794-206951 X-HE-Meta: U2FsdGVkX1/qKMroc9LJgdBhCocMwkmgMq0V7GVV0tX4NON+ERhg00s9AEctC4GdR7rUBU5IvDy0kwyuTZACfW5e3JdASsczZC4CmepXfriP6Z3TAcVY/UXohD+jzK/TSaZnGp/ir3iuQpcAjCHkdcGUOqzU/VDtYTQ0lvcu4YmU/t77gueDv5B1Nit/3iMNnGCojIkvT5q2iVMNYaEkME5FPwNrMkZR5RQeEwaem1tdMTNaFHkNxYASCIcnR/k9PCG+QO1Q9ws8xtQ1QNLhHjwEqC0UgA0+O+/BbRdF4e/zu7/0fxLtjp8WWUFUhUiQtfusYK612Vw6QDiPqgiK4pr6M/sAC7sfJLu8v58are3HK+Hx6sOnxgwetCbVvOX985s/HzUNf762b+JQ28B3lDPbc6Vn3KcMcBBmFqMFYr2UuooiwClkRGGog3hxU6VQSmEM+Sr4/+OxY1vBHuEnkRXTFZZDixz23IKV9pKFwpDsIjEEvcs5FVXQVSCwsohXjqDtbRkJ0Pb7ulP7hllMxNjhdzscNK6inlSZBHupKsbOXv+pPHRXJ00eBdt51RuQ+P64aBG95u+VczpCHwHNTBsOLqWrfomf5xPK2qFkZd0b0Kkewwe/S6mpQNHw9Bdrw4OT68HpW4F+GGz359OBJ02q3Shw6ukvJWamx2rFREUVbSa/Y3KlZlaP3okdCWqbvZgqMV0mrTVLxB+i1CRr/5yz8UZ+irAmMO63KgJAj4J5VnQP3MkwrfYndi6cqJZAEbrUfipXKsChmo66H3D3zoX9WitWf3T/rnflZor1diF3fkqBF316TKka2IggA+ctp+dR535MXA9ipSUioGnwDv6sVFjnWrVvJcL458BCOSy0tnRVCZxQtZ2EYJgo0QLsFRPZHtQ6okr7wOPhv9QHA9IcWL9wIW4I3Izg5Gmht0AcDeL90+8ygqJm1kU/ZBgsxSnxaHRZLagNaa75Z9D TK4IpR3j ISows3B7BtviLa4YHmLno5aKcPFXBS8ZaIQjb4LzVzp9B3w2qzLXiiSqr8aX0CsMbf6dI2HuuuoYM6tPP8fO62o1PdnWw98d0QsmsP3GoPV+kjk1xBgqPJ/eVCqrhszlN3lpVkYYDXug9KVz/jkNP9X8MQODKFP8zMGEw0AMn37kYnUEiLWPTv/A+YwwqZVqHcISA0GBwG2E35VuGY7rDmfoBSd/dcycY8NPQLYFe2ef6Qydn/7JryLaNmUrGDrTt1tYSGZRe9qBie5cppZmtOwpSFsMHCeaGayid2oU6wEVByMWlShgL4uiBZeXqflUPRFoE4KIL8sM1FtwTMqxv9nmITuuzhnzMBRbMXaE15ggTTDWKuZfI0U+cnvO4i+Ma9Jw5pqylh442L84RI6RyL1ZuL5Mt14C8XbkrEJbylHMGiDreKPTrx5B/rC/twa42cTKP9SeJyPEc/Hj/q/CPvfleGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Thu Jul 27 20:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8693CC00528 for ; Thu, 27 Jul 2023 20:46:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97BBC6B0078; Thu, 27 Jul 2023 16:46:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B9806B007B; Thu, 27 Jul 2023 16:46:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 66D5C6B007D; Thu, 27 Jul 2023 16:46:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5882A6B0078 for ; Thu, 27 Jul 2023 16:46:38 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 23F78C0378 for ; Thu, 27 Jul 2023 20:46:38 +0000 (UTC) X-FDA: 81058575276.04.9426962 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by imf03.hostedemail.com (Postfix) with ESMTP id 230F720010 for ; Thu, 27 Jul 2023 20:46:35 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="BuN5xyW/"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=4P+aiaNG7QjThvkHmooYk7/QhfcCp9AiZTYlopsjh9rXBurzQi4pNGjYcHjPyLtD8gPmfF zMphq5LrQKrKTZZviooPBkUd6VW6BwevzTPaT0MIlYgVBwjAn6Y3VJZBnC9ZZQxCEMM04c 2yan5qIWqrNyULICM/yO0o/lJpgZXcw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="BuN5xyW/"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490796; a=rsa-sha256; cv=none; b=r/LD2fzbkiCZ7N1ncj+AE+TsGmo0lGGmSRxHHm6IJGIgxQ8+NPU3nx+KYVvlYEorYPv3X/ TzO1imZf/FBsKtjBoRa972gL5v1GIHE8erTHgQ5zNjFiki0vghx5rmcpHhti6maHPKMtAS c6hVC0vSU5PAZTmAjIaPPtq14IbPSAo= Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-3fbea14700bso14132795e9.3 for ; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=BuN5xyW/6weGNXRlIjeHAIrAgftiJaDdwcAFdinE25QOj+gk0KxhfbPi6NDrWScFMY VMJS1fp/fwlHj4io7fNeedGnL3Vq6LPtuewysLC8Sjo6xuSSwb5C7A3QVA6pUp3BuGal peUgBBXdkLuyAG0r0MtOAu8bJxWihdBSiUkx7+JUoxmU92FDCSc3+a203D99Te3djvrE GFjNEAyNZJnsGcpwIVLN7dd8/TClS2qfeDsn0scV9uZaGKj9iuXLvZP5q+GVR9jZNOl4 uoPcfh3+yJsGI/CKwLu45iuV0qqJWtVosbUaOpv+ylvrytQ2cUfc7YMZnclJk3oiesVt HmMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490794; x=1691095594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R/2kTGP5bmExpEckq+8Rz1vqcaG4uRUICK5uorcV/EQ=; b=TbfUAppZvZdT/Dz2E//q52WGCRWCeuWGNjtLwSx7yApkfhvbUnTTlTc9msYRBxwYgD Uk0h6RniC4hMyrblom5bACdWTtQEeWltJEZReYAwabO9PN0w+7PLL9+mQ73hM+rgVsL5 wcRcljlazZg13cZbnQrBJHwWbj/SzFZXLb4RDx062piCzXHrrbrsJkdeU9IJ00NC7k/d +T8w8gNDc1OL1iSuf0Y8oIoUp7dt8rbIc37QkOXaHDxAsBfKnIpc5coQifZ/xnlauWax Ma/j9w2nAYfMlxcrg9kTmjZZVdmzJShCbrwvH741Cf7Rj0t+pB70vB6WAiAYL9y7Qe5U UeAA== X-Gm-Message-State: ABy/qLaynQk0/h4ydsrx6CcRAGqEpklmtLS3iorsupAcz1iKpYG0JeGo aN/3DxswIFeJRpNem8GKBn/QO0AoU5HiqOtiRSU= X-Google-Smtp-Source: APBJJlFsQ9jv1pf3Fn+r5CYo5PNZoF1WnIrPiX/ItiAceHqurCAQzDyFXe3uUVsJ5LM8vIz8zQ9d8g== X-Received: by 2002:a05:600c:21cf:b0:3fd:ee50:d6a4 with SMTP id x15-20020a05600c21cf00b003fdee50d6a4mr141098wmj.17.1690490794119; Thu, 27 Jul 2023 13:46:34 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:33 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 3/6] memblock: add parameter to memblock_setclr_flag for selecting memblock_type Date: Thu, 27 Jul 2023 21:46:21 +0100 Message-Id: <20230727204624.1942372-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 230F720010 X-Stat-Signature: ijqmg7tzfuyqzzwcfsugcep6qun76bmj X-HE-Tag: 1690490795-559953 X-HE-Meta: U2FsdGVkX1+QeaycMItyDEU0OXprhd3Pvgm4gfFe5NruqAmmPfozXcGuZhcppcFkA+n9T8g/hLuZm0xcjZ6XlHgNzflUmKHkTuHNXmsFfEAmZ4Cl75nr7kPn7u6fC+dwA/t11mYVmdrDuO3tCzDw5JKelGzhS5OymjEaOqFWMLoifTMfplmcxtW35bN8/N6RbCUpr9BwXe7OFLrpQPyAQtgyzZ4MB1LJUl6LhILpsvBegWhEZJ+AK2rd2D4njKejzAjWBUgSgNNaXoI4czJnm3B7kzZQZ4ieZPZj04okN2OYK7ojq9IuiIrC06VYEDVjslJDcSML6UW5KRdt3iBPBD6N4YLjf6Wq7dvnWr1CcceCbga2Oi2GpC8SoMIzW8F6QHtUzqYl7p4PNIL7OK5VvPpNGpd+SAc2g12bKuEOOE5smZwflFtcQPtD2uMT3dAq/GEvOHGhj1DtsAhpvkWsc9joe4Z7jbv4QE3Uke4S8Dgv6m7NjIzREqhT99ETRHry8PBzOZ/hl8odW7pN+CMsqG6FemHOt+xnsSdPWPNBKdrr1c2qotRSmQL4zJbNRBKNR0jLdbfNaVkK4NInWGRakocmqWix/OfaT81F2zPuurKwD66noT17ytfn+szXf+TtDf4EsYKqPI11nUAIVpKJp3uL9vG+A8KWQsDSGWWwWKUfarUvaGa4HNZXR+VabUJ69Dt2ETrM1VYXfEcAxc9BYPL9oJ1gSWreTA8RkFP4BLaDCGoKYvApGZdp5r7emWYzWzsYmpZvA4T0uYVlWhj2EK1HcpVvKjZ5Cqs5xdPS7opXrRP+ABE0I3KEL+tjFRfQlQA5KPj0/fF1KQzJr+EgY4EQ134yVCm3DKa+PfbtVTK2U7OAugtg9u4C/gNIQ14o05dXXTnfy897gpmTReU+pXZiaRzkh55mG6w9xvm6k15Y800m6CUkUdHeMeTRNatJshN5QxK0pIGMV5NRsUX qLad38FK IIN7XuPFYQN2T9lUf1H+MdwTDCrX6weTP1XEDexcSciZRwwW5+tdsYiHsbeYJztujVzuA/mJJQrTkyFpLJrGrr1ubBg1w6c1J3n2N+ZgaE5TmKnj+5I/UPp4ygRNCIPiii+MqFgS7yr5Rp2I9aDeMjEirxdOJz/8cO9s6+M7GsFZhqe0jFrK3tJaRFSlU8U45IKOi28dvWW50yrWRLLkeHbtpbGxaibRv2w1gRnwcE72hDEDzpOmElW6oSJmK3KrUCsB4yKAv53dKXzEODWQL7IxfblCEfELL93+lqHphcCHgBWlEHVzlzVN9nswsFrHETFvybc62z0tM7GbT1vdRppIvkr7zj1kwxHCTjtro2m6OxqM8hosv60m9gqzZPlx/QDULgsSbi3wyd35m/AQwmMNxk7OBKfULSBkFZA5MeSlRN5cuJD8qmy9WIacRnKM8fgA9NyB1lgp3/LG4NbYAz5lL9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..4fd431d16ef2 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -897,11 +897,16 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) * Return: 0 on success, -errno on failure. */ static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) + phys_addr_t size, int set, int flag, bool reserved) { - struct memblock_type *type = &memblock.memory; + struct memblock_type *type; int i, ret, start_rgn, end_rgn; + if (reserved) + type = &memblock.reserved; + else + type = &memblock.memory; + ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); if (ret) return ret; @@ -928,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG, 0); } /** @@ -940,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG, 0); } /** @@ -957,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) system_has_some_mirror = true; - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR, 0); } /** @@ -977,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP, 0); } /** @@ -989,7 +994,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } static bool should_skip_region(struct memblock_type *type, From patchwork Thu Jul 27 20:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8770FC001DC for ; Thu, 27 Jul 2023 20:46:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D0D46B007B; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 580486B007D; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FA7B6B007E; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30C196B007B for ; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DAB921203AC for ; Thu, 27 Jul 2023 20:46:38 +0000 (UTC) X-FDA: 81058575276.29.C4E9E37 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf11.hostedemail.com (Postfix) with ESMTP id D392940021 for ; Thu, 27 Jul 2023 20:46:36 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gVT8uA85; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490797; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=JzwotKNBUcJn2RFN8EoRTY4Vojtlw0mEqisJjzzRA3VVc4wdTDTkvoUUVM059ANppo45eH DZffyiNk15PlpNwUtN8K+YoDYyAElGzcXfYmT2BY28ExuD83LOnKmXageEvkgKxotybsPR WcuMEgCBGvLwrJBMVvCkJ+Ikqq1L2cM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=gVT8uA85; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490797; a=rsa-sha256; cv=none; b=R0ugRYy5bq7qc7eNmrOrJIK7EjyAttrrOZ/OwcCzL27wzw+rn2uF7YaLJrWrejbv7PkN7d J9/qDFWfmIkiaVzjaEhp5yc2XhpVpxHNmKeAvm0FbPoNozewM33IdCGZsRB3oQrzhooACI KJlTcTk4t4vTbTulAB4LG+WxJhq2UaA= Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2b701e1ca63so21960011fa.1 for ; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=gVT8uA85Q/2RybadFL1WMGqwI8huoiPfSEh3Ga80yhJTkx2wqYddCczcvHucvrLvi3 8LyY62f1ojKZNCIJ1RJJ6AJm/H/D5kfULuGeQgjkEhp/sdQz5z18I7dL/szLmlIfJpYD u61EVQdgSt8wbKxI6DB5hsAcwBuLB2hgvn0jrCvRww9Z0lGEXVNLysQBc2Fe36p46uCT jl2OXbNIY2R4lV5NY2Ex7WgCxOm/ALG64TE59hb/8mRc6i3LDTDk1ah1xfi+hOTG8MYd 4LLgZYdnHkKk+A17Ls6+GXL7nzPHfRkZeRmwu0Uv3hqpWoCBuyANT80yrLWtGgfm1gge gtcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490795; x=1691095595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JQe+LnlORYwelNDP2OagFjwF2/T/1IMvjhocyHCh0HM=; b=VTNUmXwVZB2MYWtdWUTf1m4kl7g3ieYfS6k0o/L8x5QTnaLy36kvQJ6DE/AVyp0bfl LEfqdqW3anSQLkAGnWsJ4xiM7VWv9NXBiOksGFuu72+BkqISPd99qt1KHMiqGPhjR/ag 4c3TTP/3LIfNjV9En3RERx6u2YhVTtdjSIWKMWMBf4wQFMybLte89Eo3NP3DyY6l2wRQ xwqWSIpP4z1Zu6P4l0jA6YaKG7eptrYJ8/Wupt+GC0EcLlqAtdeL9UobOHU/IUeD55t8 qK4R81FXk8hrWCzfcNLURSMwzUTCqrnxR0a2ixSi3zbnAZ+SrShIE41TiEVWvFBbISCI 4dDw== X-Gm-Message-State: ABy/qLb8QQ9Zk7+qSEcLAV45WQsOeIoNaXegXKSUY+pnk7RMgG4gEXMN 5bOiGxb+f3QKMjJJ7+JNtmPstqAXXBUJ/9JgEQA= X-Google-Smtp-Source: APBJJlFfgB46RLeDSeeOkYepjrGae0ZQ1eLESbLa8jxhbH1/PX3Edir8uGhF64v2uGzIFXEGRina+Q== X-Received: by 2002:a2e:3003:0:b0:2b5:974f:385 with SMTP id w3-20020a2e3003000000b002b5974f0385mr129660ljw.9.1690490795035; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:34 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Thu, 27 Jul 2023 21:46:22 +0100 Message-Id: <20230727204624.1942372-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D392940021 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: qaqbh513kr8sffkdbby8mrqmzhxjuyh8 X-HE-Tag: 1690490796-738402 X-HE-Meta: U2FsdGVkX1+rMC/+XRJgbXeezQKESnmnMOWOXAW6J43SJpbo3HS8WrW1C2uPpCOt96j0R2/cu16ML2D1NCHksgMnc4upC7uMwZQxqlIYCTR+c2BCfTzFEVQnmINCQAUUAwgBBODCksoOhqJpdvZ+Nwi5OLtVYmQ69Izc8cse7wXDzHF+g7Ie2uzyx4rBIdI/DVdqAnLhdNtKmgI5x5L7G2S7SCL4kk3jC2f2nNerMcno92rP0BlnkQ1qyIONZUd14l3KO++WaQut4Gkh4ge96C8BDVWE/fc42LorI7envVkDecLfu06//Dzq8nEVutZPoX4kxcgbeQAVYHKJqrgZEOk9zAr5B/V9jO2kVGVRE2lwt9cmpDfwSnUt5Uri+yvvHBo9X2WaMi7db4iba3soFCQOK4nSEBNxeSxy4+qyfdTgyfxli0lFOsbiIyNVaQHmnzQ3IuanfjHyDjpTQQWZU4wd1cG4FiHPV7yf2Pb4nJhymuAhCOSnJr0xfacbby5Sc9sgcGy3J+EN6KDy5RSox/SxMJWhfJA9gJPxuf27Fe757QlHvZbqUnZqfFBgtauqNh7a947x5HvkM+nnWulX7sFn1wljfdSsP7PRx7yq97SXIA+He51JY2UMMyE7ngp6/40wFwZnrWLJMfCWgm/f/dNw+53H/+QMlCemwG4fn3Nbrgul8fe4/+aE49iov5boKqdF9GyhokmmDuwTGVLbknisWQZNoyokZVt5acZltcdix2+5Ntkszpq/qDmhaFEGjLN67mvcQwypwVrkIDT+eMjioG+sQp077muH2wAQqdZx6dr4Yonwu3DRT2sOd9QcY5VEpi99Sws32V/eMHtjF22qAXvJmtHSHfGTjjr19vUTF7XS70ZzQLJh30XAcsLXnnmf9mfuuX1k3f05VPg7nDPCyoMkeuGAT89QIxVCiBI7ycP2aGYllVE0Z6NKwRnaFGgRDJNGqIrdExwVRzB Cwf+XySM h4yMCRtApxOcRCOoGPhE8iT/hyoxmOET/SvFV3PbKEjCs0e40rqmlplQBSOTpGR0iIQdQbEZjNT1C8iKEuCZ2k8G/cF5Sv1hyl3mj5v+PJqsm2XL858USdpY0S48i0nDGBkXCGIuYcZ1UoN3fibKYZK+6EVgvq9c+Pzr41Ex9lz780wwlzuGwANdj0ESKO/4NSVxhnNi+0Ud2HYi17S0nLgUdyHQSJODrxdhly5B+DSa7dtussGjuuacBPK/S7xjcg9UquKSkLmhGpNd9JkGtO3IYBbtxivdgoORrHqt5lCyVoBJYdjvTNb4Uzqn6e1a+Ji7fpxkwvHEYZv+vTjUiEVr83QSArcs3Sn7qOM4MVkI3wIPnz0LztPZ6ZH/vXyPIbzmUjyMJrdpccFtGtY2OSfgeS8CBfCbUhK/qoRzFHnLeLRcz8jkrhKfTq4eIu6MlZLE+4YkIamJBznanpCQ4HdVaYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 7 +++++++ mm/memblock.c | 32 ++++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..7f9d06c08592 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -47,6 +47,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't call reserve_bootmem_region for this region */ }; /** @@ -125,6 +126,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +261,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) return m->flags & MEMBLOCK_NOMAP; } +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 4fd431d16ef2..3a15708af3b6 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -997,6 +997,22 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP, 0); } +/** + * memblock_rsrv_mark_noinit - Mark a reserved memory region with flag MEMBLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * For memory regions marked with %MEMBLOCK_RSRV_NOINIT, reserve_bootmem_region + * is not called during memmap_init_reserved_pages, hence struct pages are not + * initialized for this region. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_rsrv_mark_noinit(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(base, size, 1, MEMBLOCK_RSRV_NOINIT, 1); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2113,13 +2129,17 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; - - reserve_bootmem_region(start, end, nid); + if (!memblock_is_noinit(region)) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; + reserve_bootmem_region(start, end, nid); + } } } From patchwork Thu Jul 27 20:46:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9B02C00528 for ; Thu, 27 Jul 2023 20:46:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF8C86B007D; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAA376B007E; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFCFB6B0080; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B9D8B6B007D for ; Thu, 27 Jul 2023 16:46:39 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6A495160444 for ; Thu, 27 Jul 2023 20:46:39 +0000 (UTC) X-FDA: 81058575318.24.1936BB4 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf06.hostedemail.com (Postfix) with ESMTP id 90DE818000A for ; Thu, 27 Jul 2023 20:46:37 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MYvinfuv; spf=pass (imf06.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490797; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=lw2mXTkKGxzZr0QitUWLY1oWmslfHTpjUTHQ8YZzHxkZ589jtstho6yMs/1+4/IrUc+hfI Lz5tf9p3C8Ozs9ezE56EIOQbgB5ShaZOoj3eOA+NNYeQ8GOW99IaOBIPv4fAwktrQ6lxTQ eGjsFFWJ/rXnMRl6SFqeM7lrz3fb8jU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490797; a=rsa-sha256; cv=none; b=HZfXqanU0f6NamOZoj30SEyVKbaqjXSx3QGhEXhNTcdnmaF7dXRryrWYDiIcqknOynDn4z in5e6C+PxVXsaCCZlj0FU+2huncI1VEj2iUMee7HowX7NR8tgktXPvuttS5dCussJtKGMy ko60Qj+FoppMwzkKuwPxDjpMi78BIdc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MYvinfuv; spf=pass (imf06.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-3fbc5d5746cso15093875e9.2 for ; Thu, 27 Jul 2023 13:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=MYvinfuvPzSR6QZNB+kO061wB6tpem3LnUdQ68duY2x9Lf58oM1yxzxSQNGFKohFk/ EI1cawtc+AbWjfvWpBvppmYxUriHX3hcCjpQz4l+rQHliweF9u1TU856lSFvZen+b0ac /Iqd2lMerRuef36wCLT08cM3aet4zD4TwG6rd3HjnQ3e8jECc7SC98cUtcttUQ97dfkH 2TCsQq6Rp2S9nECR41e32NdAk0O1RVp9RR8BBusJoiRVB5fSCtegpCAa22SLN6tNpX0M IR0L8UXTdcMg4LsESCJXNbMtMfHWQw91m2aMdBO9Amm4VRFmS4e2CIzdeIHkjpK5IJLK fqUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490796; x=1691095596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cuTqSOqGGusC4+STWjmE7AEvtPXrFRozZqvR7YJO9tM=; b=RoBWuzvCfIfwGkHQKsdJU6LV/baq7N6m+R+fp/3BBRXgzanT3SDEF/YbBrCv2b65fu 0oEiSwUEZgXJmdRSe10m0EWgYhBI0uvIyjVx4l9AuoqNQxzuB5GGFXY8NbCa0fXAPa5J oqPgIVFOId2fwhGPEvE1ZeiYzmyPlFLKKEnSSKUsxUAI3mNL2aHzh681ilxe6bE4X0d3 9u8ddqwts1AV5AYRFfOKMpfDwmH9Me45/9TImQYqDOl6NuEw95BHorRXRT8HVQXoWo/j htEtwboj6LBJblUp6bKV1f9d5nQB1Ihnx4ZqK9omh5ABNXvLA+zmlYcDJAiUY80N0dDp GxpQ== X-Gm-Message-State: ABy/qLbTm7pcpEB9PiBRljb+1GErOGAMX9TsvQBmDK7MJzVN/VxATvSw tov5xERNjResVXfHjUvyh3XbhH8TJMrylgYz8bc= X-Google-Smtp-Source: APBJJlEOKaj+kNcenpgjxlaFx5BKYk7IpRTqAJr1UfKl04jzi5rThxvhDJdy8KNbbbphmnOSEh7YMw== X-Received: by 2002:a7b:ce16:0:b0:3f9:b748:ff3f with SMTP id m22-20020a7bce16000000b003f9b748ff3fmr151792wmc.1.1690490795841; Thu, 27 Jul 2023 13:46:35 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:35 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Thu, 27 Jul 2023 21:46:23 +0100 Message-Id: <20230727204624.1942372-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 90DE818000A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: dc8x78kh7tsj4dj1jzs3t6a4ku6yjbsz X-HE-Tag: 1690490797-938060 X-HE-Meta: U2FsdGVkX1+DFaFCJXYVzLAN8wo5XVI/7SRTXfkFdgBSIAg9cN5Ny9wBFca981Pw3miLTEMvyUIaYNqSPlxPtD9kdCcNqtYQ9wuAKtNj1ReWKrTk3okP7TJyfr1/ilVsS5G5kBbWiEc4UrYqEdeRZlCGQnF2Qk3UJkc2HbI/qgqJbieInpnyno9cxI/L0puXSyU1XIr1nPqQg1+iW3POSjL37f3dYePf0QKDc28WqAip2TrnU6P4LiRGwaKgb0bW2OUajW0vkB49IkOyGPPhKB1Zp3gvoWytJe4lOs2Flta4q/UBB3tc6B80D7VPfPoaj8IKU5ZGjz2YExZ+UfqX4fU5AoRtEF3iPJG3SStwRbfklIxtgpCJGa2EOepLEnteByeRCC53xyFVAhF3WnZy2KAWT3IURhvKB4N/qrhpUcSyj1SV22+uYCui1UB4klvOtkj7kFZT4b15H7G8WKujD6O2doDqsYlCsf0aWuIQd9NxepTvkFHndjPv7y5h9/z6/+Uu07e4atpT/Mg2qAmSbnjZu6cRqohPTn92/8vpAAPRVLELIj4IIoEwoVkRrnve0Kqez6x6760m6lXIjjjvDa5Uw1hJyN9TYBHk224fllfW03Bx0TU9GMaPFxx/0gGk2UIIUl6w2Nykf3mllLIXFbcGBWMUNDRqvA7gidalXKhtNJ6cwkG1cAQFJnVq99Zn5k3SkNWFGxYAvQyf4GrrYUHspy//Gur3wOIEzU/3BiLMZUVP8in3WiSaQYBbXpSb78p6p8VzIMsB9ADHTsC2fmQdMfm0zSfq3PjDHXEjXvLaFzP2MdmGrUDYyb5i6Gf5DEi5tPFgbvAKaoP2UiTEl0LZoUjzBoNe5T2JVDT6I1CFZbDCKPeQcu9p6o+bMiOntVKHGjCQsEZLdu0qnj8RuZBtnV3Xccvdz6bhp+ahuZQkomU2a22if4nfq1JLGdx+hLsA9XdZbcWwUhIUHc8 +ivz6TtU c5mBXWvQ77bhDzfD6vkJq0v/WEeOESbzLi4/hneMu3HebYY2PXxwCwpqH1cf8Lg/yDVSVYCY5DHF+KXjp7a5BeAz8dImm9swLJKFZdCF73vkcVMW3aTg0ZR0V/OlucDw38VG+YIddz3NTLPC2j2qc46JoUBwmNl/o5o4sfMk/STLQ6wW8iXX+XXVpZ3IL8UZ/K9IBuIuBPldyAEK5zBaUf0fzNzPbVEipmNQ+W8+AgtUKPNV1XDHLtZsjEDcUf4owOQyPY1DafUbD2tSx0kSS2y3WXqgzxNysPko22Sp2bBvfQ/u6djnGJE/1x9Rk+l/NPGbcMBKubCIOMPl9gCop+zHNqRqNaAuViW94KC2Cm7+LHkEjuJ+TFMOLwBET6pvXya1kTWKnJISsIf8KS63vngv2AdT1kCDBU8nZkFdWqXG5lrzqwFuKLBos6d9Mloh4lESPZcXXzjfK5SFkvGkjPI54jA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 18 ++++++++++-------- mm/mm_init.c | 4 ++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..2b20553deef3 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1271,4 +1271,5 @@ hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) return huge_pte_offset(vma->vm_mm, addr, sz); } +void __init hugetlb_hstate_alloc_gigantic_pages(void); #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 58cf5978bee1..c1fcf2af591a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4418,14 +4418,6 @@ static int __init hugepages_setup(char *s) } } - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp = mhp; return 1; @@ -4437,6 +4429,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=", hugepages_setup); +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i = 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..5585c66c3c42 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,9 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { +#ifdef CONFIG_HUGETLBFS + hugetlb_hstate_alloc_gigantic_pages(); +#endif /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); From patchwork Thu Jul 27 20:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13330732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8255EB64DD for ; Thu, 27 Jul 2023 20:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C7FC6B007E; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 951EB6B0080; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CA886B0081; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6B6BF6B007E for ; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 426E81201A2 for ; Thu, 27 Jul 2023 20:46:40 +0000 (UTC) X-FDA: 81058575360.11.B7CFF38 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf07.hostedemail.com (Postfix) with ESMTP id 6193B4000F for ; Thu, 27 Jul 2023 20:46:38 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MHW+96et; spf=pass (imf07.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490798; a=rsa-sha256; cv=none; b=GmIkbmHiUe/+lXecf3cGRgpL5lshLy1xmlPvOiOfRUTGXI2wiowU/ULJ3DyxA9+7LvgymS NwkuoTpvEj6RamThFag6XN4jjfdmVdPt57GPx6kttlkF0WDCzbWLKnOUJkwacX9eL7Ytne N8EN/5VCbrqRLCNMWpPi8BPBXvFEti4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MHW+96et; spf=pass (imf07.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490798; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=W29hwWzXCgxffehWhgYsSb3UgRhrUJ1zlsiv5DQMDVwPvovl2aLL0D9VR9oOMTq3u6ppZ4 VuVAXgkwzvLeTZxfbhGhKs2gwFdrxDMkfNPi7cDtrjJVBCEZy2qMrQtx0R9YEukTSvyCfm FYqF8qRWTUilg9y3Vny2eDPm3sgMExg= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31427ddd3fbso1440879f8f.0 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=MHW+96etwxTrHcysNxNv4Jt5Keu/21p5ZeNuDIbQx9GnIkEA1kq7sTTqw6Q/8BayM4 1Acht/ZPiVv8tOChy+mgBTfHG6ixCBKorlj2dJl7YTk0/sTWgbMWE9WJCFXy+OQ9Iijl F797zrYkH1WHN8+kZpWGtO4V6mpRCt3ijeyEEL/x0uHMVkx2JVs2E0JF144vy0BqQwCm brRLxremJQxfBWLsYq+wA+OhATOq4v3hK/HHE03k794S7TLjgAEkUPQNfZuGNCFBakRv 3NI8Jepy+Nx2mW0XqXn6WjNkB/tP3+1AiL7EGltSxqblDgE7Db3FfhwIHlnqEs7zbPpH c/9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=YDJuMot4JCrGxGZGg9u7f5F2mkB5654ATH5ID1FJScWviuMd0gbih1I6OOz3bNhjKg e4gg4DWP+GLN7CMgC2nP0WJ0nCBmfEvYgtbIUqcLc5sJ2NX3EmBYAOx3ZNRVTZg/uGH9 0RbDTdFee056HMPNuTV0N5jX1kf+6BmZ/hA5DRBD9MSsUhpd1PoYd/jRB4X0z0MNmg4q S9OGL9ZeqNHXFZvSqtcO3heaNnf0ZKipeY7Q8Q9MYOGEo3sJtM0S/KzEqrYIZKynTK28 G9RdpwuNuPAnFmqH5ZGQRPSziIaMASuZvxrAXUsOrlyAbEfbmZgDFF/UFNdKzOskBIDa nu9Q== X-Gm-Message-State: ABy/qLbzMXXYujy6ZAl7JCIBhqSqcu5l6oJjH0Jo2Z84mrD4CLrBDh1K CpJTM+g5as12fmx0M82HeefjPCjIhiSV2SFFQoA= X-Google-Smtp-Source: APBJJlEO7+YO7D/m+lNz0Wmsb1Fis9T5rZ6fiE8hYl22nZAClbV/pTs8BoCD41J0AwcBpujlXdeyTA== X-Received: by 2002:adf:db44:0:b0:314:34dd:aaec with SMTP id f4-20020adfdb44000000b0031434ddaaecmr229166wrj.8.1690490796779; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:36 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Thu, 27 Jul 2023 21:46:24 +0100 Message-Id: <20230727204624.1942372-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6193B4000F X-Stat-Signature: wb1f8918ro4ub3pugta3yzpgk7maystb X-Rspam-User: X-HE-Tag: 1690490798-696374 X-HE-Meta: U2FsdGVkX1+Z6LBYLgT7x72yXyYtJ11O79v+95PUAToll6GW6/W7De1EOAPmHf/2naNc0Oe+3IdMwa51MSbPs191rSh7pnqCGrGTIhZ6BmO6MgFp33XssOmOaHs43YZgkIEpwEeHxRIyyGmcwOCR8Bnug07KrPA+Ue5miI+ASlLrjkPtBx4PnyCBDUrT2HQqpats8fosM/3dDX+9QqpAM19sHe4v+2QtS7sSBTkFtNpjAFYMGKGIASFod4b5hxNUUDKHYnCztLoP5og79NOql/0BVb56AGZtkXW02MNldiMjgD6X0jboKrpqCDwPug/vcm+Jb4jo96AY/Pod1UDu/hx/ouq/uJUHcZN6rN9aNETTy8KAXBrbSVZ9cRNxYrN89tMl8r5hDjYBJj78PHmjLyMCsfkTk82+KAMscbCJ4vzFBzznBTedJc12WL6IDCWBPJKOO2w59iQCzIOs0htn6K7T1fg4TDFzX93KvptrB15KxZ974aj89GM8GEgAPCUIL7XBGYzL8R+mPOfy6Oaf3gsUPj6M/RlY+oolWf/rvoMMHA3QF6lVvW7G4Q1b1nM7j3EyxMMcJxeA/t/csxHlyRWagEVhkN2Hm6A9YiUTf0xuiWh7TrW29U8HAIrMkGchEu2i6+8WVHgeUqxDJWHewQwPpoPIK656GT0sz/ILWx74qWhu5RMzDFq3gV2N+687pZz1DTIUMUm06qfNLam0ya/aul6NQjmLFCS+JHE1ywxYvQu4v5PiCIzIkr6hwYHmLv7rn50w1v2/TKpvJWpfYMfEaBDryzyQzyFMN5LNoPWFcMZUXJWdLabXIdfw5YBCJXg7uicxLZQ/Dejb/FVvQrCTTwukPUX/vlRXWAdPvldktIjWv7RThI1pD/StW0qiz22x2m0eaJyRdaR/4Tk94Aiislvd3KoFaSsPmWk5AUSn2uDG4qWw2lYUUMAOBK3q90NPKid9WKonLAoDCk4 B7ZhxkC7 GL8yDy700JcdMsRKEzXBk2xW15No4N1CiZ9apB6AYOuqsJKnXLayUv77DwVpqkggDkEdAnvKNLpfgSb6+JpMxusURDNHxvWqqXMqSiM4eacBZKvVEzeRO36sWsU1wn9xn5kW9TrCI3jdIzKbSYFAj/v8RFZaUiXmUIYcD0kAHQeJd+1NxUm4oFcHwTClA0hGRVK2JkwSRr50cfmjgeN0GjzdLjN2OjlhKgwS6amUuV1XknbqK1WEPRA7yFZh5xRJfI7E+zzL52yxlPRveQs3qFTmsM14AN5VIsWmJ/74cSrnfLmhFO+IrrhfN2+S5dUsFjiazXxx1qB9MSp1fdL7JIeVuflRnd1v/JYTKoyI6Y2Tm+Z7i1ZA8Y1FQ3QG3Ntz/egTwgkc9FRVOwa/6UBzy/deVqD6LuCxE2X628eGezIxcLNw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1fcf2af591a..bb2b12f41026 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3166,6 +3166,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3173,6 +3176,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3187,6 +3199,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 07555d2dc0cb..cb5171abe683 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */