From patchwork Sun Jul 30 15:16:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3928C04A94 for ; Sun, 30 Jul 2023 15:16:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4C688D0001; Sun, 30 Jul 2023 11:16:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD78C900002; Sun, 30 Jul 2023 11:16:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A03008D0001; Sun, 30 Jul 2023 11:16:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8385E8D0001 for ; Sun, 30 Jul 2023 11:16:14 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5584A12093C for ; Sun, 30 Jul 2023 15:16:14 +0000 (UTC) X-FDA: 81068629068.22.05722F6 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf02.hostedemail.com (Postfix) with ESMTP id 52FC28001D for ; Sun, 30 Jul 2023 15:16:12 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LhfXuIze; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730172; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=Y4Qnw914Bqg9f+L6IPhxixzbBVBoXwXP+/Miu0PvqKTNlbKW26uug2kQe3i/tSHKO0vsc5 /CTKRxX/q/3g0jLlBeYLffOFZ2WHVRKZGQlWXmg27Yte2TqyZoLe8IUWwUfgAlOx+7Qvum tvUWRKoUOH2MDo4uY0DvH90YucjnSOY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=LhfXuIze; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730172; a=rsa-sha256; cv=none; b=8UooUum3CJS5bGsJSkWr/8iYsIGK+FU7YrGXJ30/Eusd9JLSmK+I8y2ycOamg7C8kg7fiq uGwA9+cQZq/H/YdW7ZuMlYnGJjnc4ecZhUoPL6yl4tTV9seT9dL8HkclVRDjlQkoGx4J2n 6N9BdQhS76UWQW02Riyn1O86UrKUZNI= Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-3fe1d462762so3041615e9.0 for ; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=LhfXuIzeZE5fNFxToBAlvDQA5EoZbK+3WEM+AjQQxsQ0JxOdwHbtoU7CSVU9Y3ua/I YUpJ5p9ziGSrXpohlrNQkGF+742Gm7k1GzsA5f33b42pcc1fc/iAwrWonfQ7MYbv1q7h YVGa2uWWvFfRDEkTBXC8BuX87YIa3rtNxtLUF7dmSJ7zSlON6lGYhX3vs8+xJgQ7inEi v2WUTfER2YUR5IubnbmBUcgeUw1db17u0y+MPFkcMPvVpO4iH/iea96utluPPBbXOwxl JtX5YfmoepWtGVwANwedcUUAaZIqjKeijQsq21YfiLsa2xpb4FFh20dY0yu/BPDgxfy1 opbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730170; x=1691334970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P5VE1b09XGrn3hjFzsE3qaVui4TN6DlsvAzQ6XWShV0=; b=AWQrZvFO6pqdEuhfYDh8YXRrBuJT3BZOlDW1MX7cXDbQ9MMhnBm3H13hRd540uc0yM lijoGi+JkEF74RuM5X9ZeJa5gR6BzTexjwPvKn7WGUbcaEYIrWycGrmpK3ieH4/YrKCz bjFPQOTeIkC0ZY/h3HrldOWTOXAXSQLYvRj19nvbXG2VF7l9GT/Tk+T67RQrw7bGMQ3q 8Kq6KkSEoJc/Ne2gIgmni+GwTGMz+NiMddXyiEDncqt+Gs+LRJymOwFFoibdGzz2M2oE /3DRCdyVxuYUvW7k6kbIYPSHaQpIQYDFTQveGUptoeN+UUZa4RiyMzAN3mYtzeHgvy3z dQFg== X-Gm-Message-State: ABy/qLY93ujByi6Ice6Nrn1zextDFw3MceeAlwBviH0FNebVwfHX66MQ 4xpYhm5P1lM1ud2fCDTgqtPEL/n2kwL0CkmMJQk= X-Google-Smtp-Source: APBJJlGPOGrrpNvuwalf5EOZlDbgmbBWoE71UsgklAK9FFQ8DCMd6wl0g32vEm3QO8b2B4WvXFrH5A== X-Received: by 2002:a7b:cc8c:0:b0:3fc:175:ade7 with SMTP id p12-20020a7bcc8c000000b003fc0175ade7mr5638740wma.38.1690730170432; Sun, 30 Jul 2023 08:16:10 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:09 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 1/6] mm: hugetlb: Skip prep of tail pages when HVO is enabled Date: Sun, 30 Jul 2023 16:16:01 +0100 Message-Id: <20230730151606.2871391-2-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 52FC28001D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: snck8hyd9kcfbtoe4jn3u5qq5a9x8az4 X-HE-Tag: 1690730172-220406 X-HE-Meta: U2FsdGVkX18C/YUZaTltQ59hdDCRhN48x+yfqowL2gBrmJM4VKjNASMCEQGDYklAt2gJ3p0slmUlzCRS4fVBh7fE2TH7LSV3hSHyIDaVqJK5rLv4cLdpTmEIrhCDlLyUEErAilYKOrXv9q30B7hCWvZWGqNbVEOZv75iKnbf9CwvySQEP4CODIfb3N/1BGHKbbovBu98bmi/wRyx42WORqwdTUt4vbU11NI0lcu8g1fMkaMAtAqas75G/k4O3gWpZtfzT0PV21ln+sOsCnPGV/rJQUFllmehqlxAAuOcwH6bcv7wVrK2P7mO7K89oLFSKrOdwjBhU9P8xHYEHO16imQkE2RA1i9tYms1lKHN+Fp+f3sQ1taFDkdd+sZk0OmHBSN+IG6UbrnB3AnkZo8JVHxx6Daz9W79JkyL5tt6ur1oumKRKfzsueK2g3K9F8vdidhTl0zwweVxnDx/mWaHNrF/Z2EY2E9QMFzjTXJleWZ7G11e985AyPLEPy8zpC0X+g96IHMfx+EyL7Bl3ZO29+UD9ZV83Mz920xZAopd3vybVr3ZROPdUuvLjaJJ0EtfiSwPA2QDn9KYHOAE4y9c+cN9DFzQZ6qyjl/Qc0vQL1kDK+6OA1TcWIIn4GLwnfmmqUKz2Tfgoi31pOw66QYwCV1vodmv2T/AtetugAkmvUjzKnbjhj4SMtlK/SQnB8nzcP6Th3Gi1pijaCEvWLoy2mdCisYFiQPbskEj6K9LN6i2eVlYp6Ipo6thUSjHu2mhaIF6jYkhN0PNAXcqK4rMLlvOQhpVpya9nYFtV+5f/aVtXTKGLig88cLtn2Z3b/ixUJs+RTIsr5R22Wy/7qOOthwTABIDygk9CTU/n/Ox9dclJSr9Pg6RUov17pJN5jYIH2JPaU6rLgK/LXmWcBsgWqc9t1dugkyGKnuniGk5eLeEQdJdm6o1dthR6ryFG9nTwMCGJE9peFXg2ZSU4/m n24niiyb ad1X8VdIrkqpA42TugXbB3q5dUsqjUV1DNyiEe/BTXNgMRVOkK/Mqixk8SrKs5m2fwrUpuHjR4FtMi0YaNc7Hvv8XDlj/r6wghvcq6jnInBf+SByKT5/8P5sCJWwVte4X8Vf6/ZB05Bmdu+SZlZWzoIMrTMN3tfuMmdKcHCScEJtFhLtZQLN0WPrMP2/BF2IVtUxhWnBylUbhXSQC8Eia7FFUD1Na+uJtVNXiqSZ4kljzPHCF0XIkhAX1CzhbqeCuh+gdTJI4ZVkl2ou0b4hU5FovnT254UU1IobyUXC0mu6orhR4MYeDE849hLqqRXNjJk9yNtnVUiftwpVsrWY3Aer0J10V4QwOmWeBLLQjlik6/G+tyU2TUn7fkTnYmBUNsTlZwf3tbwEsiTANldA2tnOkw+FXM17r/JD/saQmL4Y2fhI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000026, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vmemmap is optimizable, it will free all the duplicated tail pages in hugetlb_vmemmap_optimize while preparing the new hugepage. Hence, there is no need to prepare them. For 1G x86 hugepages, it avoids preparing 262144 - 64 = 262080 struct pages per hugepage. The indirection of using __prep_compound_gigantic_folio is also removed, as it just creates extra functions to indicate demote which can be done with the argument. Signed-off-by: Usama Arif --- mm/hugetlb.c | 32 ++++++++++++++------------------ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 15 +++++++++++---- 3 files changed, 26 insertions(+), 23 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..541c07b6d60f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1942,14 +1942,23 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni spin_unlock_irq(&hugetlb_lock); } -static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) +static bool prep_compound_gigantic_folio(struct folio *folio, struct hstate *h, bool demote) { int i, j; + int order = huge_page_order(h); int nr_pages = 1 << order; struct page *p; __folio_clear_reserved(folio); + + /* + * No need to prep pages that will be freed later by hugetlb_vmemmap_optimize. + * Hence, reduce nr_pages to the pages that will be kept. + */ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && + vmemmap_should_optimize(h, &folio->page)) + nr_pages = HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page); + for (i = 0; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2019,18 +2028,6 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, return false; } -static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, false); -} - -static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, true); -} - /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -2185,7 +2182,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, if (!folio) return NULL; if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (!prep_compound_gigantic_folio(folio, h, false)) { /* * Rare failure to convert pages to compound page. * Free pages and try again - ONCE! @@ -3201,7 +3198,7 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { + if (prep_compound_gigantic_folio(folio, h, false)) { WARN_ON(folio_test_reserved(folio)); prep_new_hugetlb_folio(h, folio, folio_nid(folio)); free_huge_page(page); /* add to the hugepage allocator */ @@ -3624,8 +3621,7 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) subpage = folio_page(folio, i); inner_folio = page_folio(subpage); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_folio_for_demote(inner_folio, - target_hstate->order); + prep_compound_gigantic_folio(inner_folio, target_hstate, true); else prep_compound_page(subpage, target_hstate->order); folio_change_private(inner_folio, NULL); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c2007ef5e9b0..b721e87de2b3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -486,7 +486,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { if (!READ_ONCE(vmemmap_optimize_enabled)) return false; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..3e7978a9af73 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,16 +10,17 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); - /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See * Documentation/vm/vmemmap_dedup.rst. */ #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +bool vmemmap_should_optimize(const struct hstate *h, const struct page *head); + static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { return pages_per_huge_page(h) * sizeof(struct page); @@ -51,6 +52,12 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate { return 0; } + +static inline bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) +{ + return false; +} + #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) From patchwork Sun Jul 30 15:16:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68F7AC001DF for ; Sun, 30 Jul 2023 15:16:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45FBF8D000B; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E8958D000A; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B0338D000B; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1C2178D000A for ; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C7914808EE for ; Sun, 30 Jul 2023 15:16:14 +0000 (UTC) X-FDA: 81068629068.19.2EA58FD Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by imf06.hostedemail.com (Postfix) with ESMTP id E5358180011 for ; Sun, 30 Jul 2023 15:16:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JoxH10l1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=DozR6oP40928HxzVgP8STcpRbnP1ChauFL2drFdgvJ55mUx6x7bNl8fvJB+Pt/IS3DuOoJ fnGYXCsvM8rO8QvtmjBk5E3MbgsJdgwGrT56ReMttFH5nDTJUZjJG/6Mv4a8jnVQLyFBmk 977E/sAaiv0Ovkrk3smN+TPVwJm1t+I= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=JoxH10l1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf06.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730173; a=rsa-sha256; cv=none; b=S02FZK3Yivh9/gY1/MRsI/3fL0/AJaUBnRiZnKoxaLc8Ln9xST7H/Rq/vTDezRnpzRlZf8 vieoMkH1EV9uM+ja5kgdtZON265RgVvffUYLJ4PhBufrPOP/SQqDJFuTdtHfstjBedt54f 6eVKN2uOiCkHjEpoCDY+PC52+OxIMqc= Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742bso41888805e9.2 for ; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=JoxH10l13AhAr8ISmwB2i1gOf6hMfDgdag1NYZGf8eqVGmEdh4R3IE2D1prI5+wzm7 vu1QD7xvEBstHjEIbxAMlgCCO00ONBH3G5zPFMoDk4Drvk6SzML5WNLP4tfo6lnQh865 TX13dZrkIeWswvkw6xN/xmei3GAwgiUombSwNiUwcW22Vp57PvkxefJyTKdVVA+u+gzP VbGxb9p/bzQfa6R6GWeFqN5krmkkMZbJxyXakbPqdmWiMydGR5JelRYvQMAxOrQf62pU o4lWtSXohDF8KsxmkSED8zQCamat1slkfvH7Xda+Eyuy3MLasON3KHOua9umK5WY8Nob wW5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730171; x=1691334971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VUFHsk/m/GY1eTNfZx8Py//rj4lEwu1OMjpML8i4XiA=; b=gA9xHRxSuJ6L+703jJn9+b7PH+cPo/AGprTAAiU660DAS5JUjNvQRVgavL709zSxPT NAmxYk6B0S0Y7KBAFr7llDQqZyMy/RZcf7AA2h4YQC6ZWX8mv/hyxTb0Q5hmvGXdha2q 8ONe/H9YOkth+qbAWgaTu/omOamumwCLoPKpxTmJlNu+CwnWZE6frtbdbuIlfOa8szXN ykOqBzuElsnp8315dMITRwc+FkN9O14qVMUd2KoIH9yIL9YlJ9XR5pH1M32kPorY7a4A lU1YJkRH8P+hjesj/uy0BCqhuMvN80s3l2At+YxdDmLpheKyDGO3MS3GVWUekzZZk451 gZSw== X-Gm-Message-State: ABy/qLZW91LrRXJzJUZznk+hxJxxMatmLNa9/+ihjbq/kUFvT7vvSqOf aeEBeJbIuh83JoTdqXwh+AF88T1t15dV6SSEZIk= X-Google-Smtp-Source: APBJJlFtcqfY902vGXRLqPZchnNVVitcnCdU3eEgHgB4d1amxrHvfEkxn/i1iktNVcoW6QdeU+DsCA== X-Received: by 2002:a7b:cbd5:0:b0:3fe:1db2:5168 with SMTP id n21-20020a7bcbd5000000b003fe1db25168mr1370994wmi.11.1690730171283; Sun, 30 Jul 2023 08:16:11 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:10 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 2/6] mm: hugetlb_vmemmap: Use nid of the head page to reallocate it Date: Sun, 30 Jul 2023 16:16:02 +0100 Message-Id: <20230730151606.2871391-3-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E5358180011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: tcxapm14ia7695nib6xqrxmbzxcomnrc X-HE-Tag: 1690730172-424853 X-HE-Meta: U2FsdGVkX19dTkGuUUDvwLcLLZ5HkRKs15po3fO6ogQOzoLUTItcn+ZZaKxSBtSp3nmx7Roy4omryXiD0UHKCIirzMX3WWWYkRs2VOGjGuN+YroMM/CXgZCYsTblH00K5AhpN2mg9vkb8xo+kEtBHk9ChU3kNx1aUHNY2YvusC6fbGY671/v/uFgFrAake8Wy1r4EHAibprpIHDYJpo+hMVlE+iyMzfIa7ExWXK+IsCOgLV+ggJboiaCzIOPG+KVv54CZT1cv2sKZ6nvQ8TPgYIEcB2C27STOwvd9YBrfHRVSnjkfe39kFc1kK7ZAmF1qDeeVfPFCAOHIVVKVW/sJprYDNZCLAp3jaGil1BpeDl769J/tSOZjAfmf3STyAqKCrXjPX5gZmAsxXkBvEFMxLhd7iTt1zsOI4xoAZB0/3lZ8lRG8+kO106cLZWEF6CaIVLao/ICQRbCUV1tymQGnHetqGJqAimT+zG7W1FtLdYfk5TE6f7ixkfA0qUA3EKBRMoYtPtNRCnmw1YtVEYuiTj1NT9Zu0srlGub/ZCRuleuptisU9jN/DjanHp5fRNle441LdXhNUw+dvHGVdICzQQEmCcQ/aDhSO1oceCSjKKEMooLk8TrGwxA+UxAsbAY5UFo8M97Wsf6mgUN7Ft4PCLIS97wMc7p+f/dCXjVwMvLpOlN3oOGx+5JIpAo22tzzJxjtJ1WB7SNywjEsLNZfa09PJzTHqyTlgNRS57wG4yRgRPxf553YtlleNUMNE5hjIW3uAANDd9UTm0bdSvChSSzFUvQyTKoR/GGTLAWUTnzB3qW/ogNxM+7Rroe+USN9UeOObJVQeP7TAHR80QYE1oUy7Qe3Ocx6eFisi+X6uYEYnn5FD/N9hppxiMnub752ry0mPblfeEU5j1214IWZIkqXmKHlJKI/v1pmV9wTm1sH+ZgBIIi+Y+9L0rSIn97hUwT0nDmbG5nPI0MyD0 3jSS2tgx +EU/xEPI7R2NQonUm8mel2Q8wqHDbfbgily1VTl/CtyTExROYwqE2LKlsReuc7fqerHeY5plOq8QMR3VxzxbqUglmdGamPdBnF/P2y/3zGCemI1Uh9DW4kupNeBS4P+mOcTfS7cn/Fp7ViHHwewSRoj41quqURm0iOdOdwDgiFrxGyynpUYhZT1+tN2+Pvs3G5gYsNvUunZKvEQMhsn3dzIHF6G5V068PCdmO+iBQmq7xRu2HSsSqzlMJY1cAdLjlhgyNxc/VYjdAp4A7uAu+IVbt8bE8jcKwo7+CFjN96tsC84k3TRSpR8AT7fbn98w9m/dvtp7RVrtVtTDsBF77gEzNHCIBQ/HB2Ez9Hnju/MwrpGXprK7HJU8j80vYsCuMatDyR1n8whfflX55yj4PI/xXA48jsSzrrRGyvH0nAPVM1nAnybscmAvqN+oci3BkoYDnPKQBf6FbCsMhkaaog9hF1g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.004784, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If tail page prep and initialization is skipped, then the "start" page will not contain the correct nid. Use the nid from first vmemap page. Signed-off-by: Usama Arif --- mm/hugetlb_vmemmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b721e87de2b3..bdf750a4786b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -324,7 +324,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, }; - int nid = page_to_nid((struct page *)start); + int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | __GFP_NOWARN; From patchwork Sun Jul 30 15:16:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9E3FC001E0 for ; Sun, 30 Jul 2023 15:16:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23E498D000A; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1795A900002; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E96E98D000C; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D488F8D000A for ; Sun, 30 Jul 2023 11:16:15 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AA3E41C8D44 for ; Sun, 30 Jul 2023 15:16:15 +0000 (UTC) X-FDA: 81068629110.09.8BB91D6 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf30.hostedemail.com (Postfix) with ESMTP id D4B1280007 for ; Sun, 30 Jul 2023 15:16:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="Vg/2oAVG"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=Y+NGAGwGzs0+F0uqi+vnncjrsEykINJYPE4jSTQzlyDJfWTiN0YQWVA3K5e/btXuPIX1sF 2Q4R3orjEHAsKd0c3v+5A2FSVAp3U5FHdLoEj0I2H6nXFE4GwcXvSd9frgvvLDKQyeo42T bixQIPFQT0UmeVCeO77eXwHXmjimklI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="Vg/2oAVG"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730173; a=rsa-sha256; cv=none; b=0KDCA8Cwe5Ke47Nx2UQvcmlMeZuJSry7naY5QgWq4Z8Ydmy8oD8kfg5eG1YfDqlDi93Rrt NC5DhOFAt7+iLD63s/3SrX0wVEcJl7csacEAlk+f/gYprM+yPDDnaNB7zi1asmkrnjDH1b 3k9IZiu/Y2aQey+0gOeb3trR88R5uSg= Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-3fe167d4a18so14344255e9.0 for ; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=Vg/2oAVG+oc700uPemJAECQqoZhvkHC4+v5G5umPPiGRagPclA/x9n9QgA5SDk8rdg xAP9ReHyrkmHzKD+QCBzbqzaesJvP/4YCVXhkUBDq7oFIh6fXXlQ/W6sesudYok3JgtG i8bNtt0JeBjQ7J+lYkbC40TeksYu8cuU6k1rZccXxVLpTekSilTdDcASxORLzoJggl+u dRvUz2KpmJ9ZuYWecFzFZAf1jF7ZS6Fe1gbbhfyhcMe2hud5pyNq2/bry4VpryWWF7x9 5XjSiGdDOVXVOjoB9TsdMcNi/ewnPLEIYUwh7l9m4DJjOtr0WmlRT+SiTcjbgC7BRS1N Zt9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730172; x=1691334972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjF+zN6TJ1BvEvRrI5C4CD8rIkkkthqR+fjsUt4Q8CQ=; b=CqarMMI/dsD4oeAy9m3bHokbqktYm6qeDTXMyiqkxltNGJG6BJ8qgGSm1TYlU/An+Q iQpLUziwTurTDRiNy81HanRfvA15/eipqaTahKKzvTsbpwNV6crKgYo29AP20c45EOjK dEMuwbveueGXDQqvcG6D4kAsxb7nU7U1lFyvDEiFQNL7A88bTG1JryW8VLR2UguZUwHM 5v5tcuqWUTl7p6p4GQ1mu9vCfEI2h+hZgab+/FJSLq3Kj37/faGKA8UiHpiD7ZtDSVbm Ei6+51uph78lRvFmKDxv1d5c2RodkXPhJ+ljLDOOI5vY5W2WQjPEoRHkvnWg0200xQOV BZvA== X-Gm-Message-State: ABy/qLY61pVCWNeS73FYGN6HT1y9Rgtcv9JQ+0//KsfEumdsxu7NXjSD cAXI10+Es88WFz+Oef0C2Fo6q81Acw0lQ2aWd4M= X-Google-Smtp-Source: APBJJlEBTcG9vQb7sO+eVhsdznDus6PkQqwHLouS3PQ0kKj5rr+MyIEAIBdbHeu9lJsdSY/4wUqiMA== X-Received: by 2002:a7b:c450:0:b0:3fb:c990:3b2 with SMTP id l16-20020a7bc450000000b003fbc99003b2mr5960153wmi.34.1690730172214; Sun, 30 Jul 2023 08:16:12 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:11 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 3/6] memblock: pass memblock_type to memblock_setclr_flag Date: Sun, 30 Jul 2023 16:16:03 +0100 Message-Id: <20230730151606.2871391-4-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D4B1280007 X-Stat-Signature: uxz5kzp84nn35bu5c4wmyggsd8fuwqnn X-HE-Tag: 1690730173-787065 X-HE-Meta: U2FsdGVkX1/PPhJ47OskEEY1IC+c6SlvhLHq3yHtPh5qJ+CKWxaONG5AXxt23/o3VBO4vueLK7aX2RdIPvKspr3hV29ggIUbpZrKxe9nRfkausG88GBJLWo7cDW4Ff1LQbW/r97ZJ0G2PjDXh6/naKBfiso4W/iVYB184UlA21bX3VsH5F65yeHdaJlKoLzx/R/Oky3MrjcSg9NjYhMyog5mg9nV6dye68JCzgO/kJ7n1AaQ2xDIZvqhcy7uIdjyoCnJFI5PG6UaRkzUsqehj7RA7cEfQLx/EKj0tN43oPLeK3iM4+GaeXzM7QR6WQTPu8wgQNfs3HjnzNOLaoS3YGfVNshpS3zcGdR/VrRF/kgd3YNFFTE9iUuGa2VEHNHxg46ZS2L0umLJsbmELEPFIZkqtiqtSUXm+UMR4loB5vqVIQNuDMt9/ujuCWGE8xP/QssM/1j2GQ3y5sEAFGIHFX/hAe73qN/wwYec/n9tIV2k2rroiGetzYSbiekRMf0fNo8rEYXnjcB7RNRVJBc2Eiw+BLfp31wpgm/WAUHBnGDxMgmZTaV/IRA3gNF+lFA7ZigCf8HI/EhY+Dj/Ld99HhPVaLh13/WEAhgzxL+M0dHf6R/dI3RIAyWes3i3m8+llBjGllhoPeo/lfKWWNe9vYt0hO32tnmnTj/Tcj5nBFUIYHOrX1XGRd3oilZUkvffQkM84qDKYdlHZqvArZCMaKc5avJpHQa1lTUhQRMftYTMTuG5fSEOz7WPouaMopWTLUHDAF26AUCUYImCJF5SMhJKyEsbztuT2qIcV623TILiExwm9vO9jwdiPodG2b/Abqvu70PCe8vQpTK0b9FB4jcfpie4JdxocwmNkZoWtXRDLi6SZg7TLGQFNG3syi4RzYOa5OJIX5Qq23IHQSJ1xy8DZmSke/CYfLr33VA5cA6aEbZLhYe4Ufyw6gMe4g/vthIQqzjy2BzozrDazyo pHvTeKVz 16KeVObBpRCgUcgDdr3lTnKP0lE+bf0bNKoS6Bt55t1TKXptkjrSf7zGmAlZ5eRyFYkuULVI1exVtacDDlZvyG/qYo6nl4T2jtrX/A1yAN+l/VdmAB8ogBDXCGIZizk8yCwS8IJ4wECXGx02oYX15JQIl1DgAmg2gOoDwiqfvptT7BfvtfLWtdkBueKhG/EV80E/TsRT38oOmyFx0KWKMGq5Moi2CRNOaAp5Q1v1OczxO8cbzY/XhfvRihV5kKeCzHPmUHDKOp6n1Nrj5rqnfF6kP9v709yzXMgqeJR0iNWntYbBQtwKrBj230Es+/war0Yiu0dYveedlvr5cpk+g8qp2zRKeBUSeoU2EJ5UWPuAuCViDLeKb2m9uej8UVxeVIe+ZjPTzQqVAr+F/8n6+wNOcbQUVsEPbB2jNKW0Io07PjjlusmEwVb2Z054MExVx4r+d4peEa+H1TIof77eVcIZldw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001831, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows setting flags to both memblock types and is in preparation for setting flags (for e.g. to not initialize struct pages) on reserved memory region. Signed-off-by: Usama Arif --- mm/memblock.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/mm/memblock.c b/mm/memblock.c index f9e61e565a53..43cb4404d94c 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -896,10 +896,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) * * Return: 0 on success, -errno on failure. */ -static int __init_memblock memblock_setclr_flag(phys_addr_t base, - phys_addr_t size, int set, int flag) +static int __init_memblock memblock_setclr_flag(struct memblock_type *type, + phys_addr_t base, phys_addr_t size, int set, int flag) { - struct memblock_type *type = &memblock.memory; int i, ret, start_rgn, end_rgn; ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); @@ -928,7 +927,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, */ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG); } /** @@ -940,7 +939,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG); } /** @@ -957,7 +956,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) system_has_some_mirror = true; - return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR); } /** @@ -977,7 +976,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP); } /** @@ -989,7 +988,7 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) */ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) { - return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } static bool should_skip_region(struct memblock_type *type, From patchwork Sun Jul 30 15:16:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEC17C001DF for ; Sun, 30 Jul 2023 15:16:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E25D2900003; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD528900002; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C28C2900003; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B010D900002 for ; Sun, 30 Jul 2023 11:16:16 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8BC741601C7 for ; Sun, 30 Jul 2023 15:16:16 +0000 (UTC) X-FDA: 81068629152.24.79D355E Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by imf02.hostedemail.com (Postfix) with ESMTP id A40FE80015 for ; Sun, 30 Jul 2023 15:16:14 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="W/lVFu8o"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730174; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=v3dudHXarJYWzuRBKs50YKnPM0Nie5CMWqcuNKIjpIga6fKhRhmZpWMO7j6VBTGJHgsGiz dzh+3JLMEClFn1yAfHkaIeNt2qMhG1Wos6TxLPZ+u0mXfA/ymEzj0somcYr9yBj0VOMIvt iAJilgRxBwY5PE0rj1piZHAF7DFK+lg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="W/lVFu8o"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf02.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730174; a=rsa-sha256; cv=none; b=8ocWL4wiwWWfjtNfq56NCtarftvfqyveIvPVpRSYqTqBUZSrSZBOehq/CzamgQOFVhr5j2 OXdDKm5YtqRzxI7daazdARHiU/kvtFuTClNOx4pp84yaSe/jLAWhKl1qWRqmxl5/KPtyfL Q8WqdVD22rUKSODDAGLiofawC2ax2Ws= Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-3fe2048c910so2025525e9.1 for ; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=W/lVFu8oBYOHaQON2+IUAE4OeTID9l1Wz5BLxJdEu8LJBB9z/KErJPNGKn4gPKcqET TOz183J9eC9oj+adoUW1wOIA2x17GqKNs+kATE99/ZRY8Fjp8O5VirlHjPixUUm43ku4 z+wT3z1JbZFobFUIlYBs/gN3XFoPKT2xOzlfLL6MoMgEtn/SiRo+2YI4xg86K/JhVl3i MuhecVQM9cEuPwUzqT4PGXvn5nnp3Ror/whrcq0G86C37cVp/BcEGPKbi/b5/niap3Zq DtL+erRe6APzX6GQNORLFAEQj9gidto4zepL9icunXIYosg8Xv4r0ivr6Y0+U81B8I1M t8+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730173; x=1691334973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sRlgK3gsc81CRpAMRRPsywIlI5j9OUiW8IxZjNa20o0=; b=cd/xGjf7YFe4vDAVswo1b7NB/me1a3NiLyKOGuqdZ2DXSKD4uUTnkZ6RCEeHMdJgn2 NtAQlm2zcbitIzxS+TH54TRgryK2FwhIP4Sm4QknaKfoiuGXM1bu9El0MAD4cQcZGXfQ t5TrVgr0cw06NBl5eCwEhfqp9BptfrQUNcY/as0htzyoz5MNt95Sief42V7CYZQNlw/n orOXmMw/ANCv8zjUJjoEZz61A34M2jjr7zM8knz1IRa0hLMxp3Y9Ueg0Yv4nS4p84BWv fawMe0PSRioJP93CEAKzepXfgeON4MnpqOKbBiv/A0m1jwT88ddKD+50inwaLiBPNYOf /rOQ== X-Gm-Message-State: ABy/qLZ+aWc1TEiAvdxAksHIm3ixa/9V6EdorfOX47YjFC+Iy0nAYN2u UWrOuqCs6hDku9Xh87wuMUIbBK9bYnVfKKbKsLI= X-Google-Smtp-Source: APBJJlFzVJI2UxHIjpRrxyZbsv8rAC3x9G9eCF0qIIb2iLIkSv48sgus3BhU4mIBADGNAt8oxdN7wQ== X-Received: by 2002:a05:600c:2809:b0:3fa:9767:c816 with SMTP id m9-20020a05600c280900b003fa9767c816mr5021376wmb.39.1690730173023; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:12 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 4/6] memblock: introduce MEMBLOCK_RSRV_NOINIT flag Date: Sun, 30 Jul 2023 16:16:04 +0100 Message-Id: <20230730151606.2871391-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A40FE80015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: d5hbjj5jg5i9prwbsmujdpn3j1smqqc3 X-HE-Tag: 1690730174-591149 X-HE-Meta: U2FsdGVkX1/hRmCTUmePrBvRVMs4oIb6qoJGk07ZCA8FEv0b09trRM128E8rp/wampRVJf57vakSKa8GUKhMPRDN/0YyUDZ74xRCZEABmQc612/Fcm8BRrQjsP/vLTYMTIksx3MmfFEqw+9ogNwBii0ydDEDjA2IALkgl9ZfF1wRne7+cSYRxkkAk2fexZmJtcBxjCZu+SEEhN1Z6ZmH30bhnnbZqaeg2D/68B0cLUcZCSUFZ9qSw/LX3qtBaXYLBtHNuZiCVFUF2bfAon2mf5PtvtFykdn8GeS8Hwq/B+1HlsZLPHyk5rzsoXdrLrQK3ll2tB/2fNVyMKTzkZwbW0ZsGdlyK7k+S+X4PrGPQzANmoS28Mx9u7lcBz/pE9+iMhSwmQN6Zem3EAeQqjeJWg6xx0ubV/iGPA8KsAcuaXmqJYNaM/55hVLZdaN6hsz8Bd65N2YaxzJNYc+QhNJ5E3mHWpp2L2lkKNqUONCPel6+Wk9vIsdLmI7i4YGYlaVkaNsRk1oAmi0oVZwU4TCJ50DeMxmUfIVm7lhrjiXPQZte6KvWD7Mf+2MOrLD49+FoM5IKkX3TTlN9DHv3wcd+TA/gE7uyvaGGkHyNrUfaz7v+/ugDtAxTwGRNwt4SSa4rFRUNmN/kwukUi+K+zmxd1LASLvgsdry1C88lZEw+z6qG/9WBLcPGDOjBCzTH4nl8bkwQIStNvwPsUd0LBeVxnXZM3RxQ0xECracB2Dn66N8J7YkDvnn0jl+6f3vuMKVPYEeWetWcfBYdEIzCpGF5MFe/Vfgi4zoRfLr15JAdOI10TrRjN2iNyFtk54ckWvZEUAfxSrpY/ftEXRJvK9aRFErQ93IDLeXKAy4hzDGLzpnsFpGOgrrrzeDdNgxW2WV1ouHQwRHuQXUWH1uAJv4qwJ8q2v0tBb4fOZC5qCl2tYh42mefwyWHZCyX3iq1W/HFM37Pwa+HHzRMO3GKn2e 0qjHStNv LXNgc0embxHsi6xU6zwqdYnk6d6ydqVH3UhkN9bTTiy3Ils9EkKWIAk6OIpkE6A/KKZQXrNAKWIw6/Ox5YlBH4U7U5F7v2pCJyXLNq67RLL0DM0YoqwlEU3uhmiftfVALpsp+8766GA5Z6u8b5YtGde2RxRvWDWfDaPJlgPu691SdmyUJwyNaJkrGBekBVqsRWY9p5r7d4lcOoLq/8Sa7YV9AyIPMji77+U0GQ80GX3zBBE8WUVfJR8w8T6JlCvXfx2O5k7yD2UGIpwd0bJa+kJj1CtFj3SgMRX7KMZPmgXBxi3EUCAC95lLUxWRRoj10ufCI4/vd0Lg9+4wdTcSvW8p3aWVRZVpP7DcDthkfNIkbPnwC7e6GAtmX35D04snzAzkqSNuGesIdLIL49r72o2uYsS1EXa//R871Nud5WSqpGQP0k/Kaz8R8lVrY36U11ooQel6GF6KlwNMNZqj5Mc/6Wg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For reserved memory regions marked with this flag, reserve_bootmem_region is not called during memmap_init_reserved_pages. This can be used to avoid struct page initialization for regions which won't need them, for e.g. hugepages with HVO enabled. Signed-off-by: Usama Arif --- include/linux/memblock.h | 9 +++++++++ mm/memblock.c | 30 +++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index f71ff9f0ec81..de7760aeb870 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; * via a driver, and never indicated in the firmware-provided memory map as * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the * kernel resource tree. + * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are not initialized + * (only for reserved regions). */ enum memblock_flags { MEMBLOCK_NONE = 0x0, /* No special request */ @@ -47,6 +49,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */ }; /** @@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) return m->flags & MEMBLOCK_NOMAP; } +static inline bool memblock_is_noinit(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_RSRV_NOINIT; +} + static inline bool memblock_is_driver_managed(struct memblock_region *m) { return m->flags & MEMBLOCK_DRIVER_MANAGED; diff --git a/mm/memblock.c b/mm/memblock.c index 43cb4404d94c..40690ce0b06e 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -991,6 +991,21 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); } +/** + * memblock_reserved_mark_noinit - Mark a reserved memory region with flag MEMBLOCK_RSRV_NOINIT. + * @base: the base phys addr of the region + * @size: the size of the region + * + * struct pages will not be initialized for reserved memory regions marked with + * %MEMBLOCK_RSRV_NOINIT. + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(&memblock.reserved, base, size, 1, MEMBLOCK_RSRV_NOINIT); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) @@ -2107,13 +2122,18 @@ static void __init memmap_init_reserved_pages(void) memblock_set_node(start, end, &memblock.reserved, nid); } - /* initialize struct pages for the reserved regions */ + /* + * initialize struct pages for reserved regions that don't have + * the MEMBLOCK_RSRV_NOINIT flag set + */ for_each_reserved_mem_region(region) { - nid = memblock_get_region_node(region); - start = region->base; - end = start + region->size; + if (!memblock_is_noinit(region)) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; - reserve_bootmem_region(start, end, nid); + reserve_bootmem_region(start, end, nid); + } } } From patchwork Sun Jul 30 15:16:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CC40C001E0 for ; Sun, 30 Jul 2023 15:16:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A54E1900004; Sun, 30 Jul 2023 11:16:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2BD3900002; Sun, 30 Jul 2023 11:16:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87D6E900004; Sun, 30 Jul 2023 11:16:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 77D37900002 for ; Sun, 30 Jul 2023 11:16:17 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3C70D40310 for ; Sun, 30 Jul 2023 15:16:17 +0000 (UTC) X-FDA: 81068629194.21.33B99E1 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by imf26.hostedemail.com (Postfix) with ESMTP id 5EC20140007 for ; Sun, 30 Jul 2023 15:16:15 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=TkyhfbVG; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.49 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=ywkpbunMXKgo6N5t+6ggMTfsYCJXwYQA/aFJL2Y2FQaD467jT8jGH835/yGCglUdFDWfho 5+ULMU3cMZEs7TKYr8LJIcUXhEUoadJxBE7EtfzqH9wiokNhWQJIsEi9aoamS6VsA94a+9 P59qmOldkeU+0/Eqc7Yeb9teqBDQJZM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=TkyhfbVG; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.49 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730175; a=rsa-sha256; cv=none; b=kt33kv9CENl+88yXERN9rAuKWVR4LRqWlW2okmXwE6HaJNjOawGI5QjiggcGbjbdAixfWV zni0KPYKPUBLm5BwwMMxsimwEnnfBbyr5yarF9NhJJEbyvGyDIkiofB2YjidEYfNyQbrqh /tHtdTZnV+HfdM8u+cIBO44kTCmC+R4= Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-3fe167d4a18so14344375e9.0 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=TkyhfbVGe1Y7YHfHksu1wqG6hgQhYRmidCb5+Y61F1QRRs/h47YJFt0AIJh54Q0FKL gByq+10/h43JlVcCEVZjC6aUzw/Omy08uMjBB7CgI/FZQXN3z5WuJzdScj4/HnraGiyu il7QDEWZYGSpogyg92dYwrJa+Eqeb9M65dGnmL7nnCP2F14JH377zSxW/hWjC6XaZnyk O4M74hvreh6XC7+2F9vY3dYi8dCjt4haamygyIK9l6ZMqTa22DtdrGIfwCNlLo2xUbbo yHp+23v36B0XyqElADWdUdmxDLaLZYjFnCSrKJy9muMAQ+U2qOTBbbF4yWjE1y0Ngwfe EPHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SFM4pR1QacGlm/8xlIUmCJcy4VSy/s+0OM0YFxjuA8k=; b=kkjBnHLaC7c9rd1ZDqrNIvbt/HGoDFqR0krB1xvDD4wvz0/aIFVleCyw4i/9em+QXr 7O4FVBuZsXq0VFmwvqFHWQck7I+f1Q/RIhosI5FsSaPkyAPaMchJeJSbXTcIi+S3dcRX SQv861YUDi67/fgCZmT4moKG2exw4vPSVY1ySii3SxS/MisKBXzas84U8v/LsPYCpHeu MzkxzkZN3DmbiJLzK6K7QwgjkqWPNuyHa1V8xmA+qw40Qk5oA8ZfN7+rdasgCBViTt58 dhNTl/7lUFzbiMwlErL+97wVM7inI5OZTadHJYogE2oKgW8fcBzdeHSNgLR984tcw0Kh c4nQ== X-Gm-Message-State: ABy/qLbrxwhq5lCNsUhZg/3YE4PRLQmzpqQGbJNcc3wF6B16DP+BN8WZ hc5oAkMxjOBDsc2ryB9A1eTigx1PSmy40z9BcA0= X-Google-Smtp-Source: APBJJlGH28j3h1b9uEfHTlqYomCBicErb0vATQvL1NGT50ZdEoNVQZEd6E1WZNtnMG3Mxzbn8725YA== X-Received: by 2002:a05:600c:d5:b0:3fb:e206:ca5f with SMTP id u21-20020a05600c00d500b003fbe206ca5fmr5709508wmm.31.1690730173889; Sun, 30 Jul 2023 08:16:13 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:13 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 5/6] mm: move allocation of gigantic hstates to the start of mm_core_init Date: Sun, 30 Jul 2023 16:16:05 +0100 Message-Id: <20230730151606.2871391-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: rz45dj5okf1pdj49bhfjrcpscfgetp8i X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5EC20140007 X-HE-Tag: 1690730175-593719 X-HE-Meta: U2FsdGVkX18EaNSDxKnRgECReaoNN4VLzSKHvFm6QOSyNVyeKAQMA6GrXzJnF4QGqQN6nSrulV82sDA7BilPcIkKTwCVvtYxIDxR0T15Fw+aCod9/5J7KEzXA6WnxfAA3Y9Os3kMQD6XdkcP3P5zd9nVYnBoGQliXFWrFgoEu9cWgHU7M+9ConUNlqhTMDXU67yb6QSlqjtgAGkKfmjN4lNnxvxSK0u4s7N5P6HDLurTVWlSdOxsXW+ValHq4odOImbon2kEGPxivhEIBgdQyjpGmct2ZDkB5qIh9OMr+k/+FqiClapgQmCg1Arw+hT7zfoHCLa3xY2KUNqOYBICNsQYtqcLUvG9td8lI7867JV7twUw/tHFbW6K/kIiXeLoLI+0aphI0OOf50ePfwLdhqKExqCXoDyJ3rBV7wHsEZ4eRP8ep2cMK2ApuyzMSkJARI66eEwlU+5xRfWFAFuFmfMj7e4foOdhiMQZEUeuR2tdhLz7ZpTzhrB/Ukr5chJdXhSYPfh65O+rrsg5Qu0LPlOd7Lki39/dCNUxYZ+M5wOvXgefzUr+vLJrHT1dE1sLB+0UkTDdHNPJFPZ3NTO8e/Psu1Y6y1+HfYqOoWqhny7Q3vZi5Joqhl1o92RV+oxh3mmgUERNMo7aS6ABq+C+8CXZY1gLM0qVShGeUMoKto7SxLM5886IAg4naneTNfRT2TttYID0P255jXW9Sv9QDi7340adalTjXWs1nrX5S9Y+Q1nsP0S7XWTl+BLVsC78Zdx9D29EavMKL3UeolJzAkP1xURYpAmhCc/AL+Zf/Ep9EkKX93H9leG8JFpZR6NKidZ3zNBvdbyCDjV6Qzf9sVKHK1pKX8QdhC7CiXIcvcPvjXgKwsywM9bAu6ueWHXwv62JJ2s5gVkXuuU8VSQ5VOJnzkddDGBoYxAqZFSyof0LDDxpa+9dtgM/6L+s+qwyWX0H/cLO0kJ3isAHIT1 rAoPCo9N fSKtUadQwxgXVmUAoqRrrUBVCA4a+xVCIpT3QfcLw0w4JzxfHWWoc8iQsFi6uh6bOteIGy+fuxGu9mznLEUWIQCmxUoyHDsEEgZnliBYxuNybkq8CrYO8kemz3xS32/LzS9wNrceSVhGD5cbQ6L8PDgPddyA2LqxaRrbvscXq++6a7ubXBVPj0bVk+6DNaIkBKyVRyLBWH2xVOV/VgTcE5HmFkYltqJ4nvNrEnXD05P9ytMMHU5gWCjzA2fKTOmb0ukL8tRgt6KrajPFtKKMUYgPD0AdXQXba7dvd2KGudJm2FGY1HHLcf8cIpIzM3t0qk8326zNZ+yj4L9tB/fXB0p3ksSEiqI4wY+rVd2N5PhhXeQSacGgyklZ5SDnUrBblEpbfLQt3qZgJ4Kx/AlZ+n7ikwoknjCKHlL1WixunqKb08hvJq+5bq+aQSxihyh0nzKJWHfE4M4FEHW/kKhtyMm7bmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000032, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Whether the initialization of tail struct pages of a hugepage happens or not will become dependent on the commandline parameter hugetlb_free_vmemmap in the future. Hence, hugetlb_hstate_alloc_pages needs to be after command line parameters are parsed and the start of mm_core_init is a good point. Signed-off-by: Usama Arif --- mm/hugetlb.c | 18 ++++++++++-------- mm/internal.h | 9 +++++++++ mm/mm_init.c | 6 ++++++ 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 541c07b6d60f..bf60545496d7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4400,14 +4400,6 @@ static int __init hugepages_setup(char *s) } } - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp = mhp; return 1; @@ -4419,6 +4411,16 @@ static int __init hugepages_setup(char *s) } __setup("hugepages=", hugepages_setup); +void __init hugetlb_hstate_alloc_gigantic_pages(void) +{ + int i; + + for (i = 0; i < HUGE_MAX_HSTATE; i++) { + if (hstate_is_gigantic(&hstates[i])) + hugetlb_hstate_alloc_pages(&hstates[i]); + } +} + /* * hugepagesz command line processing * A specific huge page size can only be specified once with hugepagesz. diff --git a/mm/internal.h b/mm/internal.h index a7d9e980429a..692bb1136a39 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1102,4 +1102,13 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +#ifdef CONFIG_HUGETLBFS +void __init hugetlb_hstate_alloc_gigantic_pages(void); +#else +static inline void __init hugetlb_hstate_alloc_gigantic_pages(void); +{ +} +#endif /* CONFIG_HUGETLBFS */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..f2751ccd7d99 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2768,6 +2769,11 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { + /* + * We need to allocate gigantic hstates here early to still use the bootmem + * allocator. Non gigantic hstates are initialized later in hugetlb_init. + */ + hugetlb_hstate_alloc_gigantic_pages(); /* Initializations relying on SMP setup */ build_all_zonelists(NULL); page_alloc_init_cpuhp(); From patchwork Sun Jul 30 15:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 511B5C001DF for ; Sun, 30 Jul 2023 15:16:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5308900005; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6D9900002; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BB1F900005; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5E4EF900002 for ; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2C7881408ED for ; Sun, 30 Jul 2023 15:16:19 +0000 (UTC) X-FDA: 81068629278.04.25DDAA6 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf20.hostedemail.com (Postfix) with ESMTP id 2D8BB1C0021 for ; Sun, 30 Jul 2023 15:16:15 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsLvLnj1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=HLHLWNr49OnWNz8lpmyqThu4N8gTie8EwGUt8/wzrBc9s6CYvd3pvKg7Lv0fgCGkRap0e6 pTItTXmWH/g4x33FNNGFUCqO2Ed3cVUxLBgnD1r1hggRQEgq+63CLVnb2ApylPpEml46UK eti8nvscUeJ/zBH+Q3LqPa3QyZORyzI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsLvLnj1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730176; a=rsa-sha256; cv=none; b=IS1szDFGTrTBgLMrUb64cSssE+m1cGadHdOlP8bWqB2PNK/2IoRovFPGIp7P9cnw1CKh24 o55oGOBPg1/T5n0ICuaMPSqnAMijdmG63s9gsVzunHFhjYDN505Wl7oLWBtA6mJOAlHN27 85w5nU+fQvWisW0GDDLEXggiLBlD9us= Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-3fba8e2aa52so42188785e9.1 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=CsLvLnj1VBExXRAAglTYPWH2EpYCorZggOAC3yIHs0vybciyxsExVcLP/39obiFOYD FverDpvbx+BWRlvxddS1Ucch0Lo8ySBj5n4n9qsls9ncd4/U0ukC8FCsg5LU0Ps7yXpS 0x73YYFy0A18lccSHORRLeDqKaS8nvvq1tbkN/1SKvxZESadhIFOz/0oIn1npIu6dGrI 2CSL9zF3tMODmGVmKAr7ZywN5DBo03IuiGI33kAtutuURXCt6kQH4LYr2FKRdNpyhVsi OuuSaUD++enIZiLNISEKbcIDH+KKX7DGzU9igsnu/+61mXgL/I4WQRrk8Tlu5bD6f8xb lAyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=heiiv7OoJud1T/zMPDt7ab2Kwn/c0L0yjloB5U0yeTb2z75X4D6Yr2sZ6w91GfgbXg q1XuKMv5oNs5MmaL8ie9fyMxILmRcrF7dguEtweslG2hUJDbUA+qjLaiJ3uqKAso1pPF L4IE0YwoMpUiuZ+hVnPceC9b6DsLmXv24gkQXPrrlhXfkITxS71syR88VfP+8wN4sLxP +37iS4t4Ae5wK8Bj41HabSj8t8jR25JmgqfgYqI22ad4+Y5ECsoNUhgmK4oRPlUlHddi pYxAdyVd6Iredpf1RweAVOUImO5ScB1nzR4eVkL6/AN7CKLHgpL5Far0SV24OHAcVKyn Mewg== X-Gm-Message-State: ABy/qLZDefVIRhZW90s0NTQfomRGfISSmtcoFaA6xHM2jFkM5Qw9TGsD 930zUKK1VlG1twC61TE/oliqGDF4Ne66xvzqrkk= X-Google-Smtp-Source: APBJJlEvwIYdcin+SgJIaKf3f0NWjv2vOyb8TeOb/EE1tTkylJ9cPozIeSxvQxhFH8cAtZQs6UhkkA== X-Received: by 2002:a05:600c:3787:b0:3fa:97b3:7ce0 with SMTP id o7-20020a05600c378700b003fa97b37ce0mr3917250wmr.26.1690730174681; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:14 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Sun, 30 Jul 2023 16:16:06 +0100 Message-Id: <20230730151606.2871391-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2D8BB1C0021 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kyfjio8oeytk4m8q9w9xpgt85o5yj1r6 X-HE-Tag: 1690730175-158697 X-HE-Meta: U2FsdGVkX19YcptiH1qrbPN0aVpIkDNaj2X4vu+zMY6oRRB8/tYbLS2GHnYqHBximFhRrIs3bb3T/4acSI1cw8CQoio1mY+ksiJKVX9AK2cUq4AUE3YRfoYEMVATsKc5L9rVJV89MflfuSkLFVmYF1D1RKQoH2olWBxfUPGi4lWeRi7F1fOBXB1lI8YVbYTrZpoHoNp4nYRN67pNrcF1xv2Wj0nBZIOY/pKxdnpz6Kf3vWvAePfvV93EhwNWrqEh3xZAf4lZvLu8QpHVXcUem00QruRo3Vuw84sJxiBXx/8cJUUK5WvP0o6kwgg99TIB31L4YIhp5YfgDPQftoHH4xCarFXbJA7+OE+JAye3ucnZH0N3/3oQ5OMSDWY4Pb0MijFnUE1UIUBlqwn7foIfe2WDGDrt5LINll34p3rjqSS6q+E/AIa8Z1ThhZbQzbTseI9+qmGvgBPBw9Ubn/cYxwgx5T7RGQSAtE6CaEIu3NLY/Q5DeyHDvZniV3fHyptIVb/dCgiwGbCOFF9r5pKxBBFBiQ26y4XedetBWI73tZ6qN51AcLvo7FgrPZiZj3EoIET7H2W4gVNvZgzINC6ELEN23+R1DGeFD8kfVRWd+xhWiFg63dCXGMs29rH7InzX0PASOWj6vLaBebBVWTlunGThy+BQ21Gt+OzfeqXr1qb9CdHR0BuY4FTgZrWiU+xBn+bfjp7Ux0J5ak07jC7+qJISWKazlWiRzPQUhp8TezdyupfvStTWNNhNJQEZsFHHH3utpBEyaxdT74wMfFL10ek6H/1lciKNmSaVNQoPFYaZsvFHSuCcPvq76iP+blMswGqN8xRhQiQ4xh6W2yNccLeD5bvrkn5rP5SLBpzOlS5ed1yaecXmS6NWBWuPnK58cCt9cF8SR8Qyr0zEv9X3q0eAA3BYUtzWRPntHiU6G+uF2PaDMZFksJfqdgjkyGtRfuTJPLKvDN1J3OHkDCy 5HZ3uecg mPDuN1N0lz1opBWZ8u7e96uHXTYm8gOf3P6SdcDopAPAWtLPScw7PEReyHh3wc+7l6ByqpHQ5AUy3eBpSwxi4AgZQbnBO9zpjhQ1lgH+t3ik4nUuIvElvprAv9j6+hjyGp6jVzAF/1rDbpV5uFGXibVoHaotIzwfUuKTEKEndqrYx5zHqXttCcqYSPdvhp7VenWNnvpzSwmeZHoaofAEJ3TBeh6s8m47nrACMKhJy6j1Afd31wabNpYnSQSRInvGfNz09XjgvAEOZcQpaNBH9CHmli0aDIcuSt3T+Xqda1xwdb8fnqsD5jiDX2f89G0i24gAjMwGSuCOvoXS7SpWb5FhowMPWehWEIp9QQeu1aMx7wLBZYAXuqlo+HB0PIjHCrKjco8j1euiGmp7sv318da1UU0sWtw645b+1lj80IvfsR6A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf60545496d7..8434100f60ae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3151,6 +3151,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3158,6 +3161,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3172,6 +3184,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3e7978a9af73..3fff6f611c19 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */