From patchwork Sun Jul 30 15:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13333456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 511B5C001DF for ; Sun, 30 Jul 2023 15:16:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5308900005; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6D9900002; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BB1F900005; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5E4EF900002 for ; Sun, 30 Jul 2023 11:16:19 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2C7881408ED for ; Sun, 30 Jul 2023 15:16:19 +0000 (UTC) X-FDA: 81068629278.04.25DDAA6 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf20.hostedemail.com (Postfix) with ESMTP id 2D8BB1C0021 for ; Sun, 30 Jul 2023 15:16:15 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsLvLnj1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690730176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=HLHLWNr49OnWNz8lpmyqThu4N8gTie8EwGUt8/wzrBc9s6CYvd3pvKg7Lv0fgCGkRap0e6 pTItTXmWH/g4x33FNNGFUCqO2Ed3cVUxLBgnD1r1hggRQEgq+63CLVnb2ApylPpEml46UK eti8nvscUeJ/zBH+Q3LqPa3QyZORyzI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=CsLvLnj1; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690730176; a=rsa-sha256; cv=none; b=IS1szDFGTrTBgLMrUb64cSssE+m1cGadHdOlP8bWqB2PNK/2IoRovFPGIp7P9cnw1CKh24 o55oGOBPg1/T5n0ICuaMPSqnAMijdmG63s9gsVzunHFhjYDN505Wl7oLWBtA6mJOAlHN27 85w5nU+fQvWisW0GDDLEXggiLBlD9us= Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-3fba8e2aa52so42188785e9.1 for ; Sun, 30 Jul 2023 08:16:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=CsLvLnj1VBExXRAAglTYPWH2EpYCorZggOAC3yIHs0vybciyxsExVcLP/39obiFOYD FverDpvbx+BWRlvxddS1Ucch0Lo8ySBj5n4n9qsls9ncd4/U0ukC8FCsg5LU0Ps7yXpS 0x73YYFy0A18lccSHORRLeDqKaS8nvvq1tbkN/1SKvxZESadhIFOz/0oIn1npIu6dGrI 2CSL9zF3tMODmGVmKAr7ZywN5DBo03IuiGI33kAtutuURXCt6kQH4LYr2FKRdNpyhVsi OuuSaUD++enIZiLNISEKbcIDH+KKX7DGzU9igsnu/+61mXgL/I4WQRrk8Tlu5bD6f8xb lAyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690730174; x=1691334974; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ed3bTY/DQrtVo38S3b99vtsCN5ydWYuckmFt+jfpXnQ=; b=heiiv7OoJud1T/zMPDt7ab2Kwn/c0L0yjloB5U0yeTb2z75X4D6Yr2sZ6w91GfgbXg q1XuKMv5oNs5MmaL8ie9fyMxILmRcrF7dguEtweslG2hUJDbUA+qjLaiJ3uqKAso1pPF L4IE0YwoMpUiuZ+hVnPceC9b6DsLmXv24gkQXPrrlhXfkITxS71syR88VfP+8wN4sLxP +37iS4t4Ae5wK8Bj41HabSj8t8jR25JmgqfgYqI22ad4+Y5ECsoNUhgmK4oRPlUlHddi pYxAdyVd6Iredpf1RweAVOUImO5ScB1nzR4eVkL6/AN7CKLHgpL5Far0SV24OHAcVKyn Mewg== X-Gm-Message-State: ABy/qLZDefVIRhZW90s0NTQfomRGfISSmtcoFaA6xHM2jFkM5Qw9TGsD 930zUKK1VlG1twC61TE/oliqGDF4Ne66xvzqrkk= X-Google-Smtp-Source: APBJJlEvwIYdcin+SgJIaKf3f0NWjv2vOyb8TeOb/EE1tTkylJ9cPozIeSxvQxhFH8cAtZQs6UhkkA== X-Received: by 2002:a05:600c:3787:b0:3fa:97b3:7ce0 with SMTP id o7-20020a05600c378700b003fa97b37ce0mr3917250wmr.26.1690730174681; Sun, 30 Jul 2023 08:16:14 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:eda5:aa63:ce24:dac2]) by smtp.gmail.com with ESMTPSA id f17-20020a7bcc11000000b003fd2d33ea53sm9123027wmh.14.2023.07.30.08.16.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Jul 2023 08:16:14 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v2 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Sun, 30 Jul 2023 16:16:06 +0100 Message-Id: <20230730151606.2871391-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230730151606.2871391-1-usama.arif@bytedance.com> References: <20230730151606.2871391-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2D8BB1C0021 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kyfjio8oeytk4m8q9w9xpgt85o5yj1r6 X-HE-Tag: 1690730175-158697 X-HE-Meta: U2FsdGVkX19YcptiH1qrbPN0aVpIkDNaj2X4vu+zMY6oRRB8/tYbLS2GHnYqHBximFhRrIs3bb3T/4acSI1cw8CQoio1mY+ksiJKVX9AK2cUq4AUE3YRfoYEMVATsKc5L9rVJV89MflfuSkLFVmYF1D1RKQoH2olWBxfUPGi4lWeRi7F1fOBXB1lI8YVbYTrZpoHoNp4nYRN67pNrcF1xv2Wj0nBZIOY/pKxdnpz6Kf3vWvAePfvV93EhwNWrqEh3xZAf4lZvLu8QpHVXcUem00QruRo3Vuw84sJxiBXx/8cJUUK5WvP0o6kwgg99TIB31L4YIhp5YfgDPQftoHH4xCarFXbJA7+OE+JAye3ucnZH0N3/3oQ5OMSDWY4Pb0MijFnUE1UIUBlqwn7foIfe2WDGDrt5LINll34p3rjqSS6q+E/AIa8Z1ThhZbQzbTseI9+qmGvgBPBw9Ubn/cYxwgx5T7RGQSAtE6CaEIu3NLY/Q5DeyHDvZniV3fHyptIVb/dCgiwGbCOFF9r5pKxBBFBiQ26y4XedetBWI73tZ6qN51AcLvo7FgrPZiZj3EoIET7H2W4gVNvZgzINC6ELEN23+R1DGeFD8kfVRWd+xhWiFg63dCXGMs29rH7InzX0PASOWj6vLaBebBVWTlunGThy+BQ21Gt+OzfeqXr1qb9CdHR0BuY4FTgZrWiU+xBn+bfjp7Ux0J5ak07jC7+qJISWKazlWiRzPQUhp8TezdyupfvStTWNNhNJQEZsFHHH3utpBEyaxdT74wMfFL10ek6H/1lciKNmSaVNQoPFYaZsvFHSuCcPvq76iP+blMswGqN8xRhQiQ4xh6W2yNccLeD5bvrkn5rP5SLBpzOlS5ed1yaecXmS6NWBWuPnK58cCt9cF8SR8Qyr0zEv9X3q0eAA3BYUtzWRPntHiU6G+uF2PaDMZFksJfqdgjkyGtRfuTJPLKvDN1J3OHkDCy 5HZ3uecg mPDuN1N0lz1opBWZ8u7e96uHXTYm8gOf3P6SdcDopAPAWtLPScw7PEReyHh3wc+7l6ByqpHQ5AUy3eBpSwxi4AgZQbnBO9zpjhQ1lgH+t3ik4nUuIvElvprAv9j6+hjyGp6jVzAF/1rDbpV5uFGXibVoHaotIzwfUuKTEKEndqrYx5zHqXttCcqYSPdvhp7VenWNnvpzSwmeZHoaofAEJ3TBeh6s8m47nrACMKhJy6j1Afd31wabNpYnSQSRInvGfNz09XjgvAEOZcQpaNBH9CHmli0aDIcuSt3T+Xqda1xwdb8fnqsD5jiDX2f89G0i24gAjMwGSuCOvoXS7SpWb5FhowMPWehWEIp9QQeu1aMx7wLBZYAXuqlo+HB0PIjHCrKjco8j1euiGmp7sv318da1UU0sWtw645b+1lj80IvfsR6A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bf60545496d7..8434100f60ae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3151,6 +3151,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3158,6 +3161,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3172,6 +3184,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_reserved_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3e7978a9af73..3fff6f611c19 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */