From patchwork Tue Apr 12 11:14:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12810521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A864C433F5 for ; Tue, 12 Apr 2022 11:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D14C6B0081; Tue, 12 Apr 2022 07:15:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AAC58D0001; Tue, 12 Apr 2022 07:15:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E646E6B0083; Tue, 12 Apr 2022 07:15:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id D84AA6B0081 for ; Tue, 12 Apr 2022 07:15:07 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 94DE7183C8A8A for ; Tue, 12 Apr 2022 11:15:07 +0000 (UTC) X-FDA: 79347970254.21.7983E5D Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf21.hostedemail.com (Postfix) with ESMTP id 110041C0005 for ; Tue, 12 Apr 2022 11:15:06 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id k14so16986582pga.0 for ; Tue, 12 Apr 2022 04:15:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cGvHlpqOyWmrThvVtDXdOJEGvDkWgL4hlnS8c7l/1q4=; b=XRCzREANSjCoRhwvwtzhbFT/7sWxP6MhWTse7rJnbP+kIveoVTCWmnX/RahjTJeP5D eX5u6qyVSX1JFNDoIOHl9fSt31jhfat1X4HM6/dAFjGU2sz3M6dAhkjVp8ySdx3GjbcW CSkcUFAJfoAbnBe3P59WMQR/j0lOxsNuGUoy8QRS+bKS7dzmRO5ftWbAI27xsRRNruQu BZRlY5ri8DRp7W8LughczdfCIn8wy9nqaTENqQGKRwLqHb3ws0pA+ilvQpqtrfOlF6+t TfS77fNS3QpSIyy6rzpWMQYqZSTWjW08YILN5ahfU+/wNJnUMnJ6Suk+VbpsbHxMs0Zr Zthw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cGvHlpqOyWmrThvVtDXdOJEGvDkWgL4hlnS8c7l/1q4=; b=PHoxjidxiRYmihkRp0xMhEnUrG3xusjzlNtUcyZPfCEfM/2za4lVwGivHir17TqRqD ZwDv0DdC4umBIRd/keUjstXi9ah4nRDD4UR44BgM3i02sY92ty542g56PjfGCiNbz6xA S6aqNoXaJwI87KWhGWzuEyE+lnuRJYcEVzcOhSflnUpPQiULIeRVJ+RiVQuSRb0YGKCR vcn3loq4mfujhx+ZVhBM6KOy9GC/KMxdQfKh0CUqQ3FoDWyQ0VmgeKeM0FBuZpvR48eL T37OmzIFmErJ3tox1oAXnpEALDgKzC7ByglqvMngvCbS7Ro9B0Qvfey/VKjk07lK9JPv ggjA== X-Gm-Message-State: AOAM533kdNsfIDuLj5hAUHpR86J3mhO7EW5D3uYuNk3vD6oJdNZ8eF+C pRxOhPRd3e7Yh8urmcgJlErSJw== X-Google-Smtp-Source: ABdhPJyqgYwh+HcY/ipT3k10fjoiHOhWnLnKSBHPvmCUUAiOyumwr2SarcrOYKYxaoUlCOZ6PA8NjA== X-Received: by 2002:a05:6a00:1488:b0:505:9dfa:1a09 with SMTP id v8-20020a056a00148800b005059dfa1a09mr16104258pfu.43.1649762105908; Tue, 12 Apr 2022 04:15:05 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id l25-20020a635719000000b0039da6cdf82dsm402507pgb.83.2022.04.12.04.15.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Apr 2022 04:15:05 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v7 1/4] mm: hugetlb_vmemmap: introduce CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP Date: Tue, 12 Apr 2022 19:14:31 +0800 Message-Id: <20220412111434.96498-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220412111434.96498-1-songmuchun@bytedance.com> References: <20220412111434.96498-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 5irmcnkww4k7bjqzjkwq68q4t6htioj3 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=XRCzREAN; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 110041C0005 X-HE-Tag: 1649762106-410689 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the size of "struct page" is not the power of two but with the feature of minimizing overhead of struct page associated with each HugeTLB is enabled, then the vmemmap pages of HugeTLB will be corrupted after remapping (panic is about to happen in theory). But this only exists when !CONFIG_MEMCG && !CONFIG_SLUB on x86_64. However, it is not a conventional configuration nowadays. So it is not a real word issue, just the result of a code review. But we have to prevent anyone from configuring that combined configurations. In order to avoid many checks like "is_power_of_2 (sizeof(struct page))" through mm/hugetlb_vmemmap.c. Introduce a new macro CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP to represent the size of struct page is power of two and CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP is configured. Then make the codes of this feature depends on this new macro. Then we could prevent anyone do any unexpected configurations. A new autoconf_ext.h is introduced as well, which serves as an extension for autoconf.h since those special configurations (e.g. CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP here) are rely on the autoconf.h (generated from Kconfig), so we cannot embed those configurations into Kconfig. After this change, it would be easy if someone want to do the similar thing (add a new CONFIG) in the future. Signed-off-by: Muchun Song Suggested-by: Luis Chamberlain Reported-by: kernel test robot Reported-by: kernel test robot --- Kbuild | 19 +++++++++++++++++++ arch/x86/mm/init_64.c | 2 +- include/linux/hugetlb.h | 2 +- include/linux/kconfig.h | 4 ++++ include/linux/mm.h | 2 +- include/linux/page-flags.h | 2 +- kernel/autoconf_ext.c | 26 ++++++++++++++++++++++++++ mm/hugetlb_vmemmap.c | 8 ++------ mm/hugetlb_vmemmap.h | 4 ++-- mm/sparse-vmemmap.c | 4 ++-- scripts/mod/Makefile | 2 ++ 11 files changed, 61 insertions(+), 14 deletions(-) create mode 100644 kernel/autoconf_ext.c diff --git a/Kbuild b/Kbuild index fa441b98c9f6..83c0d5a418d1 100644 --- a/Kbuild +++ b/Kbuild @@ -2,6 +2,12 @@ # # Kbuild for top-level directory of the kernel +# autoconf_ext.h is generated last since it depends on other generated headers, +# however those other generated headers may include autoconf_ext.h. Use the +# following macro to avoid circular dependency. + +KBUILD_CFLAGS_KERNEL += -D__EXCLUDE_AUTOCONF_EXT_H + ##### # Generate bounds.h @@ -37,6 +43,19 @@ $(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE $(call filechk,offsets,__ASM_OFFSETS_H__) ##### +# Generate autoconf_ext.h. + +autoconf_ext-file := include/generated/autoconf_ext.h + +always-y += $(autoconf_ext-file) +targets += kernel/autoconf_ext.s + +kernel/autoconf_ext.s: $(bounds-file) $(timeconst-file) $(offsets-file) + +$(autoconf_ext-file): kernel/autoconf_ext.s FORCE + $(call filechk,offsets,__LINUX_AUTOCONF_EXT_H__) + +##### # Check for missing system calls always-y += missing-syscalls diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 4b9e0012bbbf..9b8dfa6e4da8 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1268,7 +1268,7 @@ static struct kcore_list kcore_vsyscall; static void __init register_page_bootmem_info(void) { -#if defined(CONFIG_NUMA) || defined(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) +#if defined(CONFIG_NUMA) || defined(CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP) int i; for_each_online_node(i) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ac2ece9e9c79..d42de8abd2b6 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -623,7 +623,7 @@ struct hstate { unsigned int nr_huge_pages_node[MAX_NUMNODES]; unsigned int free_huge_pages_node[MAX_NUMNODES]; unsigned int surplus_huge_pages_node[MAX_NUMNODES]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP unsigned int optimize_vmemmap_pages; #endif #ifdef CONFIG_CGROUP_HUGETLB diff --git a/include/linux/kconfig.h b/include/linux/kconfig.h index 20d1079e92b4..f2cb8be6d8d0 100644 --- a/include/linux/kconfig.h +++ b/include/linux/kconfig.h @@ -4,6 +4,10 @@ #include +#ifndef __EXCLUDE_AUTOCONF_EXT_H +#include +#endif + #ifdef CONFIG_CPU_BIG_ENDIAN #define __BIG_ENDIAN 4321 #else diff --git a/include/linux/mm.h b/include/linux/mm.h index e0ad13486035..4c36f77a5745 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3186,7 +3186,7 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) } #endif -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse); int vmemmap_remap_alloc(unsigned long start, unsigned long end, diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b70124b9c7c1..e409b10cd677 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -199,7 +199,7 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, hugetlb_optimize_vmemmap_key); diff --git a/kernel/autoconf_ext.c b/kernel/autoconf_ext.c new file mode 100644 index 000000000000..8475735c6fc9 --- /dev/null +++ b/kernel/autoconf_ext.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Generate definitions needed by the preprocessor. + * This code generates raw asm output which is post-processed + * to extract and format the required data. + */ +#include +#include +#include + +int main(void) +{ + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && + is_power_of_2(sizeof(struct page))) { + /* + * The 2nd parameter of DEFINE() will go into the comments. Do + * not pass 1 directly to it to make the generated macro more + * clear for the readers. + */ + DEFINE(CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP, + IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && + is_power_of_2(sizeof(struct page))); + } + + return 0; +} diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 2655434a946b..be73782cc1cf 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -178,6 +178,7 @@ #include "hugetlb_vmemmap.h" +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP /* * There are a lot of struct page structures associated with each HugeTLB page. * For tail pages, the value of compound_head is the same. So we can reuse first @@ -194,12 +195,6 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static int __init hugetlb_vmemmap_early_param(char *buf) { - /* We cannot optimize if a "struct page" crosses page boundaries. */ - if (!is_power_of_2(sizeof(struct page))) { - pr_warn("cannot free vmemmap pages because \"struct page\" crosses page boundaries\n"); - return 0; - } - if (!buf) return -EINVAL; @@ -300,3 +295,4 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can optimize %d vmemmap pages for %s\n", h->optimize_vmemmap_pages, h->name); } +#endif /* CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP */ diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 109b0a53b6fe..3afae3ff37fa 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,7 +10,7 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_alloc(struct hstate *h, struct page *head); void hugetlb_vmemmap_free(struct hstate *h, struct page *head); void hugetlb_vmemmap_init(struct hstate *h); @@ -41,5 +41,5 @@ static inline unsigned int hugetlb_optimize_vmemmap_pages(struct hstate *h) { return 0; } -#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ +#endif /* CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP */ #endif /* _LINUX_HUGETLB_VMEMMAP_H */ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 52f36527bab3..6c7f1a9ce2dd 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -34,7 +34,7 @@ #include #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP /** * struct vmemmap_remap_walk - walk vmemmap page table * @@ -420,7 +420,7 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, return 0; } -#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ +#endif /* CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP */ /* * Allocate a block of memory to be used to back the virtual memory map diff --git a/scripts/mod/Makefile b/scripts/mod/Makefile index c9e38ad937fd..f82ab128c086 100644 --- a/scripts/mod/Makefile +++ b/scripts/mod/Makefile @@ -1,6 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 OBJECT_FILES_NON_STANDARD := y CFLAGS_REMOVE_empty.o += $(CC_FLAGS_LTO) +# See comments in Kbuild +KBUILD_CFLAGS_KERNEL += -D__EXCLUDE_AUTOCONF_EXT_H hostprogs-always-y += modpost mk_elfconfig always-y += empty.o From patchwork Tue Apr 12 11:14:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12810522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 376B4C433EF for ; Tue, 12 Apr 2022 11:15:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF48B8D0001; Tue, 12 Apr 2022 07:15:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA4A26B0083; Tue, 12 Apr 2022 07:15:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6CAA8D0001; Tue, 12 Apr 2022 07:15:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 9B3106B0082 for ; Tue, 12 Apr 2022 07:15:12 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 72A5D805F2 for ; Tue, 12 Apr 2022 11:15:12 +0000 (UTC) X-FDA: 79347970464.13.95E4F31 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 098D9180005 for ; Tue, 12 Apr 2022 11:15:11 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id n18so16486784plg.5 for ; Tue, 12 Apr 2022 04:15:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0UA2E7Swg+iesG+ux9AwzGNddrfmiNsjUbQGq/dsMiM=; b=KyuopGe1m1WVOFP9p32hmnKBOcBz7duXJkJk9fS7PkoLF0O/RrKyn0sKqaHT7mwP2o Oemd+xORRsXBp6BQ77rF2ECHQKbcsydrcRVQF+xON0R1Tc3192LcDa9tNR+Yhg72Hcko WDNb0O+5PAynoZQW4wML+O/ykXnlIcgyVq39+gR98yIkZ48xzIJDvoZWzM+bvssO9tUm 82+Bu3gpNvE/FAoQFNukD7gOFZ1aOiNOmkU6gSOXM0qJyOIOfevULObXLuMq/Gcki9/P 1JI9GHBns+7OgvpyjHhLm8EgsSAg1TAA9L0/viQqSSKRt1/ETaLApAa7XkNdQSziZJcQ FSQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0UA2E7Swg+iesG+ux9AwzGNddrfmiNsjUbQGq/dsMiM=; b=XX3NAOiqUHqxsnPzVi5MJUownHGZ0mhjCQ0I0xNeO2klPza6j8HzixTWkMcMX29wOY W1flhW1MIC+1Uh3msDRnNZvAa9PKX6L25Vop+RXjTu60Biv81P5aalUsu48ZfGm53mOe LjqjcDAbuTvjpbhMXukGSwRJoLkUKJ9nGVwvYerUrDcZnJScn+w9q6XKFFuceTlHBWGp XC7/hgfjKDgIS+3hW2aJtQGC2T2HtX0qPETfA6xSrE0jESCrcYMZ4zP69H62R8Gm0++n hUtavP2OpJ0cmx2unDtFNY/Gc6ZkxQk9iCQmY3HlV1C5IMeuNVHRRd8ASi2V2/UzaPqM SePw== X-Gm-Message-State: AOAM5320te4j1g/myweDDzhQNGow+SgEuO38P5zrXdsseL/75e+N9OSS lktsYjklMvgS+B65xeAk1T4/ng== X-Google-Smtp-Source: ABdhPJw0PpvcK/RfW3RHG1GTMBjicwgJXBBNM/iVvZwsd6IE8JTVr0XCaF+IWKEuybeVVBGmHde9Gw== X-Received: by 2002:a17:902:c94d:b0:158:4e50:7a32 with SMTP id i13-20020a170902c94d00b001584e507a32mr12971286pla.163.1649762110989; Tue, 12 Apr 2022 04:15:10 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id l25-20020a635719000000b0039da6cdf82dsm402507pgb.83.2022.04.12.04.15.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Apr 2022 04:15:10 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v7 2/4] mm: memory_hotplug: override memmap_on_memory when hugetlb_free_vmemmap=on Date: Tue, 12 Apr 2022 19:14:32 +0800 Message-Id: <20220412111434.96498-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220412111434.96498-1-songmuchun@bytedance.com> References: <20220412111434.96498-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: mzrh7zwhu5qpybhpf6dxgxik9ongkoni Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=KyuopGe1; spf=pass (imf16.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 098D9180005 X-HE-Tag: 1649762111-774941 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When "hugetlb_free_vmemmap=on" and "memory_hotplug.memmap_on_memory" are both passed to boot cmdline, the variable of "memmap_on_memory" will be set to 1 even if the vmemmap pages will not be allocated from the hotadded memory since the former takes precedence over the latter. In the next patch, we want to enable or disable the feature of freeing vmemmap pages of HugeTLB via sysctl. We need a way to know if the feature of memory_hotplug.memmap_on_memory is enabled when enabling the feature of freeing vmemmap pages since those two features are not compatible, however, the variable of "memmap_on_memory" cannot indicate this nowadays. Do not set "memmap_on_memory" to 1 when both parameters are passed to cmdline, in this case, "memmap_on_memory" could indicate if this feature is enabled by the users. Also introduce mhp_memmap_on_memory() helper to move the definition of "memmap_on_memory" to the scope of CONFIG_MHP_MEMMAP_ON_MEMORY. In the next patch, mhp_memmap_on_memory() will also be exported to be used in hugetlb_vmemmap.c. Signed-off-by: Muchun Song --- mm/memory_hotplug.c | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 74430f88853d..f6eab03397d3 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -42,14 +42,36 @@ #include "internal.h" #include "shuffle.h" +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +static int memmap_on_memory_set(const char *val, const struct kernel_param *kp) +{ + if (hugetlb_optimize_vmemmap_enabled()) + return 0; + return param_set_bool(val, kp); +} + +static const struct kernel_param_ops memmap_on_memory_ops = { + .flags = KERNEL_PARAM_OPS_FL_NOARG, + .set = memmap_on_memory_set, + .get = param_get_bool, +}; /* * memory_hotplug.memmap_on_memory parameter */ static bool memmap_on_memory __ro_after_init; -#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY -module_param(memmap_on_memory, bool, 0444); +module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); + +static inline bool mhp_memmap_on_memory(void) +{ + return memmap_on_memory; +} +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} #endif enum { @@ -1272,9 +1294,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * altmap as an alternative source of memory, and we do not exactly * populate a single PMD. */ - return memmap_on_memory && - !hugetlb_optimize_vmemmap_enabled() && - IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && + return mhp_memmap_on_memory() && size == memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) && IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); @@ -2081,7 +2101,7 @@ static int __ref try_remove_memory(u64 start, u64 size) * We only support removing memory added with MHP_MEMMAP_ON_MEMORY in * the same granularity it was added - a single memory block. */ - if (memmap_on_memory) { + if (mhp_memmap_on_memory()) { nr_vmemmap_pages = walk_memory_blocks(start, size, NULL, get_nr_vmemmap_pages_cb); if (nr_vmemmap_pages) { From patchwork Tue Apr 12 11:14:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12810523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67462C433EF for ; Tue, 12 Apr 2022 11:15:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 034FB6B0082; Tue, 12 Apr 2022 07:15:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F26298D0002; Tue, 12 Apr 2022 07:15:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEEB26B0085; Tue, 12 Apr 2022 07:15:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id D21426B0082 for ; Tue, 12 Apr 2022 07:15:17 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7EDCA183CB07B for ; Tue, 12 Apr 2022 11:15:17 +0000 (UTC) X-FDA: 79347970674.27.1079D4C Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf25.hostedemail.com (Postfix) with ESMTP id 33C3CA0006 for ; Tue, 12 Apr 2022 11:15:17 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id n18so16486972plg.5 for ; Tue, 12 Apr 2022 04:15:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EfqzshiMxPgqo91bqRrogKECfyK/4BP/Cl7KQelj6Dg=; b=bPeubF5u9edpfnmGs04xInTA5275jlp30BzDSTI+ieNDlG4u7/tJ7Rz5RTrhAeA0DD xb3XM6ti+GrjarPym0NKYWxAGrxIlYSSVfiGUr/dIS0YAfsAa/cWKAEtZ14ZadQw81uF LUS8d+iv9X73VtIOegsMfQm/be60nBbUDvuoouSr3+xA55Dq6i6d0JF+5DrQD+VsGKca gfiglXT5A5ClCVJrA0q6LCJhwh1zeAiWiZ6yptsEyZShruJBkFXplBrDQLtEX9sL3YMz /ydrd5vHe6tt3lhL/kJ/qf7IMVRHgzErt1mIGmZMgxdZtI2MWBEHsAu/Ypq3AsqyPbJ+ cBEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EfqzshiMxPgqo91bqRrogKECfyK/4BP/Cl7KQelj6Dg=; b=DXSHnFP9Ntvb2ZiTE12oKGy0ob76gMXCE1RfJv4fhiPHyLJ8oo0yhHF3Nnj1NxCtBz 3LKADtJ4jFAlNgWj0Ag4CD2W8Nbx/6viDTCZDO81HwzCShA9kMDnpAl3h+MJbsODPp1+ gvCJqq0aRECk7l/ubOvszgTTYyiT1hLgh+xQX8IPohHArgES1p3onMWQ8D3/77UXntEM ifWziRFRH3SMWL8o0bDPiDCI4RKCpIafnhWfLQP4Y8oJx50X+PCb8mHY6iHOapPcN5zu E8t0Jts65U7jNp8v26BC/t9Ii5oI/fn3N8HeWkxy24xiL+mvnNezX7cmgZfh1EqNepjT r8ow== X-Gm-Message-State: AOAM532bPowj7pciZyRscyPCZDhGOsp7n7LrLyVX0TWIZjaM7IVONncE QRTEWNtn3KJbAeu4GHFl3ku1DA== X-Google-Smtp-Source: ABdhPJxzz1f3N9EC3y+nmsxylhXN2TisCoqArOTZmzLYhAtuo5R69JHBs9+2JAq88yjyDr3N5O/qwA== X-Received: by 2002:a17:90b:3b86:b0:1cb:c243:aed3 with SMTP id pc6-20020a17090b3b8600b001cbc243aed3mr3678637pjb.202.1649762116200; Tue, 12 Apr 2022 04:15:16 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id l25-20020a635719000000b0039da6cdf82dsm402507pgb.83.2022.04.12.04.15.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Apr 2022 04:15:15 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v7 3/4] mm: hugetlb_vmemmap: use kstrtobool for hugetlb_vmemmap param parsing Date: Tue, 12 Apr 2022 19:14:33 +0800 Message-Id: <20220412111434.96498-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220412111434.96498-1-songmuchun@bytedance.com> References: <20220412111434.96498-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 33C3CA0006 X-Stat-Signature: gn9tu35chedkic8qqw9hqqgzgjkke659 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=bPeubF5u; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspam-User: X-HE-Tag: 1649762117-956240 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use kstrtobool rather than open coding "on" and "off" parsing in mm/hugetlb_vmemmap.c, which is more powerful to handle all kinds of parameters like 'Yy1Nn0' or [oO][NnFf] for "on" and "off". Signed-off-by: Muchun Song --- Documentation/admin-guide/kernel-parameters.txt | 6 +++--- mm/hugetlb_vmemmap.c | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index f3cf9f21f6eb..6ea428023d51 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1669,10 +1669,10 @@ enabled. Allows heavy hugetlb users to free up some more memory (7 * PAGE_SIZE for each 2MB hugetlb page). - Format: { on | off (default) } + Format: { [oO][Nn]/Y/y/1 | [oO][Ff]/N/n/0 (default) } - on: enable the feature - off: disable the feature + [oO][Nn]/Y/y/1: enable the feature + [oO][Ff]/N/n/0: disable the feature Built with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, the default is on. diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index be73782cc1cf..4b6a5cf16f11 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -195,15 +195,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static int __init hugetlb_vmemmap_early_param(char *buf) { - if (!buf) + bool enable; + + if (kstrtobool(buf, &enable)) return -EINVAL; - if (!strcmp(buf, "on")) + if (enable) static_branch_enable(&hugetlb_optimize_vmemmap_key); - else if (!strcmp(buf, "off")) - static_branch_disable(&hugetlb_optimize_vmemmap_key); else - return -EINVAL; + static_branch_disable(&hugetlb_optimize_vmemmap_key); return 0; } From patchwork Tue Apr 12 11:14:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12810524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91923C433EF for ; Tue, 12 Apr 2022 11:15:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27E378D0003; Tue, 12 Apr 2022 07:15:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 231068D0002; Tue, 12 Apr 2022 07:15:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F58B8D0003; Tue, 12 Apr 2022 07:15:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 02FDD8D0002 for ; Tue, 12 Apr 2022 07:15:23 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CCABA23120 for ; Tue, 12 Apr 2022 11:15:22 +0000 (UTC) X-FDA: 79347970884.18.B57A2AF Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf08.hostedemail.com (Postfix) with ESMTP id 56101160006 for ; Tue, 12 Apr 2022 11:15:22 +0000 (UTC) Received: by mail-pf1-f170.google.com with SMTP id a42so10666291pfx.7 for ; Tue, 12 Apr 2022 04:15:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tOkU/RARB4HzUE+K871kdn+JSt0MwPZIDE1q0TKS/DQ=; b=OKnLJ4QCbkZSQSjqhcW5b0CZeXAOfSeBDKKAKfyreFoxa6pTgaQWKP0BesaFcmYzgF I/UYTZWev+8+PIKP2qq2wamq0V90RFdhK0z1JnJFHDgPJradlTJ7uBV1C1jV4EB/ht2m dKhhpsJ5c7o10KdScpnuFUpfFbGEa1sHsBQumLrPcVpJMirCCADQu6N/wFYxQYtNJRyq Jm5SEyUC7WD93CubrUlN/yY+ukPhZ12oig0i20MKS5RDTacSHxVldY+zULCDcqdmMXZ5 wzAWxYt77fWGOyd0gg3SXBmQLJzpsuZPrPoHo40qFZqB7X/3gT+gVrqsop8m3CrRNwwb 0zag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tOkU/RARB4HzUE+K871kdn+JSt0MwPZIDE1q0TKS/DQ=; b=GHmCjXohQlGg/z3jQAIwr9VXTAloBTYn2z+cJiwzJDDnrafutYGQr/JPS24gw6fxq3 4zA832VdeeA1u2ZnsrRi9gJIKdze6eE7tKodUqEa+7NyY/q/dnVpE1CFcbAGDrqdU3f6 NwCwZ0i6PzIrQA+ZMrtwCOrmBwEPu6LEUbpqv3NrpZQi8Z3ZCimDzACapl9lyb7Gkr/a 4DMuaeS++y9kgJN9n0uHQDZQQZFv5P32zYjWt9obWM16DTEPEwPwBw8kFUs2+37r4bRB kyjaZkx+S/jzVf7dckQrzUP0AEOFSdvPnvbwe0EEJMnexM1ydu+85tFVNxRy15o0Ge3k /juw== X-Gm-Message-State: AOAM5338RCSy2alqSPggw1allESMUlD37UcALIq732XMmuo9Nj1F/exe meYdc2UuX0HCwvaIqZmU/nqyBw== X-Google-Smtp-Source: ABdhPJwTtrUS2x+SQP3CaQv+UzJaK+MYHoLSMznZiPSDVhpNPWgS+Ftk27GJKqldQa/Z8kINzEsRCQ== X-Received: by 2002:a63:195f:0:b0:399:1f0e:a21d with SMTP id 31-20020a63195f000000b003991f0ea21dmr29964914pgz.393.1649762121328; Tue, 12 Apr 2022 04:15:21 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id l25-20020a635719000000b0039da6cdf82dsm402507pgb.83.2022.04.12.04.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Apr 2022 04:15:21 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com, masahiroy@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v7 4/4] mm: hugetlb_vmemmap: add hugetlb_optimize_vmemmap sysctl Date: Tue, 12 Apr 2022 19:14:34 +0800 Message-Id: <20220412111434.96498-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220412111434.96498-1-songmuchun@bytedance.com> References: <20220412111434.96498-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: orfp7pg8dsdq367xp44np5fhoa91z5m9 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=OKnLJ4QC; spf=pass (imf08.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 56101160006 X-HE-Tag: 1649762122-738948 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We must add hugetlb_free_vmemmap=on (or "off") to the boot cmdline and reboot the server to enable or disable the feature of optimizing vmemmap pages associated with HugeTLB pages. However, rebooting usually takes a long time. So add a sysctl to enable or disable the feature at runtime without rebooting. Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages from buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 pages per 1GB HugeTLB page), whereas already allocated HugeTLB pages will not be optimized. When those optimized HugeTLB pages are freed from the HugeTLB pool to the buddy allocator, the vmemmap pages representing that range needs to be remapped again and the vmemmap pages discarded earlier need to be rellocated again. If your use case is that HugeTLB pages are allocated 'on the fly' instead of being pulled from the HugeTLB pool, you should weigh the benefits of memory savings against the more overhead of allocation or freeing HugeTLB pages between the HugeTLB pool and the buddy allocator. Another behavior to note is that if the system is under heavy memory pressure, it could prevent the user from freeing HugeTLB pages from the HugeTLB pool to the buddy allocator since the allocation of vmemmap pages could be failed, you have to retry later if your system encounter this situation. Once disabled, the vmemmap pages of subsequent allocation of HugeTLB pages from buddy allocator will not be optimized, whereas already optimized HugeTLB pages will not be affected. If you want to make sure there is no optimized HugeTLB pages, you can set "nr_hugepages" to 0 first and then disable this. Signed-off-by: Muchun Song --- Documentation/admin-guide/sysctl/vm.rst | 27 ++++++++++ include/linux/memory_hotplug.h | 9 ++++ mm/hugetlb_vmemmap.c | 90 +++++++++++++++++++++++++++++---- mm/hugetlb_vmemmap.h | 4 +- mm/memory_hotplug.c | 7 +-- 5 files changed, 121 insertions(+), 16 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 747e325ebcd0..5c804ada85c5 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -562,6 +562,33 @@ Change the minimum size of the hugepage pool. See Documentation/admin-guide/mm/hugetlbpage.rst +hugetlb_optimize_vmemmap +======================== + +Enable (set to 1) or disable (set to 0) the feature of optimizing vmemmap pages +associated with each HugeTLB page. + +Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages from +buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 pages +per 1GB HugeTLB page), whereas already allocated HugeTLB pages will not be +optimized. When those optimized HugeTLB pages are freed from the HugeTLB pool +to the buddy allocator, the vmemmap pages representing that range needs to be +remapped again and the vmemmap pages discarded earlier need to be rellocated +again. If your use case is that HugeTLB pages are allocated 'on the fly' +instead of being pulled from the HugeTLB pool, you should weigh the benefits of +memory savings against the more overhead of allocation or freeing HugeTLB pages +between the HugeTLB pool and the buddy allocator. Another behavior to note is +that if the system is under heavy memory pressure, it could prevent the user +from freeing HugeTLB pages from the HugeTLB pool to the buddy allocator since +the allocation of vmemmap pages could be failed, you have to retry later if +your system encounter this situation. + +Once disabled, the vmemmap pages of subsequent allocation of HugeTLB pages from +buddy allocator will not be optimized, whereas already optimized HugeTLB pages +will not be affected. If you want to make sure there is no optimized HugeTLB +pages, you can set "nr_hugepages" to 0 first and then disable this. + + nr_hugepages_mempolicy ====================== diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 7ab15d6fb227..93f2cbea0f9b 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -348,4 +348,13 @@ void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); #endif /* CONFIG_MEMORY_HOTPLUG */ +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +bool mhp_memmap_on_memory(void); +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} +#endif + #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4b6a5cf16f11..94571d63916d 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -176,6 +176,7 @@ */ #define pr_fmt(fmt) "HugeTLB: " fmt +#include #include "hugetlb_vmemmap.h" #ifdef CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP @@ -189,21 +190,40 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) +enum vmemmap_optimize_mode { + VMEMMAP_OPTIMIZE_OFF, + VMEMMAP_OPTIMIZE_ON, +}; + DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); +static enum vmemmap_optimize_mode vmemmap_optimize_mode = + IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); + +static void vmemmap_optimize_mode_switch(enum vmemmap_optimize_mode to) +{ + if (vmemmap_optimize_mode == to) + return; + + if (to == VMEMMAP_OPTIMIZE_OFF) + static_branch_dec(&hugetlb_optimize_vmemmap_key); + else + static_branch_inc(&hugetlb_optimize_vmemmap_key); + vmemmap_optimize_mode = to; +} + static int __init hugetlb_vmemmap_early_param(char *buf) { bool enable; + enum vmemmap_optimize_mode mode; if (kstrtobool(buf, &enable)) return -EINVAL; - if (enable) - static_branch_enable(&hugetlb_optimize_vmemmap_key); - else - static_branch_disable(&hugetlb_optimize_vmemmap_key); + mode = enable ? VMEMMAP_OPTIMIZE_ON : VMEMMAP_OPTIMIZE_OFF; + vmemmap_optimize_mode_switch(mode); return 0; } @@ -236,8 +256,10 @@ int hugetlb_vmemmap_alloc(struct hstate *h, struct page *head) */ ret = vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); - if (!ret) + if (!ret) { ClearHPageVmemmapOptimized(head); + static_branch_dec(&hugetlb_optimize_vmemmap_key); + } return ret; } @@ -251,6 +273,8 @@ void hugetlb_vmemmap_free(struct hstate *h, struct page *head) if (!vmemmap_pages) return; + static_branch_inc(&hugetlb_optimize_vmemmap_key); + vmemmap_addr += RESERVE_VMEMMAP_SIZE; vmemmap_end = vmemmap_addr + (vmemmap_pages << PAGE_SHIFT); vmemmap_reuse = vmemmap_addr - PAGE_SIZE; @@ -260,7 +284,9 @@ void hugetlb_vmemmap_free(struct hstate *h, struct page *head) * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_addr, @vmemmap_end] is mapped to. */ - if (!vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) + if (vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) + static_branch_dec(&hugetlb_optimize_vmemmap_key); + else SetHPageVmemmapOptimized(head); } @@ -277,9 +303,6 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_optimize_vmemmap_enabled()) - return; - vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; /* * The head page is not to be freed to buddy allocator, the other tail @@ -295,4 +318,53 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can optimize %d vmemmap pages for %s\n", h->optimize_vmemmap_pages, h->name); } + +#ifdef CONFIG_PROC_SYSCTL +static int hugetlb_optimize_vmemmap_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, + loff_t *ppos) +{ + int ret; + enum vmemmap_optimize_mode mode; + static DEFINE_MUTEX(sysctl_mutex); + + if (write && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + mutex_lock(&sysctl_mutex); + mode = vmemmap_optimize_mode; + table->data = &mode; + ret = proc_dointvec_minmax(table, write, buffer, length, ppos); + if (write && !ret) + vmemmap_optimize_mode_switch(mode); + mutex_unlock(&sysctl_mutex); + + return ret; +} + +static struct ctl_table hugetlb_vmemmap_sysctls[] = { + { + .procname = "hugetlb_optimize_vmemmap", + .maxlen = sizeof(enum vmemmap_optimize_mode), + .mode = 0644, + .proc_handler = hugetlb_optimize_vmemmap_handler, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { } +}; + +static __init int hugetlb_vmemmap_sysctls_init(void) +{ + /* + * The vmemmap pages cannot be optimized if + * "memory_hotplug.memmap_on_memory" is enabled. + */ + if (!mhp_memmap_on_memory()) + register_sysctl_init("vm", hugetlb_vmemmap_sysctls); + + return 0; +} +late_initcall(hugetlb_vmemmap_sysctls_init); +#endif /* CONFIG_PROC_SYSCTL */ #endif /* CONFIG_HUGETLB_PAGE_HAS_OPTIMIZE_VMEMMAP */ diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 3afae3ff37fa..63ae2766ffe0 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -21,7 +21,9 @@ void hugetlb_vmemmap_init(struct hstate *h); */ static inline unsigned int hugetlb_optimize_vmemmap_pages(struct hstate *h) { - return h->optimize_vmemmap_pages; + if (hugetlb_optimize_vmemmap_enabled()) + return h->optimize_vmemmap_pages; + return 0; } #else static inline int hugetlb_vmemmap_alloc(struct hstate *h, struct page *head) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f6eab03397d3..af844a01e956 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -63,15 +63,10 @@ static bool memmap_on_memory __ro_after_init; module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); -static inline bool mhp_memmap_on_memory(void) +bool mhp_memmap_on_memory(void) { return memmap_on_memory; } -#else -static inline bool mhp_memmap_on_memory(void) -{ - return false; -} #endif enum {