From patchwork Tue Sep 28 12:41:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12522449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 496C6C433FE for ; Tue, 28 Sep 2021 12:42:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B05D6611CA for ; Tue, 28 Sep 2021 12:42:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B05D6611CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5162B900005; Tue, 28 Sep 2021 08:42:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49FAF900002; Tue, 28 Sep 2021 08:42:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 317EF900005; Tue, 28 Sep 2021 08:42:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 1F18A900002 for ; Tue, 28 Sep 2021 08:42:50 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BAF3918017800 for ; Tue, 28 Sep 2021 12:42:49 +0000 (UTC) X-FDA: 78636946458.27.F242882 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf25.hostedemail.com (Postfix) with ESMTP id 79482B00008E for ; Tue, 28 Sep 2021 12:42:49 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id g184so21067460pgc.6 for ; Tue, 28 Sep 2021 05:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1mV+Uo2hEu6Z/m8ownrJ3eDHR3JXra4o9LXLi90WMCs=; b=pAboec8Z9r/H+WP0iIlbENqCqMmH4ubvCfHX40yht+vZi5YqbabayFsTQMNiIOT1xC 3G1ImHKYEcxXoX4dpIef05Iq1B2PGk91xxZqXPoJE/HwwkYeRFr9SFVv4V7RvP4coBDO +oNYvcwP98w2eJRrPxqKnr4P9PDD3vNtnrRpBsb+GoPqPdJYx1vh8by55mgpXmP1/kD4 i0JT7xz97Mq/aFv2eTuz6XNWzHxFZ7uV1chIKOrcmFSGb09Q+XLvM3/lj6rnsvQWCUFC uTvQQD5VE+e7ht3Y3w4w2ohDWUGMo8Dl6dJ6X9C6kDClGn+qFYyp034FiGEArwXtPXBb xrsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1mV+Uo2hEu6Z/m8ownrJ3eDHR3JXra4o9LXLi90WMCs=; b=mDoywUvWFq4Lc/gg+mRm6n/ygZ/xEfVxPOowkG1a6Y71Elpgxa4rt8oKwHk/xc+UOP htJ6AMuLP2wVktY0ndUZuFEQJ7xZAuRx6aw3eG6mgFlNQNsiTvqtCB2xfNf1Vx4wItGm dd9gPoeoMHuG8MzGu+PE+ICsprpo55D4oV4gu54aHjwLSKUd8CkCz9K/D8xtYBkuLlZ5 XjDp0PsEiXhZg29+aGGRiAkCf9BUxFF0Wju4GxWxjE9Cdp2+yMz4bw5MFY3OG/NAGF9j AfuP3ijhWTEnK3JHT6DkN0oZUStEkQV/P0rVnEUX9yeFnXhv7ZKkSfPwnNXX7jpYmwE+ QtSQ== X-Gm-Message-State: AOAM533hH8q4z4o11/GkfJn1HUTStXnRZqjhM7W7i1C4itTVhz1MWxj3 bUoDocvJSzXlNcnsKPQWjPxI1g== X-Google-Smtp-Source: ABdhPJz5Rh7kl/Qmsw94wje1ybpjrWIrJOZK6nmPmOSK0wO9kOPxrZ/uQPaOu8GHINr/LGkKYgsdZg== X-Received: by 2002:a62:ce06:0:b0:44b:b8f7:4b83 with SMTP id y6-20020a62ce06000000b0044bb8f74b83mr1067252pfg.10.1632832968519; Tue, 28 Sep 2021 05:42:48 -0700 (PDT) Received: from localhost.localdomain ([61.120.150.70]) by smtp.gmail.com with ESMTPSA id s17sm10055287pge.50.2021.09.28.05.42.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Sep 2021 05:42:48 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net, willy@infradead.org, 21cnbao@gmail.com Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, zhengqi.arch@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v5 2/5] mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key Date: Tue, 28 Sep 2021 20:41:49 +0800 Message-Id: <20210928124152.33634-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210928124152.33634-1-songmuchun@bytedance.com> References: <20210928124152.33634-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 79482B00008E X-Stat-Signature: 3rm576jdfaejw3ptt9eusukgu3seiuuw Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=pAboec8Z; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1632832969-297719 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_head_if_fake() is used throughout memory management and the conditional check requires checking a global variable, although the overhead of this check may be small, it increases when the memory cache comes under pressure. Also, the global variable will not be modified after system boot, so it is very appropriate to use static key machanism. Signed-off-by: Muchun Song Reviewed-by: Barry Song --- include/linux/hugetlb.h | 6 ------ include/linux/page-flags.h | 16 ++++++++++++++-- mm/hugetlb_vmemmap.c | 12 ++++++------ mm/memory_hotplug.c | 2 +- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3cbf60464398..a90cc88195da 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1055,12 +1055,6 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr } #endif /* CONFIG_HUGETLB_PAGE */ -#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; -#else -#define hugetlb_free_vmemmap_enabled false -#endif - static inline spinlock_t *huge_pte_lock(struct hstate *h, struct mm_struct *mm, pte_t *pte) { diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7cd386538d0c..26e540fd3393 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -185,7 +185,14 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); + +static __always_inline bool hugetlb_free_vmemmap_enabled(void) +{ + return static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key); +} /* * If the feature of freeing some vmemmap pages associated with each HugeTLB @@ -205,7 +212,7 @@ extern bool hugetlb_free_vmemmap_enabled; */ static __always_inline const struct page *page_fixed_fake_head(const struct page *page) { - if (!hugetlb_free_vmemmap_enabled) + if (!hugetlb_free_vmemmap_enabled()) return page; /* @@ -233,6 +240,11 @@ static inline const struct page *page_fixed_fake_head(const struct page *page) { return page; } + +static inline bool hugetlb_free_vmemmap_enabled(void) +{ + return false; +} #endif static __always_inline int page_is_fake_head(struct page *page) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index f4a8fca691ee..22ecb5e21686 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,9 +188,9 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled __read_mostly = - IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); -EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); +DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -204,9 +204,9 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) return -EINVAL; if (!strcmp(buf, "on")) - hugetlb_free_vmemmap_enabled = true; + static_branch_enable(&hugetlb_free_vmemmap_enabled_key); else if (!strcmp(buf, "off")) - hugetlb_free_vmemmap_enabled = false; + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); else return -EINVAL; @@ -284,7 +284,7 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled) + if (!hugetlb_free_vmemmap_enabled()) return; vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 4ea91c3ff768..66eaa4e2e76f 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1341,7 +1341,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * populate a single PMD. */ return memmap_on_memory && - !hugetlb_free_vmemmap_enabled && + !hugetlb_free_vmemmap_enabled() && IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && size == memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) &&