From patchwork Mon Jun 13 06:35:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12879052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CEAACCA47B for ; Mon, 13 Jun 2022 06:35:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD36F8D0159; Mon, 13 Jun 2022 02:35:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A83238D0142; Mon, 13 Jun 2022 02:35:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 924378D0159; Mon, 13 Jun 2022 02:35:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 810A98D0142 for ; Mon, 13 Jun 2022 02:35:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 550C0120676 for ; Mon, 13 Jun 2022 06:35:36 +0000 (UTC) X-FDA: 79572251472.17.2B4CC38 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf28.hostedemail.com (Postfix) with ESMTP id F0B30C009E for ; Mon, 13 Jun 2022 06:35:35 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id y13-20020a17090a154d00b001eaaa3b9b8dso2168078pja.2 for ; Sun, 12 Jun 2022 23:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XLBHCSl3x40n/JirLCrBjaqZu+zJJs2ufPMKi7d2Sww=; b=oyts69b4avBW6IfgVF+4QpgyMTlLyDvTK8H8cgqFmeMxrl7bKHSPH0QcLfPSUXkpFx nvC3JmFuxzPBTZ08riXpF3Hnn0WwSf0Xz5YoNlEhowi0FwIsSdFz3JNWmEg73Mu8zFZL yTHQ74HZI/9sqE+udr02AltX1Wk3vQDAL+J9nK9PLL2UcKGGnsTrwzNPqPTZC0uucGgB SIfPr0Zg0Vv+ShY5B1ecsFgDlSe6swxvd7QQA3/qQk0QLSHfPE2ri/ZSGcRQNsI0mcpp c9v+ELelY0gKRbnsL7BJ+AN8XbTC/qE5w/gmOik/HPgrmqeYOvTpOOBkPbC19WeXC67S 4ytg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XLBHCSl3x40n/JirLCrBjaqZu+zJJs2ufPMKi7d2Sww=; b=UNPIeOEG+dDEoCO0/tjXhZWJ1dp7F0u2VuPYOYJp1MvqrSsTifjcJ3Ojdvj9ev+gu6 77PqkD1RwswOhZM7d4pkHDldN5FGdPnQt9484wcDr8rS+OjYjGdtQ1DaGVi+xDfTlcVZ kbc+Xvtj8SJBlXBArSzfD9zBIESQW9chUFJprfq2STtZGOEyM1q83SUEY6zSXyPKHlON qc8PFuv5koakQhtjjPeo5aBHJZj4yozXzI8/PcIVOPPJ6qYbXPtZwFjfpxVzz/nPHHtn R1KonmbfKTVbZytekCpn+1j4QNLafN3pdctkTBf47O9uAO+UwWSQxYX3T6x1YaGLg6nh DTRw== X-Gm-Message-State: AOAM533T2Nr6ZRhuGgvZE0yraXWULPrZN8CggDGsO3DevOvGM4kGm18M IgYZxdWiXCPJU76zXgpWQOh7Gw== X-Google-Smtp-Source: ABdhPJxCusXzGwHuisl+klFQ+x+A+Evfz+qDo9pEo1G3bTqjCJLS5ohmbbAPHwTAHuGVHxld3Y5Wog== X-Received: by 2002:a17:902:e742:b0:166:4d34:3be8 with SMTP id p2-20020a170902e74200b001664d343be8mr53370044plf.140.1655102135058; Sun, 12 Jun 2022 23:35:35 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id v3-20020aa799c3000000b0051bc538baadsm4366554pfi.184.2022.06.12.23.35.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Jun 2022 23:35:34 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, david@redhat.com, akpm@linux-foundation.org, corbet@lwn.net Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Muchun Song Subject: [PATCH 2/6] mm: hugetlb_vmemmap: optimize vmemmap_optimize_mode handling Date: Mon, 13 Jun 2022 14:35:08 +0800 Message-Id: <20220613063512.17540-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220613063512.17540-1-songmuchun@bytedance.com> References: <20220613063512.17540-1-songmuchun@bytedance.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655102136; a=rsa-sha256; cv=none; b=YRr2TMnaYqgk8Ccjdq7wWGXKDgQDWIeQ0fYGWvbq5B2fvNMNy3Rsf+3KDL+97qKU/mvezt ngGBbJq8QzDmjhbyrGkXRglaghtIVTakYdj/A/ZoFy5daxRSEbpb+IllbwV4xlRTUQsjJP ufJ24OKVpE4n6g8huGGGrpQreCqiVag= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655102136; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XLBHCSl3x40n/JirLCrBjaqZu+zJJs2ufPMKi7d2Sww=; b=JSr2mkPyj0LwVvCPj3mQ3HXG6z1w5RvD+1MlDpk3wFlOPq1ERwolqBVw7y7Yli3J73X/p/ B4zYKSFjmkSxnCV1S69W4Gxh/e0TNOXxd8/hvNy1qZvW8HJxISGMWnh/ItyntpEdeww8UJ Wrjh0w+kzTVvG2QTuIkq+fkAuD0Vupg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=oyts69b4; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=oyts69b4; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf28.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: c3cek5fgpbcpa3xokbbbsh6ghufco7jk X-Rspamd-Queue-Id: F0B30C009E X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1655102135-60986 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We hold an another reference to hugetlb_optimize_vmemmap_key when making vmemmap_optimize_mode on, because we use static_key to tell memory_hotplug that memory_hotplug.memmap_on_memory should be overridden. However, this rule has gone when we have introduced SECTION_CANNOT_OPTIMIZE_VMEMMAP. Therefore, we could simplify vmemmap_optimize_mode handling by not holding an another reference to hugetlb_optimize_vmemmap_key. Signed-off-by: Muchun Song Reviewed-by: Oscar Salvador Reviewed-by: Mike Kravetz --- include/linux/page-flags.h | 6 ++--- mm/hugetlb_vmemmap.c | 65 +++++----------------------------------------- 2 files changed, 9 insertions(+), 62 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b8b992cb201c..da7ccc3b16ad 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -200,8 +200,7 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, - hugetlb_optimize_vmemmap_key); +DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); /* * If the feature of optimizing vmemmap pages associated with each HugeTLB @@ -221,8 +220,7 @@ DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, */ static __always_inline const struct page *page_fixed_fake_head(const struct page *page) { - if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, - &hugetlb_optimize_vmemmap_key)) + if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key)) return page; /* diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e20a7082f2f8..132dc83f0130 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -23,42 +23,15 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -enum vmemmap_optimize_mode { - VMEMMAP_OPTIMIZE_OFF, - VMEMMAP_OPTIMIZE_ON, -}; - -DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON, - hugetlb_optimize_vmemmap_key); +DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static enum vmemmap_optimize_mode vmemmap_optimize_mode = +static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); -static void vmemmap_optimize_mode_switch(enum vmemmap_optimize_mode to) -{ - if (vmemmap_optimize_mode == to) - return; - - if (to == VMEMMAP_OPTIMIZE_OFF) - static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - static_branch_inc(&hugetlb_optimize_vmemmap_key); - WRITE_ONCE(vmemmap_optimize_mode, to); -} - static int __init hugetlb_vmemmap_early_param(char *buf) { - bool enable; - enum vmemmap_optimize_mode mode; - - if (kstrtobool(buf, &enable)) - return -EINVAL; - - mode = enable ? VMEMMAP_OPTIMIZE_ON : VMEMMAP_OPTIMIZE_OFF; - vmemmap_optimize_mode_switch(mode); - - return 0; + return kstrtobool(buf, &vmemmap_optimize_enabled); } early_param("hugetlb_free_vmemmap", hugetlb_vmemmap_early_param); @@ -103,7 +76,7 @@ static unsigned int optimizable_vmemmap_pages(struct hstate *h, unsigned long pfn = page_to_pfn(head); unsigned long end = pfn + pages_per_huge_page(h); - if (READ_ONCE(vmemmap_optimize_mode) == VMEMMAP_OPTIMIZE_OFF) + if (!READ_ONCE(vmemmap_optimize_enabled)) return 0; for (; pfn < end; pfn += PAGES_PER_SECTION) { @@ -155,7 +128,6 @@ void __init hugetlb_vmemmap_init(struct hstate *h) if (!is_power_of_2(sizeof(struct page))) { pr_warn_once("cannot optimize vmemmap pages because \"struct page\" crosses page boundaries\n"); - static_branch_disable(&hugetlb_optimize_vmemmap_key); return; } @@ -176,36 +148,13 @@ void __init hugetlb_vmemmap_init(struct hstate *h) } #ifdef CONFIG_PROC_SYSCTL -static int hugetlb_optimize_vmemmap_handler(struct ctl_table *table, int write, - void *buffer, size_t *length, - loff_t *ppos) -{ - int ret; - enum vmemmap_optimize_mode mode; - static DEFINE_MUTEX(sysctl_mutex); - - if (write && !capable(CAP_SYS_ADMIN)) - return -EPERM; - - mutex_lock(&sysctl_mutex); - mode = vmemmap_optimize_mode; - table->data = &mode; - ret = proc_dointvec_minmax(table, write, buffer, length, ppos); - if (write && !ret) - vmemmap_optimize_mode_switch(mode); - mutex_unlock(&sysctl_mutex); - - return ret; -} - static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", - .maxlen = sizeof(enum vmemmap_optimize_mode), + .data = &vmemmap_optimize_enabled, + .maxlen = sizeof(int), .mode = 0644, - .proc_handler = hugetlb_optimize_vmemmap_handler, - .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .proc_handler = proc_dobool, }, { } };