From patchwork Fri Oct 11 08:05:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hua Su X-Patchwork-Id: 13832221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F327FCFC60F for ; Fri, 11 Oct 2024 08:05:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AD8C6B00A1; Fri, 11 Oct 2024 04:05:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85D926B00A2; Fri, 11 Oct 2024 04:05:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D68F6B00A3; Fri, 11 Oct 2024 04:05:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4BBFC6B00A1 for ; Fri, 11 Oct 2024 04:05:57 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2E70CA0F18 for ; Fri, 11 Oct 2024 08:05:49 +0000 (UTC) X-FDA: 82660587870.27.95AA831 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf18.hostedemail.com (Postfix) with ESMTP id 7663F1C0006 for ; Fri, 11 Oct 2024 08:05:54 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ED1EzXdG; spf=pass (imf18.hostedemail.com: domain of suhua.tanke@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=suhua.tanke@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728633771; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=cyTFlm4m/KDy5BUSGM2UwZPLgDsNUyTZVqJf5DcscI8=; b=02+hgWYaCOkN8B4KvuvrKy9M95+thPiPn4ktyaoC1Mq21+ZtCTaaXif9LppXt9tQqckenI kS8+jNNs7/oiCmrzSNxPEoTEkioB15vJT10FQlwNMK7rmtmg+ErdrS4FsHRPtFYUd2Duny 9hHCfURMcePgkHBpDuhRYn6PDlnUh0c= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ED1EzXdG; spf=pass (imf18.hostedemail.com: domain of suhua.tanke@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=suhua.tanke@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728633771; a=rsa-sha256; cv=none; b=WJ1ST5VkJgTYMcQSu1Au5EVqKRDC0ZE5D4OZ2xRCHNrpSZg1DN/1S90AE6W+8zcQ6sIgYE wkWe2wh4XJXOzww7IK8gZhnaVejgE1Tr1WA3CqttX+11yF29JIfJOti6hyU6OalBgYGWk3 odBw+YVEjTpViuduVto3N/pNEWSmA5I= Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-71def8abc2fso1379906b3a.1 for ; Fri, 11 Oct 2024 01:05:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728633954; x=1729238754; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=cyTFlm4m/KDy5BUSGM2UwZPLgDsNUyTZVqJf5DcscI8=; b=ED1EzXdGbB8ssC3OM9nK3sWNiJYX/eUMsVQ1gnOJL47b+pFh38m4BPEZE3ExdUFCQx h8m3s6KF4m+b08KrBSPkzhuOxQJYMlMnQZ2j1x0nsg98rJE2YfARUJrHN47XYRrbn7Tv 5DSPPf97Tbl2UM5KmQkCuxr26F/xfZSGh8Ik+z5tv0LChlvRLjm8rzejA970MCV1NdLC o4FICVxWP4HyjOya4a+WOTmCsp667ke1K47CA8bLGmxUwdAPY8D1QlEOiQu10y3m/pji sdKlHu0kamIjPhKnFZJg8kpNcxaPXCbVDwhzzYmbA/z0ziefounQLx15oojEJTQNp0Qb tpvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728633954; x=1729238754; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cyTFlm4m/KDy5BUSGM2UwZPLgDsNUyTZVqJf5DcscI8=; b=p95zIXQPTUsBDLj+udfXy+uITr6sTna3X2fKVDdcRuHC2ZpzUFeSGHWkhfD8ExiKOm VJEFvvPhox/Mn7Zxa84qUKzo5oG5jyzf3QT4RkLbYDDEhTIZARZ162/pCAkKN8XJ0QQZ w4ZbbHGN8VOG+2PKc0PZVNQ1Las3zngR2gPA0hmeaHp8DWZWhEuryp8euJqhTnONepfT hYRCa5IbpKWBuM/3dceK9g/RhM//150bAsApKKRy3mAnspANPxumdSyjnq2bTBsWvqWa Bl42AEmjRwpZbEw91O5iofGUb+T34ozLsDlv6RrLbhECpRCz6YV22mvzFdDMc2hkCLIn yM3Q== X-Gm-Message-State: AOJu0Yw0U4UDHPIyA6HaZDiWHUR4bWKUuIfPaAaL1qU/mOWRfKWqSxn6 1HcUTOn/jijYM4qH7+iQbglE8sl30oXzoPmo7LjOY03xPkC9F5oKMm1lJj9z X-Google-Smtp-Source: AGHT+IEPfa7gZvPRyo8ByepVRETQkhFpHB8/vsFkvFzynMFqH7IIfh9guYjrHr6K/nM9x22xyqa83A== X-Received: by 2002:a05:6a00:3189:b0:71e:3b8f:918 with SMTP id d2e1a72fcca58-71e3b8f11eamr1354871b3a.7.1728633953799; Fri, 11 Oct 2024 01:05:53 -0700 (PDT) Received: from suhua-virtual-machine.localdomain ([36.112.24.11]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e2ab10205sm2131409b3a.220.2024.10.11.01.05.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Oct 2024 01:05:53 -0700 (PDT) From: suhua X-Google-Original-From: suhua To: muchun.song@linux.dev, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, suhua.tanke@gmail.com, suhua Subject: [PATCH] mm/hugetlb: Concentrated releases memory when cmdline specifies node requests for large pages Date: Fri, 11 Oct 2024 16:05:43 +0800 Message-Id: <20241011080543.2422-1-suhua1@kingsoft.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7663F1C0006 X-Stat-Signature: pa9h673jd1wbw8zqo7zphduusd8zy7fk X-Rspam-User: X-HE-Tag: 1728633954-890726 X-HE-Meta: U2FsdGVkX1/4iaWoR39xfHTA3XP9ZDzjvEbgNdTfvIWgRqy5TV73yAKim8duULIDmlc+2TorY3ZwaMR43fpwT0TsnECnJo8Fb1UVR2BR6HxX6Oouo0u1BeGVHaF9RpH6c1TaMMKTOOME9K+OHTy+YDrP2IEU2SeDSVId7EwaWeaaUd7Y0lGMmfmTf+Ikcu1sz7njuyKMHw3whRmQHCbPzxYMNtJOUjiE+jqgqbSeofvJ7IfrQOz9B5ENbFKZ4q/ZzMvieJHaSyipAJA0g1s8YgYO8ESXY6ZGqjpK9q6gw5DgEc7gJ0OVlgWtHnF4mjpd3xI+FGFzofIFJ7rPqBJY/huFtVjvKckqfrjRyXkS+MQuwR24eqf0gGnZ8MbGYfStVPVN2CZPWYHgA6qEs3c0gUiGsj3AjBRZPPtlq4KZubtpGnwZwjtbn5ntGApMROrkARXfUP1RKXB1kEK8XAfn1ji7Zuu/xPzLHYcCmaHE14++LDD7N0UkjsshNW3m43/OBPo617R0IX795qcrZ5CpgNC248lbelnFjkdc+j7qB1TUnFvpEo6cm4w0lBzLVLcD+UCXiPARRX6aCu3yGwJCzL/ful+viEtMLhLRK76ifDwllRWdLYUqNj+ksHIZnNsnDe1MRFI9R2Eo9huC+JUXwvmnsjNjwIP/c8dfIk5CkhrCArxOVW9auZPQ791yvEHPXMNmcR0PMdJcFH3wi9nrGSPH3OYA9SFK6A5+3M++EA/2+eJtVUq43W9OZVGzSIINKcBpFFImZIGysyGm5dc4yWw08vOb8NAbSKRYcL6ChzMlOTF1Z2ZF35nAsubMzD6XxyJ604tDUKcZFLf2/cdPbHdF6BxCoa6gg/mReywQKIo7mXUkMV5ickA0yWeFAbTl9XcNE+Nnje5J0R8kiqGk6KBNP1T1y7RA6sKn1Dgkrk5MtOZ50pf6Cr+eomruUOfE4wEdS5JAO5KnQ9oRO8I 2LX7yEE2 tJB8zh9N1ghAGbUW+EDIvchXb6s9X0VsVgz6jULGZlPFaOt1hphmBMuQujzHT8ZQ7mj0Iz+qdd3koHMAXWrssdJNuYaGm2G/ArWgdPPlinyLSEtlobfPaHE+m3BM+53WoyOo3e+w3kGHJWacomcGJPN6PJzDW7CDpkcy1vw6lrGeKAG2vOXp+7ln2wEfTjxuYswyDwe3rY9nS/uXmUNmHMNmvuwGqhMT9ajJ/yRLz0XUQgK+jhhdTJRqIIGlRjwQ1TpyHN+mtoF8/jYqdIy64B0un6TSxXaD2e/EGZSAI1XNwPVTiZKqrqNLmeFpSCQGsi5yogTUfDRCQLfi6ajeZS3Ja9oUm6t5f4bmIybZq+8RM++ZUyXlactwl1ToRLGk9JrUTGB+yoG1KTPI63QPCR8joN34f0dPDWhX2uzsU64u3OEwuCXdSD4cLd41xheQXsz/R1iQSXrDMG5ogeNWeWSrwFjPjx4EXFY8zeiWWmWH2jEhiaEQASVS6LeEAieSFrsVC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When HVO is enabled and huge page memory allocs are made, the freed memory can be aggregated into higher order memory in the following paths, which facilitates further allocs for higher order memory. echo 200000 > /proc/sys/vm/nr_hugepages echo 200000 > /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages grub： default_hugepagesz=2M hugepagesz=2M hugepages=200000 Currently not support for releasing aggregations to higher order in the following way, which will releasing to lower order. grub: default_hugepagesz=2M hugepagesz=2M hugepages=0:100000,1:100000 This patch supports the release of huge page optimizations aggregates to higher order memory. eg: cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-xxx ... default_hugepagesz=2M hugepagesz=2M hugepages=0:100000,1:100000 Before: Free pages count per migrate type at order 0 1 2 3 4 5 6 7 8 9 10 ... Node 0, zone Normal, type Unmovable 55282 97039 99307 0 1 1 0 1 1 1 0 Node 0, zone Normal, type Movable 25 11 345 87 48 21 2 20 9 3 75061 Node 0, zone Normal, type Reclaimable 4 2 2 4 3 0 2 1 1 1 0 Node 0, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 ... Free pages count per migrate type at order 0 1 2 3 4 5 6 7 8 9 10 Node 1, zone Normal, type Unmovable 98888 99650 99679 2 3 1 2 2 2 0 0 Node 1, zone Normal, type Movable 1 1 0 1 1 0 1 0 1 1 75937 Node 1, zone Normal, type Reclaimable 0 0 0 0 0 0 0 0 0 0 0 Node 1, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 After: Free pages count per migrate type at order 0 1 2 3 4 5 6 7 8 9 10 ... Node 0, zone Normal, type Unmovable 152 158 37 2 2 0 3 4 2 6 717 Node 0, zone Normal, type Movable 1 37 53 3 55 49 16 6 2 1 75000 Node 0, zone Normal, type Reclaimable 1 4 3 1 2 1 1 1 1 1 0 Node 0, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 ... Free pages count per migrate type at order 0 1 2 3 4 5 6 7 8 9 10 Node 1, zone Normal, type Unmovable 5 3 2 1 3 4 2 2 2 0 779 Node 1, zone Normal, type Movable 1 0 1 1 1 0 1 0 1 1 75849 Node 1, zone Normal, type Reclaimable 0 0 0 0 0 0 0 0 0 0 0 Node 1, zone Normal, type HighAtomic 0 0 0 0 0 0 0 0 0 0 0 Signed-off-by: suhua --- mm/hugetlb.c | 37 +++++++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 190fa05635f4..3441d380c90b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2077,6 +2077,24 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, return folio; } +static struct folio *only_alloc_and_account_fresh_hugetlb_folio( + struct hstate *h, gfp_t gfp_mask, + int nid, nodemask_t *nmask) +{ + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); + if (!folio) + return NULL; + + spin_lock_irq(&hugetlb_lock); + h->nr_huge_pages++; + h->nr_huge_pages_node[nid]++; + spin_unlock_irq(&hugetlb_lock); + + return folio; +} + /* * Common helper to allocate a fresh hugetlb page. All specific allocators * should use this function to get new hugetlb pages @@ -3301,23 +3319,34 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { unsigned long i; char buf[32]; + LIST_HEAD(folio_list); + struct folio *folio, *tmp_f; for (i = 0; i < h->max_huge_pages_node[nid]; ++i) { if (hstate_is_gigantic(h)) { if (!alloc_bootmem_huge_page(h, nid)) break; } else { - struct folio *folio; gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, - &node_states[N_MEMORY]); + folio = only_alloc_and_account_fresh_hugetlb_folio(h, + gfp_mask, nid, &node_states[N_MEMORY]); if (!folio) break; - free_huge_folio(folio); /* free it into the hugepage allocator */ + list_add(&folio->lru, &folio_list); } cond_resched(); } + + if (!list_empty(&folio_list)) { + /* Send list for bulk vmemmap optimization processing */ + hugetlb_vmemmap_optimize_folios(h, &folio_list); + + list_for_each_entry_safe(folio, tmp_f, &folio_list, lru) { + free_huge_folio(folio); /* free it into the hugepage allocator */ + } + } + if (i == h->max_huge_pages_node[nid]) return;