From patchwork Wed Dec 4 16:55:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Koichiro Den X-Patchwork-Id: 13894101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6246E7716D for ; Wed, 4 Dec 2024 16:55:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF6456B007B; Wed, 4 Dec 2024 11:55:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA6796B0083; Wed, 4 Dec 2024 11:55:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6DC06B0085; Wed, 4 Dec 2024 11:55:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B80C46B007B for ; Wed, 4 Dec 2024 11:55:42 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 353EE81147 for ; Wed, 4 Dec 2024 16:55:42 +0000 (UTC) X-FDA: 82857877536.13.3FE8BBC Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) by imf13.hostedemail.com (Postfix) with ESMTP id 140A920008 for ; Wed, 4 Dec 2024 16:55:23 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b=ugh0U5uP; spf=pass (imf13.hostedemail.com: domain of koichiro.den@canonical.com designates 185.125.188.123 as permitted sender) smtp.mailfrom=koichiro.den@canonical.com; dmarc=pass (policy=none) header.from=canonical.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733331333; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=OfRULcgbj9969PfzAEC+2ztALegozxLXhupgdMLyKBE=; b=6tpQ8J1K8cTjiV3k1bFHzrMjtGY9J2/Sd1Os4dqA1Pm2/9YWcRPr+Ws6YqbY0ZtQXzlSo3 ODKGWRTBLDutihz7S0rXZ51w98EI7oZ95LD76B/m/ahLK4CPkMqBNLQE/9/VHCBzNVfV5w h+/mSZZG7dkr/Xtr24EO6B4pLbEllfM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b=ugh0U5uP; spf=pass (imf13.hostedemail.com: domain of koichiro.den@canonical.com designates 185.125.188.123 as permitted sender) smtp.mailfrom=koichiro.den@canonical.com; dmarc=pass (policy=none) header.from=canonical.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733331333; a=rsa-sha256; cv=none; b=LC7reFA1jSUzpY9JFzK+2So5VhzIzrHSOUoeCIE9djgB0UMRRmrtl5ZIsElNqkm+DO/REP n0Vvz7eBX3+1YI7qtPiaVGV06P6E5JnQl1KnNSrnJtOcODZTvIhanE5huKkfD7yvBNm7+i UAmQyObyy6j9BtmdCvNdmnkpYDZv7BU= Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 02AB53F215 for ; Wed, 4 Dec 2024 16:55:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1733331336; bh=OfRULcgbj9969PfzAEC+2ztALegozxLXhupgdMLyKBE=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ugh0U5uPWYJWGPZSPrIt9a84OfP9slYt/cpqVift71aEp8+i3OMEP/H5ss+U4dQAR 8bbnCHBotEg8TGpzCZF2rrkA2CbAyUrrBomDWm10qXVKK3PxeSxK50PP4rQb3TG3nk ouJ8OlWzuCYAE/42Cw3gmVno6t9h4lo9Orc6mPrljtEcUzIVzX1c3Cb8d1JCOXOqpl actCBl9LRXnL0DWjOwgyC3oZTv9bLXeoBCFu1qu4kgfAkmZQ3B/dSi3l3a9okHA3bD DIctwyhKlifzpkXxXQPiaXCbQo8VnQHdAAaO+wd2u5aA73WrYQYOuILyXbFZLIjWC5 mCxSmiJ4FU1sQ== Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-7ea8baba60dso457a12.3 for ; Wed, 04 Dec 2024 08:55:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733331333; x=1733936133; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OfRULcgbj9969PfzAEC+2ztALegozxLXhupgdMLyKBE=; b=inQLeZ2CmVR/HTRu8NBU2HV5zXpJe8Pd7bUK8/7HzjeKdG7+LgciVOgM2TYNezwNU+ Tq4CrZ+KhtZse/BwMvhzzPmSsohS7dx/2PSwFop0f5J8swPJk+DyNoWamyyys5S2kbaS Jy/IEY83nFjcw5OnQYvHcri6XQvn3chIwd50XBvvGTgYjZFddOegtdR5cKNev6uXpVnA OVgbNLGAyIpYkJRYL4lwoHk57a7QanPkt8Uhqyrht5pXYZkjKkydy1fDMlxw6i6LZhzG Yc9SyjAQZ3QO25zsRCSp20Wl9UHR7Oa3lTbgmW006pAyULmoL8u+sfX3VuQbq0ksgrim 708Q== X-Gm-Message-State: AOJu0Yy8XV2nIxLtoqhxna88qxoUWuQ+MQVw4jyXLgehKo/jl/0e9hjM QLY9fWEYFbcfYUP0XKgIsDq0KWHJmXN2cJQT30FMovllNODw7mek0dvgwEALblRT8GpZEbou1JQ Hm+uwoSWhH2F2H2guW+wfpBbO/TJj8Dj5MtxCR5Bqo9Casa/bXdYkf/miNTunjPZpGu3hRS4M X-Gm-Gg: ASbGncuWg7lIldyNQ8FwSmFzwDMEODQ/wSCDO5S+np76/D2lWiw4s7MbONnVxXok/Se WkugdyvaZDLnpO9KbH7KBmK9iFQ5xswcwh+42DQSC2PCuT5nTp6IFskoDcczSB+Xd1ULNh7pEY6 KhVZor1dKd9z/+OVGxh3jv+bQcAERGT19XIUIaXvTZJgMkydyVsO6w7EcVc/7g0jILf+XX5Ww2o SHsGyhSDQfRQi+dbk211wyOVpC1W6k88ZdfVRauqfBSBYAyexwcuBO2SMVFKpqZf49A X-Received: by 2002:a05:6a20:12c3:b0:1db:df34:a1d6 with SMTP id adf61e73a8af0-1e16540f49cmr12044207637.42.1733331332788; Wed, 04 Dec 2024 08:55:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IFbPPD/q7lir5PlOM4kjQV6wuhK7NJGmbnDCRqclPQUWXh5pL8q/YacWxwvE4iN90uUp7L8gw== X-Received: by 2002:a05:6a20:12c3:b0:1db:df34:a1d6 with SMTP id adf61e73a8af0-1e16540f49cmr12044176637.42.1733331332413; Wed, 04 Dec 2024 08:55:32 -0800 (PST) Received: from localhost.localdomain ([240f:74:7be:1:9c88:3d14:cbea:e537]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fcfac85434sm1394687a12.21.2024.12.04.08.55.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2024 08:55:32 -0800 (PST) From: Koichiro Den To: linux-mm@kvack.org Cc: muchun.song@linux.dev, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH] hugetlb: prioritize surplus allocation from current node Date: Thu, 5 Dec 2024 01:55:03 +0900 Message-ID: <20241204165503.628784-1-koichiro.den@canonical.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 140A920008 X-Stat-Signature: 9458mmti5qyxzmcnajwo8w8wgcxrprx8 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733331323-997468 X-HE-Meta: U2FsdGVkX185KP+fAxKAQtcJUNzHD08I0Fd3iOz591PQypLHgjAf+jDSzta8K5w1nnmvbcP7i68L5KQwpjforxzsL+9IcBXq3yWPY8qjL7XkN0ILdQp8O5SD7NQ1iJvkcuasyZT5NHjbsqqhi9zF8Bc1T0an4wZbd1O5Sw9aywVcZMPjBircAX6BEvb4CDc2kUdvMBmmgs62tmx3dEafoCWkuvlH0CoY06MrAa2q3etZDZDEyDD2kubpxNrzUcLAq4C+ge+s85fO88lTEevy/um7BPQW/zERNB+DYTrHsaKXTWfvUyWSLokf4MW8SBdXzEBkhnEbaiQm6RbWRKXauf1zIUVJDr8qwDmWQinUaGHP1brq4+mDeKV0QzrNK2hnJ77EejkqrNvSflRCJhTOyAWdf2ldv8Cw1MzOlvFpm167UPfgiTE347B5kOCv58WWjIsp2az2JvBLS7CiGVFM3hdPgK6F0xwmAXpNWkf9Wkd8vf1dDEQaVkwOttacnFB72RwUnZZdiDTIrWbuo+zxWQpLi9MZu03UwGNhvyRmLN6v4ntAZWRDicL/xH80gf0sqreEXUkv1wXfxD6sHn2ja0t642+/yxTKMstNQ3t2V1tSwq9knDD5bwYhfGH7B5zCBxtQpVvOjqKdgOzXvNnKpfKpn8WtPPIohU5oOMH8aLVsr+GNQdACdFgBiJVxUdxhwHEJx6te+Ar5tUzeRXvp+1yZNLSdvjhzb0+RUnK0I1zBg2ALQLlENbNxgENu9tQQ3xklTqMeXJIgNDf/29W1GfgsaarD93JiaOn94Dc3mB/taSoJ0DZAYa7evlkctivBVXdnj7QAf4pKOWF6niLobyY17XcbZCmmwlcz88UIGvkSN51AmKHUkRACRxUoqcxFOG0/YFOW3dIpGakrCljNfCwxwhMPTcMHaYJJSQm600l7RiKQmyZmh39lGmm7w1sr3U5F8vfv0ZqAiod9o4K YS9ZEfcn pmMF58E9RC4R9x+FDOAcdzdGQ1mcb8wyynY3RKrqozG2Xra+13oOH50JL5eERZm9NotWxVQ79oQhSfyvbD8qyey6Glayhox2p66WKesX3nmkoMP23g+GqUBlKjQLRAafLwkQ3GlvrorGDMlQzOrtmqDxXGR1ehvU72oYIwTKkfKxLeRAuGnIOWwkls4CE1J+tpEj/03b+0gzoNCGyDewxeR7M00r27zuLNaalJQEv+sbmQO4loAAVRehu8hdylWNXCwa5uSEW03LbchZtYSuIeAdptKetXFGkMcB4zWOkgoJRElc+nB8LjjS+FAkbcSzvjbAiu7as8NOLZkb4LJr4ZhvznLBQqEIBIhKj7wLtz5vv4RykLq6SnFtVEmTCBNwA2jfObd+O41xN7ns= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously, surplus allocations triggered by mmap were typically made from the node where the process was running. On a page fault, the area was reliably dequeued from the hugepage_freelists for that node. However, since commit 003af997c8a9 ("hugetlb: force allocating surplus hugepages on mempolicy allowed nodes"), dequeue_hugetlb_folio_vma() may fall back to other nodes unnecessarily even if there is no MPOL_BIND policy, causing folios to be dequeued from nodes other than the current one. Also, allocating from the node where the current process is running is likely to result in a performance win, as mmap-ing processes often touch the area not so long after allocation. This change minimizes surprises for users relying on the previous behavior while maintaining the benefit introduced by the commit. So, prioritize the node the current process is running on when possible. Signed-off-by: Koichiro Den Acked-by: Aristeu Rozanski --- mm/hugetlb.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5c8de0f5c760..0fa24e105202 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2463,7 +2463,13 @@ static int gather_surplus_pages(struct hstate *h, long delta) long needed, allocated; bool alloc_ok = true; int node; - nodemask_t *mbind_nodemask = policy_mbind_nodemask(htlb_alloc_mask(h)); + nodemask_t *mbind_nodemask, alloc_nodemask; + + mbind_nodemask = policy_mbind_nodemask(htlb_alloc_mask(h)); + if (mbind_nodemask) + nodes_and(alloc_nodemask, *mbind_nodemask, cpuset_current_mems_allowed); + else + alloc_nodemask = cpuset_current_mems_allowed; lockdep_assert_held(&hugetlb_lock); needed = (h->resv_huge_pages + delta) - h->free_huge_pages; @@ -2479,8 +2485,16 @@ static int gather_surplus_pages(struct hstate *h, long delta) spin_unlock_irq(&hugetlb_lock); for (i = 0; i < needed; i++) { folio = NULL; - for_each_node_mask(node, cpuset_current_mems_allowed) { - if (!mbind_nodemask || node_isset(node, *mbind_nodemask)) { + + /* Prioritize current node */ + if (node_isset(numa_mem_id(), alloc_nodemask)) + folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h), + numa_mem_id(), NULL); + + if (!folio) { + for_each_node_mask(node, alloc_nodemask) { + if (node == numa_mem_id()) + continue; folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h), node, NULL); if (folio)