From patchwork Tue Aug 20 01:29:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Lameter via B4 Relay X-Patchwork-Id: 13769249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13C1DC3DA4A for ; Tue, 20 Aug 2024 01:30:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93D916B007B; Mon, 19 Aug 2024 21:30:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ED096B0082; Mon, 19 Aug 2024 21:30:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B5436B0083; Mon, 19 Aug 2024 21:30:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5C58F6B007B for ; Mon, 19 Aug 2024 21:30:00 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C9AB2816A3 for ; Tue, 20 Aug 2024 01:29:59 +0000 (UTC) X-FDA: 82470892518.15.97B0A5F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf11.hostedemail.com (Postfix) with ESMTP id D994940014 for ; Tue, 20 Aug 2024 01:29:57 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=La57jugl; spf=pass (imf11.hostedemail.com: domain of devnull+cl.gentwo.org@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=devnull+cl.gentwo.org@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724117382; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=oLQQvI4GGHOvX8XhOug/znUOiGDLD8BHw3U9tdLDwTQ=; b=ndLJDAGHfL5tyh15TA0X639kzqmLu3ZhIMW+JmQcLCTF53Hc+kSJuTfWl79BqY2cnak/E8 OziXEiCN4y2lmXC/RTJZEzLP2uCHZdUYiSjDn1+tZzGcIp/05CFJklEMCWnREMTBsPobp+ GhNMH7L1DxBf4+eTzK2ZJQGNeNZJ41o= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=La57jugl; spf=pass (imf11.hostedemail.com: domain of devnull+cl.gentwo.org@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=devnull+cl.gentwo.org@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724117382; a=rsa-sha256; cv=none; b=La+BKl9KrNMb6p6apooeGrnvNQJlK0jpwET94NSJNwHXs+Rdzla42xp/O9lZKTFDm21W45 QhcAm/H7nAtdXche7nxzbO3PhBP7bgbG1jhtMeJQmAlPX21GYX9NCcuaBVWnR1OTUifAg1 PLlOL9MBFU8K9pCqoPI3OLtFPg5k4lg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id D5F6F60B85; Tue, 20 Aug 2024 01:29:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id 7DE22C32782; Tue, 20 Aug 2024 01:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724117396; bh=Dm831pibPUVAy22FE2OyR6ovDgb8M43y05kQ9d0Nbp0=; h=From:Date:Subject:To:Cc:Reply-To:From; b=La57jugllwsMiRz2JSf5GLxjOMRGHXs0OAfVqft5NcRVh0aXR7D/dP99dU3tGxLUq d/xKSTSsnitQruCIdQGiRpIZnGv2kHCDWsuniRn7s8rW1hs5NboIFxySUORQxmINVN NT/ORzoObN94UJ/TXP7ocS22xcsLY7f9maD4tMJHxMho9lsgivbBiUmexaD8TSxlyw +Z2Jyz8063iUQrrI9IBYFfbikzRwz0cuyOzVjSXtO9S/Qr0CJ9T/GykcJifIO9bNDt S495oHSVcG6BhtowZ3RyELg6Bwh77u7q6bLoIwb0qdNnir33PfmkAZsmLgvfE5tWhc SUNRUJFUyzSeQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68463C3DA4A; Tue, 20 Aug 2024 01:29:56 +0000 (UTC) From: Christoph Lameter via B4 Relay Date: Mon, 19 Aug 2024 18:29:45 -0700 Subject: [PATCH RFC] SLUB: Add support for per object memory policies MIME-Version: 1.0 Message-Id: <20240819-strict_numa-v1-1-ed19f278c7f7@gentwo.org> X-B4-Tracking: v=1; b=H4sIAIjxw2YC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyDHUUlJIzE vPSU3UzU4B8JSMDIxMDC0NL3eKSoszkkvi80txE3bRkU8skY2NDI+NEIyWgjoKi1LTMCrBp0Up Bbs5KsbW1AK89VbJiAAAA To: Vlastimil Babka , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Yang Shi Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Christoph Lameter (Ampere)" X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=ed25519-sha256; t=1724117396; l=2985; i=cl@gentwo.org; s=20240811; h=from:subject:message-id; bh=WCkmlka2muGf3df5xwLt0U+IUMuVYP3f3DZ+713T/TY=; b=pytFnytVJB7IrVevzcGROanOnrwdyHDWZ/XhESjVY5Qg10L8R19EM7qNT0yVSpn44ZGrEUAL3 v4iQLyirblXCnXM00TLtKmYTJhOD7PglYdV4OQpBbDKKd9v78bWf3zN X-Developer-Key: i=cl@gentwo.org; a=ed25519; pk=I7gqGwDi9drzCReFIuf2k9de1FI1BGibsshXI0DIvq8= X-Endpoint-Received: by B4 Relay for cl@gentwo.org/20240811 with auth_id=194 X-Original-From: Christoph Lameter Reply-To: cl@gentwo.org X-Rspam-User: X-Stat-Signature: h4gdq55uizubhz4xi3fju4dfo5ofdsy1 X-Rspamd-Queue-Id: D994940014 X-Rspamd-Server: rspam11 X-HE-Tag: 1724117397-409945 X-HE-Meta: U2FsdGVkX18+Hsu+c3dekRDCI040QB5zPa6BD20E1Z6aG5vOSsslVUiNeEszncpMJrOnsMl+MIfnjq5k2gZv5XfIgVte08RvswQTM/dvFnczcH3btdigmcwwvQVXWBbccSmPrFNwdiSGk5hd+SLEXBBBqI7YD9VD3WS84G2NdMA5c9VikVby/oTCiGyTdLnYaUgPhot/HXhrDpcZQNaeaKL9obMjpjyWyA9keuQO7jqeOt2wvKjJ3821DbyE9/dYeONpY+EpEabD2U54AeudHDDAXBhMdTC1w2t7kivxsHhKviJeyYtbDqR16ZAAR1FnpZ79fz4QXKp4kNu2UeEA1quCWJ6p7fL+kEaLfP+Mw63cAfq3LU6THz9V8DYb4zKHafwWrNziPgriGgIBTyyQBG1Xj6DLKnOPz/FLFsVg9S52YIghGZmSzJ2ckwQvw4LG6toyMUgjze+cGRLl2QR3WpZp2Z8/MD6Arn6nxP2YklDLcAsxYukLO4OGnkipKF02bx9spW/Z+ezlKweO1p1KBKkM+kB+CANh+tYWYh44RFZcE89phGhLe8lDiMnvaxZyJLNDY/jXFSPPRH/RldyLLOB7g0pyFni8vnxYPkmvU5+RyqSww3DR+Fi6gmMGWbgEaL9acjZtDj2raEVmE6i3Zeoa7uEReqpdZ9/4jEtCOeTw9GrPvRdOcngYTeJZDc3I5LX5xiLUNUuE+7AoDqb6RMIqTVOCjb+z6v9maaJ1ioPxUWP9n7iDHVW7ZONBiSTfLsxzo9gKP+PbugLAzdoErEHsqN9AIb88ZSuNwmjqaCRzpyowRaCWxV/fRvMt9kp7ev/92xAoLkBh1AXQpqfe62j9ZOk5G+A9dVqmmsmMwRArxwa5oQkca9LJatM0QdCrckOL/HaHf3wO50Nt89hpATbraVKL2QV4GIKQI9J4txT/fvT3ho6ucVBMWjBa+R+GM97qx1CoraqY9EyGYoa gR8rg6cc 8eY43XURJ+gg+StxLgvqGQfEa1WzCeEGHdMhj978fcdwq1t3Rgj+9WqgLOx5Eu7Knab8mTNnxrZAxAas/2b4UQnlQAoONcgBMcw3OY9sqsi+P09/XyAQf754oei7W3lsX8F0+coEc9XncdgjVf+ep+e3Ov19/qT5QM8yeBwREZBSB+IhlqW+wzASa3O7lA0y+BPBN8PVUeMF+uDXTgMiEQPGvJ59BFOzJ/VYbzkWdhYnoGtNTZPCMg5cFC4d/44dWg+cs9KZlfIfsEMXaib14T5/E62U2oHjzRO6gvm+UhhHo1ReWxc7SFG3Fes5FXnOSbFE6Ml32ceC3AXVZczEbQUoayay9zie2e+QayP4u+ERdE4KbWqsvxLhmkgBQavnL4GKo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Lameter The old SLAB allocator used to support memory policies on a per allocation bases. In SLUB the memory policies are applied on a per page frame / folio bases. Doing so avoids having to check memory policies in critical code paths for kmalloc and friends. This worked well on Intel/AMD/PowerPC because the interconnect technology is mature and can minimize the latencies through intelligent caching even if some small objects are not placed optimally. However, on ARM we have the emergence of new NUMA interconnect technology based more on embedded devices. Caching of remote content can currently be ineffective using the standard building blocks / mesh available on that platform. Such architectures benefit if each slab object is individually placed according to memory policies. This patch adds another kernel parameter slab_strict_numa If that is set then a static branch is activated that will cause the hotpaths of the allocator to evaluate the current memory allocation policy. Each object will be properly placed by paying the price of extra processing and SLUB will no longer defer to the page allocator to apply memory policies at the folio level. Signed-off-by: Christoph Lameter (Ampere) --- This is a draft to discuss the issue. It builds but was never run. --- mm/slub.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) --- base-commit: 6e4436539ae182dc86d57d13849862bcafaa4709 change-id: 20240819-strict_numa-fc59b33123a2 Best regards, diff --git a/mm/slub.c b/mm/slub.c index c9d8a2497fd6..c86e7ee9520a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -218,6 +218,10 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); #endif #endif /* CONFIG_SLUB_DEBUG */ +#ifdef CONFIG_NUMA +DEFINE_STATIC_KEY_FALSE(strict_numa); +#endif + /* Structure holding parameters for get_partial() call chain */ struct partial_context { gfp_t flags; @@ -3825,6 +3829,14 @@ static __always_inline void *__slab_alloc_node(struct kmem_cache *s, unsigned long tid; void *object; +#ifdef CONFIG_NUMA + + if (static_branch_unlikely(&strict_numa) && node == NUMA_NO_NODE) + + node = mempolicy_slab_node(); + +#endif + redo: /* * Must read kmem_cache cpu data via this cpu ptr. Preemption is @@ -5523,6 +5535,21 @@ static int __init setup_slub_min_objects(char *str) __setup("slab_min_objects=", setup_slub_min_objects); __setup_param("slub_min_objects=", slub_min_objects, setup_slub_min_objects, 0); +#ifdef CONFIG_NUMA +static int __init setup_slab_strict_numa(char *str) +{ + if (nr_node_ids > 1) + static_branch_enable(&strict_numa); + else + pr_warn("slab_strict_numa parameter set on non NUMA system.\n"); + + return 1; +} + +__setup("slab_strict_numa", setup_slab_strict_numa); +#endif + + #ifdef CONFIG_HARDENED_USERCOPY /* * Rejects incorrectly sized objects and objects that are to be copied