From patchwork Sat Mar 30 08:23:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Jun X-Patchwork-Id: 13611504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAE5CC47DD9 for ; Sat, 30 Mar 2024 08:31:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E38876B007B; Sat, 30 Mar 2024 04:31:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE8636B0082; Sat, 30 Mar 2024 04:31:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD6586B0083; Sat, 30 Mar 2024 04:31:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B04DC6B007B for ; Sat, 30 Mar 2024 04:31:56 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 291171C1516 for ; Sat, 30 Mar 2024 08:31:56 +0000 (UTC) X-FDA: 81953037432.21.F205CEB Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by imf21.hostedemail.com (Postfix) with ESMTP id E851D1C000C for ; Sat, 30 Mar 2024 08:31:52 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf21.hostedemail.com: domain of chenjun102@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=chenjun102@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711787514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references; bh=CxMUR6nqZpg7T6btCnKBhz7lQiMha/CKvGz5cOzl/VY=; b=bw4iEDJVWZzq3uN5vZfOr4mHfZsaCEOIbmlZT/Z1NnIgvnxn/2u+kZqReiD15jmv6TtIe7 5Z0hJutKrLGPbwklPkpYfEMQPmW+qOvpy7+3j+knOcWPoBM6Hf4UIzh6yYbsd8BkbW087V /b1GLvPhaMUP7Et7S6Rq/XBcBhbCOcI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf21.hostedemail.com: domain of chenjun102@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=chenjun102@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711787514; a=rsa-sha256; cv=none; b=lcvU4VLGwv4t+ZLj9rPdd0V6dJL5GvmDMgnBejBkrauYBUbXteF/ZxvnCgaahizZMe1x0J AOjjNDY0oMU7l4Z+M+hNjtEx8MNbHWYThq9sBi5lqGUYorLpIpWc+M5aDiYQc3brJ2FO+z PZb506AWS3VtGqJMG582AVNdyBxcXhA= Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4V69Tt0fSqz1GDdV; Sat, 30 Mar 2024 16:31:14 +0800 (CST) Received: from dggpemm500006.china.huawei.com (unknown [7.185.36.236]) by mail.maildlp.com (Postfix) with ESMTPS id 1F38E1A0188; Sat, 30 Mar 2024 16:31:48 +0800 (CST) Received: from mdc.huawei.com (10.175.112.208) by dggpemm500006.china.huawei.com (7.185.36.236) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Sat, 30 Mar 2024 16:31:47 +0800 From: Chen Jun To: , , , , , , , , , <42.hyeyoo@gmail.com> CC: , , Subject: [PATCH v2] mm/slub: Reduce memory consumption in extreme scenarios Date: Sat, 30 Mar 2024 16:23:35 +0800 Message-ID: <20240330082335.29710-1-chenjun102@huawei.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-Originating-IP: [10.175.112.208] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemm500006.china.huawei.com (7.185.36.236) X-Rspam-User: X-Stat-Signature: qe67jmf3puqb9u5iwhc1fnkynbbtgduf X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E851D1C000C X-HE-Tag: 1711787512-933632 X-HE-Meta: U2FsdGVkX1/hTj7uNKN6vC9RYZgY0rwp5LvVUbtR7T6aezQF58xo51lfzxYRKOQu5T5nngialMr6/QH4VKw3yw5QshBe8HNj5Z2u2thRDGgx2D2PBbg2b7XspAdqrANEcMkFQhH8EwE2pniID/nxV+AN7qT9fwpDwQYJRAXJVLKrmJmgk30DUBkbcgLbNzFfDIKq3r+8IFLea7pU3KitM/MtjYzuwqfDbLrss/GnBipufBgfLDJWISRP6JAL49vmHzMkbNDA9lIIOEMl6H6KfTRnACn48hhzlyll70oTj1WeREVytKj8KgfrKzmkDW6BQLAZR3ZFdpkuD2av5VAyb1S6OxC6fOCy4UNi5ZMke8PJ+9WB3KREZnPDDhboJ89BMiwkgOsvs1YZbCtaJPgH/FiNrWVNYzfQHTJia4ffPp0LrUHbx1eMUkfn3AJTDkUz12QkBgc/1DMhEVfhqQuw2Gp5eJVmrVTFVm/rQHbNCsEfs/ztl13gZD7z17c/JkuByiDznywoLFLOy0A6lwZq48jEWF3acgLOJreSIersOCHegBVheh86purG6TS1ns7BsGohucJPe4bAaCVSZqYnGb1GQiaVhu4LJCmMNgZWlX7IfFfwQ2uX8VT89ACQRyCkABIJ7NuU4FPGosCMLOCcnwF38e6puudYo2HN2dWj60UoppnvR9mWXAgNYVoX7cHfj9nijvwUrlP4clglJz6nyYbQfn7ztOIJxkCjKBiwNXosWbxlXNK+E77OLJqiz9H/lur0+p20SFMI416ojTUbbrjcAO11ygITT/IpPaM4AhcXqRgvVStl4ysCkO/JQydRm6DDU2YgszuUtD9XY6k7zhYfpi0n3byrmI6z+xpQRUNzxsWjBHE2zQZHvwXWZhgGKgcYO/guYd6kIeB8b7nREUtKNtR/otzPFe6RgMr4jMhOd2zPid4ZaWDl44zMG0+YgCDpoRDXSuHZakJwcqX cu7cRKUS zzLldF1xTLn3cvePbJh7f+Vbk7yTK1dAsUNPQbuog2mR7/qyK7aFqXKF+o6O2w1dlzz8x4+ih4GQfxZSx5dB4KFKRgrYTzGnUyxH2mbflgl2YXyhtTFOaazs0WwORJ3ivACYs5XAau0evcTzzWE3BOOTdizB0mL6s0/HX+2mIL6Uu39ozAttvarLiaNSpaIY2vP+fyTKt/GfihLeKk3xR3sNd4Acou6Y5IYlG6rWrqnMsCiw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When kmalloc_node() is called without __GFP_THISNODE and the target node lacks sufficient memory, SLUB allocates a folio from a different node other than the requested node, instead of taking a partial slab from it. However, since the allocated folio does not belong to the requested node, it is deactivated and added to the partial slab list of the node it belongs to. This behavior can result in excessive memory usage when the requested node has insufficient memory, as SLUB will repeatedly allocate folios from other nodes without reusing the previously allocated ones. To prevent memory wastage, when (node != NUMA_NO_NODE) && !(gfpflags & __GFP_THISNODE) is, 1) try to get a partial slab from target node with GFP_NOWAIT | __GFP_THISNODE opportunistically. 2) if 1) failed, try to allocate a new slab from target node with GFP_NOWAIT | __GFP_THISNODE opportunistically too. 3) if 2) failed, retry 1) and 2) with orignal gfpflags. when node != NUMA_NO_NODE || (gfpflags & __GFP_THISNODE), the behavior remains unchanged. On qemu with 4 numa nodes and each numa has 1G memory. Write a test ko to call kmalloc_node(196, GFP_KERNEL, 3) for (4 * 1024 + 4) * 1024 times. cat /proc/slabinfo shows: kmalloc-256 4200530 13519712 256 32 2 : tunables.. after this patch, cat /proc/slabinfo shows: kmalloc-256 4200558 4200768 256 32 2 : tunables.. Signed-off-by: Chen Jun Signed-off-by: Kefeng Wang --- v2: - try to alloc partial slab or new slab with GFP_NOWAIT(it includes __GFP_NOWARN) opportunistically, then fallback to orignal gfpflag, suggested by Vlastimil Babka, - update changelog v1: https://lore.kernel.org/linux-mm/20230314123403.100158-1-chenjun102@huawei.com/ mm/slub.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1bb2a93cf7b6..c1c51595a59f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2699,7 +2699,7 @@ static struct slab *get_partial(struct kmem_cache *s, int node, searchnode = numa_mem_id(); slab = get_partial_node(s, get_node(s, searchnode), pc); - if (slab || node != NUMA_NO_NODE) + if (slab || (node != NUMA_NO_NODE && (pc->flags & __GFP_THISNODE))) return slab; return get_any_partial(s, pc); @@ -3375,6 +3375,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, struct slab *slab; unsigned long flags; struct partial_context pc; + bool try_thisnode = true; stat(s, ALLOC_SLOWPATH); @@ -3501,6 +3502,17 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_objects: pc.flags = gfpflags; + /* + * when (node != NUMA_NO_NODE) && !(gfpflags & __GFP_THISNODE) + * 1) try to get a partial slab from target node with GPF_NOWAIT | + * __GFP_THISNODE opportunistically. + * 2) if 1) failed, try to allocate a new slab from target node with + * GPF_NOWAIT | __GFP_THISNODE opportunistically too. + * 3) if 2) failed, retry 1) and 2) with original gfpflags. + */ + if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE) && try_thisnode) + pc.flags = GFP_NOWAIT | __GFP_THISNODE; + pc.orig_size = orig_size; slab = get_partial(s, node, &pc); if (slab) { @@ -3522,10 +3534,15 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } slub_put_cpu_ptr(s->cpu_slab); - slab = new_slab(s, gfpflags, node); + slab = new_slab(s, pc.flags, node); c = slub_get_cpu_ptr(s->cpu_slab); if (unlikely(!slab)) { + if (node != NUMA_NO_NODE && !(gfpflags & __GFP_THISNODE) && + try_thisnode) { + try_thisnode = false; + goto new_objects; + } slab_out_of_memory(s, gfpflags, node); return NULL; }