From patchwork Sat Oct 15 04:34:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 13007577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 498E4C4332F for ; Sat, 15 Oct 2022 04:34:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AFD86B0072; Sat, 15 Oct 2022 00:34:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 45FB28E0001; Sat, 15 Oct 2022 00:34:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D88F6B0078; Sat, 15 Oct 2022 00:34:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 161DD6B0072 for ; Sat, 15 Oct 2022 00:34:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D05DF1206AE for ; Sat, 15 Oct 2022 04:34:37 +0000 (UTC) X-FDA: 80021917794.11.B5785BF Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf07.hostedemail.com (Postfix) with ESMTP id 831DF40027 for ; Sat, 15 Oct 2022 04:34:36 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id t10-20020a17090a4e4a00b0020af4bcae10so6469797pjl.3 for ; Fri, 14 Oct 2022 21:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:message-id:subject:cc :to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=TPDhjKmk3ApEu3jvJvKzYDjHE8g/IRJ8TwRkyx2nzjs=; b=HkV+O9seq0CJgkveVWp+jhhU3Frhx8ed3q7VPN7T7t+LSwRYfLEdktC8MT7SydK3E+ s05PH1scaQT4RHsuTElXFlfY/B3DD6fJ4txF39LzDBF20GcUIY4FF3s9wSX95sc90hWx nOE9yKjgMl8VVBPBg5l2RXAKIXOSg3zuvyd9LtlSZNZkaDR8xf+xNysEmIcvBg07vsI0 pdDWzio85rtm0aBLEN66vpHVySQDRkZUoGnxSB93bJxGtvjUM/R8I2zVE0FMlVpS1c99 vsiVUkGszV4iF+Pr6vFcMP3raFRj8MSxdwqPske/Hh+HUITESZYWd/XJCx3EmmncPRO5 O6mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:message-id:subject:cc :to:from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TPDhjKmk3ApEu3jvJvKzYDjHE8g/IRJ8TwRkyx2nzjs=; b=0LAkwnzJpA/cAsfA9QCcH0s1WMVrPvL1nXYemllVzv+D21ByoPa8gRpeRNrI9KGB/Y +Yn+GokrMDf6yddUd/xf9+YHTZ6Ph9U1lSPCwL7d4z+1xgA5sdeXFqfZ+9Ul8/GXVf4I al+R5RIh3EFS40DGKklX9NduZ2X4BRsaVJ0oR2wPhHKmIp1wIhmPjbuhcabUq7r8IGXY b0ZFzIntq11F+wMY6Vj8sfXNMA2FmGyDliXyUGsLzthPUEwi4+Bx5U5xp5CmoVJUVi0v KaF1Alu4RQCAPd61gU+Atx8N5igXZQP61841+0iTVUGYrKX59tcWuvgdIAc0yQQwtzme 8neA== X-Gm-Message-State: ACrzQf0m2JUsybeQ4QtVfn3i/6jU8noS4UDdO25AyjlMzOSfB+TE2cAp oyaivHWshS+W/S7xrKLBP94= X-Google-Smtp-Source: AMsMyM6pgV8SYoXuIxXiDSPpJ0linvrCw2sX0HpnBEglIDEFlWuS7kC0ynShA9slKGmqgws/5pEOMw== X-Received: by 2002:a17:903:234f:b0:17f:6711:1f9f with SMTP id c15-20020a170903234f00b0017f67111f9fmr1204421plh.32.1665808475299; Fri, 14 Oct 2022 21:34:35 -0700 (PDT) Received: from hyeyoo ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id e1-20020a170902784100b00172b87d9770sm2487852pln.81.2022.10.14.21.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 21:34:34 -0700 (PDT) Date: Sat, 15 Oct 2022 13:34:29 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Guenter Roeck Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] mm/slab: use kmalloc_node() for off slab freelist_idx_t array allocation Message-ID: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221014205818.GA1428667@roeck-us.net> ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HkV+O9se; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665808476; a=rsa-sha256; cv=none; b=8LecjIs4n/sr9Hh0dTmjZ++be+71P3/65Iz/jmnibfUyUGczDRA3hnJX+h8EJnUpA4X9t4 Fe001oLC26/gmdzxGLVyLSKf7hDVcgvcW1+8UUrVbhdRXHnKVKBSl4O9fQvY4zdupEb3FM xBirbGjOLZp4wsEnjH3354L6lOlOA9Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665808476; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:dkim-signature; bh=TPDhjKmk3ApEu3jvJvKzYDjHE8g/IRJ8TwRkyx2nzjs=; b=qRZUFQ7uK5tkn6AobtAz/7LaTCuetACUwEgWgyLCalSkpOFu3yGI9rjvu15lKqGoZtlbp5 lUTrS71n1lHcDN4S6GwDlEaha8i+vSys0IiXlIbkZh/NTDrHU8jwZ/rTI29S0RtM3L2U9X gyKp0aurQRIdForXkpYWfRBtp15kdys= X-Rspamd-Server: rspam05 X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HkV+O9se; spf=pass (imf07.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: pk339sxt9zkdotmncd6gtiocjfw7k8w4 X-Rspamd-Queue-Id: 831DF40027 X-HE-Tag: 1665808476-292110 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After commit d6a71648dbc0 ("mm/slab: kmalloc: pass requests larger than order-1 page to page allocator"), SLAB passes large ( > PAGE_SIZE * 2) requests to buddy like SLUB does. SLAB has been using kmalloc caches to allocate freelist_idx_t array for off slab caches. But after the commit, freelist_size can be bigger than KMALLOC_MAX_CACHE_SIZE. Instead of using pointer to kmalloc cache, use kmalloc_node() and only check if the kmalloc cache is off slab during calculate_slab_order(). If freelist_size > KMALLOC_MAX_CACHE_SIZE, no looping condition happens as it allocates freelist_idx_t array directly from buddy. Reported-by: Guenter Roeck Fixes: d6a71648dbc0 ("mm/slab: kmalloc: pass requests larger than order-1 page to page allocator") Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- @Guenter: This fixes the issue on my emulation. Can you please test this on your environment? include/linux/slab_def.h | 1 - mm/slab.c | 37 +++++++++++++++++++------------------ 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index e24c9aff6fed..f0ffad6a3365 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -33,7 +33,6 @@ struct kmem_cache { size_t colour; /* cache colouring range */ unsigned int colour_off; /* colour offset */ - struct kmem_cache *freelist_cache; unsigned int freelist_size; /* constructor func */ diff --git a/mm/slab.c b/mm/slab.c index a5486ff8362a..d1f6e2c64c2e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1619,7 +1619,7 @@ static void slab_destroy(struct kmem_cache *cachep, struct slab *slab) * although actual page can be freed in rcu context */ if (OFF_SLAB(cachep)) - kmem_cache_free(cachep->freelist_cache, freelist); + kfree(freelist); } /* @@ -1671,21 +1671,27 @@ static size_t calculate_slab_order(struct kmem_cache *cachep, if (flags & CFLGS_OFF_SLAB) { struct kmem_cache *freelist_cache; size_t freelist_size; + size_t freelist_cache_size; freelist_size = num * sizeof(freelist_idx_t); - freelist_cache = kmalloc_slab(freelist_size, 0u); - if (!freelist_cache) - continue; - - /* - * Needed to avoid possible looping condition - * in cache_grow_begin() - */ - if (OFF_SLAB(freelist_cache)) - continue; + if (freelist_size > KMALLOC_MAX_CACHE_SIZE) { + freelist_cache_size = PAGE_SIZE << get_order(freelist_size); + } else { + freelist_cache = kmalloc_slab(freelist_size, 0u); + if (!freelist_cache) + continue; + freelist_cache_size = freelist_cache->size; + + /* + * Needed to avoid possible looping condition + * in cache_grow_begin() + */ + if (OFF_SLAB(freelist_cache)) + continue; + } /* check if off slab has enough benefit */ - if (freelist_cache->size > cachep->size / 2) + if (freelist_cache_size > cachep->size / 2) continue; } @@ -2061,11 +2067,6 @@ int __kmem_cache_create(struct kmem_cache *cachep, slab_flags_t flags) cachep->flags &= ~(SLAB_RED_ZONE | SLAB_STORE_USER); #endif - if (OFF_SLAB(cachep)) { - cachep->freelist_cache = - kmalloc_slab(cachep->freelist_size, 0u); - } - err = setup_cpu_cache(cachep, gfp); if (err) { __kmem_cache_release(cachep); @@ -2292,7 +2293,7 @@ static void *alloc_slabmgmt(struct kmem_cache *cachep, freelist = NULL; else if (OFF_SLAB(cachep)) { /* Slab management obj is off-slab. */ - freelist = kmem_cache_alloc_node(cachep->freelist_cache, + freelist = kmalloc_node(cachep->freelist_size, local_flags, nodeid); } else { /* We will use last bytes at the slab for freelist */