From patchwork Tue Oct 17 15:44:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87AC5CDB483 for ; Tue, 17 Oct 2023 15:45:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 238526B020F; Tue, 17 Oct 2023 11:45:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E91A6B0213; Tue, 17 Oct 2023 11:45:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 088726B0246; Tue, 17 Oct 2023 11:45:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EB3186B020F for ; Tue, 17 Oct 2023 11:45:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BD97CC0D8D for ; Tue, 17 Oct 2023 15:45:22 +0000 (UTC) X-FDA: 81355377684.07.6260A93 Received: from out-208.mta1.migadu.com (out-208.mta1.migadu.com [95.215.58.208]) by imf26.hostedemail.com (Postfix) with ESMTP id DD1E8140013 for ; Tue, 17 Oct 2023 15:45:20 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=CcmfhoGd; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.208 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k+bY4oHxBRtJWcNO+Gp7uU0LbPEc1T0Re1GRoZLORj4=; b=CYjRrdKuubG/q2gJv1FVtSWSPeYcr92d0rX0DPGaFSIdimUv72F2ecRMfw8G2zj3rZkB2O xFx3Vn2v06z5ico2+AAoTEtuARxv8C9DLE6c5oZg2Vr9O76nSqchByymxugK1HaNiWLmG+ 6dPLA8240lOakUoBgJJpNWvJhkFXNUM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=CcmfhoGd; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.208 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557521; a=rsa-sha256; cv=none; b=V2W/OTE+1kbIDuze0qBS+7h1HoQmywknaJf46Zde5XnJlMhDzBdpPgUU6RWCB7nPkY2zRL VRwZqy+mcrP7/RTfMyWeqNQITdbFJNnTBhA6I0G3Q5n9BaMytDPkXgnPCCzhbHFhMTtJF3 aL+YO9EVJMddMQn5xHeT77mMlxHPvdo= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557519; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k+bY4oHxBRtJWcNO+Gp7uU0LbPEc1T0Re1GRoZLORj4=; b=CcmfhoGdDV5iQ65rv3mmqcXetcW5trAeYkwUs1YW3F3rWasU7mQ1KysL674T5DLYaL+6bY hvlmkEqgAGs6ijxn+CaFmKHMdHwEHXZOjokAKPisHp+6bsfBg24xWmR06X5Sv+xRkXOR31 /Kyv+0b/9DHODnwMzhKPyqiEic4J/bc= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 1/5] slub: Introduce on_partial() Date: Tue, 17 Oct 2023 15:44:35 +0000 Message-Id: <20231017154439.3036608-2-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: DD1E8140013 X-Rspam-User: X-Stat-Signature: uorjyukb5meq1t9u3ji7dq6fnthceh39 X-Rspamd-Server: rspam01 X-HE-Tag: 1697557520-376126 X-HE-Meta: U2FsdGVkX18iQoNM7TnIXBv0cplNqAbdLxFpxGUXzpTXiFDKBr/mxWf4jgL/UTjdEWg/Im1C+I7jQkKEw6ntWGG9Yaxhn3tHGAX+0a7eBdHMmi0SehC4b0zeF+jZGTYzvzNpLD4f7MLU6QA1OR8Cq7nGbUBmY/1TWgB9f2DdkuvbqusqAxwm5+7yeDd2j2Em6zVGVuTa9uuL6zgtsqRjM6oaUbqRvNtu+1NrChlyCBJbSVwslhdNDyk+kc2ZQbxxxl5T3GjF9ERmgAqTlg9w0aBISgk6PO7vysqQCPfZKTdzGpqFvj33Y4SOCgIyMpudYrmjInDM02lnxCWVt1pghR2oR8WyYo/7GxEhXjIQOT0FuCufAuHr40ZImtP03R1hbIHxoOKkwu5Tcxy9i+VHMThW9e2xXI9A9VCMXxWRSJsVUQHINAaFCfDk/38CmA0ceCAMLQG+Nsc3bsPtYIwo0gWW9bHQTL5i3I3PxOXD1enixVg3IhnbvHfPpizB43qyKxqh5OnMD+Id5XbuePy3sVzzcHlB5tV0OoSMYwO2WqcsJGWupFwNNt+k9TIRgIIiG+OuZLtq9RKI4g0YapNRjirIDi7dEFmXv4pr/dePjcjK+OvMkvQJncMzOXp4FIG5boqkvkUinOzGazy0EEBaE7qeYFkMRcut4gMFAbUS66vwO2EK0Z5glf65bnoVKzAfOtxT5ynxQf2k6ovDPCEVNiQYPfAuboh+Mt9Um0YgwWnw+sZvLM6/i1+6hBxexaVFYt2yetJZv7ShXybdko1w3tYDkRAJ92JhrFz+/Enm9y72d0s1Xa1lVrnc4C+HZVCsBkYP9kA38we8kwbaJCcqIOeP5NX6hTvz6W0++4rewqVULFGMPEk4MAkdP1I0BQoSf7QpOzepzJfjiPh1wFTy1/QvJmAY4gSFJRSj8t70QhYOT4Fj6v8fzky+nmOvValRC+AY3tglrcGDaMw1q6M xz4jC5Ni Dh3ipPUX1i+iMyAvv+U0ph0j2Z3tsgc2L+CyKhyiFrflcnQVrItsRJ4fYT8jY21I0DCUCRpBZHQc2jNpxPqX0Av2Bg3nBeolkM+Xtb9QxILPJjIm2ByHN/jcS+V5xRBlDjcc6uWh002E3KFaQBINsHDD6bdJKWWGtKIcjXmN4NtyLfLAWMwowDfHDdSa3h92Y8htZqSVbbrzapHm1+yYvHPaKYYwuklmm/OH6s4ra9TqUBkFT+ZLtNQJ9/cQlKqGP0KMj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou We change slab->__unused to slab->flags to use it as SLUB_FLAGS, which now only include SF_NODE_PARTIAL flag. It indicates whether or not the slab is on node partial list. The following patches will change to don't freeze slab when moving it from node partial list to cpu partial list. So we can't rely on frozen bit to see if we should manipulate the slab->slab_list. Instead we will rely on this SF_NODE_PARTIAL flag, which is protected by node list_lock. Signed-off-by: Chengming Zhou --- mm/slab.h | 2 +- mm/slub.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/mm/slab.h b/mm/slab.h index 8cd3294fedf5..11e9c9a0f648 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -89,7 +89,7 @@ struct slab { }; struct rcu_head rcu_head; }; - unsigned int __unused; + unsigned int flags; #else #error "Unexpected slab allocator configured" diff --git a/mm/slub.c b/mm/slub.c index 63d281dfacdb..e5356ad14951 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1993,6 +1993,12 @@ static inline bool shuffle_freelist(struct kmem_cache *s, struct slab *slab) } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +enum SLUB_FLAGS { + SF_INIT_VALUE = 0, + SF_EXIT_VALUE = -1, + SF_NODE_PARTIAL = 1 << 0, +}; + static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) { struct slab *slab; @@ -2031,6 +2037,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) slab->objects = oo_objects(oo); slab->inuse = 0; slab->frozen = 0; + slab->flags = SF_INIT_VALUE; account_slab(slab, oo_order(oo), s, flags); @@ -2077,6 +2084,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) int order = folio_order(folio); int pages = 1 << order; + slab->flags = SF_EXIT_VALUE; __slab_clear_pfmemalloc(slab); folio->mapping = NULL; /* Make the mapping reset visible before clearing the flag */ @@ -2119,9 +2127,28 @@ static void discard_slab(struct kmem_cache *s, struct slab *slab) /* * Management of partially allocated slabs. */ +static void ___add_partial(struct kmem_cache_node *n, struct slab *slab) +{ + lockdep_assert_held(&n->list_lock); + slab->flags |= SF_NODE_PARTIAL; +} + +static void ___remove_partial(struct kmem_cache_node *n, struct slab *slab) +{ + lockdep_assert_held(&n->list_lock); + slab->flags &= ~SF_NODE_PARTIAL; +} + +static inline bool on_partial(struct kmem_cache_node *n, struct slab *slab) +{ + lockdep_assert_held(&n->list_lock); + return slab->flags & SF_NODE_PARTIAL; +} + static inline void __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) { + ___add_partial(n, slab); n->nr_partial++; if (tail == DEACTIVATE_TO_TAIL) list_add_tail(&slab->slab_list, &n->partial); @@ -2142,6 +2169,7 @@ static inline void remove_partial(struct kmem_cache_node *n, lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); n->nr_partial--; + ___remove_partial(n, slab); } /* From patchwork Tue Oct 17 15:44:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 298C3CDB483 for ; Tue, 17 Oct 2023 15:45:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D9636B0246; Tue, 17 Oct 2023 11:45:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 988C36B0248; Tue, 17 Oct 2023 11:45:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82AA66B0249; Tue, 17 Oct 2023 11:45:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 72D5E6B0246 for ; Tue, 17 Oct 2023 11:45:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 52ACEA0D63 for ; Tue, 17 Oct 2023 15:45:26 +0000 (UTC) X-FDA: 81355377852.21.0401BCA Received: from out-198.mta1.migadu.com (out-198.mta1.migadu.com [95.215.58.198]) by imf29.hostedemail.com (Postfix) with ESMTP id A5E4312002E for ; Tue, 17 Oct 2023 15:45:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DaAtiZgQ; spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.198 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557523; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uepMlN8KtJ2YuJ4CH70+gbxhj8qnD/tT1k7NrYZtVVI=; b=qIAuJD4MiIx9MwdOevnuW4I++DeK8KWHifJ1+pEW1IOsi4pEp/Pxs3tOI1ND3fyX9lwcCQ 28FEaJgjVsRKyF2ikbTej7oBy+a7VhpNZTR7oADhm2G0GX7IB5egn1Zc8KZ9REcgNVyksp HikK8ecMzVqaZwlaQUMxYX4G0h0hEDk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557523; a=rsa-sha256; cv=none; b=aAe3Ng/JAFIH4VARTHUJGfMF17z1r24ullqtkiWmpVDGwHAMBzLyr/fpeOZ6CG2ybsLzrO Sd9DcGLn7koQeP0Jq0LCqngd1juZibyh+itD0tjx/W1X7uLpxTcHi8EoCtINPlTWCmm+wx Aagk8g6XeKgGNTXg2ROqP0z2vRYitiQ= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DaAtiZgQ; spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.198 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557522; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uepMlN8KtJ2YuJ4CH70+gbxhj8qnD/tT1k7NrYZtVVI=; b=DaAtiZgQCtDa1rxPzHIgjJc4DIjqm3myRMfLOwX9HUwVK+3r/TPToUXQkLJLfTM7KtEz6X P6rFQUmudCXJ4U+y638uzl0+mFMh4ihgvECPrJJA2ZQUJxMrCtLFiczP96D5Kt+GJmG0g9 TeH5WidQi5s+QCvlDqd5nD8QeqE0pNc= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 2/5] slub: Don't manipulate slab list when used by cpu Date: Tue, 17 Oct 2023 15:44:36 +0000 Message-Id: <20231017154439.3036608-3-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: d9kyayha7ff5gm6bibon4x5m1i8xpej1 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A5E4312002E X-Rspam-User: X-HE-Tag: 1697557523-177182 X-HE-Meta: U2FsdGVkX18QXKt5pIsw1A5Hkeqji71joy01Qa+HU50w4noGI/rZQ2zL2SkZ8Kk2RRzDRvckrDa0oCYpo1/w4CR1UFVWdWOQ75zk3YVdGvmlhdXjkN2PJ2/7DuSFaDctO/4+XB6O0ApvDbz1DPhpzr2PTUatZi+kgZloi1IyvjA8lg+/SIXYBjU2gAE92pOyZ1s/rI/u8pA6k1NbHc1cyY0t4y8FwkFGYh08/JX+4iQeL9hzIY421pRmiHe6oxM5aYF4LBMJVik8/mW96UINXrcZfZo4gcG9ZIahQfamS9G3MBMASZ96Ir9cYFXyRph2aMtmScZPURbsvPwI3kv5SfECwxvoSRXBoaVnKNPkeJIkt4KPKWEYIHKeCrwOKok+6NRWBZoY3tbmuTNYUfGtF3CP32C2R8QTnumi+OLTapdeyA7Yjdz0vrMAyN5LfbQ6LUauKXSEvVeea56yNkMfMZeXlhS6GFmt5bZQzozEz6qboWSdyU5OUA91CWgo0sRz7MRMUax9bcoZD9Xx6WotchTM2Gx0qK4Ebpw40i1C9HXBTYaa1/zxVBu/a9ejt3DTkP1ePVaMcGsMH1HupG+rUbJhN30Tsjiva1TPNJTUNhqOXj0kwEkhdxlfbOwBSGkneNAqIpnjVtt+wOqza8GQEJEHYohpHgQhBaNSjNWswFoWyihvPeugYuKfVEtTM7iwhrCnD+LOEi2wx2dX3WWckqcdCSWLIc+GyV46gQrst0JwIH/D5Ooxts5+B6x4CjRTDsiI4vrkfS9N8nIxjSoOwMHczY+4K99V2C2+w4smHG8J+QsK3RY+69re2/OEMD6TN0WQYq1Tt1IJWFZa/+bVfNN314bfDX0rjmElyFno2QG0uIMdqkvFt27eFf3a97uGDeJxDV37JrdlMivQC5VHke+HEyljv7dHQMVJNUM9+DeVUH0Smh0zod5HmRz+BzzdeC4HsEZ4VVDG53nxrqi +G7CgFvH SNXNclNoJVU+PD19MIVBtJqk4DDduXA09MoPp/KKrAQO0Bqy5vUMig4tgfYZ+b6XDK1gUtyiJPft+nzusVPXyJx3OIEQHqyhbHoiNS27VMw1LO83voELuvDJAceqh+qE0aMFdJKhBIUa3yedQHW7VuIpx4sk3cLScwKNZ39TbEaZCo35ZmMeeSdTwk619yWuTRz6geDSiTRaGKkE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou We will change to don't freeze slab when moving it out of node partial list in the following patch, so we can't rely on the frozen bit to indicate if we should manipulate the slab list or not. This patch use the introduced on_partial() helper, which check the slab->flags that protected by node list_lock, so we can know if the slab is on the node partial list. Signed-off-by: Chengming Zhou --- mm/slub.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index e5356ad14951..27eac93baa13 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3636,6 +3636,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, unsigned long counters; struct kmem_cache_node *n = NULL; unsigned long flags; + bool on_node_partial; stat(s, FREE_SLOWPATH); @@ -3683,6 +3684,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, */ spin_lock_irqsave(&n->list_lock, flags); + on_node_partial = on_partial(n, slab); } } @@ -3711,6 +3713,15 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, return; } + /* + * This slab was not on node partial list and not full either, + * in which case we shouldn't manipulate its list, early return. + */ + if (!on_node_partial && prior) { + spin_unlock_irqrestore(&n->list_lock, flags); + return; + } + if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) goto slab_empty; From patchwork Tue Oct 17 15:44:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C964CDB474 for ; Tue, 17 Oct 2023 15:45:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A79AE6B0248; Tue, 17 Oct 2023 11:45:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FF936B0249; Tue, 17 Oct 2023 11:45:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C6956B024A; Tue, 17 Oct 2023 11:45:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 78A346B0248 for ; Tue, 17 Oct 2023 11:45:28 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 47CC61A0D99 for ; Tue, 17 Oct 2023 15:45:28 +0000 (UTC) X-FDA: 81355377936.25.8554443 Received: from out-195.mta1.migadu.com (out-195.mta1.migadu.com [95.215.58.195]) by imf01.hostedemail.com (Postfix) with ESMTP id 8B43540021 for ; Tue, 17 Oct 2023 15:45:26 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FTuhEnnd; spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tb71WTBIRi+Tnmk2GZg+a0sbcsGECWTHjBVZ/Kgj+ow=; b=UJ8t9JkXTxEufEZWwVqb065UY+Vtbd+Hmpfs2cNB5EbL3QxDwAm4A57AcgftDc3fWP3spY IdiT8HkEy4Lbe4NGtOTvS4eNiNIRk1y7CYg7iMEFBNuq+Enc1ilqMgaUzDBxzjLVQbGjnP UnC+cwHuwfDBz3uh9olLBkY885SQxKM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FTuhEnnd; spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.195 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557526; a=rsa-sha256; cv=none; b=vmS0noPUJnspXh/8d5XUyRx7ED8nNxFfybUsmb6PEMkrmybh16IQPXHUYjef2LOYBTyTa/ /TOfoKlJIfYZ1T/RZ54eu/SravVKj3nmGbPDbM0RWfclXp1I/Gvm4cFbEt04xnMb9W4r7B wMROGfC+tnjlE+HFHq1tfohsFQwo5RE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tb71WTBIRi+Tnmk2GZg+a0sbcsGECWTHjBVZ/Kgj+ow=; b=FTuhEnndbqSCtiqr1O7e+RaC5Xz6OtTHQQeyiMhRejvTs1gPYpLuqXRStCw16NBr7Q3k3+ O4RYPhHqfkT5J7H1e8VhHiBdbBs0AckyoX/neMpbkg24WHjGGLBQOELIyx+/BM31Uj3GT8 t6UbHf9+/jBpU1VLbPxe5Kb/sPbBt9o= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 3/5] slub: Optimize deactivate_slab() Date: Tue, 17 Oct 2023 15:44:37 +0000 Message-Id: <20231017154439.3036608-4-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8B43540021 X-Rspam-User: X-Stat-Signature: uwh9ut7owmihesa3tfi4fr5xt4g9399s X-Rspamd-Server: rspam01 X-HE-Tag: 1697557526-438853 X-HE-Meta: U2FsdGVkX19LE92YkTg7z9CQvGmcUfalC0l4gaBwU+n3inGK84nQsGTfI+dehofiAVC85JZID7T2/YhaAEcfS/3tGktZA+BCb25Igcxv/YAYfB2+nJxzd5ipDPxGL/YxbOkqzNRaupMwj0/pTawJsHIeM36Ue0kFgDH+HBHKm6W+F+IKfy60u/6ObSKBL8cnacBdHxe1+Ml5WSYnpCIQbc6PJpkq3fMJwT24glyRLf2C9+kxlXoxE/g7cv4ReDIGXYCNHl8hv+S6Qbdq9kP0xEWt1xA893Qkl2liNL8ahUUDL35LAm4AHsT15YO6wzxRVElqmtqDB6MVD+nFj/2t6WD/qbsT467AL9Rr39tPBVKKAO3p+mot6o4YnVlZedvoHZIULOoT1ZJvO6ZC6yKS0LYnv3XG+piT1jPy+uJBmRATPNYsMyCSS/bpr8GwuZbgZtgWAfgw3imS7c1JFCgU7EzygSHRoSdQ1HEcT0PfHb13bbpPKmq37ohBdCXWzMdsWL6OXcuKslE7KBVVZh7OVhRydgNbl5EgZdlSdoO9vAfkqpJjyeYdQWDVWWquHqC4JcncafrG1UURQ/vIH0vN896iIcYoQNnXys4aj49lfCQRVGFuQrgOIQQYe07Lnwmckdm6fhAVGGmP9lhHa80SdK/R9ctSKF3H2vDw/rV70ukaJnf9lX1fNi+aZbUy1s6ayvcQwfVGopJBkGTw6G/DHJUtVDPjP7BxYJqk97EWgbDdIDiVjy5RgjTRNp4q+xIWbzuPIRU3Flr/JwymhhSXzxGb1usGIsYFN+uskBx22jmRGUz2ZaJXKjJ1gBWxJDIaHZZTDpGP/jqfxbxtQvinMCaEOONJPC/SB3Wt7DY9Kj1ibPFQvjqpg+FKOiUD2tS2ThF9Q4salRd8h/OelQriYFHxV1xnF7gfMzPcwC9K2HkpFcL07ohsARm2y1ykT6OiOCqRraciffJunaXeFfj H5rg5INC UbtexJNTVs5QQjt1EueiMtsTWPUaUXluZFaiSpXqpy+xA833TorKhEIr4p+UsRUxSsQyLDM95UcIEtImWiRHHyB9W9bapTzWvtgpIdx5i8Je5HLclJZritpFL+EjKzd9j2NN8pOoY+NADewZj2IQMMUBN6bjNS0f/Cp8l6HDNSUhUy/sw6M0sNpg0N/O5Rscjg3nKSPw9Hy+PNi8EmkLMzVl6BUZ5CLuSni9apXipCamGai9QoeS2XmwOHHWV4L3jb3hn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou Since the introduce of unfrozen slabs on cpu partial list, we don't need to synchronize the slab frozen state under the node list_lock. The caller of deactivate_slab() and the caller of __slab_free() won't manipulate the slab list concurrently. So we can get node list_lock in the stage three if we need to manipulate the slab list in this path. Signed-off-by: Chengming Zhou --- mm/slub.c | 70 ++++++++++++++++++++----------------------------------- 1 file changed, 25 insertions(+), 45 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 27eac93baa13..5a9711b35c74 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2520,10 +2520,8 @@ static void init_kmem_cache_cpus(struct kmem_cache *s) static void deactivate_slab(struct kmem_cache *s, struct slab *slab, void *freelist) { - enum slab_modes { M_NONE, M_PARTIAL, M_FREE, M_FULL_NOLIST }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); int free_delta = 0; - enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2570,58 +2568,40 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, * unfrozen and number of objects in the slab may have changed. * Then release lock and retry cmpxchg again. */ -redo: - - old.freelist = READ_ONCE(slab->freelist); - old.counters = READ_ONCE(slab->counters); - VM_BUG_ON(!old.frozen); - - /* Determine target state of the slab */ - new.counters = old.counters; - if (freelist_tail) { - new.inuse -= free_delta; - set_freepointer(s, freelist_tail, old.freelist); - new.freelist = freelist; - } else - new.freelist = old.freelist; + do { + old.freelist = READ_ONCE(slab->freelist); + old.counters = READ_ONCE(slab->counters); + VM_BUG_ON(!old.frozen); + + /* Determine target state of the slab */ + new.counters = old.counters; + new.frozen = 0; + if (freelist_tail) { + new.inuse -= free_delta; + set_freepointer(s, freelist_tail, old.freelist); + new.freelist = freelist; + } else + new.freelist = old.freelist; - new.frozen = 0; + } while (!slab_update_freelist(s, slab, + old.freelist, old.counters, + new.freelist, new.counters, + "unfreezing slab")); + /* + * Stage three: Manipulate the slab list based on the updated state. + */ if (!new.inuse && n->nr_partial >= s->min_partial) { - mode = M_FREE; + stat(s, DEACTIVATE_EMPTY); + discard_slab(s, slab); + stat(s, FREE_SLAB); } else if (new.freelist) { - mode = M_PARTIAL; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ spin_lock_irqsave(&n->list_lock, flags); - } else { - mode = M_FULL_NOLIST; - } - - - if (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")) { - if (mode == M_PARTIAL) - spin_unlock_irqrestore(&n->list_lock, flags); - goto redo; - } - - - if (mode == M_PARTIAL) { add_partial(n, slab, tail); spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - } else if (mode == M_FREE) { - stat(s, DEACTIVATE_EMPTY); - discard_slab(s, slab); - stat(s, FREE_SLAB); - } else if (mode == M_FULL_NOLIST) { + } else stat(s, DEACTIVATE_FULL); - } } #ifdef CONFIG_SLUB_CPU_PARTIAL From patchwork Tue Oct 17 15:44:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 461BDCDB474 for ; Tue, 17 Oct 2023 15:45:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5E3B6B024B; Tue, 17 Oct 2023 11:45:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0BE66B024C; Tue, 17 Oct 2023 11:45:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B382A6B0250; Tue, 17 Oct 2023 11:45:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A042F6B024B for ; Tue, 17 Oct 2023 11:45:31 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 73BBE80A8F for ; Tue, 17 Oct 2023 15:45:31 +0000 (UTC) X-FDA: 81355378062.10.994CEB7 Received: from out-209.mta1.migadu.com (out-209.mta1.migadu.com [95.215.58.209]) by imf21.hostedemail.com (Postfix) with ESMTP id 97C311C0022 for ; Tue, 17 Oct 2023 15:45:29 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QTQRTtR4; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557529; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SwVUZ0CSOlUV0/uHFVqKY/DwonXk9nrbYMCg7m1DgMw=; b=bVn4oXI8CMaBoXM+Ai8V2o1bg2sG098j+PEVjJZSyYzXx5auI/4gPudGAmyhj2bdFHuzoW xw3qUhs/V0olGkBwDB9oNhutPOoe4SM5SgDa5sBJXpjK2NlPKYqXYUZeJwlPO42BshdqZ8 fWO/cfTAPXfTqZFvsHddldc+Zf3mxC4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QTQRTtR4; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557529; a=rsa-sha256; cv=none; b=AOaRmGpYM7dnt4vB3fA6K/s15MiHwl6B+V1CFeaWzHmc8SvLky+Iqpu/rGtxbo26j/zv8T DI5w4oAKQXd8sqQWC3mnck5n15jloOhbHFnG50qslIdJ/MbrWlojyaXNYXacf1PCSOwtVm 8azggsvmtMmQkMudaqpebGSTzC+TSp0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SwVUZ0CSOlUV0/uHFVqKY/DwonXk9nrbYMCg7m1DgMw=; b=QTQRTtR4TDMx815F2FxWR0MO4r4tizfmzdquqGgIm6079a5vKw/+Kv5swHYxfc+xr4bwXq Qc4wl110RF7gV0nbPrV0c1kK94qUoAu4HC34L59IfxeRumqBLP9gWTX7ryMGjzSz0nXUJR 3HtIn99NiO2gQzsXZGKH13+VO/pQ1ig= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 4/5] slub: Don't freeze slabs for cpu partial Date: Tue, 17 Oct 2023 15:44:38 +0000 Message-Id: <20231017154439.3036608-5-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 97C311C0022 X-Stat-Signature: x4byeczajiihicep7nu7gke94w86t5if X-HE-Tag: 1697557529-155892 X-HE-Meta: U2FsdGVkX1+SvcTkMChBfAxhkdZBzZIOQPDXPrObizXonV0W5LKxXEbVMw+PkxwG/yYT/SJ+9/fTEww0QQrVXzYN/vrqNky4+r7qiQTAt1vVHkHPqRRzHEaztq8T3V6QQnQDeBv7O/XlahnyouEwsUl/pwLepkwaZYayrlbmrvnj1oUxspLVfAVnOjnW/wAcOFXlfVrZU/D/bOmdqdZ4D4nPivyHG7Laqb9kjDFppTYXF3Mt4FcLGK5zYhShjR0aBIaMO2bXFHiUnBXlqj9ebYpwTfPpCKUGMk01Zr19NXXOgIuYZW9/u3rly3QMRigocxCNmOdHTvFLBwhyrzfZk+kyqKPwyJ0+hyLmpu9qojC5tUCglOHF0HuTJCF6FqF1HPUIhftPtDuwum9Gn7MTOmkG/PUWp+8Hi18iovwcmN5Own3Tg+HF0X/B7ADvlobtavhYaaJadvk7Y+26/uetIOv6esyHnVd3IGR72qsBzvoqydA7P/O8e4zmk4ixoBFlFDdFw3wzJbx8PbKd4Nh9R3MVtkU1nfQ8cEbKks3dk7BQIrYUweF915cJCii4vWepeq1W2LE+EzNij2ck4mxnVb8WDZdS86VmQxd9nKlKzmzFNSoF/nACoPnLs+v9Wy8EGkJkgts+kH2uoA9SHjPpz1q+KMBoXcw7fedTPbVjcirzA+M118APVZLevRteC3uuSMHLz4WBbuHjEQ84hIJtx67EgBhvapET3JEqdjsHl0R+d/daohAh25ZBOXfQbeJBwiIQKp4kdmt6TBtDeK5jeDjgqu9BYBKS7SrJivIzGUtBQBpfljpsuVWZMdrFIynRIpI0AK8EF9/pl09mZvrFMA4NlQhjkACVkkiJ2jyaKoFryG0PkBs9u/yjZ1s4AmFFaVg6ZZQfHcf6+5AfOafyuS8zsy7zW7Otc/utxTDNhg/IPipbEQRbo9Vk1M3mhvdQW5iknfOPDzUemUur9zt kmTckKz5 7H+63Cg+Odr618i6PgrVGPJrhaF2ZyFRXbejmxrKykOlnmifmcQCWoEY8oQ8DWutFy/Rnz3RxdLLOauloT2GwqOsB+QxGSbh0U0322LZ46WZnd8e5/ofmK38jXzGsnIreJaXldDFaR5u7Ti3RLXk0Z+Pnqoo36U45jlN3r3F4HPBD16rwX5zNH+UqDDHi9E8CbdMSRf20UptsLwY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freeze to get slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: 1. The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. 2. The node list_lock contention would be less, since we only need to freeze one slab under the node list_lock. (In fact, we can first move slabs out of node partial list, don't need to freeze any slab at all, so the contention on slab won't transfer to the node list_lock contention.) We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is serialized using the new introduced slab->flags. Note this patch just change the part of moving the partial slabs for easy code review, we will fix other parts in the following patches. Signed-off-by: Chengming Zhou --- mm/slub.c | 61 ++++++++++++++++--------------------------------------- 1 file changed, 17 insertions(+), 44 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5a9711b35c74..044235bd8a45 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2329,19 +2329,21 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, continue; } - t = acquire_slab(s, n, slab, object == NULL); - if (!t) - break; - if (!object) { - *pc->slab = slab; - stat(s, ALLOC_FROM_PARTIAL); - object = t; - } else { - put_cpu_partial(s, slab, 0); - stat(s, CPU_PARTIAL_NODE); - partial_slabs++; + t = acquire_slab(s, n, slab, object == NULL); + if (t) { + *pc->slab = slab; + stat(s, ALLOC_FROM_PARTIAL); + object = t; + continue; + } } + + remove_partial(n, slab); + put_cpu_partial(s, slab, 0); + stat(s, CPU_PARTIAL_NODE); + partial_slabs++; + #ifdef CONFIG_SLUB_CPU_PARTIAL if (!kmem_cache_has_cpu_partial(s) || partial_slabs > s->cpu_partial_slabs / 2) @@ -2612,9 +2614,6 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) unsigned long flags = 0; while (partial_slab) { - struct slab new; - struct slab old; - slab = partial_slab; partial_slab = slab->next; @@ -2627,23 +2626,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } - do { - - old.freelist = slab->freelist; - old.counters = slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters = old.counters; - new.freelist = old.freelist; - - new.frozen = 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; slab_to_discard = slab; } else { @@ -3640,18 +3623,8 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, was_frozen = new.frozen; new.inuse -= cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen = 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { n = get_node(s, slab_nid(slab)); /* @@ -3681,7 +3654,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* * If we just froze the slab then put it onto the * per cpu partial list. From patchwork Tue Oct 17 15:44:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7C2CDB482 for ; Tue, 17 Oct 2023 15:45:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8570A6B0250; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80ABA6B0251; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6566D6B0252; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 52A6D6B0250 for ; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1D1D11206C7 for ; Tue, 17 Oct 2023 15:45:34 +0000 (UTC) X-FDA: 81355378188.06.1B3706F Received: from out-209.mta1.migadu.com (out-209.mta1.migadu.com [95.215.58.209]) by imf21.hostedemail.com (Postfix) with ESMTP id 4E3B41C0015 for ; Tue, 17 Oct 2023 15:45:32 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=t7itMpfW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/rqGINGA/f9CjHGz6bcXHgniUddOArpzlJ7hGX+P1rU=; b=Ty0SjR7BqBub7YeUu8PrrTbhMjMOV8sFVy32aExkMSj/IyqTP3XTTSZKbRTHjHb+MVXU85 F5j1RglCoz3TfhyzSCIytC0Jl3ezjeA7ne0Jo4cfya0xAKYTogmgkHH2nzqeCKh32CIlUo uGytlXgpmiGxNWshsHvwOQHTVVheDE4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=t7itMpfW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557532; a=rsa-sha256; cv=none; b=jSXDnVI1xyslxtL9Fr1eiGYYxZiXT8+dK2a3IT5U/9+7Aukxjc7A7ebcSxAny+ErmSjmpz NGVaglHqESOfKg98U0zsJ8NGNic//7EHzpIl0Yy8cant+RNcTiYxfx5EI6qntn+Hhgisgd GPuPkTzKAJtPoWShYdMpDQ/hAKrKHMs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/rqGINGA/f9CjHGz6bcXHgniUddOArpzlJ7hGX+P1rU=; b=t7itMpfWMoYVC3j6sv4EJs1ydOPU1zP3XISTj8W27fYulJ37dQyEN+Pa322OfNaStXpJZV 36vAPklwKga0vw8d+1eVptg8XqL89bkZHme5rmr/8afd3FtJyx000aND1uE1U0lpxztDa8 d7+B+Qhwf+dLLjvv182AjauHrt2yObw= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 5/5] slub: Introduce get_cpu_partial() Date: Tue, 17 Oct 2023 15:44:39 +0000 Message-Id: <20231017154439.3036608-6-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Stat-Signature: bc9hpgcnqomza9ekr3iwcd37fi3jsqpo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4E3B41C0015 X-HE-Tag: 1697557532-792975 X-HE-Meta: U2FsdGVkX182y8aaNYYGaGLCRfhbxCjsFz0rAHmIuUsiR1UnXwqw9SyUV039C0TlivFMwbQ9lRoBtJHY9ACvnNUU1xknsUVB2FovjA6EuH5yVyOeNqFf5zXCsqYFEScbCOa/8yGREX+T93mpzXk2M5ynz2IwhV3EvOAyRio7yauud4vOK2OSfBHeVrs5kJB1KnyGrdhAD21laxZmkzcH7OVpi3wUAQaNE0ZUmW/q+lK9A3cXtRmGDBmBOE9zcTjtaz1KglrvXKdxTTr/T6FAx45nA5De7eUBhCjsLiYEz2lpKrWShtxalwLE/LFfnqllpjfcGj9jXsH3NBdN0BtRRGatV892Cb0XXZvx9+TFpgBHcteNF1UPorfFcich5AvtgjUnFprjkCUmKE2aJT65GHCySdhACoC1Mk64UFXpwtfxm+3oBgzvw5p1ygQH8wKvqxWyQOyg1gjmiAsE8ChiODhhwel6bQoH13ikvr1jfDIBTO6KCHKwTR0Ex+FgJrzOiJBOmzGa3ZLQrvDVctgisX7SvXDjsb6yXeMH3HXVl/tC/wUdm3mrFuFlSVMHr4rHD0R58EgnAbSZusZ/ODF9Rn73MW/yX0H9MAK98JcazYFoBSCKoG40gvXYESO+XexhUlW81kVvd6Cim4RCXBUVcboUCyD6hSGJ5wyNSJ0IE1kz/PoFNZrvUwizJNLZyEsiD8L3TRrOrmLqYPoLCY4ATW8FRLd5gb5qvB94vATsu9jdpxV80EiaO/FCQguTl+B3eatiCJwclOUmdCVHm7gKRT/2Sp/xBlBxY07BdZyplKRFZnqbimPrEOEAjO2uuRipeygjJY+cJhcRGerQrJqye3+JhDDf/rgkoherslAOFMF4BRQYNVDxGlHMlUCjdsHIjVPrjbmZIYrJ2ROcYHKxIQWOiZZcR+HM2Jnm+BDPSa8PBd/jSNH1fTzWCJiYzOxZ8UkKkMVNGQywSg9rmTp klnGJVu9 yFuxZGXB4+xdl1fkyifFoD/6nf0AH9aMcviHARG+PZPBVnvXfusZ1+xKiLJd4hy+b8dizvQT5vfKwKGM6gYIqmhHNSpnCxU6Q3AulwX5ubwngCmRiae6q0FVswrtM1fuyZ5ClDglDeDxKPZp/2IkN/MB6E45THc3hdu/AKcUaOnmjpWQ43YlxpPYcnQrP/EBU6ABJvwXW3rZ8ZQr/kArQfkkwivS8tWm0qqvtD+4zFEr5WHU3DM8V1xNLNQIxHPjzaTPX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou Since the slabs on cpu partial list are not frozen anymore, we introduce get_cpu_partial() to get a frozen slab with its freelist from cpu partial list. It's now much like getting a frozen slab with its freelist from node partial list. Another change is about get_partial(), which can return no frozen slab when all slabs are failed when acquire_slab(), but get some unfreeze slabs in its cpu partial list, so we need to check this rare case to avoid allocating a new slab. Signed-off-by: Chengming Zhou --- mm/slub.c | 87 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 19 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 044235bd8a45..d58eaf8447fd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3064,6 +3064,68 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) return freelist; } +#ifdef CONFIG_SLUB_CPU_PARTIAL + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + unsigned long flags; + struct slab *slab; + struct slab new; + unsigned long counters; + void *freelist; + + while (slub_percpu_partial(c)) { + local_lock_irqsave(&s->cpu_slab->lock, flags); + if (unlikely(!slub_percpu_partial(c))) { + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + /* we were preempted and partial list got empty */ + return NULL; + } + + slab = slub_percpu_partial(c); + slub_set_percpu_partial(c, slab); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + stat(s, CPU_PARTIAL_ALLOC); + + if (unlikely(!node_match(slab, node) || + !pfmemalloc_match(slab, gfpflags))) { + slab->next = NULL; + __unfreeze_partials(s, slab); + continue; + } + + do { + freelist = slab->freelist; + counters = slab->counters; + + new.counters = counters; + VM_BUG_ON(new.frozen); + + new.inuse = slab->objects; + new.frozen = 1; + } while (!__slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + "get_cpu_partial")); + + *slabptr = slab; + return freelist; + } + + return NULL; +} + +#else /* CONFIG_SLUB_CPU_PARTIAL */ + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + return NULL; +} + +#endif + /* * Slow path. The lockless freelist is empty or we need to perform * debugging duties. @@ -3106,7 +3168,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, node = NUMA_NO_NODE; goto new_slab; } -redo: if (unlikely(!node_match(slab, node))) { /* @@ -3182,24 +3243,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_slab: - if (slub_percpu_partial(c)) { - local_lock_irqsave(&s->cpu_slab->lock, flags); - if (unlikely(c->slab)) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - goto reread_slab; - } - if (unlikely(!slub_percpu_partial(c))) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - /* we were preempted and partial list got empty */ - goto new_objects; - } - - slab = c->slab = slub_percpu_partial(c); - slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); - goto redo; - } + freelist = get_cpu_partial(s, c, &slab, node, gfpflags); + if (freelist) + goto retry_load_slab; new_objects: @@ -3210,6 +3256,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto check_new_slab; + if (slub_percpu_partial(c)) + goto new_slab; + slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); c = slub_get_cpu_ptr(s->cpu_slab);