From patchwork Tue Oct 17 15:44:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13425540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7C2CDB482 for ; Tue, 17 Oct 2023 15:45:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8570A6B0250; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80ABA6B0251; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6566D6B0252; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 52A6D6B0250 for ; Tue, 17 Oct 2023 11:45:34 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1D1D11206C7 for ; Tue, 17 Oct 2023 15:45:34 +0000 (UTC) X-FDA: 81355378188.06.1B3706F Received: from out-209.mta1.migadu.com (out-209.mta1.migadu.com [95.215.58.209]) by imf21.hostedemail.com (Postfix) with ESMTP id 4E3B41C0015 for ; Tue, 17 Oct 2023 15:45:32 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=t7itMpfW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697557532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/rqGINGA/f9CjHGz6bcXHgniUddOArpzlJ7hGX+P1rU=; b=Ty0SjR7BqBub7YeUu8PrrTbhMjMOV8sFVy32aExkMSj/IyqTP3XTTSZKbRTHjHb+MVXU85 F5j1RglCoz3TfhyzSCIytC0Jl3ezjeA7ne0Jo4cfya0xAKYTogmgkHH2nzqeCKh32CIlUo uGytlXgpmiGxNWshsHvwOQHTVVheDE4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=t7itMpfW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.209 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697557532; a=rsa-sha256; cv=none; b=jSXDnVI1xyslxtL9Fr1eiGYYxZiXT8+dK2a3IT5U/9+7Aukxjc7A7ebcSxAny+ErmSjmpz NGVaglHqESOfKg98U0zsJ8NGNic//7EHzpIl0Yy8cant+RNcTiYxfx5EI6qntn+Hhgisgd GPuPkTzKAJtPoWShYdMpDQ/hAKrKHMs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697557531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/rqGINGA/f9CjHGz6bcXHgniUddOArpzlJ7hGX+P1rU=; b=t7itMpfWMoYVC3j6sv4EJs1ydOPU1zP3XISTj8W27fYulJ37dQyEN+Pa322OfNaStXpJZV 36vAPklwKga0vw8d+1eVptg8XqL89bkZHme5rmr/8afd3FtJyx000aND1uE1U0lpxztDa8 d7+B+Qhwf+dLLjvv182AjauHrt2yObw= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH 5/5] slub: Introduce get_cpu_partial() Date: Tue, 17 Oct 2023 15:44:39 +0000 Message-Id: <20231017154439.3036608-6-chengming.zhou@linux.dev> In-Reply-To: <20231017154439.3036608-1-chengming.zhou@linux.dev> References: <20231017154439.3036608-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Stat-Signature: bc9hpgcnqomza9ekr3iwcd37fi3jsqpo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4E3B41C0015 X-HE-Tag: 1697557532-792975 X-HE-Meta: U2FsdGVkX182y8aaNYYGaGLCRfhbxCjsFz0rAHmIuUsiR1UnXwqw9SyUV039C0TlivFMwbQ9lRoBtJHY9ACvnNUU1xknsUVB2FovjA6EuH5yVyOeNqFf5zXCsqYFEScbCOa/8yGREX+T93mpzXk2M5ynz2IwhV3EvOAyRio7yauud4vOK2OSfBHeVrs5kJB1KnyGrdhAD21laxZmkzcH7OVpi3wUAQaNE0ZUmW/q+lK9A3cXtRmGDBmBOE9zcTjtaz1KglrvXKdxTTr/T6FAx45nA5De7eUBhCjsLiYEz2lpKrWShtxalwLE/LFfnqllpjfcGj9jXsH3NBdN0BtRRGatV892Cb0XXZvx9+TFpgBHcteNF1UPorfFcich5AvtgjUnFprjkCUmKE2aJT65GHCySdhACoC1Mk64UFXpwtfxm+3oBgzvw5p1ygQH8wKvqxWyQOyg1gjmiAsE8ChiODhhwel6bQoH13ikvr1jfDIBTO6KCHKwTR0Ex+FgJrzOiJBOmzGa3ZLQrvDVctgisX7SvXDjsb6yXeMH3HXVl/tC/wUdm3mrFuFlSVMHr4rHD0R58EgnAbSZusZ/ODF9Rn73MW/yX0H9MAK98JcazYFoBSCKoG40gvXYESO+XexhUlW81kVvd6Cim4RCXBUVcboUCyD6hSGJ5wyNSJ0IE1kz/PoFNZrvUwizJNLZyEsiD8L3TRrOrmLqYPoLCY4ATW8FRLd5gb5qvB94vATsu9jdpxV80EiaO/FCQguTl+B3eatiCJwclOUmdCVHm7gKRT/2Sp/xBlBxY07BdZyplKRFZnqbimPrEOEAjO2uuRipeygjJY+cJhcRGerQrJqye3+JhDDf/rgkoherslAOFMF4BRQYNVDxGlHMlUCjdsHIjVPrjbmZIYrJ2ROcYHKxIQWOiZZcR+HM2Jnm+BDPSa8PBd/jSNH1fTzWCJiYzOxZ8UkKkMVNGQywSg9rmTp klnGJVu9 yFuxZGXB4+xdl1fkyifFoD/6nf0AH9aMcviHARG+PZPBVnvXfusZ1+xKiLJd4hy+b8dizvQT5vfKwKGM6gYIqmhHNSpnCxU6Q3AulwX5ubwngCmRiae6q0FVswrtM1fuyZ5ClDglDeDxKPZp/2IkN/MB6E45THc3hdu/AKcUaOnmjpWQ43YlxpPYcnQrP/EBU6ABJvwXW3rZ8ZQr/kArQfkkwivS8tWm0qqvtD+4zFEr5WHU3DM8V1xNLNQIxHPjzaTPX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou Since the slabs on cpu partial list are not frozen anymore, we introduce get_cpu_partial() to get a frozen slab with its freelist from cpu partial list. It's now much like getting a frozen slab with its freelist from node partial list. Another change is about get_partial(), which can return no frozen slab when all slabs are failed when acquire_slab(), but get some unfreeze slabs in its cpu partial list, so we need to check this rare case to avoid allocating a new slab. Signed-off-by: Chengming Zhou --- mm/slub.c | 87 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 19 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 044235bd8a45..d58eaf8447fd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3064,6 +3064,68 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) return freelist; } +#ifdef CONFIG_SLUB_CPU_PARTIAL + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + unsigned long flags; + struct slab *slab; + struct slab new; + unsigned long counters; + void *freelist; + + while (slub_percpu_partial(c)) { + local_lock_irqsave(&s->cpu_slab->lock, flags); + if (unlikely(!slub_percpu_partial(c))) { + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + /* we were preempted and partial list got empty */ + return NULL; + } + + slab = slub_percpu_partial(c); + slub_set_percpu_partial(c, slab); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + stat(s, CPU_PARTIAL_ALLOC); + + if (unlikely(!node_match(slab, node) || + !pfmemalloc_match(slab, gfpflags))) { + slab->next = NULL; + __unfreeze_partials(s, slab); + continue; + } + + do { + freelist = slab->freelist; + counters = slab->counters; + + new.counters = counters; + VM_BUG_ON(new.frozen); + + new.inuse = slab->objects; + new.frozen = 1; + } while (!__slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + "get_cpu_partial")); + + *slabptr = slab; + return freelist; + } + + return NULL; +} + +#else /* CONFIG_SLUB_CPU_PARTIAL */ + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + return NULL; +} + +#endif + /* * Slow path. The lockless freelist is empty or we need to perform * debugging duties. @@ -3106,7 +3168,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, node = NUMA_NO_NODE; goto new_slab; } -redo: if (unlikely(!node_match(slab, node))) { /* @@ -3182,24 +3243,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_slab: - if (slub_percpu_partial(c)) { - local_lock_irqsave(&s->cpu_slab->lock, flags); - if (unlikely(c->slab)) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - goto reread_slab; - } - if (unlikely(!slub_percpu_partial(c))) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - /* we were preempted and partial list got empty */ - goto new_objects; - } - - slab = c->slab = slub_percpu_partial(c); - slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); - goto redo; - } + freelist = get_cpu_partial(s, c, &slab, node, gfpflags); + if (freelist) + goto retry_load_slab; new_objects: @@ -3210,6 +3256,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto check_new_slab; + if (slub_percpu_partial(c)) + goto new_slab; + slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); c = slub_get_cpu_ptr(s->cpu_slab);