From patchwork Wed Mar 17 07:54:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 12145065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A906FC433E0 for ; Wed, 17 Mar 2021 07:55:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2893E64F8F for ; Wed, 17 Mar 2021 07:55:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2893E64F8F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 50C966B0071; Wed, 17 Mar 2021 03:55:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D17D6B0073; Wed, 17 Mar 2021 03:55:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 372AF6B0074; Wed, 17 Mar 2021 03:55:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 206576B0071 for ; Wed, 17 Mar 2021 03:55:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C5653181AF5E1 for ; Wed, 17 Mar 2021 07:54:59 +0000 (UTC) X-FDA: 77928605118.06.FA3DE9B Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf21.hostedemail.com (Postfix) with ESMTP id 11683E0011C5 for ; Wed, 17 Mar 2021 07:54:57 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R841e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0USESt7y_1615967693; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0USESt7y_1615967693) by smtp.aliyun-inc.com(127.0.0.1); Wed, 17 Mar 2021 15:54:53 +0800 From: Xunlei Pang To: Christoph Lameter , Christoph Lameter , Pekka Enberg , Vlastimil Babka , Roman Gushchin , Konstantin Khlebnikov , David Rientjes , Matthew Wilcox , Shu Ming , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Wen Yang , James Wang , Xunlei Pang Subject: [PATCH v4 1/3] mm/slub: Introduce two counters for partial objects Date: Wed, 17 Mar 2021 15:54:50 +0800 Message-Id: <1615967692-80524-2-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> References: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> X-Stat-Signature: ai1uspxyxsk5rr3bzbuen7z663c6psha X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 11683E0011C5 Received-SPF: none (linux.alibaba.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=out30-44.freemail.mail.aliyun.com; client-ip=115.124.30.44 X-HE-DKIM-Result: none/none X-HE-Tag: 1615967697-725074 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The node list_lock in count_partial() spends long time iterating in case of large amount of partial page lists, which can cause thunder herd effect to the list_lock contention. We have HSF RT(High-speed Service Framework Response-Time) monitors, the RT figures fluctuated randomly, then we deployed a tool detecting "irq off" and "preempt off" to dump the culprit's calltrace, capturing the list_lock cost nearly 100ms with irq off issued by "ss", this also caused network timeouts. This patch introduces two counters to maintain the actual number of partial objects dynamically instead of iterating the partial page lists with list_lock held. New counters of kmem_cache_node: partial_free_objs, partial_total_objs. The main operations are under list_lock in slow path, its performance impact is expected to be minimal except the __slab_free() path. The only concern of introducing partial counter is that partial_free_objs may cause cacheline contention and false sharing issues in case of same SLUB concurrent __slab_free(), so define it to be a percpu counter and places it carefully. As "Vlastimil Babka" and "Christoph Lameter" both suggested that we don't care about the accurate partial objects, and in case that somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Thus a new CONFIG_SLUB_DEBUG_PARTIAL is introduced to maintain accurate partial objects, which is default off to avoid unexpected performance degradation. When CONFIG_SLUB_DEBUG_PARTIAL is not selected, the sysfs or proc files related to the partial list (/proc/slabinfo, sysfs objects_partial, etc) assume their zero usage data, e.g., Value of "active_objs" and "num_objs" fields of "/proc/slabinfo" equal. "cat /sys/kernel/slab//partial" displays "0". Tested-by: James Wang Signed-off-by: Xunlei Pang --- init/Kconfig | 13 +++++++++++++ mm/slab.h | 6 ++++++ mm/slub.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 78 insertions(+), 4 deletions(-) diff --git a/init/Kconfig b/init/Kconfig index 22946fe..686851b 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1867,6 +1867,19 @@ config SLUB_DEBUG SLUB sysfs support. /sys/slab will not exist and there will be no support for cache validation etc. +config SLUB_DEBUG_PARTIAL + default n + bool "Enable SLUB debugging for the node partial list" if EXPERT + depends on SLUB && SYSFS + help + Maintain runtime counters for the node partial list debugging. + When CONFIG_SLUB_DEBUG_PARTIAL is not selected, the sysfs or proc + files related to the partial list assume zero data usage, e.g., + value of "active_objs" and "num_objs" of "/proc/slabinfo" equals. + "cat /sys/kernel/slab//partial" displays "0". + + It might have slight performance impact for production use. + config COMPAT_BRK bool "Disable heap randomization" default y diff --git a/mm/slab.h b/mm/slab.h index 076582f..f47c811 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -546,12 +546,18 @@ struct kmem_cache_node { #ifdef CONFIG_SLUB unsigned long nr_partial; +#ifdef CONFIG_SLUB_DEBUG_PARTIAL + unsigned long partial_total_objs; +#endif struct list_head partial; #ifdef CONFIG_SLUB_DEBUG atomic_long_t nr_slabs; atomic_long_t total_objects; struct list_head full; #endif +#ifdef CONFIG_SLUB_DEBUG_PARTIAL + unsigned long __percpu *partial_free_objs ____cacheline_aligned_in_smp; +#endif #endif }; diff --git a/mm/slub.c b/mm/slub.c index e26c274..856aea4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1890,10 +1890,31 @@ static void discard_slab(struct kmem_cache *s, struct page *page) /* * Management of partially allocated slabs. */ +#ifdef CONFIG_SLUB_DEBUG_PARTIAL +static inline void +__update_partial_free(struct kmem_cache_node *n, long delta) +{ + this_cpu_add(*n->partial_free_objs, delta); +} + +static inline void +__update_partial_total(struct kmem_cache_node *n, long delta) +{ + n->partial_total_objs += delta; +} +#else +static inline void +__update_partial_free(struct kmem_cache_node *n, long delta) { } + +static inline void +__update_partial_total(struct kmem_cache_node *n, long delta) { } +#endif + static inline void __add_partial(struct kmem_cache_node *n, struct page *page, int tail) { n->nr_partial++; + __update_partial_total(n, page->objects); if (tail == DEACTIVATE_TO_TAIL) list_add_tail(&page->slab_list, &n->partial); else @@ -1913,6 +1934,7 @@ static inline void remove_partial(struct kmem_cache_node *n, lockdep_assert_held(&n->list_lock); list_del(&page->slab_list); n->nr_partial--; + __update_partial_total(n, -page->objects); } /* @@ -1957,6 +1979,7 @@ static inline void *acquire_slab(struct kmem_cache *s, return NULL; remove_partial(n, page); + __update_partial_free(n, -*objects); WARN_ON(!freelist); return freelist; } @@ -2286,8 +2309,11 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, "unfreezing slab")) goto redo; - if (lock) + if (lock) { + if (m == M_PARTIAL) + __update_partial_free(n, new.objects - new.inuse); spin_unlock(&n->list_lock); + } if (m == M_PARTIAL) stat(s, tail); @@ -2353,6 +2379,7 @@ static void unfreeze_partials(struct kmem_cache *s, discard_page = page; } else { add_partial(n, page, DEACTIVATE_TO_TAIL); + __update_partial_free(n, new.objects - new.inuse); stat(s, FREE_ADD_PARTIAL); } } @@ -3039,6 +3066,13 @@ static void __slab_free(struct kmem_cache *s, struct page *page, head, new.counters, "__slab_free")); + if (!was_frozen && prior) { + if (n) + __update_partial_free(n, cnt); + else + __update_partial_free(get_node(s, page_to_nid(page)), cnt); + } + if (likely(!n)) { if (likely(was_frozen)) { @@ -3069,6 +3103,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, if (!kmem_cache_has_cpu_partial(s) && unlikely(!prior)) { remove_full(s, n, page); add_partial(n, page, DEACTIVATE_TO_TAIL); + __update_partial_free(n, cnt); stat(s, FREE_ADD_PARTIAL); } spin_unlock_irqrestore(&n->list_lock, flags); @@ -3080,6 +3115,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, * Slab on the partial list. */ remove_partial(n, page); + __update_partial_free(n, -page->objects); stat(s, FREE_REMOVE_PARTIAL); } else { /* Slab must be on the full list */ @@ -3514,17 +3550,25 @@ static inline int calculate_order(unsigned int size) return -ENOSYS; } -static void +static int init_kmem_cache_node(struct kmem_cache_node *n) { n->nr_partial = 0; spin_lock_init(&n->list_lock); INIT_LIST_HEAD(&n->partial); +#ifdef CONFIG_SLUB_DEBUG_PARTIAL + n->partial_free_objs = alloc_percpu(unsigned long); + if (!n->partial_free_objs) + return -ENOMEM; + n->partial_total_objs = 0; +#endif #ifdef CONFIG_SLUB_DEBUG atomic_long_set(&n->nr_slabs, 0); atomic_long_set(&n->total_objects, 0); INIT_LIST_HEAD(&n->full); #endif + + return 0; } static inline int alloc_kmem_cache_cpus(struct kmem_cache *s) @@ -3584,7 +3628,7 @@ static void early_kmem_cache_node_alloc(int node) page->inuse = 1; page->frozen = 0; kmem_cache_node->node[node] = n; - init_kmem_cache_node(n); + BUG_ON(init_kmem_cache_node(n) < 0); inc_slabs_node(kmem_cache_node, node, page->objects); /* @@ -3592,6 +3636,7 @@ static void early_kmem_cache_node_alloc(int node) * initialized and there is no concurrent access. */ __add_partial(n, page, DEACTIVATE_TO_HEAD); + __update_partial_free(n, page->objects - page->inuse); } static void free_kmem_cache_nodes(struct kmem_cache *s) @@ -3601,6 +3646,9 @@ static void free_kmem_cache_nodes(struct kmem_cache *s) for_each_kmem_cache_node(s, node, n) { s->node[node] = NULL; +#ifdef CONFIG_SLUB_DEBUG_PARTIAL + free_percpu(n->partial_free_objs); +#endif kmem_cache_free(kmem_cache_node, n); } } @@ -3631,7 +3679,11 @@ static int init_kmem_cache_nodes(struct kmem_cache *s) return 0; } - init_kmem_cache_node(n); + if (init_kmem_cache_node(n) < 0) { + free_kmem_cache_nodes(s); + return 0; + } + s->node[node] = n; } return 1; @@ -3922,6 +3974,7 @@ static void free_partial(struct kmem_cache *s, struct kmem_cache_node *n) list_for_each_entry_safe(page, h, &n->partial, slab_list) { if (!page->inuse) { remove_partial(n, page); + __update_partial_free(n, -page->objects); list_add(&page->slab_list, &discard); } else { list_slab_objects(s, page, @@ -4263,6 +4316,8 @@ int __kmem_cache_shrink(struct kmem_cache *s) if (free == page->objects) { list_move(&page->slab_list, &discard); n->nr_partial--; + __update_partial_free(n, -free); + __update_partial_total(n, -free); } else if (free <= SHRINK_PROMOTE_MAX) list_move(&page->slab_list, promote + free - 1); } From patchwork Wed Mar 17 07:54:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 12145063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3BF2C433DB for ; Wed, 17 Mar 2021 07:55:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3D20664F8F for ; Wed, 17 Mar 2021 07:55:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D20664F8F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C4C0B6B0070; Wed, 17 Mar 2021 03:54:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C22756B0071; Wed, 17 Mar 2021 03:54:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B12076B0073; Wed, 17 Mar 2021 03:54:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 9A8906B0070 for ; Wed, 17 Mar 2021 03:54:59 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 56DFA52D5 for ; Wed, 17 Mar 2021 07:54:59 +0000 (UTC) X-FDA: 77928605118.05.6FEC126 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf03.hostedemail.com (Postfix) with ESMTP id 18841C0007C8 for ; Wed, 17 Mar 2021 07:54:57 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0USESt84_1615967693; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0USESt84_1615967693) by smtp.aliyun-inc.com(127.0.0.1); Wed, 17 Mar 2021 15:54:53 +0800 From: Xunlei Pang To: Christoph Lameter , Christoph Lameter , Pekka Enberg , Vlastimil Babka , Roman Gushchin , Konstantin Khlebnikov , David Rientjes , Matthew Wilcox , Shu Ming , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Wen Yang , James Wang , Xunlei Pang Subject: [PATCH v4 2/3] percpu: Export per_cpu_sum() Date: Wed, 17 Mar 2021 15:54:51 +0800 Message-Id: <1615967692-80524-3-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> References: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> X-Stat-Signature: d8rf9bqfxy6zcxc187h8emkxwwikjfq8 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 18841C0007C8 Received-SPF: none (linux.alibaba.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=out30-54.freemail.mail.aliyun.com; client-ip=115.124.30.54 X-HE-DKIM-Result: none/none X-HE-Tag: 1615967697-999725 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: per_cpu_sum() is useful, and deserves to be exported. Tested-by: James Wang Signed-off-by: Xunlei Pang --- include/linux/percpu-defs.h | 10 ++++++++++ kernel/locking/percpu-rwsem.c | 10 ---------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index dff7040..0e71b68 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -220,6 +220,16 @@ (void)__vpp_verify; \ } while (0) +#define per_cpu_sum(var) \ +({ \ + typeof(var) __sum = 0; \ + int cpu; \ + compiletime_assert_atomic_type(__sum); \ + for_each_possible_cpu(cpu) \ + __sum += per_cpu(var, cpu); \ + __sum; \ +}) + #ifdef CONFIG_SMP /* diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index 70a32a5..0980e51 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -178,16 +178,6 @@ bool __percpu_down_read(struct percpu_rw_semaphore *sem, bool try) } EXPORT_SYMBOL_GPL(__percpu_down_read); -#define per_cpu_sum(var) \ -({ \ - typeof(var) __sum = 0; \ - int cpu; \ - compiletime_assert_atomic_type(__sum); \ - for_each_possible_cpu(cpu) \ - __sum += per_cpu(var, cpu); \ - __sum; \ -}) - /* * Return true if the modular sum of the sem->read_count per-CPU variable is * zero. If this sum is zero, then it is stable due to the fact that if any From patchwork Wed Mar 17 07:54:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 12145069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB2D2C433DB for ; Wed, 17 Mar 2021 07:55:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 34BC264F99 for ; Wed, 17 Mar 2021 07:55:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34BC264F99 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BB0EB6B0074; Wed, 17 Mar 2021 03:55:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B85D96B0075; Wed, 17 Mar 2021 03:55:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4F266B0078; Wed, 17 Mar 2021 03:55:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 8D5786B0074 for ; Wed, 17 Mar 2021 03:55:31 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 568CA62C0 for ; Wed, 17 Mar 2021 07:55:31 +0000 (UTC) X-FDA: 77928606462.16.2D96FB0 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf05.hostedemail.com (Postfix) with ESMTP id 9968AE0001B2 for ; Wed, 17 Mar 2021 07:55:29 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=15;SR=0;TI=SMTPD_---0USEIes2_1615967694; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0USEIes2_1615967694) by smtp.aliyun-inc.com(127.0.0.1); Wed, 17 Mar 2021 15:54:54 +0800 From: Xunlei Pang To: Christoph Lameter , Christoph Lameter , Pekka Enberg , Vlastimil Babka , Roman Gushchin , Konstantin Khlebnikov , David Rientjes , Matthew Wilcox , Shu Ming , Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Wen Yang , James Wang , Xunlei Pang Subject: [PATCH v4 3/3] mm/slub: Get rid of count_partial() Date: Wed, 17 Mar 2021 15:54:52 +0800 Message-Id: <1615967692-80524-4-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> References: <1615967692-80524-1-git-send-email-xlpang@linux.alibaba.com> X-Stat-Signature: ubhei7uui1xps1hb38ak47t4oc5khaym X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9968AE0001B2 Received-SPF: none (linux.alibaba.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=out30-130.freemail.mail.aliyun.com; client-ip=115.124.30.130 X-HE-DKIM-Result: none/none X-HE-Tag: 1615967729-693783 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the partial counters are ready, let's use them to get rid of count_partial(). The partial counters will involve in to calculate the accurate partial usage when CONFIG_SLUB_DEBUG_PARTIAL is on, otherwise simply assume their zero usage statistics. Tested-by: James Wang Signed-off-by: Xunlei Pang --- mm/slub.c | 64 +++++++++++++++++++++++++++++++-------------------------------- 1 file changed, 31 insertions(+), 33 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 856aea4..9bff669 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2533,11 +2533,6 @@ static inline int node_match(struct page *page, int node) } #ifdef CONFIG_SLUB_DEBUG -static int count_free(struct page *page) -{ - return page->objects - page->inuse; -} - static inline unsigned long node_nr_objs(struct kmem_cache_node *n) { return atomic_long_read(&n->total_objects); @@ -2545,18 +2540,33 @@ static inline unsigned long node_nr_objs(struct kmem_cache_node *n) #endif /* CONFIG_SLUB_DEBUG */ #if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS) -static unsigned long count_partial(struct kmem_cache_node *n, - int (*get_count)(struct page *)) +enum partial_item { PARTIAL_FREE, PARTIAL_INUSE, PARTIAL_TOTAL, PARTIAL_SLAB }; + +static unsigned long partial_counter(struct kmem_cache_node *n, + enum partial_item item) { - unsigned long flags; - unsigned long x = 0; - struct page *page; + unsigned long ret = 0; - spin_lock_irqsave(&n->list_lock, flags); - list_for_each_entry(page, &n->partial, slab_list) - x += get_count(page); - spin_unlock_irqrestore(&n->list_lock, flags); - return x; +#ifdef CONFIG_SLUB_DEBUG_PARTIAL + if (item == PARTIAL_FREE) { + ret = per_cpu_sum(*n->partial_free_objs); + if ((long)ret < 0) + ret = 0; + } else if (item == PARTIAL_TOTAL) { + ret = n->partial_total_objs; + } else if (item == PARTIAL_INUSE) { + ret = per_cpu_sum(*n->partial_free_objs); + if ((long)ret < 0) + ret = 0; + ret = n->partial_total_objs - ret; + if ((long)ret < 0) + ret = 0; + } else { /* item == PARTIAL_SLAB */ + ret = n->nr_partial; + } +#endif + + return ret; } #endif /* CONFIG_SLUB_DEBUG || CONFIG_SYSFS */ @@ -2587,7 +2597,7 @@ static unsigned long count_partial(struct kmem_cache_node *n, unsigned long nr_objs; unsigned long nr_free; - nr_free = count_partial(n, count_free); + nr_free = partial_counter(n, PARTIAL_FREE); nr_slabs = node_nr_slabs(n); nr_objs = node_nr_objs(n); @@ -4654,18 +4664,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, EXPORT_SYMBOL(__kmalloc_node_track_caller); #endif -#ifdef CONFIG_SYSFS -static int count_inuse(struct page *page) -{ - return page->inuse; -} - -static int count_total(struct page *page) -{ - return page->objects; -} -#endif - #ifdef CONFIG_SLUB_DEBUG static void validate_slab(struct kmem_cache *s, struct page *page) { @@ -5102,7 +5100,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s, x = atomic_long_read(&n->total_objects); else if (flags & SO_OBJECTS) x = atomic_long_read(&n->total_objects) - - count_partial(n, count_free); + partial_counter(n, PARTIAL_FREE); else x = atomic_long_read(&n->nr_slabs); total += x; @@ -5116,11 +5114,11 @@ static ssize_t show_slab_objects(struct kmem_cache *s, for_each_kmem_cache_node(s, node, n) { if (flags & SO_TOTAL) - x = count_partial(n, count_total); + x = partial_counter(n, PARTIAL_TOTAL); else if (flags & SO_OBJECTS) - x = count_partial(n, count_inuse); + x = partial_counter(n, PARTIAL_INUSE); else - x = n->nr_partial; + x = partial_counter(n, PARTIAL_SLAB); total += x; nodes[node] += x; } @@ -5884,7 +5882,7 @@ void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo) for_each_kmem_cache_node(s, node, n) { nr_slabs += node_nr_slabs(n); nr_objs += node_nr_objs(n); - nr_free += count_partial(n, count_free); + nr_free += partial_counter(n, PARTIAL_FREE); } sinfo->active_objs = nr_objs - nr_free;