From patchwork Mon Aug 10 12:17:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 11707349 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05088913 for ; Mon, 10 Aug 2020 12:18:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D040620709 for ; Mon, 10 Aug 2020 12:18:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D040620709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C12FA6B0007; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BC2006B0008; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADAA08D0001; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 99E9C6B0007 for ; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 602058248047 for ; Mon, 10 Aug 2020 12:18:01 +0000 (UTC) X-FDA: 77134560762.18.pear63_4a0e46b26fda Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 40E47100EC69B for ; Mon, 10 Aug 2020 12:17:59 +0000 (UTC) X-Spam-Summary: 1,0,0,d97f49c6078727b0,d41d8cd98f00b204,xlpang@linux.alibaba.com,,RULES_HIT:2:41:355:379:541:800:960:966:968:973:988:989:1260:1261:1345:1359:1437:1535:1605:1606:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2693:2740:2900:2901:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:4118:4250:4321:4384:4385:4395:5007:6119:6261:7875:7903:8603:9010:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:13161:13229:13869:14096:14394:14877:14915:21080:21433:21451:21627:21740:21990:30005:30034:30054,0,RBL:115.124.30.57:@linux.alibaba.com:.lbl8.mailshell.net-64.201.201.201 62.20.2.100;04y8n8hjzn77o5p1sr6bdpe687nudopws6togs1tf17xje18wq3k7amdtinqq7p.iuhy3z9tkcjmz39u96igfaodru4tf1dfzs5wji81a8dut5yo54qr4zfw13zukwq.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: pear63_4a0e46b26fda X-Filterd-Recvd-Size: 7223 Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Mon, 10 Aug 2020 12:17:56 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04397;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0U5MlrY9_1597061872; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0U5MlrY9_1597061872) by smtp.aliyun-inc.com(127.0.0.1); Mon, 10 Aug 2020 20:17:53 +0800 From: Xunlei Pang To: Vlastimil Babka , Christoph Lameter , Wen Yang , Roman Gushchin , Pekka Enberg , Konstantin Khlebnikov , David Rientjes , Xunlei Pang Cc: linux-kernel@vger.kernel.org, "linux-mm@kvack.org" Subject: [PATCH v2 1/3] mm/slub: Introduce two counters for partial objects Date: Mon, 10 Aug 2020 20:17:50 +0800 Message-Id: <1597061872-58724-2-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> References: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> X-Rspamd-Queue-Id: 40E47100EC69B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The node list_lock in count_partial() spends long time iterating in case of large amount of partial page lists, which can cause thunder herd effect to the list_lock contention. We have HSF RT(High-speed Service Framework Response-Time) monitors, the RT figures fluctuated randomly, then we deployed a tool detecting "irq off" and "preempt off" to dump the culprit's calltrace, capturing the list_lock cost nearly 100ms with irq off issued by "ss", this also caused network timeouts. This patch introduces two counters to maintain the actual number of partial objects dynamically instead of iterating the partial page lists with list_lock held. New counters of kmem_cache_node: partial_free_objs, partial_total_objs. The main operations are under list_lock in slow path, its performance impact should be minimal except the __slab_free() path which will be addressed later. Acked-by: Pekka Enberg Co-developed-by: Wen Yang Signed-off-by: Xunlei Pang --- mm/slab.h | 2 ++ mm/slub.c | 37 ++++++++++++++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/mm/slab.h b/mm/slab.h index 7e94700..c85e2fa 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -616,6 +616,8 @@ struct kmem_cache_node { #ifdef CONFIG_SLUB unsigned long nr_partial; struct list_head partial; + atomic_long_t partial_free_objs; + atomic_long_t partial_total_objs; #ifdef CONFIG_SLUB_DEBUG atomic_long_t nr_slabs; atomic_long_t total_objects; diff --git a/mm/slub.c b/mm/slub.c index 6589b41..6708d96 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1775,10 +1775,24 @@ static void discard_slab(struct kmem_cache *s, struct page *page) /* * Management of partially allocated slabs. */ + +static inline void +__update_partial_free(struct kmem_cache_node *n, long delta) +{ + atomic_long_add(delta, &n->partial_free_objs); +} + +static inline void +__update_partial_total(struct kmem_cache_node *n, long delta) +{ + atomic_long_add(delta, &n->partial_total_objs); +} + static inline void __add_partial(struct kmem_cache_node *n, struct page *page, int tail) { n->nr_partial++; + __update_partial_total(n, page->objects); if (tail == DEACTIVATE_TO_TAIL) list_add_tail(&page->slab_list, &n->partial); else @@ -1798,6 +1812,7 @@ static inline void remove_partial(struct kmem_cache_node *n, lockdep_assert_held(&n->list_lock); list_del(&page->slab_list); n->nr_partial--; + __update_partial_total(n, -page->objects); } /* @@ -1842,6 +1857,7 @@ static inline void *acquire_slab(struct kmem_cache *s, return NULL; remove_partial(n, page); + __update_partial_free(n, -*objects); WARN_ON(!freelist); return freelist; } @@ -2174,8 +2190,11 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, "unfreezing slab")) goto redo; - if (lock) + if (lock) { + if (m == M_PARTIAL) + __update_partial_free(n, page->objects - page->inuse); spin_unlock(&n->list_lock); + } if (m == M_PARTIAL) stat(s, tail); @@ -2241,6 +2260,7 @@ static void unfreeze_partials(struct kmem_cache *s, discard_page = page; } else { add_partial(n, page, DEACTIVATE_TO_TAIL); + __update_partial_free(n, page->objects - page->inuse); stat(s, FREE_ADD_PARTIAL); } } @@ -2915,6 +2935,13 @@ static void __slab_free(struct kmem_cache *s, struct page *page, head, new.counters, "__slab_free")); + if (!was_frozen && prior) { + if (n) + __update_partial_free(n, cnt); + else + __update_partial_free(get_node(s, page_to_nid(page)), cnt); + } + if (likely(!n)) { /* @@ -2944,6 +2971,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, if (!kmem_cache_has_cpu_partial(s) && unlikely(!prior)) { remove_full(s, n, page); add_partial(n, page, DEACTIVATE_TO_TAIL); + __update_partial_free(n, page->objects - page->inuse); stat(s, FREE_ADD_PARTIAL); } spin_unlock_irqrestore(&n->list_lock, flags); @@ -2955,6 +2983,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, * Slab on the partial list. */ remove_partial(n, page); + __update_partial_free(n, page->inuse - page->objects); stat(s, FREE_REMOVE_PARTIAL); } else { /* Slab must be on the full list */ @@ -3364,6 +3393,8 @@ static inline int calculate_order(unsigned int size) n->nr_partial = 0; spin_lock_init(&n->list_lock); INIT_LIST_HEAD(&n->partial); + atomic_long_set(&n->partial_free_objs, 0); + atomic_long_set(&n->partial_total_objs, 0); #ifdef CONFIG_SLUB_DEBUG atomic_long_set(&n->nr_slabs, 0); atomic_long_set(&n->total_objects, 0); @@ -3437,6 +3468,7 @@ static void early_kmem_cache_node_alloc(int node) * initialized and there is no concurrent access. */ __add_partial(n, page, DEACTIVATE_TO_HEAD); + __update_partial_free(n, page->objects - page->inuse); } static void free_kmem_cache_nodes(struct kmem_cache *s) @@ -3747,6 +3779,7 @@ static void free_partial(struct kmem_cache *s, struct kmem_cache_node *n) list_for_each_entry_safe(page, h, &n->partial, slab_list) { if (!page->inuse) { remove_partial(n, page); + __update_partial_free(n, page->objects - page->inuse); list_add(&page->slab_list, &discard); } else { list_slab_objects(s, page, @@ -4045,6 +4078,8 @@ int __kmem_cache_shrink(struct kmem_cache *s) if (free == page->objects) { list_move(&page->slab_list, &discard); n->nr_partial--; + __update_partial_free(n, -free); + __update_partial_total(n, -free); } else if (free <= SHRINK_PROMOTE_MAX) list_move(&page->slab_list, promote + free - 1); } From patchwork Mon Aug 10 12:17:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 11707353 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81494109A for ; Mon, 10 Aug 2020 12:18:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 573A52070B for ; Mon, 10 Aug 2020 12:18:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 573A52070B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 92C646B0003; Mon, 10 Aug 2020 08:18:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8DC0F6B0005; Mon, 10 Aug 2020 08:18:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F2806B000A; Mon, 10 Aug 2020 08:18:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id 6A1656B0003 for ; Mon, 10 Aug 2020 08:18:43 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 26D91180AD806 for ; Mon, 10 Aug 2020 12:18:43 +0000 (UTC) X-FDA: 77134562526.01.wren64_3604c6226fda Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id DCD701004FEA0 for ; Mon, 10 Aug 2020 12:18:42 +0000 (UTC) X-Spam-Summary: 1,0,0,2a6a35ebd9526eb7,d41d8cd98f00b204,xlpang@linux.alibaba.com,,RULES_HIT:41:69:355:379:541:960:966:973:988:989:1260:1261:1345:1359:1437:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3868:4321:4385:5007:6261:7875:7904:8603:9592:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:14096:14110:14181:14394:14721:14915:21080:21451:21627:21990:30054,0,RBL:115.124.30.133:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201;04ygwubyyyy6dwcwq1kndne53tsr3ycfee35xwz8j9wbh3whobedhiwpua347we.57qbynjnkqgdchx1woenprwcdizihb5mb4zogkud5nx54buutnw5jswitdo5ddo.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: wren64_3604c6226fda X-Filterd-Recvd-Size: 5191 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Mon, 10 Aug 2020 12:18:41 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04407;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0U5MbCCX_1597061873; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0U5MbCCX_1597061873) by smtp.aliyun-inc.com(127.0.0.1); Mon, 10 Aug 2020 20:17:53 +0800 From: Xunlei Pang To: Vlastimil Babka , Christoph Lameter , Wen Yang , Roman Gushchin , Pekka Enberg , Konstantin Khlebnikov , David Rientjes , Xunlei Pang Cc: linux-kernel@vger.kernel.org, "linux-mm@kvack.org" Subject: [PATCH v2 2/3] mm/slub: Get rid of count_partial() Date: Mon, 10 Aug 2020 20:17:51 +0800 Message-Id: <1597061872-58724-3-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> References: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> X-Rspamd-Queue-Id: DCD701004FEA0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000041, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the partial counters are ready, let's use them directly and get rid of count_partial(). Co-developed-by: Wen Yang Signed-off-by: Xunlei Pang --- mm/slub.c | 57 ++++++++++++++++++++++++--------------------------------- 1 file changed, 24 insertions(+), 33 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6708d96..25a4421 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2414,11 +2414,6 @@ static inline int node_match(struct page *page, int node) } #ifdef CONFIG_SLUB_DEBUG -static int count_free(struct page *page) -{ - return page->objects - page->inuse; -} - static inline unsigned long node_nr_objs(struct kmem_cache_node *n) { return atomic_long_read(&n->total_objects); @@ -2426,19 +2421,27 @@ static inline unsigned long node_nr_objs(struct kmem_cache_node *n) #endif /* CONFIG_SLUB_DEBUG */ #if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS) -static unsigned long count_partial(struct kmem_cache_node *n, - int (*get_count)(struct page *)) -{ - unsigned long flags; - unsigned long x = 0; - struct page *page; +enum partial_item { PARTIAL_FREE, PARTIAL_INUSE, PARTIAL_TOTAL }; + +static unsigned long partial_counter(struct kmem_cache_node *n, + enum partial_item item) +{ + unsigned long ret = 0; + + if (item == PARTIAL_FREE) { + ret = atomic_long_read(&n->partial_free_objs); + } else if (item == PARTIAL_TOTAL) { + ret = atomic_long_read(&n->partial_total_objs); + } else if (item == PARTIAL_INUSE) { + ret = atomic_long_read(&n->partial_total_objs) - + atomic_long_read(&n->partial_free_objs); + if ((long)ret < 0) + ret = 0; + } - spin_lock_irqsave(&n->list_lock, flags); - list_for_each_entry(page, &n->partial, slab_list) - x += get_count(page); - spin_unlock_irqrestore(&n->list_lock, flags); - return x; + return ret; } + #endif /* CONFIG_SLUB_DEBUG || CONFIG_SYSFS */ static noinline void @@ -2468,7 +2471,7 @@ static unsigned long count_partial(struct kmem_cache_node *n, unsigned long nr_objs; unsigned long nr_free; - nr_free = count_partial(n, count_free); + nr_free = partial_counter(n, PARTIAL_FREE); nr_slabs = node_nr_slabs(n); nr_objs = node_nr_objs(n); @@ -4444,18 +4447,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags, } #endif -#ifdef CONFIG_SYSFS -static int count_inuse(struct page *page) -{ - return page->inuse; -} - -static int count_total(struct page *page) -{ - return page->objects; -} -#endif - #ifdef CONFIG_SLUB_DEBUG static void validate_slab(struct kmem_cache *s, struct page *page) { @@ -4912,7 +4903,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s, x = atomic_long_read(&n->total_objects); else if (flags & SO_OBJECTS) x = atomic_long_read(&n->total_objects) - - count_partial(n, count_free); + partial_counter(n, PARTIAL_FREE); else x = atomic_long_read(&n->nr_slabs); total += x; @@ -4926,9 +4917,9 @@ static ssize_t show_slab_objects(struct kmem_cache *s, for_each_kmem_cache_node(s, node, n) { if (flags & SO_TOTAL) - x = count_partial(n, count_total); + x = partial_counter(n, PARTIAL_TOTAL); else if (flags & SO_OBJECTS) - x = count_partial(n, count_inuse); + x = partial_counter(n, PARTIAL_INUSE); else x = n->nr_partial; total += x; @@ -5961,7 +5952,7 @@ void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo) for_each_kmem_cache_node(s, node, n) { nr_slabs += node_nr_slabs(n); nr_objs += node_nr_objs(n); - nr_free += count_partial(n, count_free); + nr_free += partial_counter(n, PARTIAL_FREE); } sinfo->active_objs = nr_objs - nr_free; From patchwork Mon Aug 10 12:17:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xunlei Pang X-Patchwork-Id: 11707347 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9054B913 for ; Mon, 10 Aug 2020 12:18:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B5E62070B for ; Mon, 10 Aug 2020 12:18:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B5E62070B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4975B6B0006; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 448D96B0007; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3373D6B0008; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id 1D0D16B0006 for ; Mon, 10 Aug 2020 08:18:01 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C423E181AEF00 for ; Mon, 10 Aug 2020 12:18:00 +0000 (UTC) X-FDA: 77134560720.07.sense75_5e0e8fd26fda Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 96F7D1803F9B8 for ; Mon, 10 Aug 2020 12:18:00 +0000 (UTC) X-Spam-Summary: 1,0,0,85605366811aaf95,d41d8cd98f00b204,xlpang@linux.alibaba.com,,RULES_HIT:41:355:379:541:960:965:966:973:988:989:1260:1261:1345:1359:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2553:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:3874:4250:4321:4384:4385:4390:4395:5007:6261:7903:8603:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:13161:13229:14110:14181:14394:14721:14915:21080:21451:21627:21990:30054:30090,0,RBL:115.124.30.57:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201;04ygxmhuoxnfejib8izjztret67qnoc6g1z5dthg7sjz8bbdpdhzfjcr3azh63g.b7nb6w1yoc55exz64uw3aezyjshw89t7yoyjdyizntdwcx9t9hsndyj8a75cf6m.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: sense75_5e0e8fd26fda X-Filterd-Recvd-Size: 5127 Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Mon, 10 Aug 2020 12:17:58 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07484;MF=xlpang@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0U5N4-L-_1597061873; Received: from localhost(mailfrom:xlpang@linux.alibaba.com fp:SMTPD_---0U5N4-L-_1597061873) by smtp.aliyun-inc.com(127.0.0.1); Mon, 10 Aug 2020 20:17:54 +0800 From: Xunlei Pang To: Vlastimil Babka , Christoph Lameter , Wen Yang , Roman Gushchin , Pekka Enberg , Konstantin Khlebnikov , David Rientjes , Xunlei Pang Cc: linux-kernel@vger.kernel.org, "linux-mm@kvack.org" Subject: [PATCH v2 3/3] mm/slub: Use percpu partial free counter Date: Mon, 10 Aug 2020 20:17:52 +0800 Message-Id: <1597061872-58724-4-git-send-email-xlpang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> References: <1597061872-58724-1-git-send-email-xlpang@linux.alibaba.com> X-Rspamd-Queue-Id: 96F7D1803F9B8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only concern of introducing partial counter is that, partial_free_objs may cause atomic operation contention in case of same SLUB concurrent __slab_free(). This patch changes it to be a percpu counter to avoid that. Co-developed-by: Wen Yang Signed-off-by: Xunlei Pang --- mm/slab.h | 2 +- mm/slub.c | 38 +++++++++++++++++++++++++++++++------- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index c85e2fa..a709a70 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -616,7 +616,7 @@ struct kmem_cache_node { #ifdef CONFIG_SLUB unsigned long nr_partial; struct list_head partial; - atomic_long_t partial_free_objs; + atomic_long_t __percpu *partial_free_objs; atomic_long_t partial_total_objs; #ifdef CONFIG_SLUB_DEBUG atomic_long_t nr_slabs; diff --git a/mm/slub.c b/mm/slub.c index 25a4421..f6fc60b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1775,11 +1775,21 @@ static void discard_slab(struct kmem_cache *s, struct page *page) /* * Management of partially allocated slabs. */ +static inline long get_partial_free(struct kmem_cache_node *n) +{ + long nr = 0; + int cpu; + + for_each_possible_cpu(cpu) + nr += atomic_long_read(per_cpu_ptr(n->partial_free_objs, cpu)); + + return nr; +} static inline void __update_partial_free(struct kmem_cache_node *n, long delta) { - atomic_long_add(delta, &n->partial_free_objs); + atomic_long_add(delta, this_cpu_ptr(n->partial_free_objs)); } static inline void @@ -2429,12 +2439,12 @@ static unsigned long partial_counter(struct kmem_cache_node *n, unsigned long ret = 0; if (item == PARTIAL_FREE) { - ret = atomic_long_read(&n->partial_free_objs); + ret = get_partial_free(n); } else if (item == PARTIAL_TOTAL) { ret = atomic_long_read(&n->partial_total_objs); } else if (item == PARTIAL_INUSE) { ret = atomic_long_read(&n->partial_total_objs) - - atomic_long_read(&n->partial_free_objs); + get_partial_free(n); if ((long)ret < 0) ret = 0; } @@ -3390,19 +3400,28 @@ static inline int calculate_order(unsigned int size) return -ENOSYS; } -static void +static int init_kmem_cache_node(struct kmem_cache_node *n) { + int cpu; + n->nr_partial = 0; spin_lock_init(&n->list_lock); INIT_LIST_HEAD(&n->partial); - atomic_long_set(&n->partial_free_objs, 0); + + n->partial_free_objs = alloc_percpu(atomic_long_t); + if (!n->partial_free_objs) + return -ENOMEM; + for_each_possible_cpu(cpu) + atomic_long_set(per_cpu_ptr(n->partial_free_objs, cpu), 0); atomic_long_set(&n->partial_total_objs, 0); #ifdef CONFIG_SLUB_DEBUG atomic_long_set(&n->nr_slabs, 0); atomic_long_set(&n->total_objects, 0); INIT_LIST_HEAD(&n->full); #endif + + return 0; } static inline int alloc_kmem_cache_cpus(struct kmem_cache *s) @@ -3463,7 +3482,7 @@ static void early_kmem_cache_node_alloc(int node) page->inuse = 1; page->frozen = 0; kmem_cache_node->node[node] = n; - init_kmem_cache_node(n); + BUG_ON(init_kmem_cache_node(n) < 0); inc_slabs_node(kmem_cache_node, node, page->objects); /* @@ -3481,6 +3500,7 @@ static void free_kmem_cache_nodes(struct kmem_cache *s) for_each_kmem_cache_node(s, node, n) { s->node[node] = NULL; + free_percpu(n->partial_free_objs); kmem_cache_free(kmem_cache_node, n); } } @@ -3511,7 +3531,11 @@ static int init_kmem_cache_nodes(struct kmem_cache *s) return 0; } - init_kmem_cache_node(n); + if (init_kmem_cache_node(n) < 0) { + free_kmem_cache_nodes(s); + return 0; + } + s->node[node] = n; } return 1;