[023/147] mm, slub: detach whole partial list at once in unfreeze_partials()

Commit Message

Andrew Morton Sept. 8, 2021, 2:54 a.m. UTC

From: Vlastimil Babka <[e-mail]>
Subject: mm, slub: detach whole partial list at once in unfreeze_partials()

Instead of iterating through the live percpu partial list, detach it from
the kmem_cache_cpu at once.  This is simpler and will allow further
optimization.

Link: https://lkml.kernel.org/r/[e-mail]
Signed-off-by: Vlastimil Babka <[e-mail]>
Cc: Christoph Lameter <[e-mail]>
Cc: David Rientjes <[e-mail]>
Cc: Jann Horn <[e-mail]>
Cc: Jesper Dangaard Brouer <[e-mail]>
Cc: Joonsoo Kim <[e-mail]>
Cc: Mel Gorman <[e-mail]>
Cc: Mike Galbraith <[e-mail]>
Cc: Pekka Enberg <[e-mail]>
Cc: Qian Cai <[e-mail]>
Cc: Sebastian Andrzej Siewior <[e-mail]>
Cc: Thomas Gleixner <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 mm/slub.c |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

Patch

--- a/mm/slub.c~mm-slub-detach-whole-partial-list-at-once-in-unfreeze_partials
+++ a/mm/slub.c
@@ -2358,16 +2358,20 @@  static void unfreeze_partials(struct kme
 {
 #ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct kmem_cache_node *n = NULL, *n2 = NULL;
-	struct page *page, *discard_page = NULL;
+	struct page *page, *partial_page, *discard_page = NULL;
 	unsigned long flags;
 
 	local_irq_save(flags);
 
-	while ((page = slub_percpu_partial(c))) {
+	partial_page = slub_percpu_partial(c);
+	c->partial = NULL;
+
+	while (partial_page) {
 		struct page new;
 		struct page old;
 
-		slub_set_percpu_partial(c, page);
+		page = partial_page;
+		partial_page = page->next;
 
 		n2 = get_node(s, page_to_nid(page));
 		if (n != n2) {