[027/147] mm, slab: split out the cpu offline variant of flush_slab()

Commit Message

Andrew Morton Sept. 8, 2021, 2:54 a.m. UTC

From: Vlastimil Babka <[e-mail]>
Subject: mm, slab: split out the cpu offline variant of flush_slab()

flush_slab() is called either as part IPI handler on given live cpu, or as
a cleanup on behalf of another cpu that went offline.  The first case
needs to protect updating the kmem_cache_cpu fields with disabled irqs. 
Currently the whole call happens with irqs disabled by the IPI handler,
but the following patch will change from IPI to workqueue, and
flush_slab() will have to disable irqs (to be replaced with a local lock
later) in the critical part.

To prepare for this change, replace the call to flush_slab() for the dead
cpu handling with an opencoded variant that will not disable irqs nor take
a local lock.

Link: https://lkml.kernel.org/r/[e-mail]
Suggested-by: Mike Galbraith <[e-mail]>
Signed-off-by: Vlastimil Babka <[e-mail]>
Cc: Christoph Lameter <[e-mail]>
Cc: David Rientjes <[e-mail]>
Cc: Jann Horn <[e-mail]>
Cc: Jesper Dangaard Brouer <[e-mail]>
Cc: Joonsoo Kim <[e-mail]>
Cc: Mel Gorman <[e-mail]>
Cc: Pekka Enberg <[e-mail]>
Cc: Qian Cai <[e-mail]>
Cc: Sebastian Andrzej Siewior <[e-mail]>
Cc: Thomas Gleixner <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 mm/slub.c |   12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

Patch

--- a/mm/slub.c~mm-slab-split-out-the-cpu-offline-variant-of-flush_slab
+++ a/mm/slub.c
@@ -2511,9 +2511,17 @@  static inline void flush_slab(struct kme
 static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu)
 {
 	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
+	void *freelist = c->freelist;
+	struct page *page = c->page;
 
-	if (c->page)
-		flush_slab(s, c);
+	c->page = NULL;
+	c->freelist = NULL;
+	c->tid = next_tid(c->tid);
+
+	if (page) {
+		deactivate_slab(s, page, freelist);
+		stat(s, CPUSLAB_FLUSH);
+	}
 
 	unfreeze_partials_cpu(s, c);
 }