[RFC,04/26] mm, slub: simplify kmem_cache_cpu and tid setup

Commit Message

Vlastimil Babka May 24, 2021, 11:39 p.m. UTC

In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
that's done by reading the tid first using this_cpu_read(), then the
kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then reading
tid using the pointer. That guarantees they are from the same cpu. We don't
need to read the tid using this_cpu_read() because the value will be validated
by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
freelist didn't change by anyone preempting us since reading the tid.

Signed-off-by: Vlastimil Babka <[e-mail]>
---
 mm/slub.c | 22 +++++++++-------------
 1 file changed, 9 insertions(+), 13 deletions(-)

Comments

Mel Gorman May 25, 2021, 11:47 a.m. UTC | #1

On Tue, May 25, 2021 at 01:39:24AM +0200, Vlastimil Babka wrote:
> In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
> our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
> that's done by reading the tid first using this_cpu_read(), then the
> kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
> plain READ_ONCE().
> 
> This can be simplified to just fetching kmem_cache_cpu pointer and then reading
> tid using the pointer. That guarantees they are from the same cpu. We don't
> need to read the tid using this_cpu_read() because the value will be validated
> by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
> freelist didn't change by anyone preempting us since reading the tid.
> 
> Signed-off-by: Vlastimil Babka <[e-mail]>

Wow, that's a fun approach to avoiding disabling preemption but the
validation check against preemption remains the same so;

Acked-by: Mel Gorman <[e-mail]>

Patch

diff --git a/mm/slub.c b/mm/slub.c
index 83ad64c1d9da..7b4cdc59b9ff 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2840,15 +2840,14 @@  static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 	 * reading from one cpu area. That does not matter as long
 	 * as we end up on the original cpu again when doing the cmpxchg.
 	 *
-	 * We should guarantee that tid and kmem_cache are retrieved on
-	 * the same cpu. It could be different if CONFIG_PREEMPTION so we need
-	 * to check if it is matched or not.
+	 * We must guarantee that tid and kmem_cache_cpu are retrieved on the
+	 * same cpu. We read first the kmem_cache_cpu pointer and use it to read
+	 * the tid. If we are preempted and switched to another cpu between the
+	 * two reads, it's OK as the two are still associated with the same cpu
+	 * and cmpxchg later will validate the cpu.
 	 */
-	do {
-		tid = this_cpu_read(s->cpu_slab->tid);
-		c = raw_cpu_ptr(s->cpu_slab);
-	} while (IS_ENABLED(CONFIG_PREEMPTION) &&
-		 unlikely(tid != READ_ONCE(c->tid)));
+	c = raw_cpu_ptr(s->cpu_slab);
+	tid = READ_ONCE(c->tid);
 
 	/*
 	 * Irqless object alloc/free algorithm used here depends on sequence
@@ -3122,11 +3121,8 @@  static __always_inline void do_slab_free(struct kmem_cache *s,
 	 * data is retrieved via this pointer. If we are on the same cpu
 	 * during the cmpxchg then the free will succeed.
 	 */
-	do {
-		tid = this_cpu_read(s->cpu_slab->tid);
-		c = raw_cpu_ptr(s->cpu_slab);
-	} while (IS_ENABLED(CONFIG_PREEMPTION) &&
-		 unlikely(tid != READ_ONCE(c->tid)));
+	c = raw_cpu_ptr(s->cpu_slab);
+	tid = READ_ONCE(c->tid);
 
 	/* Same with comment on barrier() in slab_alloc_node() */
 	barrier();