[RFC,10/26] mm, slub: do initial checks in ___slab_alloc() with irqs enabled

Commit Message

Vlastimil Babka May 24, 2021, 11:39 p.m. UTC

As another step of shortening irq disabled sections in ___slab_alloc(), don't
disable irqs until doing initial checks if there is a cached percpu slab and
it's suitable for our allocation.

Now we have to recheck c->page after actually disabling irqs as an allocation
in irq might have replaced it.

Signed-off-by: Vlastimil Babka <[e-mail]>
---
 mm/slub.c | 34 +++++++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 9 deletions(-)

Comments

Mel Gorman May 25, 2021, 1:04 p.m. UTC | #1

On Tue, May 25, 2021 at 01:39:30AM +0200, Vlastimil Babka wrote:
> As another step of shortening irq disabled sections in ___slab_alloc(), don't
> disable irqs until doing initial checks if there is a cached percpu slab and
> it's suitable for our allocation.
> 
> Now we have to recheck c->page after actually disabling irqs as an allocation
> in irq might have replaced it.
> 
> Signed-off-by: Vlastimil Babka <[e-mail]>

Minor nit only -- consider adding a comment at the new_slab label that
IRQs must be disabled already.

Acked-by: Mel Gorman <[e-mail]>

Vlastimil Babka June 8, 2021, 12:13 p.m. UTC | #2

On 5/25/21 3:04 PM, Mel Gorman wrote:
> On Tue, May 25, 2021 at 01:39:30AM +0200, Vlastimil Babka wrote:
>> As another step of shortening irq disabled sections in ___slab_alloc(), don't
>> disable irqs until doing initial checks if there is a cached percpu slab and
>> it's suitable for our allocation.
>> 
>> Now we have to recheck c->page after actually disabling irqs as an allocation
>> in irq might have replaced it.
>> 
>> Signed-off-by: Vlastimil Babka <[e-mail]>
> 
> Minor nit only -- consider adding a comment at the new_slab label that
> IRQs must be disabled already.

Good point, will use lockdep_assert_irqs_disabled() as that's a functional comment.

> Acked-by: Mel Gorman <[e-mail]>

Thanks.

Patch

diff --git a/mm/slub.c b/mm/slub.c
index c5f4f9282496..e243a991b15b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2643,8 +2643,8 @@  static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 
 	stat(s, ALLOC_SLOWPATH);
 
-	local_irq_save(flags);
-	page = c->page;
+reread_page:
+	page = READ_ONCE(c->page);
 	if (!page) {
 		/*
 		 * if the node is not online or has no normal memory, just
@@ -2653,6 +2653,11 @@  static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 		if (unlikely(node != NUMA_NO_NODE &&
 			     !node_isset(node, slab_nodes)))
 			node = NUMA_NO_NODE;
+		local_irq_save(flags);
+		if (unlikely(c->page)) {
+			local_irq_restore(flags);
+			goto reread_page;
+		}
 		goto new_slab;
 	}
 redo:
@@ -2667,8 +2672,7 @@  static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 			goto redo;
 		} else {
 			stat(s, ALLOC_NODE_MISMATCH);
-			deactivate_slab(s, page, c->freelist, c);
-			goto new_slab;
+			goto deactivate_slab;
 		}
 	}
 
@@ -2677,12 +2681,15 @@  static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	 * PFMEMALLOC but right now, we are losing the pfmemalloc
 	 * information when the page leaves the per-cpu allocator
 	 */
-	if (unlikely(!pfmemalloc_match(page, gfpflags))) {
-		deactivate_slab(s, page, c->freelist, c);
-		goto new_slab;
-	}
+	if (unlikely(!pfmemalloc_match(page, gfpflags)))
+		goto deactivate_slab;
 
-	/* must check again c->freelist in case of cpu migration or IRQ */
+	/* must check again c->page in case of IRQ */
+	local_irq_save(flags);
+	if (unlikely(page != c->page)) {
+		local_irq_restore(flags);
+		goto reread_page;
+	}
 	freelist = c->freelist;
 	if (freelist)
 		goto load_freelist;
@@ -2709,11 +2716,20 @@  static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
 	local_irq_restore(flags);
 	return freelist;
 
+deactivate_slab:
+	local_irq_save(flags);
+	if (page != c->page) {
+		local_irq_restore(flags);
+		goto reread_page;
+	}
+	deactivate_slab(s, page, c->freelist, c);
+
 new_slab:
 
 	if (slub_percpu_partial(c)) {
 		page = c->page = slub_percpu_partial(c);
 		slub_set_percpu_partial(c, page);
+		local_irq_restore(flags);
 		stat(s, CPU_PARTIAL_ALLOC);
 		goto redo;
 	}