[169/178] mm/page_alloc: optimize code layout for __alloc_pages_bulk

Commit Message

Andrew Morton April 30, 2021, 6:01 a.m. UTC

From: Jesper Dangaard Brouer <[e-mail]>
Subject: mm/page_alloc: optimize code layout for __alloc_pages_bulk

Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear
that the code activated is suboptimal.  The compiler guesses wrong and
places unlikely code at the beginning.  Due to the use of WARN_ON_ONCE()
macro the UD2 asm instruction is added to the code, which confuse the
I-cache prefetcher in the CPU.

[[e-mail]: minor changes and rebasing]
Link: https://lkml.kernel.org/r/[e-mail]
Signed-off-by: Jesper Dangaard Brouer <[e-mail]>
Signed-off-by: Mel Gorman <[e-mail]>
Reviewed-by: Alexander Lobakin <[e-mail]>
Acked-By: Vlastimil Babka <[e-mail]>
Cc: Alexander Duyck <[e-mail]>
Cc: Christoph Hellwig <[e-mail]>
Cc: Chuck Lever <[e-mail]>
Cc: David Miller <[e-mail]>
Cc: Ilias Apalodimas <[e-mail]>
Cc: Matthew Wilcox (Oracle) <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 mm/page_alloc.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Patch

--- a/mm/page_alloc.c~mm-page_alloc-optimize-code-layout-for-__alloc_pages_bulk
+++ a/mm/page_alloc.c
@@ -5042,7 +5042,7 @@  unsigned long __alloc_pages_bulk(gfp_t g
 	unsigned int alloc_flags = ALLOC_WMARK_LOW;
 	int nr_populated = 0;
 
-	if (WARN_ON_ONCE(nr_pages <= 0))
+	if (unlikely(nr_pages <= 0))
 		return 0;
 
 	/*
@@ -5089,7 +5089,7 @@  unsigned long __alloc_pages_bulk(gfp_t g
 	 * If there are no allowed local zones that meets the watermarks then
 	 * try to allocate a single page and reclaim if necessary.
 	 */
-	if (!zone)
+	if (unlikely(!zone))
 		goto failed;
 
 	/* Attempt the batch allocation */
@@ -5107,7 +5107,7 @@  unsigned long __alloc_pages_bulk(gfp_t g
 
 		page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags,
 								pcp, pcp_list);
-		if (!page) {
+		if (unlikely(!page)) {
 			/* Try and get at least one page */
 			if (!nr_populated)
 				goto failed_irq;