[076/200] arm64: mremap speedup - enable HAVE_MOVE_PUD

Commit Message

Andrew Morton Dec. 15, 2020, 3:07 a.m. UTC

From: Kalesh Singh <[e-mail]>
Subject: arm64: mremap speedup - enable HAVE_MOVE_PUD

HAVE_MOVE_PUD enables remapping pages at the PUD level if both the source
and destination addresses are PUD-aligned.

With HAVE_MOVE_PUD enabled it can be inferred that there is approximately
a 19x improvement in performance on arm64.  (See data below).

------- Test Results ---------

The following results were obtained using a 5.4 kernel, by remapping a
PUD-aligned, 1GB sized region to a PUD-aligned destination.  The results
from 10 iterations of the test are given below:

Total mremap times for 1GB data on arm64. All times are in nanoseconds.

Control          HAVE_MOVE_PUD

1247761          74271
1219896          46771
1094792          59687
1227760          48385
1043698          76666
1101771          50365
1159896          52500
1143594          75261
1025833          61354
1078125          48697

1134312.6        59395.7    <-- Mean time in nanoseconds

A 1GB mremap completion time drops from ~1.1 milliseconds to ~59
microseconds on arm64.  (~19x speed up).

Link: https://lkml.kernel.org/r/[e-mail]
Signed-off-by: Kalesh Singh <[e-mail]>
Acked-by: Kirill A. Shutemov <[e-mail]>
Cc: Catalin Marinas <[e-mail]>
Cc: Will Deacon <[e-mail]>
Cc: Aneesh Kumar K.V <[e-mail]>
Cc: Anshuman Khandual <[e-mail]>
Cc: Arnd Bergmann <[e-mail]>
Cc: Borislav Petkov <[e-mail]>
Cc: Brian Geffon <[e-mail]>
Cc: Christian Brauner <[e-mail]>
Cc: Dave Hansen <[e-mail]>
Cc: Frederic Weisbecker <[e-mail]>
Cc: Gavin Shan <[e-mail]>
Cc: Hassan Naveed <[e-mail]>
Cc: "H. Peter Anvin" <[e-mail]>
Cc: Ingo Molnar <[e-mail]>
Cc: Jia He <[e-mail]>
Cc: John Hubbard <[e-mail]>
Cc: Kees Cook <[e-mail]>
Cc: Krzysztof Kozlowski <[e-mail]>
Cc: Lokesh Gidra <[e-mail]>
Cc: Mark Rutland <[e-mail]>
Cc: Masahiro Yamada <[e-mail]>
Cc: Masami Hiramatsu <[e-mail]>
Cc: Mike Rapoport <[e-mail]>
Cc: Mina Almasry <[e-mail]>
Cc: Minchan Kim <[e-mail]>
Cc: Peter Zijlstra (Intel) <[e-mail]>
Cc: Ralph Campbell <[e-mail]>
Cc: Ram Pai <[e-mail]>
Cc: Sami Tolvanen <[e-mail]>
Cc: Sandipan Das <[e-mail]>
Cc: SeongJae Park <[e-mail]>
Cc: Shuah Khan <[e-mail]>
Cc: Steven Price <[e-mail]>
Cc: Suren Baghdasaryan <[e-mail]>
Cc: Thomas Gleixner <[e-mail]>
Cc: Zi Yan <[e-mail]>
Signed-off-by: Andrew Morton <[e-mail]>
---

 arch/arm64/Kconfig               |    1 +
 arch/arm64/include/asm/pgtable.h |    1 +
 2 files changed, 2 insertions(+)

Patch

--- a/arch/arm64/include/asm/pgtable.h~arm64-mremap-speedup-enable-have_move_pud
+++ a/arch/arm64/include/asm/pgtable.h
@@ -462,6 +462,7 @@  static inline pmd_t pmd_mkdevmap(pmd_t p
 #define pfn_pud(pfn,prot)	__pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
 
 #define set_pmd_at(mm, addr, pmdp, pmd)	set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
+#define set_pud_at(mm, addr, pudp, pud)	set_pte_at(mm, addr, (pte_t *)pudp, pud_pte(pud))
 
 #define __p4d_to_phys(p4d)	__pte_to_phys(p4d_pte(p4d))
 #define __phys_to_p4d_val(phys)	__phys_to_pte_val(phys)
--- a/arch/arm64/Kconfig~arm64-mremap-speedup-enable-have_move_pud
+++ a/arch/arm64/Kconfig
@@ -125,6 +125,7 @@  config ARM64
 	select HANDLE_DOMAIN_IRQ
 	select HARDIRQS_SW_RESEND
 	select HAVE_MOVE_PMD
+	select HAVE_MOVE_PUD
 	select HAVE_PCI
 	select HAVE_ACPI_APEI if (ACPI && EFI)
 	select HAVE_ALIGNED_STRUCT_PAGE if SLUB