| Message ID | 20200706122447.696786-5-linus.walleij@linaro.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | KASan for Arm | expand |
On Mon, Jul 06, 2020 at 02:24:46PM +0200, Linus Walleij wrote: > This patch initializes KASan shadow region's page table and memory. > There are two stage for KASan initializing: > > 1. At early boot stage the whole shadow region is mapped to just > one physical page (kasan_zero_page). It is finished by the function > kasan_early_init which is called by __mmap_switched(arch/arm/kernel/ > head-common.S) > > 2. After the calling of paging_init, we use kasan_zero_page as zero > shadow for some memory that KASan does not need to track, and we > allocate a new shadow space for the other memory that KASan need to > track. These issues are finished by the function kasan_init which is > call by setup_arch. > > When using KASan we also need to increase the THREAD_SIZE_ORDER > from 1 to 2 as the extra calls for shadow memory uses quite a bit > of stack. > > As we need to make a temporary copy of the PGD when setting up > shadow memory we create a helpful PGD_SIZE definition for both > LPAE and non-LPAE setups. > > The KASan core code unconditionally calls pud_populate() so this > needs to be changed from BUG() to do {} while (0) when building > with KASan enabled. > > After the initial development by Andre Ryabinin several modifications > have been made to this code: > > Abbott Liu <liuwenliang@huawei.com> > - Add support ARM LPAE: If LPAE is enabled, KASan shadow region's > mapping table need be copied in the pgd_alloc() function. > - Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate, > kasan_pgd_populate from .meminit.text section to .init.text section. > Reported by Florian Fainelli <f.fainelli@gmail.com> > > Linus Walleij <linus.walleij@linaro.org>: > - Drop the custom mainpulation of TTBR0 and just use > cpu_switch_mm() to switch the pgd table. > - Adopt to handle 4th level page tabel folding. > - Rewrite the entire page directory and page entry initialization > sequence to be recursive based on ARM64:s kasan_init.c. > > Cc: Alexander Potapenko <glider@google.com> > Cc: Dmitry Vyukov <dvyukov@google.com> > Cc: kasan-dev@googlegroups.com > Cc: Mike Rapoport <rppt@linux.ibm.com> > Co-developed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> > Co-developed-by: Abbott Liu <liuwenliang@huawei.com> > Co-developed-by: Ard Biesheuvel <ardb@kernel.org> > Reviewed-by: Ard Biesheuvel <ardb@kernel.org> > Tested-by: Ard Biesheuvel <ardb@kernel.org> # QEMU/KVM/mach-virt/LPAE/8G > Reported-by: Russell King - ARM Linux <linux@armlinux.org.uk> > Reported-by: Florian Fainelli <f.fainelli@gmail.com> > Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> > Signed-off-by: Abbott Liu <liuwenliang@huawei.com> > Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> > Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Looks ok to me, so FWIW Acked-by: Mike Rapoport <rppt@linux.ibm.com> > --- > ChangeLog v11->v12: > - Do not try to shadow highmem memory blocks. (Ard) > - Provoke a build bug if the entire shadow memory doesn't fit > inside a single pgd_index() (Ard) > - Move the pointer to (unsigned long) casts into the create_mapping() > function. (Ard) > - After setting up the shadow memory make sure to issue > local_flush_tlb_all() so that we refresh all the global mappings. (Ard) > - Simplify pte_populate() (Ard) > - Skip over pud population as well as p4d. (Ard) > - Drop the stop condition pmd_none(*pmdp) in the pmd population > loop. (Ard) > - Stop passing around the node (NUMA) parameter in the init code, > we are not expecting any NUMA architectures to be introduced into > ARM32 so just hardcode NUMA_NO_NODE when calling > memblock_alloc_try_nid(). > ChangeLog v10->v11: > - Fix compilation on LPAE systems. > - Move the check for valid pgdp, pudp and pmdp into the loop for > each level moving over the directory pointers: we were just lucky > that we just needed one directory for each level so this fixes > the pmdp issue with LPAE and KASan now works like a charm on > LPAE as well. > - Fold fourth level page directory (p4d) into the global page directory > pgd and just skip into the page upper directory (pud) directly. We > do not anticipate that ARM32 will every use 5-level page tables. > - Simplify the ifdeffery around the temporary pgd. > - Insert a comment about pud_populate() that is unconditionally called > by the KASan core code. > ChangeLog v9->v10: > - Rebase onto v5.8-rc1 > - add support for folded p4d page tables, use the primitives necessary > for the 4th level folding, add (empty) walks of p4d level. > - Use the <linux/pgtable.h> header file that has now appeared as part > of the VM consolidation series. > - Use a recursive method to walk pgd/p4d/pud/pmd/pte instead of the > separate early/main calls and the flat call structure used in the > old code. This was inspired by the ARM64 KASan init code. > - Assume authorship of this code, I have now written the majority of > it so the blame is on me and noone else. > ChangeLog v8->v9: > - Drop the custom CP15 manipulation and cache flushing for swapping > TTBR0 and instead just use cpu_switch_mm(). > - Collect Ard's tags. > ChangeLog v7->v8: > - Rebased. > ChangeLog v6->v7: > - Use SPDX identifer for the license. > - Move the TTBR0 accessor calls into this patch. > --- > arch/arm/include/asm/kasan.h | 32 ++++ > arch/arm/include/asm/pgalloc.h | 8 +- > arch/arm/include/asm/thread_info.h | 8 + > arch/arm/kernel/head-common.S | 3 + > arch/arm/kernel/setup.c | 2 + > arch/arm/mm/Makefile | 3 + > arch/arm/mm/kasan_init.c | 264 +++++++++++++++++++++++++++++ > arch/arm/mm/pgd.c | 16 +- > 8 files changed, 334 insertions(+), 2 deletions(-) > create mode 100644 arch/arm/include/asm/kasan.h > create mode 100644 arch/arm/mm/kasan_init.c > > diff --git a/arch/arm/include/asm/kasan.h b/arch/arm/include/asm/kasan.h > new file mode 100644 > index 000000000000..56b954db160e > --- /dev/null > +++ b/arch/arm/include/asm/kasan.h > @@ -0,0 +1,32 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * arch/arm/include/asm/kasan.h > + * > + * Copyright (c) 2015 Samsung Electronics Co., Ltd. > + * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com> > + * > + */ > + > +#ifndef __ASM_KASAN_H > +#define __ASM_KASAN_H > + > +#ifdef CONFIG_KASAN > + > +#include <asm/kasan_def.h> > + > +#define KASAN_SHADOW_SCALE_SHIFT 3 > + > +/* > + * The compiler uses a shadow offset assuming that addresses start > + * from 0. Kernel addresses don't start from 0, so shadow > + * for kernel really starts from 'compiler's shadow offset' + > + * ('kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT) > + */ > + > +extern void kasan_init(void); > + > +#else > +static inline void kasan_init(void) { } > +#endif > + > +#endif > diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h > index 069da393110c..3bf1905df9c3 100644 > --- a/arch/arm/include/asm/pgalloc.h > +++ b/arch/arm/include/asm/pgalloc.h > @@ -21,6 +21,7 @@ > #define _PAGE_KERNEL_TABLE (PMD_TYPE_TABLE | PMD_BIT4 | PMD_DOMAIN(DOMAIN_KERNEL)) > > #ifdef CONFIG_ARM_LPAE > +#define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) > > static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) > { > @@ -39,14 +40,19 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) > } > > #else /* !CONFIG_ARM_LPAE */ > +#define PGD_SIZE (PAGE_SIZE << 2) > > /* > * Since we have only two-level page tables, these are trivial > */ > #define pmd_alloc_one(mm,addr) ({ BUG(); ((pmd_t *)2); }) > #define pmd_free(mm, pmd) do { } while (0) > +#ifdef CONFIG_KASAN > +/* The KASan core unconditionally calls pud_populate() on all architectures */ > +#define pud_populate(mm,pmd,pte) do { } while (0) > +#else > #define pud_populate(mm,pmd,pte) BUG() > - > +#endif > #endif /* CONFIG_ARM_LPAE */ > > extern pgd_t *pgd_alloc(struct mm_struct *mm); > diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h > index 3609a6980c34..02813a5d9e10 100644 > --- a/arch/arm/include/asm/thread_info.h > +++ b/arch/arm/include/asm/thread_info.h > @@ -13,7 +13,15 @@ > #include <asm/fpstate.h> > #include <asm/page.h> > > +#ifdef CONFIG_KASAN > +/* > + * KASan uses a lot of extra stack space so the thread size order needs to > + * be increased. > + */ > +#define THREAD_SIZE_ORDER 2 > +#else > #define THREAD_SIZE_ORDER 1 > +#endif > #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) > #define THREAD_START_SP (THREAD_SIZE - 8) > > diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S > index 6840c7c60a85..89c80154b9ef 100644 > --- a/arch/arm/kernel/head-common.S > +++ b/arch/arm/kernel/head-common.S > @@ -111,6 +111,9 @@ __mmap_switched: > str r8, [r2] @ Save atags pointer > cmp r3, #0 > strne r10, [r3] @ Save control register values > +#ifdef CONFIG_KASAN > + bl kasan_early_init > +#endif > mov lr, #0 > b start_kernel > ENDPROC(__mmap_switched) > diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c > index d8e18cdd96d3..b0820847bb92 100644 > --- a/arch/arm/kernel/setup.c > +++ b/arch/arm/kernel/setup.c > @@ -58,6 +58,7 @@ > #include <asm/unwind.h> > #include <asm/memblock.h> > #include <asm/virt.h> > +#include <asm/kasan.h> > > #include "atags.h" > > @@ -1130,6 +1131,7 @@ void __init setup_arch(char **cmdline_p) > early_ioremap_reset(); > > paging_init(mdesc); > + kasan_init(); > request_standard_resources(mdesc); > > if (mdesc->restart) > diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile > index 99699c32d8a5..4536159bc8fa 100644 > --- a/arch/arm/mm/Makefile > +++ b/arch/arm/mm/Makefile > @@ -113,3 +113,6 @@ obj-$(CONFIG_CACHE_L2X0_PMU) += cache-l2x0-pmu.o > obj-$(CONFIG_CACHE_XSC3L2) += cache-xsc3l2.o > obj-$(CONFIG_CACHE_TAUROS2) += cache-tauros2.o > obj-$(CONFIG_CACHE_UNIPHIER) += cache-uniphier.o > + > +KASAN_SANITIZE_kasan_init.o := n > +obj-$(CONFIG_KASAN) += kasan_init.o > diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c > new file mode 100644 > index 000000000000..b040f4de6b15 > --- /dev/null > +++ b/arch/arm/mm/kasan_init.c > @@ -0,0 +1,264 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * This file contains kasan initialization code for ARM. > + * > + * Copyright (c) 2018 Samsung Electronics Co., Ltd. > + * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com> > + * Author: Linus Walleij <linus.walleij@linaro.org> > + */ > + > +#define pr_fmt(fmt) "kasan: " fmt > +#include <linux/kasan.h> > +#include <linux/kernel.h> > +#include <linux/memblock.h> > +#include <linux/sched/task.h> > +#include <linux/start_kernel.h> > +#include <linux/pgtable.h> > +#include <asm/cputype.h> > +#include <asm/highmem.h> > +#include <asm/mach/map.h> > +#include <asm/memory.h> > +#include <asm/page.h> > +#include <asm/pgalloc.h> > +#include <asm/procinfo.h> > +#include <asm/proc-fns.h> > + > +#include "mm.h" > + > +static pgd_t tmp_pgd_table[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); > + > +pmd_t tmp_pmd_table[PTRS_PER_PMD] __page_aligned_bss; > + > +static __init void *kasan_alloc_block(size_t size) > +{ > + return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS), > + MEMBLOCK_ALLOC_KASAN, NUMA_NO_NODE); > +} > + > +static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, > + unsigned long end, bool early) > +{ > + unsigned long next; > + pte_t *ptep = pte_offset_kernel(pmdp, addr); > + > + do { > + pte_t entry; > + > + next = addr + PAGE_SIZE; > + > + if (!early) { > + void *p = kasan_alloc_block(PAGE_SIZE); > + if (!p) { > + panic("%s failed to alloc pte for address 0x%lx\n", > + __func__, addr); > + return; > + } > + memset(p, KASAN_SHADOW_INIT, PAGE_SIZE); > + entry = pfn_pte(virt_to_pfn(p), > + __pgprot(pgprot_val(PAGE_KERNEL))); > + } else if (pte_none(READ_ONCE(*ptep))) { > + /* > + * The early shadow memory is mapping all KASan > + * operations to one and the same page in memory, > + * "kasan_early_shadow_page" so that the instrumentation > + * will work on a scratch area until we can set up the > + * proper KASan shadow memory. > + */ > + entry = pfn_pte(virt_to_pfn(kasan_early_shadow_page), > + __pgprot(_L_PTE_DEFAULT | L_PTE_DIRTY | L_PTE_XN)); > + } else { > + /* > + * Early shadow mappings are PMD_SIZE aligned, so if the > + * first entry is already set, they must all be set. > + */ > + return; > + } > + > + set_pte_at(&init_mm, addr, ptep, entry); > + } while (ptep++, addr = next, addr != end); > +} > + > +/* > + * The pmd (page middle directory) is only used on LPAE > + */ > +static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, > + unsigned long end, bool early) > +{ > + unsigned long next; > + pmd_t *pmdp = pmd_offset(pudp, addr); > + > + do { > + if (pmd_none(*pmdp)) { > + void *p = early ? kasan_early_shadow_pte : > + kasan_alloc_block(PAGE_SIZE); > + > + if (!p) { > + panic("%s failed to allocate pmd for address 0x%lx\n", > + __func__, addr); > + return; > + } > + pmd_populate_kernel(&init_mm, pmdp, p); > + flush_pmd_entry(pmdp); > + } > + > + next = pmd_addr_end(addr, end); > + kasan_pte_populate(pmdp, addr, next, early); > + } while (pmdp++, addr = next, addr != end); > +} > + > +static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, > + bool early) > +{ > + unsigned long next; > + pgd_t *pgdp; > + p4d_t *p4dp; > + pud_t *pudp; > + > + pgdp = pgd_offset_k(addr); > + > + do { > + /* Allocate and populate the PGD if it doesn't already exist */ > + if (!early && pgd_none(*pgdp)) { > + void *p = kasan_alloc_block(PAGE_SIZE); > + > + if (!p) { > + panic("%s failed to allocate pgd for address 0x%lx\n", > + __func__, addr); > + return; > + } > + pgd_populate(&init_mm, pgdp, p); > + } > + > + next = pgd_addr_end(addr, end); > + /* > + * We just immediately jump over the p4d and pud page > + * directories since we believe ARM32 will never gain four > + * nor five level page tables. > + */ > + p4dp = p4d_offset(pgdp, addr); > + pudp = pud_offset(p4dp, addr); > + > + kasan_pmd_populate(pudp, addr, next, early); > + } while (pgdp++, addr = next, addr != end); > +} > + > +extern struct proc_info_list *lookup_processor_type(unsigned int); > + > +void __init kasan_early_init(void) > +{ > + struct proc_info_list *list; > + > + /* > + * locate processor in the list of supported processor > + * types. The linker builds this table for us from the > + * entries in arch/arm/mm/proc-*.S > + */ > + list = lookup_processor_type(read_cpuid_id()); > + if (list) { > +#ifdef MULTI_CPU > + processor = *list->proc; > +#endif > + } > + > + BUILD_BUG_ON((KASAN_SHADOW_END - (1UL << 29)) != KASAN_SHADOW_OFFSET); > + /* > + * We walk the page table and set all of the shadow memory to point > + * to the scratch page. > + */ > + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, true); > +} > + > +static void __init clear_pgds(unsigned long start, > + unsigned long end) > +{ > + for (; start && start < end; start += PMD_SIZE) > + pmd_clear(pmd_off_k(start)); > +} > + > +static int __init create_mapping(void *start, void *end) > +{ > + pr_info("populating shadow for %px to %px\n", start, end); > + kasan_pgd_populate((unsigned long)start & PAGE_MASK, > + (unsigned long)end, false); > + return 0; > +} > + > +void __init kasan_init(void) > +{ > + struct memblock_region *reg; > + int i; > + > + /* > + * We are going to perform proper setup of shadow memory. > + * > + * At first we should unmap early shadow (clear_pgds() call bellow). > + * However, instrumented code couldn't execute without shadow memory. > + * > + * To keep the early shadow memory MMU tables around while setting up > + * the proper shadow memory, we copy swapper_pg_dir (the initial page > + * table) to tmp_pgd_table and use that to keep the early shadow memory > + * mapped until the full shadow setup is finished. Then we swap back > + * to the proper swapper_pg_dir. > + */ > + > + memcpy(tmp_pgd_table, swapper_pg_dir, sizeof(tmp_pgd_table)); > +#ifdef CONFIG_ARM_LPAE > + /* We need to be in the same PGD or this won't work */ > + BUILD_BUG_ON(pgd_index(KASAN_SHADOW_START) != > + pgd_index(KASAN_SHADOW_END)); > + memcpy(tmp_pmd_table, > + pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_START)), > + sizeof(tmp_pmd_table)); > + set_pgd(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)], > + __pgd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER)); > +#endif > + cpu_switch_mm(tmp_pgd_table, &init_mm); > + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); > + > + kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START), > + kasan_mem_to_shadow((void *)-1UL) + 1); > + > + for_each_memblock(memory, reg) { > + void *start = __va(reg->base); > + void *end = __va(reg->base + reg->size); > + > + /* Do not attempt to shadow highmem */ > + if (reg->base >= arm_lowmem_limit) > + continue; > + if (reg->base + reg->size > arm_lowmem_limit) > + end = __va(arm_lowmem_limit); > + if (start >= end) > + continue; > + > + create_mapping(kasan_mem_to_shadow(start), > + kasan_mem_to_shadow(end)); > + } > + > + /* > + * 1. The module global variables are in MODULES_VADDR ~ MODULES_END, > + * so we need to map this area. > + * 2. PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR > + * ~ MODULES_END's shadow is in the same PMD_SIZE, so we can't > + * use kasan_populate_zero_shadow. > + */ > + create_mapping( > + kasan_mem_to_shadow((void *)MODULES_VADDR), > + kasan_mem_to_shadow((void *)(PKMAP_BASE + PMD_SIZE))); > + > + /* > + * KAsan may reuse the contents of kasan_early_shadow_pte directly, so > + * we should make sure that it maps the zero page read-only. > + */ > + for (i = 0; i < PTRS_PER_PTE; i++) > + set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE, > + &kasan_early_shadow_pte[i], > + pfn_pte(virt_to_pfn(kasan_early_shadow_page), > + __pgprot(pgprot_val(PAGE_KERNEL) > + | L_PTE_RDONLY))); > + local_flush_tlb_all(); > + > + memset(kasan_early_shadow_page, 0, PAGE_SIZE); > + cpu_switch_mm(swapper_pg_dir, &init_mm); > + pr_info("Kernel address sanitizer initialized\n"); > + init_task.kasan_depth = 0; > +} > diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c > index c5e1b27046a8..f8e9bc58a84f 100644 > --- a/arch/arm/mm/pgd.c > +++ b/arch/arm/mm/pgd.c > @@ -66,7 +66,21 @@ pgd_t *pgd_alloc(struct mm_struct *mm) > new_pmd = pmd_alloc(mm, new_pud, 0); > if (!new_pmd) > goto no_pmd; > -#endif > +#ifdef CONFIG_KASAN > + /* > + * Copy PMD table for KASAN shadow mappings. > + */ > + init_pgd = pgd_offset_k(TASK_SIZE); > + init_p4d = p4d_offset(init_pgd, TASK_SIZE); > + init_pud = pud_offset(init_p4d, TASK_SIZE); > + init_pmd = pmd_offset(init_pud, TASK_SIZE); > + new_pmd = pmd_offset(new_pud, TASK_SIZE); > + memcpy(new_pmd, init_pmd, > + (pmd_index(MODULES_VADDR) - pmd_index(TASK_SIZE)) > + * sizeof(pmd_t)); > + clean_dcache_area(new_pmd, PTRS_PER_PMD * sizeof(pmd_t)); > +#endif /* CONFIG_KASAN */ > +#endif /* CONFIG_LPAE */ > > if (!vectors_high()) { > /* > -- > 2.25.4 >
diff --git a/arch/arm/include/asm/kasan.h b/arch/arm/include/asm/kasan.h new file mode 100644 index 000000000000..56b954db160e --- /dev/null +++ b/arch/arm/include/asm/kasan.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * arch/arm/include/asm/kasan.h + * + * Copyright (c) 2015 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com> + * + */ + +#ifndef __ASM_KASAN_H +#define __ASM_KASAN_H + +#ifdef CONFIG_KASAN + +#include <asm/kasan_def.h> + +#define KASAN_SHADOW_SCALE_SHIFT 3 + +/* + * The compiler uses a shadow offset assuming that addresses start + * from 0. Kernel addresses don't start from 0, so shadow + * for kernel really starts from 'compiler's shadow offset' + + * ('kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT) + */ + +extern void kasan_init(void); + +#else +static inline void kasan_init(void) { } +#endif + +#endif diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index 069da393110c..3bf1905df9c3 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -21,6 +21,7 @@ #define _PAGE_KERNEL_TABLE (PMD_TYPE_TABLE | PMD_BIT4 | PMD_DOMAIN(DOMAIN_KERNEL)) #ifdef CONFIG_ARM_LPAE +#define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { @@ -39,14 +40,19 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) } #else /* !CONFIG_ARM_LPAE */ +#define PGD_SIZE (PAGE_SIZE << 2) /* * Since we have only two-level page tables, these are trivial */ #define pmd_alloc_one(mm,addr) ({ BUG(); ((pmd_t *)2); }) #define pmd_free(mm, pmd) do { } while (0) +#ifdef CONFIG_KASAN +/* The KASan core unconditionally calls pud_populate() on all architectures */ +#define pud_populate(mm,pmd,pte) do { } while (0) +#else #define pud_populate(mm,pmd,pte) BUG() - +#endif #endif /* CONFIG_ARM_LPAE */ extern pgd_t *pgd_alloc(struct mm_struct *mm); diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 3609a6980c34..02813a5d9e10 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -13,7 +13,15 @@ #include <asm/fpstate.h> #include <asm/page.h> +#ifdef CONFIG_KASAN +/* + * KASan uses a lot of extra stack space so the thread size order needs to + * be increased. + */ +#define THREAD_SIZE_ORDER 2 +#else #define THREAD_SIZE_ORDER 1 +#endif #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) #define THREAD_START_SP (THREAD_SIZE - 8) diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S index 6840c7c60a85..89c80154b9ef 100644 --- a/arch/arm/kernel/head-common.S +++ b/arch/arm/kernel/head-common.S @@ -111,6 +111,9 @@ __mmap_switched: str r8, [r2] @ Save atags pointer cmp r3, #0 strne r10, [r3] @ Save control register values +#ifdef CONFIG_KASAN + bl kasan_early_init +#endif mov lr, #0 b start_kernel ENDPROC(__mmap_switched) diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index d8e18cdd96d3..b0820847bb92 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -58,6 +58,7 @@ #include <asm/unwind.h> #include <asm/memblock.h> #include <asm/virt.h> +#include <asm/kasan.h> #include "atags.h" @@ -1130,6 +1131,7 @@ void __init setup_arch(char **cmdline_p) early_ioremap_reset(); paging_init(mdesc); + kasan_init(); request_standard_resources(mdesc); if (mdesc->restart) diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 99699c32d8a5..4536159bc8fa 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -113,3 +113,6 @@ obj-$(CONFIG_CACHE_L2X0_PMU) += cache-l2x0-pmu.o obj-$(CONFIG_CACHE_XSC3L2) += cache-xsc3l2.o obj-$(CONFIG_CACHE_TAUROS2) += cache-tauros2.o obj-$(CONFIG_CACHE_UNIPHIER) += cache-uniphier.o + +KASAN_SANITIZE_kasan_init.o := n +obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c new file mode 100644 index 000000000000..b040f4de6b15 --- /dev/null +++ b/arch/arm/mm/kasan_init.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file contains kasan initialization code for ARM. + * + * Copyright (c) 2018 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin <ryabinin.a.a@gmail.com> + * Author: Linus Walleij <linus.walleij@linaro.org> + */ + +#define pr_fmt(fmt) "kasan: " fmt +#include <linux/kasan.h> +#include <linux/kernel.h> +#include <linux/memblock.h> +#include <linux/sched/task.h> +#include <linux/start_kernel.h> +#include <linux/pgtable.h> +#include <asm/cputype.h> +#include <asm/highmem.h> +#include <asm/mach/map.h> +#include <asm/memory.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/procinfo.h> +#include <asm/proc-fns.h> + +#include "mm.h" + +static pgd_t tmp_pgd_table[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); + +pmd_t tmp_pmd_table[PTRS_PER_PMD] __page_aligned_bss; + +static __init void *kasan_alloc_block(size_t size) +{ + return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS), + MEMBLOCK_ALLOC_KASAN, NUMA_NO_NODE); +} + +static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, + unsigned long end, bool early) +{ + unsigned long next; + pte_t *ptep = pte_offset_kernel(pmdp, addr); + + do { + pte_t entry; + + next = addr + PAGE_SIZE; + + if (!early) { + void *p = kasan_alloc_block(PAGE_SIZE); + if (!p) { + panic("%s failed to alloc pte for address 0x%lx\n", + __func__, addr); + return; + } + memset(p, KASAN_SHADOW_INIT, PAGE_SIZE); + entry = pfn_pte(virt_to_pfn(p), + __pgprot(pgprot_val(PAGE_KERNEL))); + } else if (pte_none(READ_ONCE(*ptep))) { + /* + * The early shadow memory is mapping all KASan + * operations to one and the same page in memory, + * "kasan_early_shadow_page" so that the instrumentation + * will work on a scratch area until we can set up the + * proper KASan shadow memory. + */ + entry = pfn_pte(virt_to_pfn(kasan_early_shadow_page), + __pgprot(_L_PTE_DEFAULT | L_PTE_DIRTY | L_PTE_XN)); + } else { + /* + * Early shadow mappings are PMD_SIZE aligned, so if the + * first entry is already set, they must all be set. + */ + return; + } + + set_pte_at(&init_mm, addr, ptep, entry); + } while (ptep++, addr = next, addr != end); +} + +/* + * The pmd (page middle directory) is only used on LPAE + */ +static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, + unsigned long end, bool early) +{ + unsigned long next; + pmd_t *pmdp = pmd_offset(pudp, addr); + + do { + if (pmd_none(*pmdp)) { + void *p = early ? kasan_early_shadow_pte : + kasan_alloc_block(PAGE_SIZE); + + if (!p) { + panic("%s failed to allocate pmd for address 0x%lx\n", + __func__, addr); + return; + } + pmd_populate_kernel(&init_mm, pmdp, p); + flush_pmd_entry(pmdp); + } + + next = pmd_addr_end(addr, end); + kasan_pte_populate(pmdp, addr, next, early); + } while (pmdp++, addr = next, addr != end); +} + +static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, + bool early) +{ + unsigned long next; + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp; + + pgdp = pgd_offset_k(addr); + + do { + /* Allocate and populate the PGD if it doesn't already exist */ + if (!early && pgd_none(*pgdp)) { + void *p = kasan_alloc_block(PAGE_SIZE); + + if (!p) { + panic("%s failed to allocate pgd for address 0x%lx\n", + __func__, addr); + return; + } + pgd_populate(&init_mm, pgdp, p); + } + + next = pgd_addr_end(addr, end); + /* + * We just immediately jump over the p4d and pud page + * directories since we believe ARM32 will never gain four + * nor five level page tables. + */ + p4dp = p4d_offset(pgdp, addr); + pudp = pud_offset(p4dp, addr); + + kasan_pmd_populate(pudp, addr, next, early); + } while (pgdp++, addr = next, addr != end); +} + +extern struct proc_info_list *lookup_processor_type(unsigned int); + +void __init kasan_early_init(void) +{ + struct proc_info_list *list; + + /* + * locate processor in the list of supported processor + * types. The linker builds this table for us from the + * entries in arch/arm/mm/proc-*.S + */ + list = lookup_processor_type(read_cpuid_id()); + if (list) { +#ifdef MULTI_CPU + processor = *list->proc; +#endif + } + + BUILD_BUG_ON((KASAN_SHADOW_END - (1UL << 29)) != KASAN_SHADOW_OFFSET); + /* + * We walk the page table and set all of the shadow memory to point + * to the scratch page. + */ + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, true); +} + +static void __init clear_pgds(unsigned long start, + unsigned long end) +{ + for (; start && start < end; start += PMD_SIZE) + pmd_clear(pmd_off_k(start)); +} + +static int __init create_mapping(void *start, void *end) +{ + pr_info("populating shadow for %px to %px\n", start, end); + kasan_pgd_populate((unsigned long)start & PAGE_MASK, + (unsigned long)end, false); + return 0; +} + +void __init kasan_init(void) +{ + struct memblock_region *reg; + int i; + + /* + * We are going to perform proper setup of shadow memory. + * + * At first we should unmap early shadow (clear_pgds() call bellow). + * However, instrumented code couldn't execute without shadow memory. + * + * To keep the early shadow memory MMU tables around while setting up + * the proper shadow memory, we copy swapper_pg_dir (the initial page + * table) to tmp_pgd_table and use that to keep the early shadow memory + * mapped until the full shadow setup is finished. Then we swap back + * to the proper swapper_pg_dir. + */ + + memcpy(tmp_pgd_table, swapper_pg_dir, sizeof(tmp_pgd_table)); +#ifdef CONFIG_ARM_LPAE + /* We need to be in the same PGD or this won't work */ + BUILD_BUG_ON(pgd_index(KASAN_SHADOW_START) != + pgd_index(KASAN_SHADOW_END)); + memcpy(tmp_pmd_table, + pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_START)), + sizeof(tmp_pmd_table)); + set_pgd(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)], + __pgd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER)); +#endif + cpu_switch_mm(tmp_pgd_table, &init_mm); + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + + kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START), + kasan_mem_to_shadow((void *)-1UL) + 1); + + for_each_memblock(memory, reg) { + void *start = __va(reg->base); + void *end = __va(reg->base + reg->size); + + /* Do not attempt to shadow highmem */ + if (reg->base >= arm_lowmem_limit) + continue; + if (reg->base + reg->size > arm_lowmem_limit) + end = __va(arm_lowmem_limit); + if (start >= end) + continue; + + create_mapping(kasan_mem_to_shadow(start), + kasan_mem_to_shadow(end)); + } + + /* + * 1. The module global variables are in MODULES_VADDR ~ MODULES_END, + * so we need to map this area. + * 2. PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR + * ~ MODULES_END's shadow is in the same PMD_SIZE, so we can't + * use kasan_populate_zero_shadow. + */ + create_mapping( + kasan_mem_to_shadow((void *)MODULES_VADDR), + kasan_mem_to_shadow((void *)(PKMAP_BASE + PMD_SIZE))); + + /* + * KAsan may reuse the contents of kasan_early_shadow_pte directly, so + * we should make sure that it maps the zero page read-only. + */ + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE, + &kasan_early_shadow_pte[i], + pfn_pte(virt_to_pfn(kasan_early_shadow_page), + __pgprot(pgprot_val(PAGE_KERNEL) + | L_PTE_RDONLY))); + local_flush_tlb_all(); + + memset(kasan_early_shadow_page, 0, PAGE_SIZE); + cpu_switch_mm(swapper_pg_dir, &init_mm); + pr_info("Kernel address sanitizer initialized\n"); + init_task.kasan_depth = 0; +} diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index c5e1b27046a8..f8e9bc58a84f 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -66,7 +66,21 @@ pgd_t *pgd_alloc(struct mm_struct *mm) new_pmd = pmd_alloc(mm, new_pud, 0); if (!new_pmd) goto no_pmd; -#endif +#ifdef CONFIG_KASAN + /* + * Copy PMD table for KASAN shadow mappings. + */ + init_pgd = pgd_offset_k(TASK_SIZE); + init_p4d = p4d_offset(init_pgd, TASK_SIZE); + init_pud = pud_offset(init_p4d, TASK_SIZE); + init_pmd = pmd_offset(init_pud, TASK_SIZE); + new_pmd = pmd_offset(new_pud, TASK_SIZE); + memcpy(new_pmd, init_pmd, + (pmd_index(MODULES_VADDR) - pmd_index(TASK_SIZE)) + * sizeof(pmd_t)); + clean_dcache_area(new_pmd, PTRS_PER_PMD * sizeof(pmd_t)); +#endif /* CONFIG_KASAN */ +#endif /* CONFIG_LPAE */ if (!vectors_high()) { /*
This patch initializes KASan shadow region's page table and memory. There are two stage for KASan initializing: 1. At early boot stage the whole shadow region is mapped to just one physical page (kasan_zero_page). It is finished by the function kasan_early_init which is called by __mmap_switched(arch/arm/kernel/ head-common.S) 2. After the calling of paging_init, we use kasan_zero_page as zero shadow for some memory that KASan does not need to track, and we allocate a new shadow space for the other memory that KASan need to track. These issues are finished by the function kasan_init which is call by setup_arch. When using KASan we also need to increase the THREAD_SIZE_ORDER from 1 to 2 as the extra calls for shadow memory uses quite a bit of stack. As we need to make a temporary copy of the PGD when setting up shadow memory we create a helpful PGD_SIZE definition for both LPAE and non-LPAE setups. The KASan core code unconditionally calls pud_populate() so this needs to be changed from BUG() to do {} while (0) when building with KASan enabled. After the initial development by Andre Ryabinin several modifications have been made to this code: Abbott Liu <liuwenliang@huawei.com> - Add support ARM LPAE: If LPAE is enabled, KASan shadow region's mapping table need be copied in the pgd_alloc() function. - Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate, kasan_pgd_populate from .meminit.text section to .init.text section. Reported by Florian Fainelli <f.fainelli@gmail.com> Linus Walleij <linus.walleij@linaro.org>: - Drop the custom mainpulation of TTBR0 and just use cpu_switch_mm() to switch the pgd table. - Adopt to handle 4th level page tabel folding. - Rewrite the entire page directory and page entry initialization sequence to be recursive based on ARM64:s kasan_init.c. Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: kasan-dev@googlegroups.com Cc: Mike Rapoport <rppt@linux.ibm.com> Co-developed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Co-developed-by: Abbott Liu <liuwenliang@huawei.com> Co-developed-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Ard Biesheuvel <ardb@kernel.org> # QEMU/KVM/mach-virt/LPAE/8G Reported-by: Russell King - ARM Linux <linux@armlinux.org.uk> Reported-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Signed-off-by: Abbott Liu <liuwenliang@huawei.com> Signed-off-by: Florian Fainelli <f.fainelli@gmail.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> --- ChangeLog v11->v12: - Do not try to shadow highmem memory blocks. (Ard) - Provoke a build bug if the entire shadow memory doesn't fit inside a single pgd_index() (Ard) - Move the pointer to (unsigned long) casts into the create_mapping() function. (Ard) - After setting up the shadow memory make sure to issue local_flush_tlb_all() so that we refresh all the global mappings. (Ard) - Simplify pte_populate() (Ard) - Skip over pud population as well as p4d. (Ard) - Drop the stop condition pmd_none(*pmdp) in the pmd population loop. (Ard) - Stop passing around the node (NUMA) parameter in the init code, we are not expecting any NUMA architectures to be introduced into ARM32 so just hardcode NUMA_NO_NODE when calling memblock_alloc_try_nid(). ChangeLog v10->v11: - Fix compilation on LPAE systems. - Move the check for valid pgdp, pudp and pmdp into the loop for each level moving over the directory pointers: we were just lucky that we just needed one directory for each level so this fixes the pmdp issue with LPAE and KASan now works like a charm on LPAE as well. - Fold fourth level page directory (p4d) into the global page directory pgd and just skip into the page upper directory (pud) directly. We do not anticipate that ARM32 will every use 5-level page tables. - Simplify the ifdeffery around the temporary pgd. - Insert a comment about pud_populate() that is unconditionally called by the KASan core code. ChangeLog v9->v10: - Rebase onto v5.8-rc1 - add support for folded p4d page tables, use the primitives necessary for the 4th level folding, add (empty) walks of p4d level. - Use the <linux/pgtable.h> header file that has now appeared as part of the VM consolidation series. - Use a recursive method to walk pgd/p4d/pud/pmd/pte instead of the separate early/main calls and the flat call structure used in the old code. This was inspired by the ARM64 KASan init code. - Assume authorship of this code, I have now written the majority of it so the blame is on me and noone else. ChangeLog v8->v9: - Drop the custom CP15 manipulation and cache flushing for swapping TTBR0 and instead just use cpu_switch_mm(). - Collect Ard's tags. ChangeLog v7->v8: - Rebased. ChangeLog v6->v7: - Use SPDX identifer for the license. - Move the TTBR0 accessor calls into this patch. --- arch/arm/include/asm/kasan.h | 32 ++++ arch/arm/include/asm/pgalloc.h | 8 +- arch/arm/include/asm/thread_info.h | 8 + arch/arm/kernel/head-common.S | 3 + arch/arm/kernel/setup.c | 2 + arch/arm/mm/Makefile | 3 + arch/arm/mm/kasan_init.c | 264 +++++++++++++++++++++++++++++ arch/arm/mm/pgd.c | 16 +- 8 files changed, 334 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/kasan.h create mode 100644 arch/arm/mm/kasan_init.c