From patchwork Tue Mar 24 14:49:04 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Ryabinin X-Patchwork-Id: 6080171 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id C7BF19F2A9 for ; Tue, 24 Mar 2015 14:52:29 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 4AB03201D3 for ; Tue, 24 Mar 2015 14:52:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C438E2015E for ; Tue, 24 Mar 2015 14:52:26 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1YaQ9j-0006hF-Mk; Tue, 24 Mar 2015 14:50:11 +0000 Received: from mailout3.w1.samsung.com ([210.118.77.13]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1YaQ9J-000605-I4 for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2015 14:49:48 +0000 Received: from eucpsbgm1.samsung.com (unknown [203.254.199.244]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7u4-24.01(7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTP id <0NLQ00L5I1D0NQ60@mailout3.w1.samsung.com> for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2015 14:53:24 +0000 (GMT) X-AuditID: cbfec7f4-b7f126d000001e9a-97-551178c8b5c6 Received: from eusync1.samsung.com ( [203.254.199.211]) by eucpsbgm1.samsung.com (EUCPMTA) with SMTP id 18.9B.07834.8C871155; Tue, 24 Mar 2015 14:46:32 +0000 (GMT) Received: from localhost.localdomain ([106.109.129.143]) by eusync1.samsung.com (Oracle Communications Messaging Server 7u4-23.01 (7.0.4.23.0) 64bit (built Aug 10 2011)) with ESMTPA id <0NLQ00C1D15YIE60@eusync1.samsung.com>; Tue, 24 Mar 2015 14:49:20 +0000 (GMT) From: Andrey Ryabinin To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 2/2] arm64: add KASan support Date: Tue, 24 Mar 2015 17:49:04 +0300 Message-id: <1427208544-8232-3-git-send-email-a.ryabinin@samsung.com> X-Mailer: git-send-email 2.3.3 In-reply-to: <1427208544-8232-1-git-send-email-a.ryabinin@samsung.com> References: <1427208544-8232-1-git-send-email-a.ryabinin@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJJMWRmVeSWpSXmKPExsVy+t/xy7onKgRDDZ4+YLHY9usRm8Wc9WvY LN4v62G02PT4GqvF5V1z2CzurfnPavHy4wkWB3aPNfPWMHps+jSJ3ePEjN8sHpuX1Hv0bVnF 6PF5k1wAWxSXTUpqTmZZapG+XQJXxpE1a1gKVpVXNHz8wdLAeC+xi5GTQ0LARGLmmVXsELaY xIV769m6GLk4hASWMkq8OrCFFcLpY5L48mULG0gVm4CexL9Z28FsEQENiSldj9lBipgF7jJK fJv1nQUkISygK/F37zFmEJtFQFXiw9R7jF2MHBy8Aq4Syw9ygpgSAnISC9+Lg1RwCrhJfHz8 CqxaCKhiw7YDzBMYeRcwMqxiFE0tTS4oTkrPNdQrTswtLs1L10vOz93ECAmuLzsYFx+zOsQo wMGoxMMbUc8fKsSaWFZcmXuIUYKDWUmEd0OxYKgQb0piZVVqUX58UWlOavEhRiYOTqkGRufX x1i2Pd0+5/OdMMNV14V8zzud2NK8c1l6zy8JVeFPvPmehX8blTQDv5/UFZZybTc+vF/89wNJ tqp/wjHPbr1+zMSwVHHSY96syccllXZdmqsRXdlyb/WDRX+ygw6clgy8d5IhPdt0Cavx8186 oZ8nl19qLlB3kIkte1fVKC7y/OXWRoYCMyWW4oxEQy3mouJEAJY25DQMAgAA X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20150324_074945_912415_42F2A181 X-CRM114-Status: GOOD ( 20.52 ) X-Spam-Score: -5.0 (-----) Cc: Andrey Ryabinin , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch adds arch specific code for kernel address sanitizer (see Documentation/kasan.txt). 1/8 of kernel addresses reserved for shadow memory. There was no big enough hole for this, so virtual addresses for shadow were stolen from vmalloc area. At early boot stage the whole shadow region populated with just one physical page (kasan_zero_page). Later, this page reused as readonly zero shadow for some memory that KASan currently don't track (vmalloc). After mapping the physical memory, pages for shadow memory are allocated and mapped. KASan's stack instrumentation significantly increases stack's consumption, so CONFIG_KASAN doubles THREAD_SIZE. Functions like memset/memmove/memcpy do a lot of memory accesses. If bad pointer passed to one of these function it is important to catch this. Compiler's instrumentation cannot do this since these functions are written in assembly. KASan replaces memory functions with manually instrumented variants. Original functions declared as weak symbols so strong definitions in mm/kasan/kasan.c could replace them. Original functions have aliases with '__' prefix in name, so we could call non-instrumented variant if needed. Some files built without kasan instrumentation (e.g. mm/slub.c). Original mem* function replaced (via #define) with prefixed variants to disable memory access checks for such files. Signed-off-by: Andrey Ryabinin --- arch/arm64/Kconfig | 7 ++ arch/arm64/include/asm/pgtable.h | 3 +- arch/arm64/include/asm/string.h | 16 +++ arch/arm64/include/asm/thread_info.h | 8 ++ arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/module.c | 16 ++- arch/arm64/kernel/setup.c | 2 + arch/arm64/lib/memcpy.S | 3 + arch/arm64/lib/memmove.S | 7 +- arch/arm64/lib/memset.S | 3 + arch/arm64/mm/Makefile | 3 + arch/arm64/mm/kasan_init.c | 211 +++++++++++++++++++++++++++++++++++ 12 files changed, 276 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/mm/kasan_init.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 4085df1..10bbd71 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -41,6 +41,7 @@ config ARM64 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_KASAN if SPARSEMEM_VMEMMAP select HAVE_ARCH_KGDB select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK @@ -116,6 +117,12 @@ config GENERIC_CSUM config GENERIC_CALIBRATE_DELAY def_bool y +config KASAN_SHADOW_OFFSET + hex + default 0xdfff200000000000 if ARM64_VA_BITS_48 + default 0xdffffc8000000000 if ARM64_VA_BITS_42 + default 0xdfffff9000000000 if ARM64_VA_BITS_39 + config ZONE_DMA def_bool y diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index bd5db28..f5ce010 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -40,7 +40,7 @@ * fixed mappings and modules */ #define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT)) * sizeof(struct page), PUD_SIZE) -#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS) +#define VMALLOC_START ((UL(0xffffffffffffffff) << VA_BITS) + (UL(1) << (VA_BITS - 3))) #define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K) #define vmemmap ((struct page *)(VMALLOC_END + SZ_64K)) @@ -73,6 +73,7 @@ extern void __pgd_error(const char *file, int line, unsigned long val); #define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE) #define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE) +#define PAGE_KERNEL_RO __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY) #define PAGE_HYP __pgprot(_PAGE_DEFAULT | PTE_HYP) #define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP) diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h index 64d2d48..bff522c 100644 --- a/arch/arm64/include/asm/string.h +++ b/arch/arm64/include/asm/string.h @@ -36,17 +36,33 @@ extern __kernel_size_t strnlen(const char *, __kernel_size_t); #define __HAVE_ARCH_MEMCPY extern void *memcpy(void *, const void *, __kernel_size_t); +extern void *__memcpy(void *, const void *, __kernel_size_t); #define __HAVE_ARCH_MEMMOVE extern void *memmove(void *, const void *, __kernel_size_t); +extern void *__memmove(void *, const void *, __kernel_size_t); #define __HAVE_ARCH_MEMCHR extern void *memchr(const void *, int, __kernel_size_t); #define __HAVE_ARCH_MEMSET extern void *memset(void *, int, __kernel_size_t); +extern void *__memset(void *, int, __kernel_size_t); #define __HAVE_ARCH_MEMCMP extern int memcmp(const void *, const void *, size_t); + +#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__) + +/* + * For files that not instrumented (e.g. mm/slub.c) we + * should use not instrumented version of mem* functions. + */ + +#define memcpy(dst, src, len) __memcpy(dst, src, len) +#define memmove(dst, src, len) __memmove(dst, src, len) +#define memset(s, c, n) __memset(s, c, n) +#endif + #endif diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 702e1e6..4a2c11e 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -24,10 +24,18 @@ #include #ifndef CONFIG_ARM64_64K_PAGES +#ifndef CONFIG_KASAN #define THREAD_SIZE_ORDER 2 +#else +#define THREAD_SIZE_ORDER 3 +#endif #endif +#ifndef CONFIG_KASAN #define THREAD_SIZE 16384 +#else +#define THREAD_SIZE 32768 +#endif #define THREAD_START_SP (THREAD_SIZE - 16) #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 51c9811..1a99e95 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -482,6 +482,9 @@ __mmap_switched: str_l x21, __fdt_pointer, x5 // Save FDT pointer str_l x24, memstart_addr, x6 // Save PHYS_OFFSET mov x29, #0 +#ifdef CONFIG_KASAN + b kasan_early_init +#endif b start_kernel ENDPROC(__mmap_switched) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 67bf410..7d90c0f 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -34,9 +35,18 @@ void *module_alloc(unsigned long size) { - return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, - GFP_KERNEL, PAGE_KERNEL_EXEC, 0, - NUMA_NO_NODE, __builtin_return_address(0)); + void *p; + + p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR, MODULES_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, 0, + NUMA_NO_NODE, __builtin_return_address(0)); + + if (p && (kasan_module_alloc(p, size) < 0)) { + vfree(p); + return NULL; + } + + return p; } enum aarch64_reloc_op { diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 51ef972..f197f45 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -396,6 +397,7 @@ void __init setup_arch(char **cmdline_p) arm64_memblock_init(); paging_init(); + kasan_init(); request_standard_resources(); early_ioremap_reset(); diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S index 8a9a96d..845e40a 100644 --- a/arch/arm64/lib/memcpy.S +++ b/arch/arm64/lib/memcpy.S @@ -56,6 +56,8 @@ C_h .req x12 D_l .req x13 D_h .req x14 +.weak memcpy +ENTRY(__memcpy) ENTRY(memcpy) mov dst, dstin cmp count, #16 @@ -199,3 +201,4 @@ ENTRY(memcpy) b.ne .Ltail63 ret ENDPROC(memcpy) +ENDPROC(__memcpy) diff --git a/arch/arm64/lib/memmove.S b/arch/arm64/lib/memmove.S index 57b19ea..48074d2 100644 --- a/arch/arm64/lib/memmove.S +++ b/arch/arm64/lib/memmove.S @@ -57,12 +57,14 @@ C_h .req x12 D_l .req x13 D_h .req x14 +.weak memmove +ENTRY(__memmove) ENTRY(memmove) cmp dstin, src - b.lo memcpy + b.lo __memcpy add tmp1, src, count cmp dstin, tmp1 - b.hs memcpy /* No overlap. */ + b.hs __memcpy /* No overlap. */ add dst, dstin, count add src, src, count @@ -195,3 +197,4 @@ ENTRY(memmove) b.ne .Ltail63 ret ENDPROC(memmove) +ENDPROC(__memmove) diff --git a/arch/arm64/lib/memset.S b/arch/arm64/lib/memset.S index 7c72dfd..4ab2594 100644 --- a/arch/arm64/lib/memset.S +++ b/arch/arm64/lib/memset.S @@ -54,6 +54,8 @@ dst .req x8 tmp3w .req w9 tmp3 .req x9 +.weak memset +ENTRY(__memset) ENTRY(memset) mov dst, dstin /* Preserve return value. */ and A_lw, val, #255 @@ -214,3 +216,4 @@ ENTRY(memset) b.ne .Ltail_maybe_long ret ENDPROC(memset) +ENDPROC(__memset) diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 773d37a..e17703c 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -4,3 +4,6 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ context.o proc.o pageattr.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_ARM64_PTDUMP) += dump.o + +KASAN_SANITIZE_kasan_init.o := n +obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c new file mode 100644 index 0000000..df537da --- /dev/null +++ b/arch/arm64/mm/kasan_init.c @@ -0,0 +1,211 @@ +#include +#include +#include +#include + +#include +#include +#include + +static char kasan_zero_page[PAGE_SIZE] __page_aligned_bss; +static pgd_t tmp_page_table[PTRS_PER_PGD] __initdata __aligned(PAGE_SIZE); + +#if CONFIG_PGTABLE_LEVELS > 3 +static pud_t kasan_zero_pud[PTRS_PER_PUD] __page_aligned_bss; +#endif +#if CONFIG_PGTABLE_LEVELS > 2 +static pmd_t kasan_zero_pmd[PTRS_PER_PMD] __page_aligned_bss; +#endif +static pte_t kasan_zero_pte[PTRS_PER_PTE] __page_aligned_bss; + +static void __init init_kasan_page_tables(void) +{ + int i; + +#if CONFIG_PGTABLE_LEVELS > 3 + for (i = 0; i < PTRS_PER_PUD; i++) + set_pud(&kasan_zero_pud[i], __pud(__pa(kasan_zero_pmd) + | PAGE_KERNEL)); +#endif +#if CONFIG_PGTABLE_LEVELS > 2 + for (i = 0; i < PTRS_PER_PMD; i++) + set_pmd(&kasan_zero_pmd[i], __pmd(__pa(kasan_zero_pte) + | PAGE_KERNEL)); +#endif + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte(&kasan_zero_pte[i], __pte(__pa(kasan_zero_page) + | PAGE_KERNEL)); +} + +void __init kasan_map_early_shadow(pgd_t *pgdp) +{ + int i; + unsigned long start = KASAN_SHADOW_START; + unsigned long end = KASAN_SHADOW_END; + pgd_t pgd; + +#if CONFIG_PGTABLE_LEVELS > 3 + pgd = __pgd(__pa(kasan_zero_pud) | PAGE_KERNEL); +#elif CONFIG_PGTABLE_LEVELS > 2 + pgd = __pgd(__pa(kasan_zero_pmd) | PAGE_KERNEL); +#else + pgd = __pgd(__pa(kasan_zero_pte) | PAGE_KERNEL); +#endif + + for (i = pgd_index(start); start < end; i++) { + set_pgd(&pgdp[i], pgd); + start += PGDIR_SIZE; + } +} + +void __init kasan_early_init(void) +{ + init_kasan_page_tables(); + kasan_map_early_shadow(swapper_pg_dir); + kasan_map_early_shadow(idmap_pg_dir); + flush_tlb_all(); + start_kernel(); +} + +static void __init clear_pgds(unsigned long start, + unsigned long end) +{ + for (; start && start < end; start += PGDIR_SIZE) + set_pgd(pgd_offset_k(start), __pgd(0)); +} + +static int __init zero_pte_populate(pmd_t *pmd, unsigned long addr, + unsigned long end) +{ + pte_t *pte = pte_offset_kernel(pmd, addr); + + while (addr + PAGE_SIZE <= end) { + set_pte(pte, __pte(__pa(kasan_zero_page) + | PAGE_KERNEL_RO)); + addr += PAGE_SIZE; + pte = pte_offset_kernel(pmd, addr); + } + return 0; +} + +static int __init zero_pmd_populate(pud_t *pud, unsigned long addr, + unsigned long end) +{ + int ret = 0; + pmd_t *pmd = pmd_offset(pud, addr); + + while (IS_ALIGNED(addr, PMD_SIZE) && addr + PMD_SIZE <= end) { + set_pmd(pmd, __pmd(__pa(kasan_zero_pte) + | PAGE_KERNEL_RO)); + addr += PMD_SIZE; + pmd++; + } + + if (addr < end) { + if (pmd_none(*pmd)) { + void *p = vmemmap_alloc_block(PAGE_SIZE, NUMA_NO_NODE); + if (!p) + return -ENOMEM; + set_pmd(pmd, __pmd(__pa(p) | PAGE_KERNEL)); + } + ret = zero_pte_populate(pmd, addr, end); + } + return ret; +} + +static int __init zero_pud_populate(pgd_t *pgd, unsigned long addr, + unsigned long end) +{ + int ret = 0; + pud_t *pud = pud_offset(pgd, addr); + +#if CONFIG_PGTABLE_LEVELS > 2 + while (IS_ALIGNED(addr, PUD_SIZE) && addr + PUD_SIZE <= end) { + set_pud(pud, __pud(__pa(kasan_zero_pmd) + | PAGE_KERNEL_RO)); + addr += PUD_SIZE; + pud++; + } +#endif + + if (addr < end) { + if (pud_none(*pud)) { + void *p = vmemmap_alloc_block(PAGE_SIZE, NUMA_NO_NODE); + if (!p) + return -ENOMEM; + set_pud(pud, __pud(__pa(p) | PAGE_KERNEL)); + } + ret = zero_pmd_populate(pud, addr, end); + } + return ret; +} + +static int __init zero_pgd_populate(unsigned long addr, unsigned long end) +{ + int ret = 0; + pgd_t *pgd = pgd_offset_k(addr); + +#if CONFIG_PGTABLE_LEVELS > 3 + while (IS_ALIGNED(addr, PGDIR_SIZE) && addr + PGDIR_SIZE <= end) { + set_pgd(pgd, __pgd(__pa(kasan_zero_pud) + | PAGE_KERNEL_RO)); + addr += PGDIR_SIZE; + pgd++; + } +#endif + + if (addr < end) { + if (pgd_none(*pgd)) { + void *p = vmemmap_alloc_block(PAGE_SIZE, NUMA_NO_NODE); + if (!p) + return -ENOMEM; + set_pgd(pgd, __pgd(__pa(p) | PAGE_KERNEL)); + } + ret = zero_pud_populate(pgd, addr, end); + } + return ret; +} + +static void __init populate_zero_shadow(unsigned long start, unsigned long end) +{ + if (zero_pgd_populate(start, end)) + panic("kasan: unable to map zero shadow!"); +} + +static void cpu_set_ttbr1(unsigned long ttbr1) +{ + asm( + " msr ttbr1_el1, %0\n" + " isb" + : + : "r" (ttbr1)); +} + +void __init kasan_init(void) +{ + struct memblock_region *reg; + + memcpy(tmp_page_table, swapper_pg_dir, sizeof(tmp_page_table)); + cpu_set_ttbr1(__pa(tmp_page_table)); + + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + + populate_zero_shadow(KASAN_SHADOW_START, + (unsigned long)kasan_mem_to_shadow((void *)MODULES_VADDR)); + + for_each_memblock(memory, reg) { + void *start = (void *)__phys_to_virt(reg->base); + void *end = (void *)__phys_to_virt(reg->base + reg->size); + + if (start >= end) + break; + + vmemmap_populate((unsigned long)kasan_mem_to_shadow(start), + (unsigned long)kasan_mem_to_shadow(end), + pfn_to_nid(virt_to_pfn(start))); + } + + memset(kasan_zero_page, 0, PAGE_SIZE); + cpu_set_ttbr1(__pa(swapper_pg_dir)); + init_task.kasan_depth = 0; +}