From patchwork Mon Jul 6 12:24:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Walleij X-Patchwork-Id: 11645587 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5163760D for ; Mon, 6 Jul 2020 12:29:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 19ADE206E6 for ; Mon, 6 Jul 2020 12:29:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hCVx2oZs"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="bmSgSD/K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 19ADE206E6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5aGwHs4H1O/ZUKiTjlPV+IK5Spxixm5Je94MXlZwUlk=; b=hCVx2oZsHzTvLotyfTITaLyZ5 KRvTj9YXNxt75JSAAObf0QBhj5KbZDFuHLesKVuNeiu13k2BqHYJZT0OSwfyFOoGZq0I3WJGX2C1W tyCYnlNbGaRdtqfDwTSQyVTL6GWOuJsnCOdqjtYyriQFzh/hwCg4h8KMhsVFgveqQFlJ4E5j8J8i9 tJvxqGiIfIPidOSvP/uihZ52ms1iONb5094KDMaJ965PU1NCDqXpAXnAoZfy4KbFNp7XSviqt4Uss FXyyKQke/zNKPAsr1UWpPGQU9LBiCvc5Xf8uk6QFm4OwbZ4I5my5d30kk0WZvCsy2DH8lutmQqqR1 Y7a3hpt8Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jsQDm-00072L-PT; Mon, 06 Jul 2020 12:27:43 +0000 Received: from mail-lj1-x242.google.com ([2a00:1450:4864:20::242]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jsQDY-0006xD-8Z for linux-arm-kernel@lists.infradead.org; Mon, 06 Jul 2020 12:27:31 +0000 Received: by mail-lj1-x242.google.com with SMTP id t25so40467836lji.12 for ; Mon, 06 Jul 2020 05:27:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f/C71RN3h4v4uYwnG7UoSsXOtVMsTKlvllSWIOMLtDE=; b=bmSgSD/Kg5Mj9uU3Pjm94EyhVuXhyVdt/MR11GFIABHE4WDZq6l5Iu/eXtSDmfKxzH DRBmi/lt/TI1LmaM2H2VBoSWQ7L/2xY//hvGFT8kpshkPF6Er0vu4hQ8/6xNdCaDWLwz rnscWbdEsgqi4j5zUL9Hb9s80S2o+YNfmlRj01HC0bBYU1W3n5HcwgZ8/e5TDAdj/lhM i0Vqk8E9c/MgXbaXq43v46QJFZI6rfLnpOpdrxdQkctEhVNug+JKoKIdoVklr2gfDfxO JcrHoBunLW7GTxqud23r6Q1GjihfadY0ehhsSAS7+3PJAIsMv+RbE/QBpn3XyyEvVhjI zLtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f/C71RN3h4v4uYwnG7UoSsXOtVMsTKlvllSWIOMLtDE=; b=hOXByL0HNZ/ola1Pu7x6h5qwLl1mXv7va4aufIBtQelyLBlbkKtTCzF5J+36WdGfwj Bb9s3LArzKwP5ZD8AUnPkbJB//+zV6fFNgppu+k8SvuYbQ6jpafpVJ4Gw71UITiuiLXW 04C2a35N+B9g5tPWlQNjFxpsSF+yXefL14eWNLbsUed34+UZIHvAJqa1Z5oyhstLLMie BMl5OSndn12lvyvLu8gjtzhb1Aw1cDmLymBfQv1FmRT5gvn1aHJKxQFdr9fU3iSO5MO7 Dha+0KFHwVeSqSCx8vXQvVCba388CsfbG/aAqsHyW8M7bzV+VagBO/RTfzOD61BXSVd+ B79Q== X-Gm-Message-State: AOAM532an4U8Y5vwq16yH4t1lln1YABAwcjII4l0kvRcYO7jmOsvDuQR 9S+DIA278daK+cBC5pJT1ErVAw== X-Google-Smtp-Source: ABdhPJyEHVKMlRCeT7KniEtc7sMH+1GminbKE/h1p1ngK5HykNRZK2/HrfxeDxwBj4nMRWwfA/qFgA== X-Received: by 2002:a2e:91ca:: with SMTP id u10mr21806516ljg.101.1594038446983; Mon, 06 Jul 2020 05:27:26 -0700 (PDT) Received: from localhost.localdomain (c-92d7225c.014-348-6c756e10.bbcust.telenor.se. [92.34.215.146]) by smtp.gmail.com with ESMTPSA id v20sm8534223lfr.74.2020.07.06.05.27.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jul 2020 05:27:26 -0700 (PDT) From: Linus Walleij To: Florian Fainelli , Abbott Liu , Russell King , Ard Biesheuvel , Andrey Ryabinin , Mike Rapoport Subject: [PATCH 4/5 v12] ARM: Initialize the mapping of KASan shadow memory Date: Mon, 6 Jul 2020 14:24:46 +0200 Message-Id: <20200706122447.696786-5-linus.walleij@linaro.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200706122447.696786-1-linus.walleij@linaro.org> References: <20200706122447.696786-1-linus.walleij@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200706_082729_068305_27966E95 X-CRM114-Status: GOOD ( 37.59 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:242 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Arnd Bergmann , Linus Walleij , kasan-dev@googlegroups.com, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This patch initializes KASan shadow region's page table and memory. There are two stage for KASan initializing: 1. At early boot stage the whole shadow region is mapped to just one physical page (kasan_zero_page). It is finished by the function kasan_early_init which is called by __mmap_switched(arch/arm/kernel/ head-common.S) 2. After the calling of paging_init, we use kasan_zero_page as zero shadow for some memory that KASan does not need to track, and we allocate a new shadow space for the other memory that KASan need to track. These issues are finished by the function kasan_init which is call by setup_arch. When using KASan we also need to increase the THREAD_SIZE_ORDER from 1 to 2 as the extra calls for shadow memory uses quite a bit of stack. As we need to make a temporary copy of the PGD when setting up shadow memory we create a helpful PGD_SIZE definition for both LPAE and non-LPAE setups. The KASan core code unconditionally calls pud_populate() so this needs to be changed from BUG() to do {} while (0) when building with KASan enabled. After the initial development by Andre Ryabinin several modifications have been made to this code: Abbott Liu - Add support ARM LPAE: If LPAE is enabled, KASan shadow region's mapping table need be copied in the pgd_alloc() function. - Change kasan_pte_populate,kasan_pmd_populate,kasan_pud_populate, kasan_pgd_populate from .meminit.text section to .init.text section. Reported by Florian Fainelli Linus Walleij : - Drop the custom mainpulation of TTBR0 and just use cpu_switch_mm() to switch the pgd table. - Adopt to handle 4th level page tabel folding. - Rewrite the entire page directory and page entry initialization sequence to be recursive based on ARM64:s kasan_init.c. Cc: Alexander Potapenko Cc: Dmitry Vyukov Cc: kasan-dev@googlegroups.com Cc: Mike Rapoport Co-developed-by: Andrey Ryabinin Co-developed-by: Abbott Liu Co-developed-by: Ard Biesheuvel Reviewed-by: Ard Biesheuvel Tested-by: Ard Biesheuvel # QEMU/KVM/mach-virt/LPAE/8G Reported-by: Russell King - ARM Linux Reported-by: Florian Fainelli Signed-off-by: Andrey Ryabinin Signed-off-by: Abbott Liu Signed-off-by: Florian Fainelli Signed-off-by: Linus Walleij Acked-by: Mike Rapoport --- ChangeLog v11->v12: - Do not try to shadow highmem memory blocks. (Ard) - Provoke a build bug if the entire shadow memory doesn't fit inside a single pgd_index() (Ard) - Move the pointer to (unsigned long) casts into the create_mapping() function. (Ard) - After setting up the shadow memory make sure to issue local_flush_tlb_all() so that we refresh all the global mappings. (Ard) - Simplify pte_populate() (Ard) - Skip over pud population as well as p4d. (Ard) - Drop the stop condition pmd_none(*pmdp) in the pmd population loop. (Ard) - Stop passing around the node (NUMA) parameter in the init code, we are not expecting any NUMA architectures to be introduced into ARM32 so just hardcode NUMA_NO_NODE when calling memblock_alloc_try_nid(). ChangeLog v10->v11: - Fix compilation on LPAE systems. - Move the check for valid pgdp, pudp and pmdp into the loop for each level moving over the directory pointers: we were just lucky that we just needed one directory for each level so this fixes the pmdp issue with LPAE and KASan now works like a charm on LPAE as well. - Fold fourth level page directory (p4d) into the global page directory pgd and just skip into the page upper directory (pud) directly. We do not anticipate that ARM32 will every use 5-level page tables. - Simplify the ifdeffery around the temporary pgd. - Insert a comment about pud_populate() that is unconditionally called by the KASan core code. ChangeLog v9->v10: - Rebase onto v5.8-rc1 - add support for folded p4d page tables, use the primitives necessary for the 4th level folding, add (empty) walks of p4d level. - Use the header file that has now appeared as part of the VM consolidation series. - Use a recursive method to walk pgd/p4d/pud/pmd/pte instead of the separate early/main calls and the flat call structure used in the old code. This was inspired by the ARM64 KASan init code. - Assume authorship of this code, I have now written the majority of it so the blame is on me and noone else. ChangeLog v8->v9: - Drop the custom CP15 manipulation and cache flushing for swapping TTBR0 and instead just use cpu_switch_mm(). - Collect Ard's tags. ChangeLog v7->v8: - Rebased. ChangeLog v6->v7: - Use SPDX identifer for the license. - Move the TTBR0 accessor calls into this patch. --- arch/arm/include/asm/kasan.h | 32 ++++ arch/arm/include/asm/pgalloc.h | 8 +- arch/arm/include/asm/thread_info.h | 8 + arch/arm/kernel/head-common.S | 3 + arch/arm/kernel/setup.c | 2 + arch/arm/mm/Makefile | 3 + arch/arm/mm/kasan_init.c | 264 +++++++++++++++++++++++++++++ arch/arm/mm/pgd.c | 16 +- 8 files changed, 334 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/kasan.h create mode 100644 arch/arm/mm/kasan_init.c diff --git a/arch/arm/include/asm/kasan.h b/arch/arm/include/asm/kasan.h new file mode 100644 index 000000000000..56b954db160e --- /dev/null +++ b/arch/arm/include/asm/kasan.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * arch/arm/include/asm/kasan.h + * + * Copyright (c) 2015 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin + * + */ + +#ifndef __ASM_KASAN_H +#define __ASM_KASAN_H + +#ifdef CONFIG_KASAN + +#include + +#define KASAN_SHADOW_SCALE_SHIFT 3 + +/* + * The compiler uses a shadow offset assuming that addresses start + * from 0. Kernel addresses don't start from 0, so shadow + * for kernel really starts from 'compiler's shadow offset' + + * ('kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT) + */ + +extern void kasan_init(void); + +#else +static inline void kasan_init(void) { } +#endif + +#endif diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index 069da393110c..3bf1905df9c3 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -21,6 +21,7 @@ #define _PAGE_KERNEL_TABLE (PMD_TYPE_TABLE | PMD_BIT4 | PMD_DOMAIN(DOMAIN_KERNEL)) #ifdef CONFIG_ARM_LPAE +#define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { @@ -39,14 +40,19 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) } #else /* !CONFIG_ARM_LPAE */ +#define PGD_SIZE (PAGE_SIZE << 2) /* * Since we have only two-level page tables, these are trivial */ #define pmd_alloc_one(mm,addr) ({ BUG(); ((pmd_t *)2); }) #define pmd_free(mm, pmd) do { } while (0) +#ifdef CONFIG_KASAN +/* The KASan core unconditionally calls pud_populate() on all architectures */ +#define pud_populate(mm,pmd,pte) do { } while (0) +#else #define pud_populate(mm,pmd,pte) BUG() - +#endif #endif /* CONFIG_ARM_LPAE */ extern pgd_t *pgd_alloc(struct mm_struct *mm); diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h index 3609a6980c34..02813a5d9e10 100644 --- a/arch/arm/include/asm/thread_info.h +++ b/arch/arm/include/asm/thread_info.h @@ -13,7 +13,15 @@ #include #include +#ifdef CONFIG_KASAN +/* + * KASan uses a lot of extra stack space so the thread size order needs to + * be increased. + */ +#define THREAD_SIZE_ORDER 2 +#else #define THREAD_SIZE_ORDER 1 +#endif #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) #define THREAD_START_SP (THREAD_SIZE - 8) diff --git a/arch/arm/kernel/head-common.S b/arch/arm/kernel/head-common.S index 6840c7c60a85..89c80154b9ef 100644 --- a/arch/arm/kernel/head-common.S +++ b/arch/arm/kernel/head-common.S @@ -111,6 +111,9 @@ __mmap_switched: str r8, [r2] @ Save atags pointer cmp r3, #0 strne r10, [r3] @ Save control register values +#ifdef CONFIG_KASAN + bl kasan_early_init +#endif mov lr, #0 b start_kernel ENDPROC(__mmap_switched) diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index d8e18cdd96d3..b0820847bb92 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -58,6 +58,7 @@ #include #include #include +#include #include "atags.h" @@ -1130,6 +1131,7 @@ void __init setup_arch(char **cmdline_p) early_ioremap_reset(); paging_init(mdesc); + kasan_init(); request_standard_resources(mdesc); if (mdesc->restart) diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 99699c32d8a5..4536159bc8fa 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -113,3 +113,6 @@ obj-$(CONFIG_CACHE_L2X0_PMU) += cache-l2x0-pmu.o obj-$(CONFIG_CACHE_XSC3L2) += cache-xsc3l2.o obj-$(CONFIG_CACHE_TAUROS2) += cache-tauros2.o obj-$(CONFIG_CACHE_UNIPHIER) += cache-uniphier.o + +KASAN_SANITIZE_kasan_init.o := n +obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c new file mode 100644 index 000000000000..b040f4de6b15 --- /dev/null +++ b/arch/arm/mm/kasan_init.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file contains kasan initialization code for ARM. + * + * Copyright (c) 2018 Samsung Electronics Co., Ltd. + * Author: Andrey Ryabinin + * Author: Linus Walleij + */ + +#define pr_fmt(fmt) "kasan: " fmt +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "mm.h" + +static pgd_t tmp_pgd_table[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); + +pmd_t tmp_pmd_table[PTRS_PER_PMD] __page_aligned_bss; + +static __init void *kasan_alloc_block(size_t size) +{ + return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS), + MEMBLOCK_ALLOC_KASAN, NUMA_NO_NODE); +} + +static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, + unsigned long end, bool early) +{ + unsigned long next; + pte_t *ptep = pte_offset_kernel(pmdp, addr); + + do { + pte_t entry; + + next = addr + PAGE_SIZE; + + if (!early) { + void *p = kasan_alloc_block(PAGE_SIZE); + if (!p) { + panic("%s failed to alloc pte for address 0x%lx\n", + __func__, addr); + return; + } + memset(p, KASAN_SHADOW_INIT, PAGE_SIZE); + entry = pfn_pte(virt_to_pfn(p), + __pgprot(pgprot_val(PAGE_KERNEL))); + } else if (pte_none(READ_ONCE(*ptep))) { + /* + * The early shadow memory is mapping all KASan + * operations to one and the same page in memory, + * "kasan_early_shadow_page" so that the instrumentation + * will work on a scratch area until we can set up the + * proper KASan shadow memory. + */ + entry = pfn_pte(virt_to_pfn(kasan_early_shadow_page), + __pgprot(_L_PTE_DEFAULT | L_PTE_DIRTY | L_PTE_XN)); + } else { + /* + * Early shadow mappings are PMD_SIZE aligned, so if the + * first entry is already set, they must all be set. + */ + return; + } + + set_pte_at(&init_mm, addr, ptep, entry); + } while (ptep++, addr = next, addr != end); +} + +/* + * The pmd (page middle directory) is only used on LPAE + */ +static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, + unsigned long end, bool early) +{ + unsigned long next; + pmd_t *pmdp = pmd_offset(pudp, addr); + + do { + if (pmd_none(*pmdp)) { + void *p = early ? kasan_early_shadow_pte : + kasan_alloc_block(PAGE_SIZE); + + if (!p) { + panic("%s failed to allocate pmd for address 0x%lx\n", + __func__, addr); + return; + } + pmd_populate_kernel(&init_mm, pmdp, p); + flush_pmd_entry(pmdp); + } + + next = pmd_addr_end(addr, end); + kasan_pte_populate(pmdp, addr, next, early); + } while (pmdp++, addr = next, addr != end); +} + +static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, + bool early) +{ + unsigned long next; + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp; + + pgdp = pgd_offset_k(addr); + + do { + /* Allocate and populate the PGD if it doesn't already exist */ + if (!early && pgd_none(*pgdp)) { + void *p = kasan_alloc_block(PAGE_SIZE); + + if (!p) { + panic("%s failed to allocate pgd for address 0x%lx\n", + __func__, addr); + return; + } + pgd_populate(&init_mm, pgdp, p); + } + + next = pgd_addr_end(addr, end); + /* + * We just immediately jump over the p4d and pud page + * directories since we believe ARM32 will never gain four + * nor five level page tables. + */ + p4dp = p4d_offset(pgdp, addr); + pudp = pud_offset(p4dp, addr); + + kasan_pmd_populate(pudp, addr, next, early); + } while (pgdp++, addr = next, addr != end); +} + +extern struct proc_info_list *lookup_processor_type(unsigned int); + +void __init kasan_early_init(void) +{ + struct proc_info_list *list; + + /* + * locate processor in the list of supported processor + * types. The linker builds this table for us from the + * entries in arch/arm/mm/proc-*.S + */ + list = lookup_processor_type(read_cpuid_id()); + if (list) { +#ifdef MULTI_CPU + processor = *list->proc; +#endif + } + + BUILD_BUG_ON((KASAN_SHADOW_END - (1UL << 29)) != KASAN_SHADOW_OFFSET); + /* + * We walk the page table and set all of the shadow memory to point + * to the scratch page. + */ + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, true); +} + +static void __init clear_pgds(unsigned long start, + unsigned long end) +{ + for (; start && start < end; start += PMD_SIZE) + pmd_clear(pmd_off_k(start)); +} + +static int __init create_mapping(void *start, void *end) +{ + pr_info("populating shadow for %px to %px\n", start, end); + kasan_pgd_populate((unsigned long)start & PAGE_MASK, + (unsigned long)end, false); + return 0; +} + +void __init kasan_init(void) +{ + struct memblock_region *reg; + int i; + + /* + * We are going to perform proper setup of shadow memory. + * + * At first we should unmap early shadow (clear_pgds() call bellow). + * However, instrumented code couldn't execute without shadow memory. + * + * To keep the early shadow memory MMU tables around while setting up + * the proper shadow memory, we copy swapper_pg_dir (the initial page + * table) to tmp_pgd_table and use that to keep the early shadow memory + * mapped until the full shadow setup is finished. Then we swap back + * to the proper swapper_pg_dir. + */ + + memcpy(tmp_pgd_table, swapper_pg_dir, sizeof(tmp_pgd_table)); +#ifdef CONFIG_ARM_LPAE + /* We need to be in the same PGD or this won't work */ + BUILD_BUG_ON(pgd_index(KASAN_SHADOW_START) != + pgd_index(KASAN_SHADOW_END)); + memcpy(tmp_pmd_table, + pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_START)), + sizeof(tmp_pmd_table)); + set_pgd(&tmp_pgd_table[pgd_index(KASAN_SHADOW_START)], + __pgd(__pa(tmp_pmd_table) | PMD_TYPE_TABLE | L_PGD_SWAPPER)); +#endif + cpu_switch_mm(tmp_pgd_table, &init_mm); + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + + kasan_populate_early_shadow(kasan_mem_to_shadow((void *)VMALLOC_START), + kasan_mem_to_shadow((void *)-1UL) + 1); + + for_each_memblock(memory, reg) { + void *start = __va(reg->base); + void *end = __va(reg->base + reg->size); + + /* Do not attempt to shadow highmem */ + if (reg->base >= arm_lowmem_limit) + continue; + if (reg->base + reg->size > arm_lowmem_limit) + end = __va(arm_lowmem_limit); + if (start >= end) + continue; + + create_mapping(kasan_mem_to_shadow(start), + kasan_mem_to_shadow(end)); + } + + /* + * 1. The module global variables are in MODULES_VADDR ~ MODULES_END, + * so we need to map this area. + * 2. PKMAP_BASE ~ PKMAP_BASE+PMD_SIZE's shadow and MODULES_VADDR + * ~ MODULES_END's shadow is in the same PMD_SIZE, so we can't + * use kasan_populate_zero_shadow. + */ + create_mapping( + kasan_mem_to_shadow((void *)MODULES_VADDR), + kasan_mem_to_shadow((void *)(PKMAP_BASE + PMD_SIZE))); + + /* + * KAsan may reuse the contents of kasan_early_shadow_pte directly, so + * we should make sure that it maps the zero page read-only. + */ + for (i = 0; i < PTRS_PER_PTE; i++) + set_pte_at(&init_mm, KASAN_SHADOW_START + i*PAGE_SIZE, + &kasan_early_shadow_pte[i], + pfn_pte(virt_to_pfn(kasan_early_shadow_page), + __pgprot(pgprot_val(PAGE_KERNEL) + | L_PTE_RDONLY))); + local_flush_tlb_all(); + + memset(kasan_early_shadow_page, 0, PAGE_SIZE); + cpu_switch_mm(swapper_pg_dir, &init_mm); + pr_info("Kernel address sanitizer initialized\n"); + init_task.kasan_depth = 0; +} diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index c5e1b27046a8..f8e9bc58a84f 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -66,7 +66,21 @@ pgd_t *pgd_alloc(struct mm_struct *mm) new_pmd = pmd_alloc(mm, new_pud, 0); if (!new_pmd) goto no_pmd; -#endif +#ifdef CONFIG_KASAN + /* + * Copy PMD table for KASAN shadow mappings. + */ + init_pgd = pgd_offset_k(TASK_SIZE); + init_p4d = p4d_offset(init_pgd, TASK_SIZE); + init_pud = pud_offset(init_p4d, TASK_SIZE); + init_pmd = pmd_offset(init_pud, TASK_SIZE); + new_pmd = pmd_offset(new_pud, TASK_SIZE); + memcpy(new_pmd, init_pmd, + (pmd_index(MODULES_VADDR) - pmd_index(TASK_SIZE)) + * sizeof(pmd_t)); + clean_dcache_area(new_pmd, PTRS_PER_PMD * sizeof(pmd_t)); +#endif /* CONFIG_KASAN */ +#endif /* CONFIG_LPAE */ if (!vectors_high()) { /*