From patchwork Fri Jul 1 13:04:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12903286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC83BC43334 for ; Fri, 1 Jul 2022 13:09:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cf6LOfVadjK+IIf2oJjfoVxX2an/7w4e6Waq9GmEvdg=; b=xrMVA79XFjK8ti hS+cKI+n9MGHu+tiJFRbb4o6hJmmQoI+norEGPPMIUn2fqfnAXOO5S2bdb5U6NBZ4kcDjI+Yh2OC5 Xunme7AmP6hgalYUiuhueZiwLW0E3D00Bssy5FFZMx9u+u0it1dPT0HC9IIzYcy78z41MeU4FVR5X phQazD8DwPdckKhRXkbPVL5iwLkkWgsfQpJ4f3T3X/VS2710tzTZYOE3oVdDYqNo7ERW00AxN9W5Q NbCF6Gn7cHyK/WL/5dZQ9AwbIQbCAhGw+LIARYd5O/6AEs3Kpq6qemcJa9Tk/pXYZBj+cUDxB1qMJ em1ZctP8VsSFmRNvXn4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o7GNH-004wxF-Kq; Fri, 01 Jul 2022 13:07:57 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o7GKg-004vmz-Rn for linux-arm-kernel@lists.infradead.org; Fri, 01 Jul 2022 13:05:17 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E4D88618EA; Fri, 1 Jul 2022 13:05:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 633FEC341CB; Fri, 1 Jul 2022 13:05:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656680713; bh=++CT+Sjb8g2KWnNyKa+UaDsKHyFYhtruAzxxX5Dm6YI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F7pQabnUhxtJJyRd1WbSACUkN8hWN6pcU/NLYN69BIpwEumgiT5W0vWhSMncIiDHC niJZ+UK3nLH53xPgxK63ROgBI1rtkviCmbxUHgUpFZ4ZFODfucb9o3RJEXRM9cO21k j1YwpM1dWJp1X7EhQjfBFMMEZjXF14eVgl7uqW8eyx7mtZoq1NXBwtMANgMJfHmaLq Oxe4xJcBfLEkLxRyoZji+GYIyxUWxwIyntOiXOfjrKMrB5VfKX8+/DXuZi7hZf89RK pgJKd8E3+Q4o1Sg0IJCudYU3GynyIRpLzvAPdidvqUHJAVp0Spg6KYnnuqDKlgzpy/ A+eWVwv1yCWwA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v6 7/9] arm64: mm: omit redundant remap of kernel image Date: Fri, 1 Jul 2022 15:04:42 +0200 Message-Id: <20220701130444.2945106-8-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220701130444.2945106-1-ardb@kernel.org> References: <20220701130444.2945106-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8863; h=from:subject; bh=++CT+Sjb8g2KWnNyKa+UaDsKHyFYhtruAzxxX5Dm6YI=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBivvDoeas40JQzA3C9/kTm1/OncNdG+Z5Pj3mGx6hH 9g9LBB+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYr7w6AAKCRDDTyI5ktmPJM44DA CeSYrMlxZOblnuZlr8ok5xaDHteWXVOOryY/qr7dNMRJ2ldDtUKjhMeuuUyupBsan0ckKRMpwHQHx0 aCN4XmETHqLogYqbMbZ0z/+YV1MoS2yFMQa0D0TXc8GLoQMq7j6eS7TkiYGK4u6SJzu2wAMCFHEO7Q ARR3g7qI2h9Ekmft6UHi2qtiCgaoeHWoKTDF2NXeXwVXZnF5foz21DxN+mMDzXQUQysyNkbLhRsenM VIZ2IXp+LS4lICCVKWUMi5y9L3zvg97L1vl/NFq9LsJeY9L4XBX8+nQhXhB6YOnNxBns1Qhe62Ii6e gdcvgxjoJY6nvdwCHtYXIo8zmSOUylBUZgsLQ7G9M2KZE0InI9MDSmdP86ZackntiwH0o390mJEWJQ 5qNWt4vLVJNHXq+gQwVppPhu5pPvHcYkSMQnriuPMjDEKJDpIz8kk4aaUNYX58T7xMvgLCdJShxcOC NHTDJBKz0Dvz1qZ/47umYs/QtciAtNHsdzOrw+RhiB8+E= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220701_060515_010130_259A3AF4 X-CRM114-Status: GOOD ( 26.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/early_map_kernel.c | 4 + arch/arm64/mm/kasan_init.c | 15 --- arch/arm64/mm/mmu.c | 101 +++----------------- 6 files changed, 17 insertions(+), 108 deletions(-) diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 12d5f47f7dbe..ab52688ac4bd 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -36,12 +36,10 @@ void kasan_init(void); #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow(pgd_t *pgdir); asmlinkage void kasan_early_init(void); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 48f8466a4be9..a93d495d6e8c 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -73,7 +73,7 @@ extern void mark_linear_text_alias_ro(void); extern bool kaslr_requires_kpti(void); #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 032c3e1aff20..b010cba48916 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -59,6 +59,7 @@ PROVIDE(__pi___memset = __pi_memset); */ PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/early_map_kernel.c b/arch/arm64/kernel/pi/early_map_kernel.c index 4199548584fb..b2e660a1d003 100644 --- a/arch/arm64/kernel/pi/early_map_kernel.c +++ b/arch/arm64/kernel/pi/early_map_kernel.c @@ -335,6 +335,10 @@ static void map_kernel(void *fdt, u64 kaslr_offset, u64 va_offset) text_prot, false); dsb(ishst); } + + // Copy the root page table to its final location + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } asmlinkage u64 early_map_kernel(void *fdt) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index e969e68de005..df98f496539f 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 8965f621e2c3..0a23f4f14f99 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -633,9 +633,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(void *va_start, void *va_end, + struct vm_struct *vma, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -643,9 +643,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -708,87 +705,18 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, vmlinux_initdata, vmlinux_data; - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); - - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdp, FIXADDR_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdp, FIXADDR_START), - READ_ONCE(*pgd_offset_k(FIXADDR_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdp, FIXADDR_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } - - kasan_copy_shadow(pgdp); + declare_vma(_stext, _etext, &vmlinux_text, VM_NO_GUARD); + declare_vma(__start_rodata, __inittext_begin, &vmlinux_rodata, VM_NO_GUARD); + declare_vma(__inittext_begin, __inittext_end, &vmlinux_inittext, VM_NO_GUARD); + declare_vma(__initdata_begin, __initdata_end, &vmlinux_initdata, VM_NO_GUARD); + declare_vma(_data, _end, &vmlinux_data, 0); } static void __init create_idmap(void) @@ -824,24 +752,17 @@ static void __init create_idmap(void) void __init paging_init(void) { pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); map_mem(pgdp); pgd_clear_fixmap(); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); - memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } /*