From patchwork Fri Jun 24 15:06:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1FA25C43334 for ; Fri, 24 Jun 2022 15:15:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CWNqazggIztN9vf3X88U1F4wlKyWmFqw7jH7j1DVc40=; b=AZTuVfguvlAmam BI5DybMgDpqSaAOUQE9znt5dHhEjW20HSJEUAxo7NpE/JcWD08z2tpAisuwQu+AuiPa878XuRYqL1 vZ6jocfmuBjOm8U3ouSY2szRuM06t/GkbOpLgDrRpzsJKZqPV/9JTzmxZLwjo2Bi0bheNb9TO8Cik Y4DrBubtCLK3wkwZzUpP7EiXf+rwiXWBouX+2FxY5T7aNS7HWLkbHKp0mhlEMAs3DjhuPmTq9Tj8W qJ6NDhfxrYkVVxlMpxiPu0AOTNwKTSOgOL39sgPkJ+n5Ot79FItDb+8Jw+zhEH5DsI7WLSC4cFyt9 3E9qkKmoOF0zol0vkORQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l0Y-002lkd-9x; Fri, 24 Jun 2022 15:14:06 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuT-002iWo-Pj for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:51 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5CCD961FE1; Fri, 24 Jun 2022 15:07:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E1C70C36AEA; Fri, 24 Jun 2022 15:07:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083268; bh=XPxRHNk/ubPTm/4oa5ivGOXgzzl0krSBgTYBVHZzn/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jiUCva+QQ+Vna37qcxIzP+wtHu5m2pa3EHAuRPl+gUUDqkUgXvTLPbYVwvWLsTnR1 z8wU5xPPUbYvApJKAhxTS2EfPZfVDdSoN6DE/ojdpm5Njn6KayFssHh6SNAZix+R2E 25MZBVrAdDH2Zo27QxGX63fFmK1JUzXfHA/ZeAyWUPEpe2J3HhHNjrachpuqI1d6u7 onU7Tw2hl8P/zInM5M4F9wmCrz5VNF4HrxIPMQkpxO6+gFWBdLPIinBw8+HDErbyuv 0LOypCgBFzrSSz7cju0Uw+1/2yWFRXUNaCp50DFcWaOsq6ptRXgtmK4CXl+pjGKyij a5IYhEMh6ZC9g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 17/21] arm64: head: populate kernel page tables with MMU and caches on Date: Fri, 24 Jun 2022 17:06:47 +0200 Message-Id: <20220624150651.1358849-18-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4295; h=from:subject; bh=XPxRHNk/ubPTm/4oa5ivGOXgzzl0krSBgTYBVHZzn/w=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMDwPEE9uMR6lvdSMLCOrfG7zwhmejFVjDKWlLy Lvmjm3WJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTAwAKCRDDTyI5ktmPJCqEDA CefRTkTMyqOeaRdyqZ5ASJCwL+e4kLONFpAOv7y2smlDhz2IY/+H17m5AEmtWnKWf3sVycS/cFv1aS +q0rlQ4b8ZEa+7BYC7HkyZEa4mCqjAAnn6S+ygMlndmk67glOucWNH5PHYKfIqtZAjoPbcDuH37s8X vPYppLqoybhynYpJbdR7w8+TI/JnMxjwID2POZhN4tpVzei9Ew84elc/2znRwVJ/nD5d3GbCpytsuB O9pSs0wFGNUpumomF3F/7EmiwwuuR9YM3Flgsg4FlUpMZTp6LLz/l3MQChnra4gRFdhVKjgVLKC+Df moJ3uvshQBgCGoNMhNV4oJhYv3uzJnG1haYe4sV90BwkGdmHBta5J7vZlNmkdCTqkzY0lzxexnp93V XRSM4oQA3kAzc70Zlrkr9Eq7/YOSyYelm5ethS2FWvFpH4dY9dLkFRI2HjHTBvzlxYcH1KxEYeAt6n 4tJDl3jd29BzKvP/UDQ71L52Npj0bgfWqWHsRoPt44hII= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080749_972248_E3522A53 X-CRM114-Status: GOOD ( 15.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we can access the entire kernel image via the ID map, we can execute the page table population code with the MMU and caches enabled. The only thing we need to ensure is that translations via TTBR1 remain disabled while we are updating the page tables the second time around, in case KASLR wants them to be randomized. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 62 +++++--------------- 1 file changed, 16 insertions(+), 46 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index d704d0bd8ffc..583cbea865e1 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -85,8 +85,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset - * x28 clear_page_tables() callee preserved temp register - * x19/x20 __primary_switch() callee preserved temp registers * x24 __primary_switch() .. relocate_kernel() current RELR displacement * x28 create_idmap() callee preserved temp register */ @@ -96,9 +94,7 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag - bl clear_page_tables bl create_idmap - bl create_kernel_mapping /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -128,32 +124,14 @@ SYM_CODE_START_LOCAL(preserve_boot_args) SYM_CODE_END(preserve_boot_args) SYM_FUNC_START_LOCAL(clear_page_tables) - mov x28, lr - - /* - * Invalidate the init page tables to avoid potential dirty cache lines - * being evicted. Other page tables are allocated in rodata as part of - * the kernel image, and thus are clean to the PoC per the boot - * protocol. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - bl dcache_inval_poc - /* * Clear the init page tables. */ adrp x0, init_pg_dir adrp x1, init_pg_end - sub x1, x1, x0 -1: stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - subs x1, x1, #64 - b.ne 1b - - ret x28 + sub x2, x1, x0 + mov x1, xzr + b __pi_memset // tail call SYM_FUNC_END(clear_page_tables) /* @@ -399,16 +377,8 @@ SYM_FUNC_START_LOCAL(create_kernel_mapping) map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - dmb sy - - adrp x0, init_pg_dir - adrp x1, init_pg_end - b dcache_inval_poc // tail call + dsb ishst // sync with page table walker + ret SYM_FUNC_END(create_kernel_mapping) /* @@ -863,14 +833,15 @@ SYM_FUNC_END(__relocate_kernel) #endif SYM_FUNC_START_LOCAL(__primary_switch) -#ifdef CONFIG_RANDOMIZE_BASE - mov x19, x0 // preserve new SCTLR_EL1 value - mrs x20, sctlr_el1 // preserve old SCTLR_EL1 value -#endif - - adrp x1, init_pg_dir + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + bl clear_page_tables + bl create_kernel_mapping + + adrp x1, init_pg_dir + load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE #ifdef CONFIG_RELR mov x24, #0 // no RELR displacement yet @@ -886,9 +857,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) * to take into account by discarding the current kernel mapping and * creating a new one. */ - pre_disable_mmu_workaround - msr sctlr_el1, x20 // disable the MMU - isb + adrp x1, reserved_pg_dir // Disable translations via TTBR1 + load_ttbr1 x1, x1, x2 bl clear_page_tables bl create_kernel_mapping // Recreate kernel mapping @@ -896,8 +866,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) dsb nsh isb - set_sctlr_el1 x19 // re-enable the MMU - + adrp x1, init_pg_dir // Re-enable translations via TTBR1 + load_ttbr1 x1, x1, x2 bl __relocate_kernel #endif #endif