From patchwork Fri Jun 24 15:06:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39497C433EF for ; Fri, 24 Jun 2022 15:15:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cmIOCwTHggcJJSJcIMyBD7RXT8R5w57azCGxd/plSW8=; b=cfnc1alMVyRypp eOYklCoM8xHcNnBvc6x+bFhzMdwLlRMG7b0gk4eFv/aq2d1MvsdM3FKYRPkWewAv/T2+MVHtnjJ/v 5XvFP0NQ5bmoOTrUwaWdXlZ3ps0vHqDQeqAtq8dwKVLiWpknCBlGehAotJt5dp50oAPEaTIVY0ynN ISCsTt2fa/cHaZRAGP2las1fDjbNBMTyWfWoaOu4mZIO20lZwhTFC1Z3wak+XqZl/rvZ+Moaoqo87 v2YkMxzrLjTbY6VYSe14qVGvhVkbY51DekLFesElgmWKLitOi1LqYCPn7fyzObkVamq5ArNdBTj8z U2TRl6TbCF/0mcY89BrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l16-002m0c-KF; Fri, 24 Jun 2022 15:14:41 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuX-002iXy-NO for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:55 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6C039B8293D; Fri, 24 Jun 2022 15:07:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39CFFC341C6; Fri, 24 Jun 2022 15:07:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083271; bh=/E6XrsNXjjeer9NBBlG34L/h2YChbTIYfexHwIflbPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YD43GMQ8f564A7hM2PttmQ9Gy1q79FedJgPCUsYL1KK9Pm2MHlJ7MO+Iznj0e1PHV sExUR/uWMvwVnO+lRvUQoohUiruCLuNXDTg862kDzJPudYgXwCKbB7gcv6vUnfvQL3 wiOTXuAhoQ154XhcE2JI2g7zwqT9Dj4scYYtxPWxB+tf4ESAlNZgtdnLZu7oRD8Eiv BqVGrWKJhtj96T1W4d1VKZkkvXywaIvx+7US4HAH0BBoaGBVAcvGMTL3zwayh/NWvJ bzefVeBOs1PktRdTNEeBjnb5riup/qsmtxRgy25MyGn7AabgQBIDqd6le/eXn2sOoP HD6hF8ZYDFUlg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 18/21] arm64: head: record CPU boot mode after enabling the MMU Date: Fri, 24 Jun 2022 17:06:48 +0200 Message-Id: <20220624150651.1358849-19-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5885; h=from:subject; bh=/E6XrsNXjjeer9NBBlG34L/h2YChbTIYfexHwIflbPQ=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMFCBWo5bs9hbAAudtVFUyluBkT4K9TRYdc8x/W KBqyrjKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTBQAKCRDDTyI5ktmPJBQXC/ 9aNqhEzNPrQJsilJ1rguCd9n5lskGC+ufVwWkmBD+31n2vMMcjTpBXi+PCp3mDUdYXghUo/5ZnrYzn BRIyRiP10wff855Ky7LcaovtQgFqbaTKezeN2JZSdXohST5z1fOToGWigl7Vsj+vYcGx8iIKYdBaw7 30omdx+K8BQHDpGrWHknLuxYvm3K2XrE29YWbljdrm7XnD6YiFTUfaumECfcnjPncgcVCeNpKTdngG nnMqZrl7ZxEDW98JEIC1fA1RIdeVtuSdt4xFvsZ2K/Ea1JeGK3rC//67tMcMGY1vabieykApSJLfSV hCdCii533mdGI5WcSqWdJiTjq3jORR/HmONCeiblEfWOwDttvqlXoYfic0rWTmkrIc7AYPyTl3EJHf sOmpwQINrEiwI1XTkGLAbjNqzUx1MKHLt6YHhu3G1ryCAcG/20xecSprO9VHoK4hVD9rJ8L+bSH37s /98toi5bRB8UHjenIBQMTg8wTe6UEASwIx1tkkZ0dzdpE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080754_119435_A2B2196C X-CRM114-Status: GOOD ( 18.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to avoid having to touch memory with the MMU and caches disabled, and therefore having to invalidate it from the caches explicitly, just defer storing the value until after the MMU has been turned on, unless we are giving up with an error. While at it, move the associated variable definitions into C code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 50 +++++--------------- arch/arm64/kernel/hyp-stub.S | 4 +- arch/arm64/mm/mmu.c | 8 ++++ 3 files changed, 23 insertions(+), 39 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 583cbea865e1..8de346dd4470 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,6 +82,7 @@ * primary lowlevel boot path: * * Register Scope Purpose + * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset @@ -91,9 +92,9 @@ SYM_CODE_START(primary_entry) bl preserve_boot_args bl init_kernel_el // w0=cpu_boot_mode + mov x20, x0 adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 - bl set_cpu_boot_mode_flag bl create_idmap /* @@ -429,6 +430,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappings + mov x0, x20 + bl set_cpu_boot_mode_flag + // Clear BSS adr_l x0, __bss_start mov x1, xzr @@ -454,6 +458,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) ret // to __primary_switch() 0: #endif + mov x0, x20 bl switch_to_vhe // Prefer VHE if possible ldp x29, x30, [sp], #16 bl start_kernel @@ -553,52 +558,21 @@ SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) b.ne 1f add x1, x1, #4 1: str w0, [x1] // Save CPU boot mode - dmb sy - dc ivac, x1 // Invalidate potentially stale cache line ret SYM_FUNC_END(set_cpu_boot_mode_flag) -/* - * These values are written with the MMU off, but read with the MMU on. - * Writers will invalidate the corresponding address, discarding up to a - * 'Cache Writeback Granule' (CWG) worth of data. The linker script ensures - * sufficient alignment that the CWG doesn't overlap another section. - */ - .pushsection ".mmuoff.data.write", "aw" -/* - * We need to find out the CPU boot mode long after boot, so we need to - * store it in a writable variable. - * - * This is not in .bss, because we set it sufficiently early that the boot-time - * zeroing of .bss would clobber it. - */ -SYM_DATA_START(__boot_cpu_mode) - .long BOOT_CPU_MODE_EL2 - .long BOOT_CPU_MODE_EL1 -SYM_DATA_END(__boot_cpu_mode) -/* - * The booting CPU updates the failed status @__early_cpu_boot_status, - * with MMU turned off. - */ -SYM_DATA_START(__early_cpu_boot_status) - .quad 0 -SYM_DATA_END(__early_cpu_boot_status) - - .popsection - /* * This provides a "holding pen" for platforms to hold all secondary * cores are held until we're ready for them to initialise. */ SYM_FUNC_START(secondary_holding_pen) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag - mrs x0, mpidr_el1 + mrs x2, mpidr_el1 mov_q x1, MPIDR_HWID_BITMASK - and x0, x0, x1 + and x2, x2, x1 adr_l x3, secondary_holding_pen_release pen: ldr x4, [x3] - cmp x4, x0 + cmp x4, x2 b.eq secondary_startup wfe b pen @@ -610,7 +584,6 @@ SYM_FUNC_END(secondary_holding_pen) */ SYM_FUNC_START(secondary_entry) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag b secondary_startup SYM_FUNC_END(secondary_entry) @@ -618,6 +591,7 @@ SYM_FUNC_START_LOCAL(secondary_startup) /* * Common entry point for secondary CPUs. */ + mov x20, x0 // preserve boot mode bl switch_to_vhe bl __cpu_secondary_check52bitva bl __cpu_setup // initialise processor @@ -629,6 +603,9 @@ SYM_FUNC_START_LOCAL(secondary_startup) SYM_FUNC_END(secondary_startup) SYM_FUNC_START_LOCAL(__secondary_switched) + mov x0, x20 + bl set_cpu_boot_mode_flag + str_l xzr, __early_cpu_boot_status, x3 adr_l x5, vectors msr vbar_el1, x5 isb @@ -691,7 +668,6 @@ SYM_FUNC_START(__enable_mmu) b.lt __no_granule_support cmp x3, #ID_AA64MMFR0_TGRAN_SUPPORTED_MAX b.gt __no_granule_support - update_early_cpu_boot_status 0, x3, x4 phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 load_ttbr1 x1, x1, x3 diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 43d212618834..5bafb53fafb4 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -223,11 +223,11 @@ SYM_FUNC_END(__hyp_reset_vectors) /* * Entry point to switch to VHE if deemed capable + * + * w0: boot mode, as returned by init_kernel_el() */ SYM_FUNC_START(switch_to_vhe) // Need to have booted at EL2 - adr_l x1, __boot_cpu_mode - ldr w0, [x1] cmp w0, #BOOT_CPU_MODE_EL2 b.ne 1f diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e17eacfa048c..a723bd2cfc27 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -56,6 +56,14 @@ EXPORT_SYMBOL(kimage_vaddr); u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); +u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; + +/* + * The booting CPU updates the failed status @__early_cpu_boot_status, + * with MMU turned off. + */ +long __section(".mmuoff.data.write") __early_cpu_boot_status; + /* * Empty_zero_page is a special page that is used for zero-initialized data * and COW.