From patchwork Tue Nov 29 16:14:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1E62C4167B for ; Tue, 29 Nov 2022 16:17:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uzZJ2R7uQMyTDaa/GTvvYumhW4SfEDJW/oNC26RK8Z0=; b=JH8rtHY2BPPWDx 7fQSjb5JlhUFBFAm2hY4UhQFOjV1LDyRu12feIprUxHb2rbQV/upg14tukyschA9fKm+4fy54DRgZ 3pCiGK6a9mIevhM5hnhxEHZpmIZWoNKBNeyKTTLmhOy7xriJ5P4qf8rUaTuXMPgdELsydRieFFdQs 6YFQCTlJuYHWF+1muuvsr233+EOFNMzNFmAExblTpV5YuqUeolOetk0aQjVgj85XBp3a/e9Q6Bdt0 UqSeI7XJOvUHdrvBkSh8LBpTJ0rMjKqdYEUe+No3QcPV4ABk3PYYetzeDI7mrcEEuFjej2wMg6/Dy rvTL5IUiSwYBJACC9BSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03H7-00A0zB-CA; Tue, 29 Nov 2022 16:16:01 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Fa-00A0Wm-LE for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:28 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3DB49617BA; Tue, 29 Nov 2022 16:14:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57DAAC4347C; Tue, 29 Nov 2022 16:14:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738465; bh=DQUF2BaKxU7I7wgW2wQ8NneWctq1uRwv3omzNQ8G4FE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JbvFYT41c131d5vc0rplgNWlhE8Nb9vpxg5Hfu12Z0sXGspYh/CJ+jdqZMTNc9A1f 5PK0Cafa3LLITYnkOhyuRMdP9uApo55LGAdA+5fexmlnK5rKC9Ggc1AW21Bd1vX/wH JeVX5uIX6w1p3E2Wnclncqm1Yixt9DstzeCEPQzCbHc2lBAdOIC+bBB1cxdS8DsvR1 WDSE3ePTourtuuZPAyiwIN2t5b1NWMg1WOYiqm7a3kuU1S77q+XPBdnDYcq62zJiWI +cZ3LwsKll3SFFWXcyJpe98cZ3UD/v+VS7RsRhhSBSz1va3pWYmA4IT04kL9LetFjS P1uSsNz0qHgMg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 1/6] arm64: head: Move all finalise_el2 calls to after __enable_mmu Date: Tue, 29 Nov 2022 17:14:13 +0100 Message-Id: <20221129161418.1968319-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2013; i=ardb@kernel.org; h=from:subject; bh=DQUF2BaKxU7I7wgW2wQ8NneWctq1uRwv3omzNQ8G4FE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/QUc2/m1NXlUs2ES7zPr7Q5sHP0pY1zRsNmdpq lXusGvCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv0AAKCRDDTyI5ktmPJLfFDA CRjlF+skkF3Y0fYfmUUB8yCZJrvzB37TCBEPt0pQkPusb7c3ptns8ZWUax4s49IuJ5iD+Co5n74F/p U4/8FTpSI6yeCn+WM68oGwKVTy1sOVN5Z9mQcY4N6Xj0vOa0tG+mY9ICHxpqWb2mgvTPNLZvnbh8OL be7CefFE890+zPLjoDhcLFHaJ5iJmv3NwRXl5n1ommHW5e1tDIuarS3C6le7++M+bZRGk3O2DY/00/ Hv5WIgzDMaLtx3BjIm4ZsBNTvwYE1dLfA90DEnWmKJM5JAZvncksf3JHhw7zQpHUIS3Tx4uuFcDCYS PUX3WORhBTF0CWkUPO38/Vc/HQrbJ6la2eC+TohW7R/TFlX1r+G3RmccC96wiQK85CwrivBkPnnPXu YyiGwX1dsD9DUHsbJd4vCkHBObwBQUB2AhVHsYDLYnhYntN1n6GSRL64ABixrZGzyQZn6cWymRjiC0 uMXbngjtfVJy4rBDD4bGr3VUrA1ljXP1cBhSG3ecu55eE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081426_821059_BD616557 X-CRM114-Status: GOOD ( 14.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the primary boot path, finalise_el2() is called much later than on the secondary boot or resume-from-suspend paths, and this does not appear to be intentional. Since we aim to do as little as possible before enabling the MMU and caches, align secondary and resume with primary boot, and defer the call to after the MMU is turned on. This also removes the need to clean finalise_el2() to the PoC once we enable support for booting with the MMU on. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 5 ++++- arch/arm64/kernel/sleep.S | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 952e17bd1c0b4f91..c4e12d466a5f35f0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -587,7 +587,6 @@ SYM_FUNC_START_LOCAL(secondary_startup) * Common entry point for secondary CPUs. */ mov x20, x0 // preserve boot mode - bl finalise_el2 bl __cpu_secondary_check52bitva #if VA_BITS > 48 ldr_l x0, vabits_actual @@ -603,6 +602,10 @@ SYM_FUNC_END(secondary_startup) SYM_FUNC_START_LOCAL(__secondary_switched) mov x0, x20 bl set_cpu_boot_mode_flag + + mov x0, x20 + bl finalise_el2 + str_l xzr, __early_cpu_boot_status, x3 adr_l x5, vectors msr vbar_el1, x5 diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 97c9de57725dfddb..7b7c56e048346e97 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -100,7 +100,7 @@ SYM_FUNC_END(__cpu_suspend_enter) .pushsection ".idmap.text", "awx" SYM_CODE_START(cpu_resume) bl init_kernel_el - bl finalise_el2 + mov x19, x0 // preserve boot mode #if VA_BITS > 48 ldr_l x0, vabits_actual #endif @@ -116,6 +116,9 @@ SYM_CODE_END(cpu_resume) .popsection SYM_FUNC_START(_cpu_resume) + mov x0, x19 + bl finalise_el2 + mrs x1, mpidr_el1 adr_l x8, mpidr_hash // x8 = struct mpidr_hash virt address From patchwork Tue Nov 29 16:14:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B7CE8C433FE for ; Tue, 29 Nov 2022 16:18:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UdbPA0f8agb0MhbMS+RzqKnKeGju6+nsbCcgYgAXnt4=; b=ecaXZ1vEK38wO0 CdjJplfWPryqhL5gabiruJS/s/WdCd5/g6km6YNqcw+NvPYC1OkYMGNs1dcmLtLo3I2Kom1Uaakcx lIdmmf8/8RN150dYJ54jTB6x9LMOBDfGRvKLsuB8JlTyWgpW5RCTTl87tVWMyqp69bbUhg33/dM5y vAfHmXH4ZndOBdEEq2DAgxy2gLb9S72CHVv+LVQS1NcdNIY/aT8UceRy984zRIfsueAhUwlZGMg1z 6S5Xj78BKl9LKiuWEN7XUOxW9j6ODx7Etp1rUk5g7RUY6FGXTJrXxWhfNNkX4xUzwGTcUBg55Qoi1 qtCuyc2v7QuRulYNfDUA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03I7-00A1LH-K6; Tue, 29 Nov 2022 16:17:04 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Ff-00A0Xn-0Z for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:32 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id AC946B811B8; Tue, 29 Nov 2022 16:14:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 190DDC43470; Tue, 29 Nov 2022 16:14:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738467; bh=U2iqnVkuhwtUu/qXPBf/2GNjjB270ekS914+aIpYPEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FuZlmyujS1i8rwA8Tfj/9d6TBfTkhYZdnEt8WizSuHihX0SCcgl3cWFb4E5hd6jNv 84ZzeZNhTANzTjClBgrnDyWdcRsHJP8PfPIF1CkcNszkG9qJNbkjtZn5a7PvmAL8zB /1jNIK2Fd3ELYutzV7EcrhpBeKANBYQR2R+aS8AIShmvUxt80qlM26k/ujBIGtxewJ wixwEK/a9bwrcbY3qsiMGoPhlVoHBtb2meRMCfZJPOOiTYXXMiS1WUqc8nmBKyq6DH JBDX3BuhZTMGoiSxkUKBqyj5anrnTPz1kqAnsEuLo47Gxdz+F+2+wyYRb2sbyAN80o V8bJTFMrTIgNQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 2/6] arm64: kernel: move identity map out of .text mapping Date: Tue, 29 Nov 2022 17:14:14 +0100 Message-Id: <20221129161418.1968319-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3476; i=ardb@kernel.org; h=from:subject; bh=U2iqnVkuhwtUu/qXPBf/2GNjjB270ekS914+aIpYPEc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/ShyvMBIlSTQM481D7o9xscX5bWYIvJ8NQL2UN h7H6lBWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv0gAKCRDDTyI5ktmPJCy0C/ 47GESyz9hlte82zWGZkKhoCwqE2ZfqsZ6iTSLe5JeU7/eF31/PhKrKYGtoHDknSjKII0f0LiopKh5x s5+d0sCteS2ClurgvR8LSr5w1Os72fUw9DSk7VFOOuJz0gA0uyoCWOcHjRi7zEohvvmsERu4Bdf5bi sVS8D8q65LZuf1nDUECj+CNgA00X7TjDZPp+V06m1oxgRE9KF+E2sql+E9hK9Mn/FS8I6t7Afk8w7e zTillIcbl14L2VMW8lYBKLVj7eCsUFXac4xa1DUA3HRq6XEYKwp8/W2UVajbIMn5jRefTgi5k7VS/H DNZ+1FrSTwGD8seFVpoyX1fN8q0XK/4vtoj2AYFXcMYMaYBhoI20Hptz4jM0NsF/A2pVHzx0vnlO2V Xujugiv1F/PRr2jDG6NqLeGprZ02+sCSH6BXDnA7MfpNQOR93RYVH3ktxLkA0Yi5j8Fcvnw4BF6dpc zO1elEHETz5gsfS1kBXlsLvQuhnPnnIPv8T8mArxA9OZc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081431_403026_5AFA0FF8 X-CRM114-Status: GOOD ( 14.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Reorganize the ID map slightly so that only code that is executed with the MMU off or via the 1:1 mapping remains. This allows us to move the identity map out of the .text segment, as it will no longer need executable permissions via the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 28 +++++++++++--------- arch/arm64/kernel/vmlinux.lds.S | 2 +- arch/arm64/mm/proc.S | 2 -- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index c4e12d466a5f35f0..bec97aad092c2b43 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -543,19 +543,6 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) eret SYM_FUNC_END(init_kernel_el) -/* - * Sets the __boot_cpu_mode flag depending on the CPU boot mode passed - * in w0. See arch/arm64/include/asm/virt.h for more info. - */ -SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) - adr_l x1, __boot_cpu_mode - cmp w0, #BOOT_CPU_MODE_EL2 - b.ne 1f - add x1, x1, #4 -1: str w0, [x1] // Save CPU boot mode - ret -SYM_FUNC_END(set_cpu_boot_mode_flag) - /* * This provides a "holding pen" for platforms to hold all secondary * cores are held until we're ready for them to initialise. @@ -599,6 +586,7 @@ SYM_FUNC_START_LOCAL(secondary_startup) br x8 SYM_FUNC_END(secondary_startup) + .text SYM_FUNC_START_LOCAL(__secondary_switched) mov x0, x20 bl set_cpu_boot_mode_flag @@ -631,6 +619,19 @@ SYM_FUNC_START_LOCAL(__secondary_too_slow) b __secondary_too_slow SYM_FUNC_END(__secondary_too_slow) +/* + * Sets the __boot_cpu_mode flag depending on the CPU boot mode passed + * in w0. See arch/arm64/include/asm/virt.h for more info. + */ +SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) + adr_l x1, __boot_cpu_mode + cmp w0, #BOOT_CPU_MODE_EL2 + b.ne 1f + add x1, x1, #4 +1: str w0, [x1] // Save CPU boot mode + ret +SYM_FUNC_END(set_cpu_boot_mode_flag) + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -662,6 +663,7 @@ SYM_FUNC_END(__secondary_too_slow) * Checks if the selected granule size is supported by the CPU. * If it isn't, park the CPU */ + .section ".idmap.text","awx" SYM_FUNC_START(__enable_mmu) mrs x3, ID_AA64MMFR0_EL1 ubfx x3, x3, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 4c13dafc98b8400f..407415a5163ab62f 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -179,7 +179,6 @@ SECTIONS LOCK_TEXT KPROBES_TEXT HYPERVISOR_TEXT - IDMAP_TEXT *(.gnu.warning) . = ALIGN(16); *(.got) /* Global offset table */ @@ -206,6 +205,7 @@ SECTIONS TRAMP_TEXT HIBERNATE_TEXT KEXEC_TEXT + IDMAP_TEXT . = ALIGN(PAGE_SIZE); } diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 066fa60b93d24827..91410f48809000a0 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -110,7 +110,6 @@ SYM_FUNC_END(cpu_do_suspend) * * x0: Address of context pointer */ - .pushsection ".idmap.text", "awx" SYM_FUNC_START(cpu_do_resume) ldp x2, x3, [x0] ldp x4, x5, [x0, #16] @@ -166,7 +165,6 @@ alternative_else_nop_endif isb ret SYM_FUNC_END(cpu_do_resume) - .popsection #endif .pushsection ".idmap.text", "awx" From patchwork Tue Nov 29 16:14:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C85FC433FE for ; Tue, 29 Nov 2022 16:17:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yd1fuPcMg1lsiFjtCuAti6p3qtcm7wAhonkgXouVhak=; b=j8epWs4gnMSlMx 56fp4CQ9FAXB91FqXOo5EkZF8/fl5uJ9WDiMf0H9AcE8BMtNV/5RVB5/dJ1gi5nUFUu3BG/1nbtrd TCMnaj4PLUUPha6UtKmnMG/JDxgMUNuDd1OsYqiV4XtCzQu54B40Da6saB35ewmeUJvPYYe45A+ss 1Zh4u0cvwq43cap8NJs+hE3w6eV7xIm/NjjmEuv35C/53SfL3lMZa0SSUS8O/QqU9o2J7rwW5tYDn kj3FwxSz/vVqJtJCqTKWJS6TGkHmKmlAwZ5lyuxLn0P7lsx/IoU4ACVibCGfC/CLTG+sfuJnfkzC1 5XwnhZmPpSQ4Sr0M/sbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Hf-00A1Ag-CP; Tue, 29 Nov 2022 16:16:35 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Fd-00A0Y1-Vh for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 56CB7617BA; Tue, 29 Nov 2022 16:14:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CF271C433D7; Tue, 29 Nov 2022 16:14:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738469; bh=2Olt+kmxs4NdbHYZ6IbTcuFcKJShiSGgmZUFx4t4lTU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CMnxJfnpVc2+2sTR0JxTSypfzWIKxDuBe6eoU0qTjLnAMur2MG8ZlFGdEzgFm/dWz ZkYx6p4KajYXSNtCDxmXGECvca1eqkk5CdzW/jjvmvxCvt5cNkz7RQ8B5DHg1Cz9wX B49L01HmOIL+sZO7XnOb/DLoSFhpjwEYGsLBP7ffGla9RIz9fkdZ1HhqKw6LuXvN8T n8EdopOlETOzpzNUdT8XT9OvDJ5qwchF9cUAYe4arOyQ0vo4agmKPwxv9jqtfoGMk0 IxszDgQxkeR8VBwGdVZqhp7zjIH1pYFf6xH+EiQ+Vj2DOo75u4HzBN1tAlZPYI6h5a 46pde5b0jrmtg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 3/6] arm64: head: record the MMU state at primary entry Date: Tue, 29 Nov 2022 17:14:15 +0100 Message-Id: <20221129161418.1968319-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5596; i=ardb@kernel.org; h=from:subject; bh=2Olt+kmxs4NdbHYZ6IbTcuFcKJShiSGgmZUFx4t4lTU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/TG7WYkzHsQnU4InkgB+IbQqcf8WBJJeyroATV JKD51RmJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv0wAKCRDDTyI5ktmPJBEEC/ 0SuhhpExln4DX5B4JAgLwoCQCo0P06tpyaO7jaqlcDhORkcxH6VfZOBF4cNgZoBQxdhRK4P3+f2HSR FJAGQfLoJunUtbKT6lpRHSsfExvZ6VkzR8oXqflVFWEcfuVxvGt6VggJQxX0h5d1PdP3h48xQye3Es n9HYgJYbhROXvr8AqtbvUCs6p/brQGK3UdeN8xwfL4eL9XyxVaNmrcAdUjtr60LDsj3suJS+xydKjQ J6gH1FVtfc961aU+61Dlyws3iO5BOVjUYJ0DfQDCQ8tvEy5ao4yhHPW8rq8Xpn6eRjppA3Ia+BNsjM meXINLi+bAWM6wXU7wLPtg9FlD95KAvyFtu/c+xKhcqaxZKsYdH+c9gG6Nwb8xvlZyXlaM9VcjPEWx p3RuQhT66r24YjiXN8ZYR3cVHeBmVHzbKa/V6DnSkug2fWg9rsMghIZfC6i0UbMt+/4IoiMaMvMj/W SKjGHkWXQCfmbyY/x2OA6EBSrnQmDFavz1BW1mOwo6o84= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081430_160197_B80F29EB X-CRM114-Status: GOOD ( 19.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare for being able to deal with primary entry with the MMU and caches enabled, by recording whether or not we entered with the MMU on in register x19 and in a global variable. (Note that writing to this variable does not require cache invalidation, nor is it required for storing the bootargs in that case, so omit the cache maintenance). Since boot with the MMU enabled is not permitted by the bare metal boot protocol, ensure that a diagnostic is emitted and a taint bit set if the MMU was found to be enabled on a non-EFI boot. We will make an exception for EFI boot later, which has strict requirements for the mapping of system memory, permitting us to relax the boot protocol and hand over from the EFI stub to the core kernel with MMU and caches left enabled. To reduce the likelihood that bare metal boot will violate this requirement, introduce a separate entry point for EFI boot, which is different from the one that is invoked by branching to offset #0 in the image. While at it, add 'pre_disable_mmu_workaround' macro invocations to init_kernel_el, as its manipulation of SCTLR_ELx may amount to disabling of the MMU after subsequent patches. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 28 ++++++++++++++++++-- arch/arm64/kernel/image-vars.h | 2 +- arch/arm64/kernel/setup.c | 9 +++++-- 3 files changed, 34 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index bec97aad092c2b43..c3b97f4ae6d769f7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -77,6 +77,7 @@ * primary lowlevel boot path: * * Register Scope Purpose + * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob @@ -85,8 +86,13 @@ * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ -SYM_CODE_START(primary_entry) - bl preserve_boot_args +SYM_CODE_START(efi_primary_entry) + bl record_mmu_state + b 0f + +SYM_INNER_LABEL(primary_entry, SYM_L_LOCAL) + mov x19, xzr // MMU must be off on bare metal boot +0: bl preserve_boot_args bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 bl create_idmap @@ -109,6 +115,18 @@ SYM_CODE_START(primary_entry) b __primary_switch SYM_CODE_END(primary_entry) +SYM_CODE_START_LOCAL(record_mmu_state) + mrs x19, CurrentEL + cmp x19, #CurrentEL_EL2 + mrs x19, sctlr_el1 + b.ne 0f + mrs x19, sctlr_el2 +0: tst x19, #SCTLR_ELx_C // Z := (C == 0) + and x19, x19, #SCTLR_ELx_M // isolate M bit + csel x19, xzr, x19, eq // clear x19 if Z + ret +SYM_CODE_END(record_mmu_state) + /* * Preserve the arguments passed by the bootloader in x0 .. x3 */ @@ -119,11 +137,14 @@ SYM_CODE_START_LOCAL(preserve_boot_args) stp x21, x1, [x0] // x0 .. x3 at kernel entry stp x2, x3, [x0, #16] + cbnz x19, 0f // skip cache invalidation if MMU is on dmb sy // needed before dc ivac with // MMU off add x1, x0, #0x20 // 4 x 8 bytes b dcache_inval_poc // tail call +0: str_l x19, mmu_enabled_at_boot, x0 + ret SYM_CODE_END(preserve_boot_args) SYM_FUNC_START_LOCAL(clear_page_tables) @@ -497,6 +518,7 @@ SYM_FUNC_START(init_kernel_el) SYM_INNER_LABEL(init_el1, SYM_L_LOCAL) mov_q x0, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround msr sctlr_el1, x0 isb mov_q x0, INIT_PSTATE_EL1 @@ -529,11 +551,13 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) cbz x0, 1f /* Set a sane SCTLR_EL1, the VHE way */ + pre_disable_mmu_workaround msr_s SYS_SCTLR_EL12, x1 mov x2, #BOOT_CPU_FLAG_E2H b 2f 1: + pre_disable_mmu_workaround msr sctlr_el1, x1 mov x2, xzr 2: diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8151412653de209c..b925094692983734 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -11,7 +11,7 @@ #endif PROVIDE(__efistub_kernel_size = _edata - _text); -PROVIDE(__efistub_primary_entry_offset = primary_entry - _text); +PROVIDE(__efistub_primary_entry_offset = efi_primary_entry - _text); /* * The EFI stub has its own symbol namespace prefixed by __efistub_, to diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 12cfe9d0d3fac10d..0ac8605a8efe00a5 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -58,6 +58,7 @@ static int num_standard_resources; static struct resource *standard_resources; phys_addr_t __fdt_pointer __initdata; +u64 mmu_enabled_at_boot __initdata; /* * Standard memory resources @@ -332,8 +333,12 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) xen_early_init(); efi_init(); - if (!efi_enabled(EFI_BOOT) && ((u64)_text % MIN_KIMG_ALIGN) != 0) - pr_warn(FW_BUG "Kernel image misaligned at boot, please fix your bootloader!"); + if (!efi_enabled(EFI_BOOT)) { + if ((u64)_text % MIN_KIMG_ALIGN) + pr_warn(FW_BUG "Kernel image misaligned at boot, please fix your bootloader!"); + WARN_TAINT(mmu_enabled_at_boot, TAINT_FIRMWARE_WORKAROUND, + FW_BUG "Booted with MMU enabled!"); + } arm64_memblock_init(); From patchwork Tue Nov 29 16:14:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BAB3AC433FE for ; Tue, 29 Nov 2022 16:18:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BCWUszCKt4iOob1rfckZTwAyiFFRxtr16saf+MrcAMY=; b=jqs7HN00VS53Pc XyoaOigIzp6jAxLmOKg2kT3+3jFQy8y4xyaJmaxH91lceiuJDt50HYP1KYWFYFEQP2Y2SZs6QYcy5 Nd3aKCW1qWCzgO2EYqSDqmpwK/3oId4x7zYjqAPwowtbMu6FuQXOlTpWRAcuNH8sijwd1lJugzJFR c1HhTREHlnd9uYd9GIUPd57bJjSBfXsiGesK4eGWRc+PtwnWEzxT6blE4ukVqZZTdBy98MZg19tr+ ckWmmmNqIlpm+6l9BPb5QV0rJK5TI5kRAjZeIW1wSr18dmu7s/1jCt+gj5hCchwU7aH3z6b1dMXO7 Xlgx7imJIlRuQ/wTsY+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03In-00A1cH-7f; Tue, 29 Nov 2022 16:17:46 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Fh-00A0Z1-O8 for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:35 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 28128B816CF; Tue, 29 Nov 2022 16:14:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 90426C433D7; Tue, 29 Nov 2022 16:14:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738470; bh=Ml+Ny+IJGRPmjIFrsKAsVGEU9ak8prI+CI1vJoHufHo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mMzv7MRidRCQRzRB4gazXYhYVa6fgtwb07ud5qOZ7MiihpYiYQqy2LADaCKHUpmuj 98uvgdnXTvOPAJEkfHN9tPMD362XUlvi/BNF55c6ITqyezujOOWwInMJZhEEBW5t9b vPSFlfvfQXuGtS7epaO/k4LxD1Zruce6+HXcBRsm+vb/RXe7Q9g2pEdFbF9a6zm/2c 62VEonxAVF6aQV3cB2+do8fUylvTyz8WNR1evGi62Xz6X452bMedSDf+uh4mt7R+sW C3UcvUHayELxmC0fg9nlrwaxobc0591L41yBUyas/9r8SIvQzdxWlX92VZ6MZ8M9+t mdTwkazQbZQJw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 4/6] arm64: head: avoid cache invalidation when entering with the MMU on Date: Tue, 29 Nov 2022 17:14:16 +0100 Message-Id: <20221129161418.1968319-5-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1441; i=ardb@kernel.org; h=from:subject; bh=Ml+Ny+IJGRPmjIFrsKAsVGEU9ak8prI+CI1vJoHufHo=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/VX50O/LzTDrteHK0oAAH4EkQvG4SrQFt56fyb dW5nz6eJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv1QAKCRDDTyI5ktmPJDjJDA CnZ8zqVpHeQxKWGN3LQwwBy+7hbPon5oX9GRfgL8OLqW2akw8PN2uFMaYHNkhFoF4G7f8hU8f43Waz EpnsvXfWdS9jLtacSkjZln2K81I6al5kAHoNelh0sY0XNgVKgGyXXUWHWmsF/z3ihTabbICTSa0lzP 1VOtkb5/uSZo9Q2X6iKGFqsLDgFyeYyrHsPMvOS1XD3rKDDzTZSl7ZQSOSn7g5P6/0DRQXaFy28hmw pGEjVoFNJvybhnOCWa9ngYvRtKi9vcj8Ea6Klonn3/phrBRQW/cpFYgnwO41s5/+qZ+Hky2Y1fPKM9 YI+jkHJupIqqGyOi5+cVdw8kSGa9vLqvQQUBqqPEoNEghV2xma/s4DBTZVJfCRG8sapIUrAOmwS3yi GGL7fxVBiP9pwrSBZDh8YcWZ2sZujg11frVspRurTSITc9pR/suOQVBHtTI4xHvRH1xuR7z+m10VGi VsnWGptoeHjD6/u9d1AdNK86QCUtSqOk4rNVyDtn5KDE0= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081433_966273_37A7B046 X-CRM114-Status: GOOD ( 13.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we enter with the MMU on, there is no need for explicit cache invalidation for stores to memory, as they will be coherent with the caches. Let's take advantage of this, and create the ID map with the MMU still enabled if that is how we entered, and avoid any cache invalidation calls in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index c3b97f4ae6d769f7..5abf8f9fdd97b673 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -93,9 +93,9 @@ SYM_CODE_START(efi_primary_entry) SYM_INNER_LABEL(primary_entry, SYM_L_LOCAL) mov x19, xzr // MMU must be off on bare metal boot 0: bl preserve_boot_args + bl create_idmap bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 - bl create_idmap /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -381,12 +381,13 @@ SYM_FUNC_START_LOCAL(create_idmap) * accesses (MMU disabled), invalidate those tables again to * remove any speculatively loaded cache lines. */ + cbnz x19, 0f // skip cache invalidation if MMU is on dmb sy adrp x0, init_idmap_pg_dir adrp x1, init_idmap_pg_end bl dcache_inval_poc - ret x28 +0: ret x28 SYM_FUNC_END(create_idmap) SYM_FUNC_START_LOCAL(create_kernel_mapping) From patchwork Tue Nov 29 16:14:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66457C4167B for ; Tue, 29 Nov 2022 16:19:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eeanUDOjs8VkD3ps3k+PAlgu1lyKwfKh7ShmuRRfVEU=; b=J1bV7IsEITQDaQ PRAsZiA8ifoAO5/JjWqTZZVqzP01L3TERqVh+b4Te80z+34vwA9HhsioLcJVJeCEZo4d6XjRJkcSm PXj8R0SIUIMYirgmao885UjrbgiNb5yimMQ339fNy3rZtm6yj7+4YmlWVPlM8JTwmuaQzKpbVgdgp AeNYYEg5CEZcv6i/s8KEgQnppJ0TfsYqbaIZSqnr6NZxpHhKck9LkWq1UIwbxX2HYVGVHxKpzoqBk EfDJVk3aR0lwhdcm8kGDmvjX+NQnOo3sXmrVUddO4wn8dmLhSfX33izwSttwVgCYpE59ccXyIJumu OJehDbVLuHLTuHGH9yKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03JG-00A1qk-K2; Tue, 29 Nov 2022 16:18:14 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Fj-00A0Zo-7s for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D6260B811B8; Tue, 29 Nov 2022 16:14:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 52030C43470; Tue, 29 Nov 2022 16:14:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738472; bh=5l50LH5mY6IjiFbDKQ8jAuQAm4lxLhhHcFErsGJlabU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gbFpQtGCfes+aGnPm5KDXi5IlIBLGf0W2bv0EkJ+8czT8T4IcpSjkJd/slIQ4nwfx bYnkPN8ayGAzECt0rPTDARR6xk4A39wQhlTwa7HjgBxlPWXbxJ01EyvFfrUIFEQi6W JcCp3PiQKrY4NDv3WQSskOZef8R5r1zIgat9BxfU2nFw/uEtbHJXowmNFzU8sLKZrx /nWu0KHZOIoU9SojiwJ8Lqlf/CuaXU8FMO+RJ4lg0w/iHEh8m/CxDjnLUSzcgSW+B9 OH1RaUGthJay12q8n96ryNG1aTswR62cJyjs7Dvzw2dtLhaLsgYQ3THlyCzU00QwS8 SjRYs2EuJhvUQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 5/6] arm64: head: Clean the ID map and the HYP text to the PoC if needed Date: Tue, 29 Nov 2022 17:14:17 +0100 Message-Id: <20221129161418.1968319-6-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3817; i=ardb@kernel.org; h=from:subject; bh=5l50LH5mY6IjiFbDKQ8jAuQAm4lxLhhHcFErsGJlabU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/Xn/Wlo7Yh6xIAq3GD8OYXYy11cMzpkaYHdSng QB5bBmqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv1wAKCRDDTyI5ktmPJGYrC/ 9Go7veaYjzF+RA81LrzU0K7rxkAkJmTlM/6AOvRqfMTiCl+lz0XvgYhG1JmHZEnJcZEXhiErn8FPYd FO+TNqngrciW5NPS6jWsGH8TV36UWekk3EFlcDbmYdSCjLHSnju0WTeLjeu0dHsvSLfnuRDzZjozlO nplXaU+a7CcnSTuF0QKWuwNvmP/yok3kc0lSC7swvyH0jpfKbXn+RgPocjNQBdvIJU02/k4Q7TN+m7 vrmKeD9NzQwaLV3jyKH4QcFAuF/kFHB3CyWtxBLDfcGZU2xV+XfI7V0yP9Ga13tBckXAN0OA3NP6xm eaWp0UR0Q/WP8/nSbG2XeoqkFOAPsCrVWQSYSE255VXJKyNOJSGozLQ9FQZfWVPfFh+jPbyBfa/QR+ r48l0bxX7Cb6g+YoPzPZlhA1Cr/dnQemq4uXAj76e1/ibcmTNbn50E9MAtcMOFcL9x6VS3dx2JYAwA TAf4YYP78QDVMYBe3TALFanjEtWIRr9Q8CQoflQ2YYtCI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081435_620368_AACEF44B X-CRM114-Status: GOOD ( 18.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we enter with the MMU and caches enabled, the bootloader may not have performed any cache maintenance to the PoC. So clean the ID mapped page to the PoC, to ensure that instruction and data accesses with the MMU off see the correct data. For similar reasons, clean all the HYP text to the PoC as well when entering at EL2 with the MMU and caches enabled. Note that this means primary_entry() itself needs to be moved into the ID map as well, as we will return from init_kernel_el() with the MMU and caches off. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 30 ++++++++++++++++++-- arch/arm64/kernel/sleep.S | 1 + 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 5abf8f9fdd97b673..d7b908c26253f7fe 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -90,10 +90,22 @@ SYM_CODE_START(efi_primary_entry) bl record_mmu_state b 0f + .section ".idmap.text","awx" SYM_INNER_LABEL(primary_entry, SYM_L_LOCAL) mov x19, xzr // MMU must be off on bare metal boot 0: bl preserve_boot_args bl create_idmap + + /* + * If we entered with the MMU and caches on, clean the ID mapped part + * of the primary boot code to the PoC so we can safely execute it with + * the MMU off. + */ + cbz x19, 1f + adrp x0, __idmap_text_start + adr_l x1, __idmap_text_end + bl dcache_clean_poc +1: mov x0, x19 bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 @@ -115,6 +127,7 @@ SYM_INNER_LABEL(primary_entry, SYM_L_LOCAL) b __primary_switch SYM_CODE_END(primary_entry) + __INIT SYM_CODE_START_LOCAL(record_mmu_state) mrs x19, CurrentEL cmp x19, #CurrentEL_EL2 @@ -511,10 +524,12 @@ SYM_FUNC_END(__primary_switched) * Returns either BOOT_CPU_MODE_EL1 or BOOT_CPU_MODE_EL2 in x0 if * booted in EL1 or EL2 respectively, with the top 32 bits containing * potential context flags. These flags are *not* stored in __boot_cpu_mode. + * + * x0: whether we are being called from the primary boot path with the MMU on */ SYM_FUNC_START(init_kernel_el) - mrs x0, CurrentEL - cmp x0, #CurrentEL_EL2 + mrs x1, CurrentEL + cmp x1, #CurrentEL_EL2 b.eq init_el2 SYM_INNER_LABEL(init_el1, SYM_L_LOCAL) @@ -529,6 +544,14 @@ SYM_INNER_LABEL(init_el1, SYM_L_LOCAL) eret SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) + msr elr_el2, lr + + // clean all HYP code to the PoC if we booted at EL2 with the MMU on + cbz x0, 0f + adrp x0, __hyp_idmap_text_start + adr_l x1, __hyp_text_end + bl dcache_clean_poc +0: mov_q x0, HCR_HOST_NVHE_FLAGS msr hcr_el2, x0 isb @@ -562,7 +585,6 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) msr sctlr_el1, x1 mov x2, xzr 2: - msr elr_el2, lr mov w0, #BOOT_CPU_MODE_EL2 orr x0, x0, x2 eret @@ -573,6 +595,7 @@ SYM_FUNC_END(init_kernel_el) * cores are held until we're ready for them to initialise. */ SYM_FUNC_START(secondary_holding_pen) + mov x0, xzr bl init_kernel_el // w0=cpu_boot_mode mrs x2, mpidr_el1 mov_q x1, MPIDR_HWID_BITMASK @@ -590,6 +613,7 @@ SYM_FUNC_END(secondary_holding_pen) * be used where CPUs are brought online dynamically by the kernel. */ SYM_FUNC_START(secondary_entry) + mov x0, xzr bl init_kernel_el // w0=cpu_boot_mode b secondary_startup SYM_FUNC_END(secondary_entry) diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 7b7c56e048346e97..2ae7cff1953aaf87 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -99,6 +99,7 @@ SYM_FUNC_END(__cpu_suspend_enter) .pushsection ".idmap.text", "awx" SYM_CODE_START(cpu_resume) + mov x0, xzr bl init_kernel_el mov x19, x0 // preserve boot mode #if VA_BITS > 48 From patchwork Tue Nov 29 16:14:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13058808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 224ECC433FE for ; Tue, 29 Nov 2022 16:19:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7DANx9wJJ6wfLPuC+eSw5N0ysNQAaL60cUjAz3ayz9A=; b=wbj3aCP3z+svKT c1aUyfzBIrRkyYvBReWsZqQmsVfngd7YxCtCr+PxApoQByMfDjH9LnJuUmbZt6kU9QgUoiHl+ZSFU B+6o75gs4+FizMku9yuiVnUGB0cbixQ/DgPGF8Rye71SbHq2EfZL3EfPK187egZZlOPQ6aVV9PicI DQ4xxs8Q3Qdvwyj+J5l10ln774aF6hFb5oY3nJqvkEI71p/ZMBvOc/zgGgNqTZjsuP7JARiHTwfSs pODAAOGkkX63PXaRQt/VIDn3t7VMgQcqnMS4nsk36XogUDtnNkLD3VF3Pb7sjVFNr5VuprWlanFkf XTP3QVjQ0/oGVeDZqG4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Jb-00A213-M7; Tue, 29 Nov 2022 16:18:35 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p03Fj-00A0aH-Em for linux-arm-kernel@lists.infradead.org; Tue, 29 Nov 2022 16:14:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EB896617D5; Tue, 29 Nov 2022 16:14:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 142D7C433C1; Tue, 29 Nov 2022 16:14:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669738474; bh=tSjYOPJsaij17TN4QuwS8cUzZGIxtcHzEAMZyGu8Xj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nSF2sIkH/TjFf8Z85Kf4Hg0CMieuBl8Qml5UO/5b56U4HgGqy8EW+tYSv/t8BOzFP Le/ppMl3y2bYTbYg6xd1BUu6W7lDR1zVkbNGJzw5nGxsPKVZ+EcjSn1lPvT0leBrrb JnMPccg7BWflO1XScRrrNHDbTXvdt7zQtBaXEwZs1whT8SXM5SM4c73vj9FuFqUY/d +qqBpLeQ9UBXiWwMIbzmfdtyyuPs85XzeKmn3+5i8DskY54+VUwmdHcmMAhfiA1ZhQ ke3IQRqaVgy5Zuwt5eALt7LxU12mgnCJ/JsdQBxYNW7IIwp+drTvcxnhtbENlQ4NHF WbC9jyAOfAO6A== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland Subject: [PATCH v6 6/6] arm64: lds: reduce effective minimum image alignment to 64k Date: Tue, 29 Nov 2022 17:14:18 +0100 Message-Id: <20221129161418.1968319-7-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221129161418.1968319-1-ardb@kernel.org> References: <20221129161418.1968319-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5121; i=ardb@kernel.org; h=from:subject; bh=tSjYOPJsaij17TN4QuwS8cUzZGIxtcHzEAMZyGu8Xj8=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjhi/YJtSt2893zV6KhY/9M7qGlR8We0E7r3zzGEDs 3wdT0w+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY4Yv2AAKCRDDTyI5ktmPJBeKC/ 9TGK4diBULm5C/HjO0jLolGQIgRpoZ0lUYLXg1c2Xptf2jz1CBf0B7G0FE1Zygho48WlelCBrtt2Hy 1WL7xbIfBP3Gh0N962vNwXLeu2q/Rfs+A5Q1jawfps4ZQDOn1FvW/aGBvlONU5tDcDtq9tQMObo17r gqmUIv6I0JC0slwzwoOV7LH82E30MNdNGYA3njCgRsco46EGhs+6HFx2+THV+UX2aeWbxMaU5+p4YO jB58J2GXMisMom0oNDpyzxtbR4p7oGZ6V10okQ/irUv91EE6ab1ULf56p37fiOp9nHcYujxsKc8FVo ws3mt/pI00X7M+28oy4P4Ml864UKKhO/F5hfYyO+CcpEOX7Y6oFuGWWVFL/hexNZvrvdTjT8YHCRWl 5Hi8C+gRl0cN5Yn2eF6MI9kytVu6L10HTTSqCFSlWtBwEqw4Dv+AoLAfQXZSlSFF5oqwTNJtATNUKK mnLNP+RHsRHZm0IdO2mocj6mMcFbw0/+tJxpI8kx5bfmg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221129_081435_646571_31A56759 X-CRM114-Status: GOOD ( 23.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Our segment alignment is 64k for all configurations, and coincidentally, this is the largest alignment supported by the PE/COFF executable format used by EFI. This means that generally, there is no need to move the image around in memory after it has been loaded by the firmware, which can be advantageous as it also permits us to rely on the memory attributes set by the firmware (R-X for [_text, __inittext_end] and RW- for [__initdata_begin, _end]. However, the minimum alignment of the image is actually 128k on 64k pages configurations with CONFIG_VMAP_STACK=y, due to the existence of a single 128k aligned object in the image, which is the stack of the init task. Let's work around this by adding some padding before the init stack allocation, so we can round down the stack pointer to a suitably aligned value if the image is not aligned to 128k in memory. Note that this does not affect the boot protocol, which still requires 2 MiB alignment for bare metal boot, but is only part of the internal contract between the EFI stub and the kernel proper. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/efi.h | 7 ------- arch/arm64/kernel/head.S | 3 +++ arch/arm64/kernel/vmlinux.lds.S | 11 ++++++++++- drivers/firmware/efi/libstub/arm64-stub.c | 2 +- include/linux/efi.h | 6 +----- 5 files changed, 15 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index d6cf535d8352b324..a2321e92644d1c66 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -62,13 +62,6 @@ efi_status_t __efi_rt_asm_wrapper(void *, const char *, ...); /* arch specific definitions used by the stub code */ -/* - * In some configurations (e.g. VMAP_STACK && 64K pages), stacks built into the - * kernel need greater alignment than we require the segments to be padded to. - */ -#define EFI_KIMG_ALIGN \ - (SEGMENT_ALIGN > THREAD_ALIGN ? SEGMENT_ALIGN : THREAD_ALIGN) - /* * On arm64, we have to ensure that the initrd ends up in the linear region, * which is a 1 GB aligned region of size '1UL << (VA_BITS_MIN - 1)' that is diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index d7b908c26253f7fe..13732e012db808bd 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -433,6 +433,9 @@ SYM_FUNC_END(create_kernel_mapping) msr sp_el0, \tsk ldr \tmp1, [\tsk, #TSK_STACK] +#if THREAD_ALIGN > SEGMENT_ALIGN + bic \tmp1, \tmp1, #THREAD_ALIGN - 1 +#endif add sp, \tmp1, #THREAD_SIZE sub sp, sp, #PT_REGS_SIZE diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 407415a5163ab62f..fe0f8a09f1cca6fd 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -287,7 +287,16 @@ SECTIONS _data = .; _sdata = .; - RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN) +#if THREAD_ALIGN > SEGMENT_ALIGN + /* + * Add some padding for the init stack so we can fix up any potential + * misalignment at runtime. In practice, this can only occur on 64k + * pages configurations with CONFIG_VMAP_STACK=y. + */ + . += THREAD_ALIGN - SEGMENT_ALIGN; + ASSERT(. == init_stack, "init_stack not at start of RW_DATA as expected") +#endif + RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, SEGMENT_ALIGN) /* * Data written with the MMU off but read with the MMU on requires diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 259e4b852d63276d..468872e07e6c171f 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -97,7 +97,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, * 2M alignment if KASLR was explicitly disabled, even if it was not * going to be activated to begin with. */ - u64 min_kimg_align = efi_nokaslr ? MIN_KIMG_ALIGN : EFI_KIMG_ALIGN; + u64 min_kimg_align = efi_nokaslr ? MIN_KIMG_ALIGN : SEGMENT_ALIGN; if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { efi_guid_t li_fixed_proto = LINUX_EFI_LOADED_IMAGE_FIXED_GUID; diff --git a/include/linux/efi.h b/include/linux/efi.h index 929d559ad41d29c6..32b83020c7a35269 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -421,11 +421,7 @@ void efi_native_runtime_setup(void); /* * This GUID may be installed onto the kernel image's handle as a NULL protocol * to signal to the stub that the placement of the image should be respected, - * and moving the image in physical memory is undesirable. To ensure - * compatibility with 64k pages kernels with virtually mapped stacks, and to - * avoid defeating physical randomization, this protocol should only be - * installed if the image was placed at a randomized 128k aligned address in - * memory. + * and moving the image in physical memory is undesirable. */ #define LINUX_EFI_LOADED_IMAGE_FIXED_GUID EFI_GUID(0xf5a37b6d, 0x3344, 0x42a5, 0xb6, 0xbb, 0x97, 0x86, 0x48, 0xc1, 0x89, 0x0a)