From patchwork Fri Jun 24 15:06:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41067C43334 for ; Fri, 24 Jun 2022 15:08:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zo5hhiWTMCkLM4dV3eiybIatFo55IoXdJasfxAPmq1A=; b=wpH7cSjdOSZBFz XxF0Um+OLPj7+9E1oWDuLZQDpQTc32U32UyAkh4pMC4mJEjrEBdS6K/8nZWa6uNFy8ptr+vJA5+Ag 0bo9PP/qANrxToR9rUaAIyWHuqr1WXCbGi0cX+lU8riGP0NBNG56QjbK1b9FXYVMD7i4mArc3wL5N /sw36sSQLl/Bgj/TyuyuG7aSl1EIfmf8Jwb1RMFcUfO2Loc+cqnStQlKXbI8QpFHYjpcO360LzuHZ TH8pbdjiLx7voMBYo8WLn8dAYe7OG4eoDUiKj70KXi9/VYfxN5a2U/tCO2utyQ2rMSWGXQJKpo8Bg m8IFIAjkr0U0QvXWeBrA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku3-002iNL-S3; Fri, 24 Jun 2022 15:07:23 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ktu-002iKW-AK for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:15 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 04874B826EE; Fri, 24 Jun 2022 15:07:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E033EC341C6; Fri, 24 Jun 2022 15:07:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083231; bh=nQj+OXIbaKKle65XIN24cL4njBHpj6xT59hY56xesN4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P0SGRMiTSYbWTrQuDse2RtbnER8pU1AEcBK7O6k2rUEO8rpI03++Vj6RvXiit5AzI rcPP2DSo+/kUmyyfLDYlzyEvMulIaGWKpx2vSvq2VZhs6lGfhKX9bk9hJm8nI0H42C qTLrg/VS7hKksMCisKKP2E/tDjrJoJAGABalgzPXUZG61SDM7EU8Pmt+cOrNqg3vVR X5F1gkDrWQgN0KBktN4Oql7T+kggdu8s5HvhzDjpQ94B52W8kWp+D5FRLAsG8MJ1tR c0nKR5vH1lngFj+3CqHJbzImHc687O//5cp4NggNoxNSI4k0mVUscA6i0E1XCDhWzE TBLSSpnb8yIDQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 01/21] arm64: head: move kimage_vaddr variable into C file Date: Fri, 24 Jun 2022 17:06:31 +0200 Message-Id: <20220624150651.1358849-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1366; h=from:subject; bh=nQj+OXIbaKKle65XIN24cL4njBHpj6xT59hY56xesN4=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLoKosfleHve3BLHvB4clHclk1ueoW+2bjkGDak BzVvWlOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS6AAKCRDDTyI5ktmPJMpWDA CNX9kIIaCvtGWcLnQGcqdnSHHAQ8We34regs/SRQZwYF82TUnSplZ2BHX260MgKwprQyHOMQWu2+05 ZpvkmDcW+nwj44xTPJsOlY48bncxN3tq6Ycx8Nn8oKqZK24OwMOdUZOD9SfHidDLhtYlv7x9WLDYi2 eb1QhFTE9jvEozXbXjeMluF6bae8AlJw5keOeCiI2L9QPx/Qq6tXuxtrhe8+DZjP89SUJQj1hNjE7J yhjYkNRC3eFdiPnVl8eUOklvHsb46DeEt8DyJg6EnY15/Ga7IcRgM72EKe/7pA3QfcQUlOTuwcnr69 bxdazF43YOV0Sqp/ZMnCee18xIwre6g5xZiTuh98S45aufW+IzOfIhIQaFd0fK8HNAAGCZ4nwQCwuv EkwR7d30/+0rXFIREL424hdc+TClIC6cEftOmYHCS4Q7+juX2CBeG05YmBrb+z6XCwTcUWpioFOpu8 exMnZCHLmUWWPlRgXVHE1rFx1k0NkSJ84ygVW4HIYGTPs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080714_559495_EA87F88C X-CRM114-Status: GOOD ( 12.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This variable definition does not need to be in head.S so move it out. Signed-off-by: Ard Biesheuvel Reviewed-by: Anshuman Khandual Acked-by: Mark Rutland --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/mm/mmu.c | 3 +++ 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 6a98f1a38c29..1cdecce552bb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -469,13 +469,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) ASM_BUG() SYM_FUNC_END(__primary_switched) - .pushsection ".rodata", "a" -SYM_DATA_START(kimage_vaddr) - .quad _text -SYM_DATA_END(kimage_vaddr) -EXPORT_SYMBOL(kimage_vaddr) - .popsection - /* * end early head section, begin head code that is also used for * hotplug and needs to have the same protections as the text region diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index c5563ff990da..7148928e3932 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -49,6 +49,9 @@ u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; u64 __section(".mmuoff.data.write") vabits_actual; EXPORT_SYMBOL(vabits_actual); +u64 kimage_vaddr __ro_after_init = (u64)&_text; +EXPORT_SYMBOL(kimage_vaddr); + u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); From patchwork Fri Jun 24 15:06:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBD33CCA473 for ; Fri, 24 Jun 2022 15:08:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WGmaxUteM7OW2o/MVwnVSWGOtLA9vNY8Nfsq2BlJVZs=; b=pljHbir7k8ldXu /ekk99ndxAuOtx++vdrWUdkXTh8RyK3pi/u1znmnBkiawtieOnx7HZJ1vMb1dyGsPHhWf+SSybKDT +O/EiYrM+EIGeGE8piqYuWkIl7rmFOTjQ40SzH2puWGzVUTvvfriqKMSg8Jhn7ZRmuD8KIID331J/ mtIP3de/gED4GhXW1ofzvDUVwNAAyCY2tb5mwDR5jnfbreJvbN6Q0PSDB36eJey/3K9RSyINcFnP9 bW0lRz8kFcALOGCHgDvgFYLluZxp6i6LbnqQBDnJOb1RyqMmPN7j6zrBpH4V2c8T41FnRCs6q0RR0 6pGbANlOd8W1y6h+jriw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuC-002iQH-8z; Fri, 24 Jun 2022 15:07:32 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ktw-002iKu-Mc for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:18 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5EE23B82904; Fri, 24 Jun 2022 15:07:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39241C341C0; Fri, 24 Jun 2022 15:07:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083234; bh=Nlk+fT8XE7NfZPpFGoVniozAoA3n+KamLs/q34Zdfco=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=olZLEfDmcVxWnhG2ubBfJT1q5jqDVxE4f2GflVogBCx8qgVcMDAtstfi9jun1mDS9 NVKVtaXAiSrgpWxaq80QZSK0sizoVvBoWV//N0N9/s6pbEyM/bv4fvHTscen1/uWvV 1TIBQIg5/06NEkm+i2ujag/xNyjKkn5Nv81mCz9gdVjmHuDFM9a33IW3AlS9BL3+6D N8Z2G8TaZd9GFeXEZpf1uhbCeDgcTr2qsTYVKaFEIIkAlFkRcU7ewuy6QjtftOwJkX 9vdp5YiLd2a5R2dsAa6ugmdyCsFY7mG9KQAwFSuJpW3PHCN8atYqSFjuVTh/yG3H/Y vZTyhZr+0mvhQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 02/21] arm64: mm: make vabits_actual a build time constant if possible Date: Fri, 24 Jun 2022 17:06:32 +0200 Message-Id: <20220624150651.1358849-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3484; h=from:subject; bh=Nlk+fT8XE7NfZPpFGoVniozAoA3n+KamLs/q34Zdfco=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLqOSnUNh+BMv8XNf4z00iv3rDH1Ne05woz9euk 4XRbTxWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS6gAKCRDDTyI5ktmPJDkeDA CyQaz6IrPejb0D9yjtDw395AGVUV9zV+ZzA0qguQ72ZDUP8LGM5PG5U1Ml+4NLVnPirR8cWcMVJ1ep VWienPHGpq8YxbltxCD+74nr+9/2ESyCCBzTCFntbsOezniGn6L1kzQPiRvTGvFGPJT839bwfc/Vwn c+bxvSOKsu0QhhKIsBMT8RFmDejgd5coMoabAsWwng14czYXVp1WmMXdiov2UvRInwsDPzvYSdfD9v /QfaANG6ld8k77PqVaf50n30RS0QHcj1yRnN/7XfGphHiQ8yWKbfYa8wXkUAeFvo7i6LoPlcRJcmuD mLXv6g60C0Xz6U9pnNSyEGujy13HlI6CPMKUTtsCWvxZwmji3lXo3so/3tuXIVBczDXZ8VdOQGMRe3 p8U/W6I9gacJmQvG3ADrGnO48/wW41441IGOfp1biSd7zSscmzK2FzWY2mi1AcGzjF8tjtfU0BFYSs BxFWEmbpBwusR1EN+V2o8SF0vO98TcF2SdwX8Jfk+3cXM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080717_095635_FFB5EFFA X-CRM114-Status: GOOD ( 15.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, we only support 52-bit virtual addressing on 64k pages configurations, and in all other cases, vabits_actual is guaranteed to equal VA_BITS (== VA_BITS_MIN). So get rid of the variable entirely in that case. While at it, move the assignment out of the asm entry code - it has no need to be there. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 4 ++++ arch/arm64/kernel/head.S | 15 +-------------- arch/arm64/mm/init.c | 15 ++++++++++++++- arch/arm64/mm/mmu.c | 4 +++- 4 files changed, 22 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 0af70d9abede..c751cd9b94f8 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -174,7 +174,11 @@ #include #include +#if VA_BITS > 48 extern u64 vabits_actual; +#else +#define vabits_actual ((u64)VA_BITS) +#endif extern s64 memstart_addr; /* PHYS_OFFSET - the physical address of the start of memory. */ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 1cdecce552bb..dc07858eb673 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -293,19 +293,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adrp x0, idmap_pg_dir adrp x3, __idmap_text_start // __pa(__idmap_text_start) -#ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s x6, SYS_ID_AA64MMFR2_EL1 - and x6, x6, #(0xf << ID_AA64MMFR2_LVA_SHIFT) - mov x5, #52 - cbnz x6, 1f -#endif - mov x5, #VA_BITS_MIN -1: - adr_l x6, vabits_actual - str x5, [x6] - dmb sy - dc ivac, x6 // Invalidate potentially stale cache line - /* * VA_BITS may be too small to allow for an ID mapping to be created * that covers system RAM if that is located sufficiently high in the @@ -713,7 +700,7 @@ SYM_FUNC_START(__enable_mmu) SYM_FUNC_END(__enable_mmu) SYM_FUNC_START(__cpu_secondary_check52bitva) -#ifdef CONFIG_ARM64_VA_BITS_52 +#if VA_BITS > 48 ldr_l x0, vabits_actual cmp x0, #52 b.ne 2f diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 339ee84e5a61..1faa6760895e 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -265,7 +265,20 @@ early_param("mem", early_mem); void __init arm64_memblock_init(void) { - s64 linear_region_size = PAGE_END - _PAGE_OFFSET(vabits_actual); + s64 linear_region_size; + +#if VA_BITS > 48 + if (cpuid_feature_extract_unsigned_field( + read_sysreg_s(SYS_ID_AA64MMFR2_EL1), + ID_AA64MMFR2_LVA_SHIFT)) + vabits_actual = VA_BITS; + + /* make the variable visible to secondaries with the MMU off */ + dcache_clean_inval_poc((u64)&vabits_actual, + (u64)&vabits_actual + sizeof(vabits_actual)); +#endif + + linear_region_size = PAGE_END - _PAGE_OFFSET(vabits_actual); /* * Corner case: 52-bit VA capable systems running KVM in nVHE mode may diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 7148928e3932..a6392656d589 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -46,8 +46,10 @@ u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; -u64 __section(".mmuoff.data.write") vabits_actual; +#if VA_BITS > 48 +u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); +#endif u64 kimage_vaddr __ro_after_init = (u64)&_text; EXPORT_SYMBOL(kimage_vaddr); From patchwork Fri Jun 24 15:06:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00D75C433EF for ; Fri, 24 Jun 2022 15:08:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fSchFtjf4oLDDYE+j+vAwyQk80vwP0er4Pphd9TcQ9w=; b=NHBjwpV+4e+Ylm 2ayiXWvn+4PDX69F4MVZyyreZUbw1nB9JGRyA1xUpeL/OC9W3jDQQmtOwL0DXg/aaaa9n40iwwOBc OvpcZgWQYH+DGmPpu3OyUnvwm393QnTRoyUEXcAZBK3injHmKgFIOZJ97orqtNhOPHA5QTwHBnkac DfH7LiIlQ9T1M6Z0gghjxIyXwpAH0BR9UV3hKbg1zIX/7IoXSUwTpOGPtuqUJp4eiYNUTfWnLHGYw LM/9fTErrcBw89zerk0rTbaU4GCN7voG2LN3cdfNPMty+zLbtcmnaTtu6ryDHJBgWHDC+8TpCjsOR dpeGRltnLc4gCHG9SrDQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuK-002iSl-VL; Fri, 24 Jun 2022 15:07:41 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ktx-002iLA-D8 for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:19 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F288962183; Fri, 24 Jun 2022 15:07:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 85BC5C36AE2; Fri, 24 Jun 2022 15:07:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083236; bh=tMQKwsx3kgYRPwIWWp9SOEYVqf/uSzFPsgbG2y0wW+U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XnCSnL1WdYUJINW3sSXrvnlyX6uf2OPtX1Li7jSFfbORpM8u2KzeVaYaMKNYNCQMJ F0XN0eHPR3yEdD7LJoVjt7T+lg1LFU1xh0MUbnbgNysacu+KwQESyFsisPkkutYQo2 V5cxG2sT2ipmU+r3boJAo0hudYbQkBI0E/nwPBWJeL51Q3FtxUcL8jA9Ju6E6ItBEF Gre5rkU8n4Lu6cVfezUix8cE/9oelji3PQCvKrENlempL5pBnMWyLNL782xp2dh+sv QXu8ShN/nLt7GTImrTibG3ubFKml/JSAYcTXS0TMb9JFWOiz1ZI9a3TBOG6BNzBTgG 2b770B1iRqN1g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 03/21] arm64: head: move assignment of idmap_t0sz to C code Date: Fri, 24 Jun 2022 17:06:33 +0200 Message-Id: <20220624150651.1358849-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4313; h=from:subject; bh=tMQKwsx3kgYRPwIWWp9SOEYVqf/uSzFPsgbG2y0wW+U=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLr1Izzi5+OlwgHRthAOZTuLISNk4KFDVt0eBJ9 FVeOadiJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS6wAKCRDDTyI5ktmPJGtEDA CHuGMsHNIb8LMYF3Y1JoTOuobEJ/dXcC1zE3bKXXJpt+ylHek78zNHgVk+srC5h8w6fTy0VpcBvxbi u0mbG0vB/UDyS8HXataIxTdu67HBzztEHckNhoe0Aea8RW2dFgnxEq+0svbELW+dMbqiWwW9chXjXz Rot3Bc63Ew5w6mVUMTIcZSdfeLLnideRnMwQKJpgW7LNC4YTBBedW8paZc+wEsoKzXVXSXpiMSOgk6 ZI6ZSPr0Wyzfaw0KL7gL4yXjt6hv2gB7ZnA4vDRzzGfOxoAobxZdkSlEHolt9RHaBFxNj250f4mOQk wW3KMK6UG+8C1bHk/wumvoYJMDBE7zmNBs7pAWdtr1RxuDD8PvgmGztHo03Ui2EjS1Sxv9zCkU1RXI 8suN2bvivNDUHr2FBUI9hnibRts+JSulgJPoN+ioCSXvhmKGgNXnnttCZostW53rm45VTjjrIviJCk xomuhr1tD+uGZyYQJ/vAaI24qjx6tkbS4X0velTXHms0A= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080717_563953_E6C0A9B0 X-CRM114-Status: GOOD ( 18.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Setting idmap_t0sz involves fiddling with the caches if done with the MMU off. Since we will be creating an initial ID map with the MMU and caches off, and the permanent ID map with the MMU and caches on, let's move this assignment of idmap_t0sz out of the startup code, and replace it with a macro that simply issues the three instructions needed to calculate the value wherever it is needed before the MMU is turned on. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland --- arch/arm64/include/asm/assembler.h | 14 ++++++++++++++ arch/arm64/include/asm/mmu_context.h | 2 +- arch/arm64/kernel/head.S | 13 +------------ arch/arm64/mm/mmu.c | 4 +++- arch/arm64/mm/proc.S | 2 +- 5 files changed, 20 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 8c5a61aeaf8e..9468f45c07a6 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -359,6 +359,20 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm +/* + * idmap_get_t0sz - get the T0SZ value needed to cover the ID map + * + * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the + * entire ID map region can be mapped. As T0SZ == (64 - #bits used), + * this number conveniently equals the number of leading zeroes in + * the physical address of _end. + */ + .macro idmap_get_t0sz, reg + adrp \reg, _end + orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 + clz \reg, \reg + .endm + /* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 6770667b34a3..6ac0086ebb1a 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -60,7 +60,7 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in * physical memory, in which case it will be smaller. */ -extern u64 idmap_t0sz; +extern int idmap_t0sz; extern u64 idmap_ptrs_per_pgd; /* diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index dc07858eb673..7f361bc72d12 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -299,22 +299,11 @@ SYM_FUNC_START_LOCAL(__create_page_tables) * physical address space. So for the ID map, use an extended virtual * range in that case, and configure an additional translation level * if needed. - * - * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the - * entire ID map region can be mapped. As T0SZ == (64 - #bits used), - * this number conveniently equals the number of leading zeroes in - * the physical address of __idmap_text_end. */ - adrp x5, __idmap_text_end - clz x5, x5 + idmap_get_t0sz x5 cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? b.ge 1f // .. then skip VA range extension - adr_l x6, idmap_t0sz - str x5, [x6] - dmb sy - dc ivac, x6 // Invalidate potentially stale cache line - #if (VA_BITS < 48) #define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) #define EXTRA_PTRS (1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a6392656d589..f875c4954e22 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -43,7 +43,7 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); +int idmap_t0sz __ro_after_init; u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; #if VA_BITS > 48 @@ -774,6 +774,8 @@ void __init paging_init(void) { pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); + idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); + map_kernel(pgdp); map_mem(pgdp); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 972ce8d7f2c5..97cd67697212 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -470,7 +470,7 @@ SYM_FUNC_START(__cpu_setup) add x9, x9, #64 tcr_set_t1sz tcr, x9 #else - ldr_l x9, idmap_t0sz + idmap_get_t0sz x9 #endif tcr_set_t0sz tcr, x9 From patchwork Fri Jun 24 15:06:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5BD2BC433EF for ; Fri, 24 Jun 2022 15:08:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bHJ5hP+6pHoKvlq9Yux9cM82pZ6FcpA7jpI9REv3RXA=; b=pshbUmiyMEzZB3 O3eVupSRfWdwCBAfKnjHlI4Oo0uo6M5vWmH7ERGDlVhd+Qt2qidyh/7Xz65WOtwf0rHmzhqhvup4c 9sndd5bYsk6GgZQq7Xc2mhcwMCLnbUMIpViq/pPffRGJlwwq7v5v7crBdX4ttReDNIOZJHDq+FB5W xVIq30w85zX/4f1dSNqV8YzY+ej4GFlY1iz1pouyMJVnLSlFpyYApnQlGQn2H8qzpkEFMu0VO6j/5 uZCgzUxSo7es4GaNvaTQXsamLyOuscxGf+ZOiN7lMqKHDnTo9wE1+PMhAmhD7WJP9UpRAPQY176xU ujHDIXuLdbOjbU83VTWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuW-002iXn-MF; Fri, 24 Jun 2022 15:07:52 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku2-002iMx-WC for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:24 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 6B75CCE2AB2; Fri, 24 Jun 2022 15:07:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D1D94C36AE3; Fri, 24 Jun 2022 15:07:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083238; bh=TvugYq/yTY2PK7+ouhMEJORa44d53Jx+xs/sbOdz64w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E55bVQgqIrkSRyXm5zzk6SeEXrkyb+Nf1V7zp0R7r0R+cFvOIJQ0iCOZiD2WGybaz AATuSXxEJJ2+npTWU/ul/2bGNHL39i+Etj26S/iq2FO+1OMVrJqbDEOE5nwU5vsIDb d0EOMLrHWHow+rttck4U+R4vCnVnCdsT4HpZF3qzGDvfCYw3JbBCkvxTUt83dTNM5A ayVm6pnb+fxnV2h/n/xOglMxoeuatMlyNp+IgLRMTfBIum/9VKm2enVKzi6oSUJk36 LgCKJa1Aii80YAq/v+IkewsIhQZVHheNdJQQqXravkgwOhjcGN29ClX8EEd6IIDygs AAz65jcW3hAiw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 04/21] arm64: head: drop idmap_ptrs_per_pgd Date: Fri, 24 Jun 2022 17:06:34 +0200 Message-Id: <20220624150651.1358849-5-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2669; h=from:subject; bh=TvugYq/yTY2PK7+ouhMEJORa44d53Jx+xs/sbOdz64w=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLteztav2iQxjxowPu10w9s/7VbxZMQvgzF+aCt MMKJ09iJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS7QAKCRDDTyI5ktmPJODIC/ 9UUJNKW8Ek1a315oLKRzkWpnRaxpFxTOagWhUq57/Biqhcrm4qnDGjQtjGrhzWjcyPf0qVKjco8RfV u4JltK0uzIrhYcw0lDW0m+pPqhtLkH5KIQwVij7SBAXOsDKNEsqEKW+seufl3oLRJ4f1IV88aMhMLR zzcmoxGNGSeb6Ll6Z8csmiFUPRQwVGhMn9CWPeyk5dZe8LbDapydRcKIIfe3Chu5acKEOIBcaPENZN 3cx5g4yon8zyoPcs0CiMeOsSukJd++WrYBIfifRqtzNtHdeWe+hZxshCiqDxYusKYIg2vn6CY6Rw4N dJJ015eiKhwrqxnJQ2ezaTwT9x/wkkROfbTOkHSFGFfIOlcTptDfnUWjmYeFooKZHtZrCTvmU/ime+ wfYiiifsaSPID4rn87Rr2zc6xdtermgLgkV96oeHVG3ySmwe5W7qPjk4e807y4x1ruwzdhgywngk/v H91TbH5SP/SQUAm3LF8rGWHKBFmSePYLPFed3RNedo6A8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080723_426207_E8594BFE X-CRM114-Status: GOOD ( 14.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The assignment of idmap_ptrs_per_pgd lacks any cache invalidation, even though it is updated with the MMU and caches disabled. However, we never bother to read the value again except in the very next instruction, and so we can just drop the variable entirely. Signed-off-by: Ard Biesheuvel Reviewed-by: Anshuman Khandual Acked-by: Mark Rutland --- arch/arm64/include/asm/mmu_context.h | 1 - arch/arm64/kernel/head.S | 7 +++---- arch/arm64/mm/mmu.c | 1 - 3 files changed, 3 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 6ac0086ebb1a..7b387c3b312a 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -61,7 +61,6 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) * physical memory, in which case it will be smaller. */ extern int idmap_t0sz; -extern u64 idmap_ptrs_per_pgd; /* * Ensure TCR.T0SZ is set to the provided value. diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7f361bc72d12..53126a35d73c 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -300,6 +300,7 @@ SYM_FUNC_START_LOCAL(__create_page_tables) * range in that case, and configure an additional translation level * if needed. */ + mov x4, #PTRS_PER_PGD idmap_get_t0sz x5 cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? b.ge 1f // .. then skip VA range extension @@ -319,18 +320,16 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #error "Mismatch between VA_BITS and page size/number of translation levels" #endif - mov x4, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 + mov x2, EXTRA_PTRS + create_table_entry x0, x3, EXTRA_SHIFT, x2, x5, x6 #else /* * If VA_BITS == 48, we don't have to configure an additional * translation level, but the top-level table has more entries. */ mov x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT) - str_l x4, idmap_ptrs_per_pgd, x5 #endif 1: - ldr_l x4, idmap_ptrs_per_pgd adr_l x6, __idmap_text_end // __pa(__idmap_text_end) map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f875c4954e22..3be0e9f69406 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -44,7 +44,6 @@ #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ int idmap_t0sz __ro_after_init; -u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; #if VA_BITS > 48 u64 vabits_actual __ro_after_init = VA_BITS_MIN; From patchwork Fri Jun 24 15:06:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0C2AC43334 for ; Fri, 24 Jun 2022 15:10:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=q2/moHkRU2CoO4GiSlVYYE9O6mGvE4jwcOgSo0GgbbI=; b=kkgkXlSwgIb7dZ QjB1CKQGOJgZq8rD6sRkS8+rXmhnd1qNwWCft1jx05jPCiLLkSELf2GCCaZKb1ANaG6K6nxv1eTrW Yf5hW6u+6Zi7Zysw9v680T3P5g67Lgxc6W4j2waBci45p+9M98jRln1+M/Zexbzw9kKTn4v+FyP9g 4slYvDrR9E/N+qOGGsQBUAVXnWmq4ME0A5DvHa36bmUU5yJTMkfQuHAOrvLul+QsjiJx93a6FgVaG ifnlxNJoIPpIiZcXU5SulCazPqZSKF3Aevbpkd/kVORpAnBNyOGAmqAXx19KTHJhqLDpzg7IwcPHP 8jvc/Nz6LHRunYFyZcag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kvz-002jIy-0x; Fri, 24 Jun 2022 15:09:23 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku2-002iMy-0F for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:23 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 99410621CE; Fri, 24 Jun 2022 15:07:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29B4EC36AEA; Fri, 24 Jun 2022 15:07:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083241; bh=P1BZpC0pdrKSsXZjI0I0b72Rj7KUsMARzBN3cnmSIus=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FbYxKlRy61o+v38R7zLSfEiSyJJD1CNyekG3ICmgl9y0MKere1JPouASCguq0EYSL tMSysyS7qCrXVHc7X9o3yTlbKfLyMhKumkgOnoOhHFk5RCIlHYiKyoUMCHwBAI4wC8 nAgVfl7ZB0Dq6HmF8Ex6zQ+Q3e82i4FTHOhN3a7SNFXnRqE71kMip5NhbdBPb907Lt 9HLMrFfvu7k4bYcRz+vlkYvIbjIp/YZQPZJvlcpc1/c9By7QLpbVvpR0jlzEO4yZ39 qUF6/t+I+MwpjhyF7TobmGZbDQ0WywJ05inOQkIhik3WABzjZT/vuShuhtX1JfgnWH k8OqmHsiERJzA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 05/21] arm64: head: simplify page table mapping macros (slightly) Date: Fri, 24 Jun 2022 17:06:35 +0200 Message-Id: <20220624150651.1358849-6-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6906; h=from:subject; bh=P1BZpC0pdrKSsXZjI0I0b72Rj7KUsMARzBN3cnmSIus=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLv+gAgFF8LaOOf3WD2XyhoP5g6OuiGOZtn97Mh nHe9VH6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS7wAKCRDDTyI5ktmPJH9HC/ 0RYWH4X9qip47qcuLKB4Y3nkKBMH0fNXTYw1ywpB9yutwpVPArZfkU9FBLv3RBnmiRnB9LJJgrO8qG i87xhRubayy5Hy0VN+9Haebh6WBpwOU26kzplb4QPbh+iJx+BT0z/kxPuQE/9Jy9wRTu6lHFPjS82W kmq5Hgo5mMaCoDBP0/nYI3OfJJTF2BORmcTjNogz1RoO5MPRcYVirkz6hMPMcwRMQrzhYDzImJfA0K 8N9ky1+Q9UW/WiaFD64dxRRA11RptabAYZ1nEQsMkLuvUkYw1KL8v3l5HDj6binGjYR9Ra3eT2RNsX AuJ4xb6rfg/mx0rmkBcbNeKznoUgOyhEjPfJwHEsJkKlqxhu9ocy4WPx09+GkeiOszlK5bPWNK8Koa yHkVf8tU6U2KHoFuc8S755tVKSMMgKEN9rxePArwoCjUAmiXKtIM3VzHS2kDUapzOfibfWVyYOcHYF lH6GS0vRGFhhUUm9m69E9DSptIKOKZgY6av8/kTc5gNqg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080722_162964_AAB5FAE3 X-CRM114-Status: GOOD ( 17.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Simplify the macros in head.S that are used to set up the early page tables, by switching to immediates for the number of bits that are interpreted as the table index at each level. This makes it much easier to infer from the instruction stream what is going on, and reduces the number of instructions emitted substantially. Note that the extended ID map for cases where no additional level needs to be configured now uses a compile time size as well, which means that we interpret up to 10 bits as the table index at the root level (for 52-bit physical addressing), without taking into account whether or not this is supported on the current system. However, those bits can only be set if we are executing the image from an address that exceeds the 48-bit PA range, and are guaranteed to be cleared otherwise, and given that we are dealing with a mapping in the lower TTBR0 range of the address space, the result is therefore the same as if we'd mask off only 6 bits. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland --- arch/arm64/kernel/head.S | 55 ++++++++------------ 1 file changed, 22 insertions(+), 33 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 53126a35d73c..9fdde2f9cc0f 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -179,31 +179,20 @@ SYM_CODE_END(preserve_boot_args) * vstart: virtual address of start of range * vend: virtual address of end of range - we map [vstart, vend] * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table + * order: #imm 2log(number of entries in page table) * istart: index in table corresponding to vstart * iend: index in table corresponding to vend * count: On entry: how many extra entries were required in previous level, scales * our end index. * On exit: returns how many extra entries required for next page table level * - * Preserves: vstart, vend, shift, ptrs + * Preserves: vstart, vend * Returns: istart, iend, count */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += count * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - + .macro compute_indices, vstart, vend, shift, order, istart, iend, count + ubfx \istart, \vstart, \shift, \order + ubfx \iend, \vend, \shift, \order + add \iend, \iend, \count, lsl \order sub \count, \iend, \istart .endm @@ -218,38 +207,39 @@ SYM_CODE_END(preserve_boot_args) * vend: virtual address of end of range - we map [vstart, vend - 1] * flags: flags to use to map last level entries * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries + * order: #imm 2log(number of entries in PGD table) * * Temporaries: istart, iend, tmp, count, sv - these need to be different registers * Preserves: vstart, flags * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv + .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv sub \vend, \vend, #1 add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count + + compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count + mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv - mov \sv, \rtbl #if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count + compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count + mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv - mov \sv, \rtbl #endif #if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count + compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count + mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv #endif - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp + compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count + bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 + populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp .endm /* @@ -300,12 +290,12 @@ SYM_FUNC_START_LOCAL(__create_page_tables) * range in that case, and configure an additional translation level * if needed. */ - mov x4, #PTRS_PER_PGD idmap_get_t0sz x5 cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? b.ge 1f // .. then skip VA range extension #if (VA_BITS < 48) +#define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) #define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) #define EXTRA_PTRS (1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT)) @@ -323,16 +313,16 @@ SYM_FUNC_START_LOCAL(__create_page_tables) mov x2, EXTRA_PTRS create_table_entry x0, x3, EXTRA_SHIFT, x2, x5, x6 #else +#define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) /* * If VA_BITS == 48, we don't have to configure an additional * translation level, but the top-level table has more entries. */ - mov x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT) #endif 1: adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14 /* * Map the kernel image (starting with PHYS_OFFSET). @@ -340,13 +330,12 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adrp x0, init_pg_dir mov_q x5, KIMAGE_VADDR // compile time __va(_text) add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD adrp x6, _end // runtime __pa(_end) adrp x3, _text // runtime __pa(_text) sub x6, x6, x3 // _end - _text add x6, x6, x5 // runtime __va(_end) - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 /* * Since the page tables have been populated with non-cacheable From patchwork Fri Jun 24 15:06:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5A8DC43334 for ; Fri, 24 Jun 2022 15:09:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zvl7u0IKdtJvlxdJPYrK1Zp+6aw8WHROzbUzEQcE0Do=; b=lw9Jh1wW5xqIZB d89FHM3y9sqpPdpBq7XFFMG/rdU1v8hCM64DStMI0ci2v+okgAUXnXVOLsfz2S2cLfAQBDFjCEHy+ sCjewh7bm2QEDAyVCIgGRhUCsjTYs8iPk9LT+e9xFNCa2LKzHOw0tac2gK4C97sU6fFd1cL7G7atP whI5EMbZBg8L/tA7nmp2fy0ASJxZ+z28qUa77Ou0pAxMaxEzsWJlPyOynj8A8Jm9nRNK/mO+iWNuu GdLERbOUbPqW/7qZifHObY9RQCCcs6W2lz1UTiiS+H56o0aLWD6+1nJVkoVmj/8+P2UyhyEVqzqzM GfjFwKsovoGwaZw8huxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuh-002idU-QV; Fri, 24 Jun 2022 15:08:03 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku5-002iNq-Sl for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9032CB826EE; Fri, 24 Jun 2022 15:07:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 758B9C341C0; Fri, 24 Jun 2022 15:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083243; bh=uR6DlXNQLjSiEeBqNN7df4CkoP+TjlQK6KQcfX40dhM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wv0G/HRB0HnRjHLqEkI5v8jMNilmKDKd648RlMFV7o5RhAPLNEappqCVLXtgb/EKr Jp5HdGJPZgO0+iCmOj01eL7kQVM44HvlZrJzzVqJVoNlCde3GpXhVXm1xlHbilrSx1 qU12DLbxWWz8+xuIluO58WtrdzIP8L9U+WyMcqIvX7KpZyAa49QCS+Tvh5MwXnUu+t pukghuOwIMaEeZA04uvUmFA2h1ZNs/iv5mhMlixdI/g0zs30A+wmelKy0jVqwKwPV4 dQ2lF6Z+55zDdcQwmX13Sblm6fELN1P/y6+K1z6IFyoomxuF1b9BegWM0wL3j+IahN EzgxXCUEQ/D3w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 06/21] arm64: head: switch to map_memory macro for the extended ID map Date: Fri, 24 Jun 2022 17:06:36 +0200 Message-Id: <20220624150651.1358849-7-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6360; h=from:subject; bh=uR6DlXNQLjSiEeBqNN7df4CkoP+TjlQK6KQcfX40dhM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLwdCRi/VuhL5KvoeTsPU+1VRO0P/nLOKhrzExp 7FJ3dg6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS8AAKCRDDTyI5ktmPJIu3C/ 9owHbJh90PiZN0+IwTjk8S3G92aDe2IBuawngzN1iKIuIcl6oeKJFOdJVc6uIFTGGU2xMQD21PY/jE WLKpGGktRodd7j2pHszCaKUVslSXbyEA/f9rs99HyQ552VzsvQ1UlgZSRneX3mOY1pIh1W0aMDmSX8 XnnrzZbzB3exUxIhZ+CHt27Tcs8C3CczeLYLkSHqpHooChX51Xk4TTS8nc5hb0S4Og6rHizucDY7+3 wn+R2QwakBKiXyoLBvMnsL52gEMZmPqxf9BqDm9diaedvNRwMvR4HXd3HXQ3ft2u36LA1xYL1KdcfV 28BvRx+57vnXpb39CI/16SoVYN9Aw/yn9CpG8izzb0rHS3a75WLXGuMtRNGHHBVUYS8RIGPXyfSSqU kaVcrqdrxnCNs0NEfTtYnjcOejz4Ye/d0LMLfjNXDAqQQVi0bhOv1bLgZfQo4WFcOP0NvRfBpUPOne ctYWPlQXtPK/Pf1ljk85te2SKnIQuC+tcaNjPj5rEgFUg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080726_262713_8A4E605B X-CRM114-Status: GOOD ( 22.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In a future patch, we will start using an ID map that covers the entire image, rather than a single page. This means that we need to deal with the pathological case of an extended ID map where the kernel image does not fit neatly inside a single entry at the root level, which means we will need to create additional table entries and map additional pages for page tables. The existing map_memory macro already takes care of most of that, so let's just extend it to deal with this case as well. While at it, drop the conditional branch on the value of T0SZ: we don't set the variable anymore in the entry code, and so we can just let the map_memory macro deal with the case where the output address exceeds VA_BITS. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 76 ++++++++++---------- 1 file changed, 37 insertions(+), 39 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 9fdde2f9cc0f..eb54c0289c8a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -122,29 +122,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b dcache_inval_poc // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -209,15 +186,27 @@ SYM_CODE_END(preserve_boot_args) * phys: physical address corresponding to vstart - physical memory is contiguous * order: #imm 2log(number of entries in PGD table) * + * If extra_shift is set, an extra level will be populated if the end address does + * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. + * * Temporaries: istart, iend, tmp, count, sv - these need to be different registers * Preserves: vstart, flags * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv + .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift sub \vend, \vend, #1 add \rtbl, \tbl, #PAGE_SIZE mov \count, #0 + .ifnb \extra_shift + tst \vend, #~((1 << (\extra_shift)) - 1) + b.eq .L_\@ + compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count + mov \sv, \rtbl + populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp + mov \tbl, \sv + .endif +.L_\@: compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp @@ -284,20 +273,32 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adrp x3, __idmap_text_start // __pa(__idmap_text_start) /* - * VA_BITS may be too small to allow for an ID mapping to be created - * that covers system RAM if that is located sufficiently high in the - * physical address space. So for the ID map, use an extended virtual - * range in that case, and configure an additional translation level - * if needed. + * The ID map carries a 1:1 mapping of the physical address range + * covered by the loaded image, which could be anywhere in DRAM. This + * means that the required size of the VA (== PA) space is decided at + * boot time, and could be more than the configured size of the VA + * space for ordinary kernel and user space mappings. + * + * There are three cases to consider here: + * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover + * the placement of the image. In this case, we configure one extra + * level of translation on the fly for the ID map only. (This case + * also covers 42-bit VA/52-bit PA on 64k pages). + * + * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can + * only happen when using 64k pages, in which case we need to extend + * the root level table rather than add a level. Note that we can + * treat this case as 'always extended' as long as we take care not + * to program an unsupported T0SZ value into the TCR register. + * + * - Combinations that would require two additional levels of + * translation are not supported, e.g., VA_BITS==36 on 16k pages, or + * VA_BITS==39/4k pages with 5-level paging, where the input address + * requires more than 47 or 48 bits, respectively. */ - idmap_get_t0sz x5 - cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? - b.ge 1f // .. then skip VA range extension - #if (VA_BITS < 48) #define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) #define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) -#define EXTRA_PTRS (1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT)) /* * If VA_BITS < 48, we have to configure an additional table level. @@ -309,20 +310,17 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif - - mov x2, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x2, x5, x6 #else #define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) +#define EXTRA_SHIFT /* * If VA_BITS == 48, we don't have to configure an additional * translation level, but the top-level table has more entries. */ #endif -1: adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14 + map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT /* * Map the kernel image (starting with PHYS_OFFSET). From patchwork Fri Jun 24 15:06:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A050C43334 for ; Fri, 24 Jun 2022 15:09:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FOurAzs4a8A6IC5S2tn6aZFIuLRWzR+9z6mRfQ+aE20=; b=bvrm1MZBrJHKvg ULbu1rPgM3v9uXkJYx4PObBc9XYrqUtZ6JApB16npRyFxzy4YO6NIPAeDcL75YvMCaZi1rUS6Vebi EAAiJLzopFlCDnmKr3XlXWPIhxKTK2WUOEh3/ULl6Q6JYwRZSIrz/Rg5BXIEqIMwZnGKZseHMLtFI kXZd4PvwP8zf08TUn7ySChw9fmiQ+2q0zsFJOfLEPsCEZIb9V2C5+OUMkIQ1XpL0pXtjGnVy4nv04 6YG7sy/iQT68WOIyuDtOkXkcOHjUFSL+vAxNAg44/NO/ksgRfbpHtYBQeJeFJeIgfi6ySCiv3aKqV 5nnsUlXpJvKLaXYFdS7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kv5-002ioi-D2; Fri, 24 Jun 2022 15:08:27 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4ku9-002iOP-Au for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EDD6EB828E0; Fri, 24 Jun 2022 15:07:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C229FC341C6; Fri, 24 Jun 2022 15:07:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083245; bh=s3JvUhBYEJDd0ZqtHxd2crGhZ2ZETuhTxYys0m9YjQo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sk0eKnSamJnfCS1h2u3YxTYiDlp6t/8/xjYxYZ+5r0CmvqF4o1APJ/OcZgPQ2JYgi wMrbv6+3QiW5iLoiI48GTljcjfR1FStYhJjVwxAlB/2DYttmHFK5Ah6ogY8G3T5Y/w I7Jd23wD9aaQQPCWQxn093FfIEjBOIhW7amQNueOJ5Yv+5RCJzx5e6BZ+woPovFJLz A9WZCCvkZiHPTkcMlVPchCh0zz/fA8peF4OXue5ZInPJiTyffbE16ANaHvcItHU2hA AEo1q8w4oo2/I2ErUdUAqlGcoVAXHJ9nNf+OiUH5FqhKwGTTCfY0iRxv/1SZ48w3VM 1IOD8QJFwnKBA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 07/21] arm64: head: split off idmap creation code Date: Fri, 24 Jun 2022 17:06:37 +0200 Message-Id: <20220624150651.1358849-8-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5970; h=from:subject; bh=s3JvUhBYEJDd0ZqtHxd2crGhZ2ZETuhTxYys0m9YjQo=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdLy+vcEnXVF4tjgr2Aqg9Q+44VVRNJUoCrjhMk3 /2PQQviJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS8gAKCRDDTyI5ktmPJLWdC/ 9NLsZYFlqBlTk6NyjyLKKFvwWQU19YXLlpk4bSZAx6q1nkw8CIqf4JjMMuyu0bI8D03ftNOPkesER/ b1mIea0ppErwpfDTSmZhrDWVDE+bayTMSQkAEUb3hB5zHp8xeWepo7SCDf4BKecSFj1CA7T+VYU9Bm 5SJBZUQ4L+PiQniflH12Wh/u4wb3RFfqrUN1R2yERNiCw4k2bwdtt4u/l4hMvi/i4Hqf/FFIdhUKrj oPW/jpalj9SJq5ltqH4Xy5q+0hcYP2M8WTUDuhLDx70nH0oOt29AoVjS32MBOqwa/IW0d7wcE5iALY pSlpcOfHRUYH0F2BOsFSAGh4fsxCwFnuaqo6M2t/c+xPX3GiYe71dQPT+VS+WyZk7LuOHPPYKcfsi3 UcDe457lasN588oRpbrvpeVKJqTN1qHRLv0nL1L+gXiMmvX5W7aoMdG4n4NnMvVzzHcfsxuH2O0sgp MqO2i5S8ttve8zAvV10ER5DbclNaME2l4oUzIH0L1rGlk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080729_707279_A2CB2538 X-CRM114-Status: GOOD ( 17.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Split off the creation of the ID map page tables, so that we can avoid running it again unnecessarily when KASLR is in effect (which only randomizes the virtual placement). This will permit us to drop some explicit cache maintenance to the PoC which was necessary because the cache invalidation being performed on some global variables might otherwise clobber unrelated variables that happen to share a cacheline. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 101 ++++++++++---------- 1 file changed, 52 insertions(+), 49 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index eb54c0289c8a..1cbc52097bf9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -84,7 +84,7 @@ * Register Scope Purpose * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset - * x28 __create_page_tables() callee preserved temp register + * x28 clear_page_tables() callee preserved temp register * x19/x20 __primary_switch() callee preserved temp registers * x24 __primary_switch() .. relocate_kernel() current RELR displacement */ @@ -94,7 +94,10 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag - bl __create_page_tables + bl clear_page_tables + bl create_idmap + bl create_kernel_mapping + /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for * details. @@ -122,6 +125,35 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b dcache_inval_poc // tail call SYM_CODE_END(preserve_boot_args) +SYM_FUNC_START_LOCAL(clear_page_tables) + mov x28, lr + + /* + * Invalidate the init page tables to avoid potential dirty cache lines + * being evicted. Other page tables are allocated in rodata as part of + * the kernel image, and thus are clean to the PoC per the boot + * protocol. + */ + adrp x0, init_pg_dir + adrp x1, init_pg_end + bl dcache_inval_poc + + /* + * Clear the init page tables. + */ + adrp x0, init_pg_dir + adrp x1, init_pg_end + sub x1, x1, x0 +1: stp xzr, xzr, [x0], #16 + stp xzr, xzr, [x0], #16 + stp xzr, xzr, [x0], #16 + stp xzr, xzr, [x0], #16 + subs x1, x1, #64 + b.ne 1b + + ret x28 +SYM_FUNC_END(clear_page_tables) + /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -231,44 +263,8 @@ SYM_CODE_END(preserve_boot_args) populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp .endm -/* - * Setup the initial page tables. We only setup the barest amount which is - * required to get the kernel running. The following sections are required: - * - identity mapping to enable the MMU (low address, TTBR0) - * - first few MB of the kernel linear mapping to jump to once the MMU has - * been enabled - */ -SYM_FUNC_START_LOCAL(__create_page_tables) - mov x28, lr - /* - * Invalidate the init page tables to avoid potential dirty cache lines - * being evicted. Other page tables are allocated in rodata as part of - * the kernel image, and thus are clean to the PoC per the boot - * protocol. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - bl dcache_inval_poc - - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x1, x1, x0 -1: stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - subs x1, x1, #64 - b.ne 1b - - mov x7, SWAPPER_MM_MMUFLAGS - - /* - * Create the identity mapping. - */ +SYM_FUNC_START_LOCAL(create_idmap) adrp x0, idmap_pg_dir adrp x3, __idmap_text_start // __pa(__idmap_text_start) @@ -319,12 +315,23 @@ SYM_FUNC_START_LOCAL(__create_page_tables) */ #endif adr_l x6, __idmap_text_end // __pa(__idmap_text_end) + mov x7, SWAPPER_MM_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT /* - * Map the kernel image (starting with PHYS_OFFSET). + * Since the page tables have been populated with non-cacheable + * accesses (MMU disabled), invalidate those tables again to + * remove any speculatively loaded cache lines. */ + dmb sy + + adrp x0, idmap_pg_dir + adrp x1, idmap_pg_end + b dcache_inval_poc // tail call +SYM_FUNC_END(create_idmap) + +SYM_FUNC_START_LOCAL(create_kernel_mapping) adrp x0, init_pg_dir mov_q x5, KIMAGE_VADDR // compile time __va(_text) add x5, x5, x23 // add KASLR displacement @@ -332,6 +339,7 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adrp x3, _text // runtime __pa(_text) sub x6, x6, x3 // _end - _text add x6, x6, x5 // runtime __va(_end) + mov x7, SWAPPER_MM_MMUFLAGS map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 @@ -342,16 +350,10 @@ SYM_FUNC_START_LOCAL(__create_page_tables) */ dmb sy - adrp x0, idmap_pg_dir - adrp x1, idmap_pg_end - bl dcache_inval_poc - adrp x0, init_pg_dir adrp x1, init_pg_end - bl dcache_inval_poc - - ret x28 -SYM_FUNC_END(__create_page_tables) + b dcache_inval_poc // tail call +SYM_FUNC_END(create_kernel_mapping) /* * Initialize CPU registers with task-specific and cpu-specific context. @@ -836,7 +838,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) pre_disable_mmu_workaround msr sctlr_el1, x20 // disable the MMU isb - bl __create_page_tables // recreate kernel mapping + bl clear_page_tables + bl create_kernel_mapping // Recreate kernel mapping tlbi vmalle1 // Remove any stale TLB entries dsb nsh From patchwork Fri Jun 24 15:06:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87A03C43334 for ; Fri, 24 Jun 2022 15:10:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9Mgb1GW2Pdp7VhpagXfffuXw55kj7ZcqEIY1yOU8uSY=; b=DEakeHdD7zqny1 rYitJ56oazMIAyBosf2+IcZdic0SffKDZVdaylGmYQb8hwx0w3gedf466jCQTIs3tVYdjlz3EwdBQ Fox4P1hjvPS8WivtIWEv3R29n1REKDBwoVbDiUvjlW1bhOTP5kM/irW3v+fxKjYrLsZUNDAh0TCQ9 XQQSrKPiMk9vkTNEH5U2gAn6syV0a7OnhcZbH5MwLyMcR+XCi7Dh/6Uzh9vBVwHYKqvVvtlakHtJh 07/Xui3REP6bro6GA768jrmXnfdg9GTrmzTkUdQP9X872fRDI8Z2GWRPc9+i3QXdpl/8hOFondu9U wqw6FjW9nDAJL/yfmbcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kve-002j9O-Md; Fri, 24 Jun 2022 15:09:03 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuA-002iPb-2U for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 939C6621CE; Fri, 24 Jun 2022 15:07:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 19F9AC341C0; Fri, 24 Jun 2022 15:07:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083248; bh=yCVJYYYykSt6KnWCMioWX1iDCDAGGrNVSzypc+CilpI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZhqYQq7z38DlJxBJEpIlaGbgRiXUk4XdPTYvr26vhRnkK3h56ayRHCQJrSe2+T9rG wz4CszyuVxJOBEGegDCkjzkwhu1a4khYzxJ1s95xq0ccQ2ZU4OALhgJ0UM7Eon15lB iwhNgSNcEEnUjkaE+MTntdekBgRLVOXVmjfvB1z0qd325rnlPVxdDW8GeCzvtZAoaA QCVEzRg1gd+jdKcVLpnb4uFd4RQJZrKYAi2vQ017fVE04M+UmqnLFS09J5ohIweAhf QtrsG4LafrD3OhhzINjPXfqfDR3RpB7V8ZduhNfEJeeF7NwyuMA609dFwWZic0+ewC YwUnEEST/ujFQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 08/21] arm64: kernel: drop unnecessary PoC cache clean+invalidate Date: Fri, 24 Jun 2022 17:06:38 +0200 Message-Id: <20220624150651.1358849-9-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2148; h=from:subject; bh=yCVJYYYykSt6KnWCMioWX1iDCDAGGrNVSzypc+CilpI=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL0HnTJXAduRed8vz+HQLHWvtUhbR0eJGPZ8t/F GSbvlKyJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS9AAKCRDDTyI5ktmPJIHSC/ 92bEDJS4/W40OF5cPmA3LMkXC4Ru/+XVz1Q3PYtKA3MuVAHwIkhF7BqEcc41iF6bhywm055cNl43yQ H321Vq4/VyYAI55a1QSkJMMIEnpIb7+23h0hNha/GIt7DMs8fRat5yYRhGciBaEkep1+OVc4LJI+Sx xqhyOVHJXAe5Sp/DHQlDqzQgx3emho5zi28vMo/mbKH7ez4s/4QhxIEG2QziWoVzpKm8hVLVeCQuv9 pRPYC4rHSnLbIxRMjvfHF7z/yvVzTc5aAYEFd9ouSt826Ws7cSXmchy5Lc3NMOIYSZko29n18kLJPf yJRYMkeSWgWPpehqKxZfCIJsz8Uck39Apnasww8Jo1GMNwuvfb2nzl1w6CU4QyVtNvKuf+enzdhXc3 0eaElXPfJFEstKbCQyCeOARy8buslOZAXjLCJAztdrHi/hvitTq3z7x/4k5dMrRhhFu6vJe0UAiGK3 88J6coUTkYJOrVoZu8pXRAsCmLT3IGqS/mIPfR9EZIBco= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080730_227301_D9FC0218 X-CRM114-Status: GOOD ( 16.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some early boot code runs before the virtual placement of the kernel is finalized, and we used to go back to the very start and recreate the ID map along with the page tables describing the virtual kernel mapping, and this involved setting some global variables with the caches off. In order to ensure that global state created by the KASLR code is not corrupted by the cache invalidation that occurs in that case, we needed to clean those global variables to the PoC explicitly. This is no longer needed now that the ID map is created only once (and the associated global variable updates are no longer repeated). So drop the cache maintenance that is no longer necessary. Signed-off-by: Ard Biesheuvel Reviewed-by: Anshuman Khandual --- arch/arm64/kernel/kaslr.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 418b2bba1521..d5542666182f 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -13,7 +13,6 @@ #include #include -#include #include #include #include @@ -72,9 +71,6 @@ u64 __init kaslr_early_init(void) * we end up running with module randomization disabled. */ module_alloc_base = (u64)_etext - MODULES_VSIZE; - dcache_clean_inval_poc((unsigned long)&module_alloc_base, - (unsigned long)&module_alloc_base + - sizeof(module_alloc_base)); /* * Try to map the FDT early. If this fails, we simply bail, @@ -174,13 +170,6 @@ u64 __init kaslr_early_init(void) module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21; module_alloc_base &= PAGE_MASK; - dcache_clean_inval_poc((unsigned long)&module_alloc_base, - (unsigned long)&module_alloc_base + - sizeof(module_alloc_base)); - dcache_clean_inval_poc((unsigned long)&memstart_offset_seed, - (unsigned long)&memstart_offset_seed + - sizeof(memstart_offset_seed)); - return offset; } From patchwork Fri Jun 24 15:06:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D44DC433EF for ; Fri, 24 Jun 2022 15:11:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YViJA9cgygjJTWBsMG5+ugpJaJyPnnapUCdgnhipYsU=; b=Mg+f8TW3bL95lI A1ee23v2os8LW8FxYCeik8UPnNFFR8vsTwGdwUsBuEWqv1IJOY7/FCkYJ2NuFhsfEs3DW1IVviZLw V8gLWIbJQrZVrHDYF3oDOTHaRxTKpMbq/jXUVSsIZ8Sl9CmNcZAiFTKC08hXHQi5cHgg/uy6llbiv L9MUHPzMC/O36uaLhTxWWITrCriPkyKO5JGW47qHDteA+CUCsIwTOvrzLM0Cn8IhbWnJ3aeowa1Uv 4yTdo7cN3qkV8Hbh2BqZgbw1K6tqdnapYsh3ZNP9Iexqw9pNyfC3SVRDv7jVRQ3qt/7WSO3u0Nq7g Qh+DQ0r+5kr/exVuLNGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kwV-002jWq-9q; Fri, 24 Jun 2022 15:09:55 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuB-002iQ5-QT for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:33 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 93695B82903; Fri, 24 Jun 2022 15:07:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66CFBC341C6; Fri, 24 Jun 2022 15:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083250; bh=nURZO8SxcJ6o/OcejTN7XMVfqtGnblUp7/ha5UnY1p0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PnBhAZB7noXFrUszBP1M/+r5HLkH792494NP0MbuSwm4rwud+I+xQlOOrXORoRvFl w1JdBshsewT3mpXR7NChWc7FeYj7XYHzAq2FJwvGkU3SlElmO3yl6M3chKpZXSQefJ F8sK8k7tbiQnkdFFrbBtimDHVonVjxGspFUzicWQ0ZDNbpo36KNfgDL6pFQrInOTfZ 76f7v2nRvZgyujJS3PeouSWSboJdGBZoZG4jFXYsWAEmTvX+xO5S1BX5rT2Car2JtE q5nyysJgE3xu4mWWJzfaGvd0OykZu8ycZ48sNJd5lFWugZa/lA2pBBANTUbUlt+W1F n8iA8V3SEvM0A== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 09/21] arm64: head: pass ID map root table address to __enable_mmu() Date: Fri, 24 Jun 2022 17:06:39 +0200 Message-Id: <20220624150651.1358849-10-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2427; h=from:subject; bh=nURZO8SxcJ6o/OcejTN7XMVfqtGnblUp7/ha5UnY1p0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL2GNtMDVXQU1a/9OpeykZkA53dK/xE5/HPcaRB YsfhWhOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS9gAKCRDDTyI5ktmPJB1lDA CSAtEmG1BljYYF1rBH8jUb+2kK7m5yNTxErauIl+Qg10vQPA3U9NaHcqs+ngaqu8zPesS7KYYfLC6g jMI+fX/BsbQ+3Dt7BILG9GLYG3PC1/fMA+ldj867w3Aug30fDiBq3w3sPeEU5aAM/DEGYEaEfD0d8k lC1062SOyJQqYtOkJsj7RJXfgy2QX2lAGhSwDiU5MkH8fNsYnLfjIgUJTVrUb9X8V6bTgldGCGYrMY Uq/nUvFjyzwTs75OvQ0dRtwb1DIjV5Qx45Knfhd7bPbUaMvJM8xHQXaSHkSlGh/zqI35yqBNU7pDhg 42xUTbwvhUnDlbiwle6ASK1+32VuDgVD9M7efgpBxY/fqu8xoNlQ6ayAMskymNHvHfuufT+kSp9fzD hGhfdinDv50HvYgRJ2BCr+kC7NcUwZuMKVButDRzV4QDGlWnZ4dHuYPxG1aJ5fsW9W9Rbd8kKGDo97 TRN10KzBirT1D3ELuOveRo3idmsydsNxCZGxQz0wFNaKI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080732_175122_CDC82BCC X-CRM114-Status: GOOD ( 14.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will be adding an initial ID map that covers the entire kernel image, so we will pass the actual ID map root table to use to __enable_mmu(), rather than hard code it. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 14 ++++++++------ arch/arm64/kernel/sleep.S | 1 + 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 1cbc52097bf9..70c462bbd6bf 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -595,6 +595,7 @@ SYM_FUNC_START_LOCAL(secondary_startup) bl __cpu_secondary_check52bitva bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir + adrp x2, idmap_pg_dir bl __enable_mmu ldr x8, =__secondary_switched br x8 @@ -648,6 +649,7 @@ SYM_FUNC_END(__secondary_too_slow) * * x0 = SCTLR_EL1 value for turning on the MMU. * x1 = TTBR1_EL1 value + * x2 = ID map root table address * * Returns to the caller via x30/lr. This requires the caller to be covered * by the .idmap.text section. @@ -656,14 +658,13 @@ SYM_FUNC_END(__secondary_too_slow) * If it isn't, park the CPU */ SYM_FUNC_START(__enable_mmu) - mrs x2, ID_AA64MMFR0_EL1 - ubfx x2, x2, #ID_AA64MMFR0_TGRAN_SHIFT, 4 - cmp x2, #ID_AA64MMFR0_TGRAN_SUPPORTED_MIN + mrs x3, ID_AA64MMFR0_EL1 + ubfx x3, x3, #ID_AA64MMFR0_TGRAN_SHIFT, 4 + cmp x3, #ID_AA64MMFR0_TGRAN_SUPPORTED_MIN b.lt __no_granule_support - cmp x2, #ID_AA64MMFR0_TGRAN_SUPPORTED_MAX + cmp x3, #ID_AA64MMFR0_TGRAN_SUPPORTED_MAX b.gt __no_granule_support - update_early_cpu_boot_status 0, x2, x3 - adrp x2, idmap_pg_dir + update_early_cpu_boot_status 0, x3, x4 phys_to_ttbr x1, x1 phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 @@ -819,6 +820,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) #endif adrp x1, init_pg_dir + adrp x2, idmap_pg_dir bl __enable_mmu #ifdef CONFIG_RELOCATABLE #ifdef CONFIG_RELR diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 4ea9392f86e0..e36b09d942f7 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -104,6 +104,7 @@ SYM_CODE_START(cpu_resume) bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir + adrp x2, idmap_pg_dir bl __enable_mmu ldr x8, =_cpu_resume br x8 From patchwork Fri Jun 24 15:06:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E14A8C433EF for ; Fri, 24 Jun 2022 15:11:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jQwZ70w+VibVCw/iNKkxqgste+SYj3ibtbpaxHPlp14=; b=nWiYfpwy5YH5kM y0NdY6Ck4Y8IegGFQEBIMUPbuqkWTHXAcJ0b2P2ukQIFPHXjUlgCwz70N/oWNhqd6wzlwehInYC3u hx9OfG3gl07TYP8BCTL/5cXQm8vHk/q1dYr5KCGDKPblOGpm2gwtV5J6o7p6rpspdzCf7YUIuH8N0 n4yHeGhF4mVTOZPbyZ1OxMaHVEJd/9BtrIFznGtdjtxq+0AxG8oQMX97XepP8DebKxnN5f5wKo7Sk NWvcb4DpplohNUarfHCS6FuG37ZW4ubc20XcoLLBNmZGMGyslFHJZhP50LReTL1WUCNJH7aI1KRgS GUysjlDzWYWFlM5yZc8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kx0-002jol-Cb; Fri, 24 Jun 2022 15:10:26 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuD-002iQn-IH for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:35 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2CE5062183; Fri, 24 Jun 2022 15:07:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B28F5C3411C; Fri, 24 Jun 2022 15:07:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083252; bh=0MsyA9t61JqcBmST0UVswIQsT/gQeOMoIkSz0EN+cIU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OIQ4A2s5yHsdi+5CnuVnGbEkMBeeRxJBN4va/KdSexQ1764/oCrIJw63f0zjnb40v A1GE70sb0p03b9dc+4GMZOnPX4torq2BNnO8p3yE6lMW24sRpsI731RKx/2JpkdCYT ABKrXRZHTPRzVA4apNs6FPcR4WA1kzZyKQ+ygPQQbo/oQbogGf5JWBReEV0QX6SuqR 3wnXx7mhgEpLnUxOE7AV48L1Ba05Cp69oxdc3bLcoXjdOor986fBTrkjPbPyVegr26 nOCJCmK+ryuc/VBMXhf9wQy5eoD86FBM1E3hcbFMDfOdLEHWHKUNpSSzWQm0I1pMO2 AgM/70y8jde+g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 10/21] arm64: mm: provide idmap pointer to cpu_replace_ttbr1() Date: Fri, 24 Jun 2022 17:06:40 +0200 Message-Id: <20220624150651.1358849-11-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4222; h=from:subject; bh=0MsyA9t61JqcBmST0UVswIQsT/gQeOMoIkSz0EN+cIU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL3hRuqafL/xIoEuXNbg3LMdFomZyNHk/7vOkny tPHyWSWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS9wAKCRDDTyI5ktmPJBEkC/ 9LVoH7MZWY4khhOa14LtF8RYh/dUIBhQQgv2+rW0HbMgmfuyai3nLedczsIqs1JM2b1ZmHifQF5487 5UMGIcem/zTYBmLjJ083QUk2FqH4N6Xz88edMbFuVmEUtyyvTOXov/dyFK+c0WgPlnXZiqhhUsVEIQ vOSaVMSYZ06Naqg8yJY2iJL43BFvgQep8+GrNx1JBMM1APNtarYUiWoN47kZ1m/0sGfkaf4Nma4gSv B8iBip4YyUoczOoAMC/k5zUUZeJ3mvzDk/kJL9wA38CqwDBZH+ylSaGndL6K75gcG1YHZsWQSsvP5Y mcGublE3hP7DObFwayoGRlXEzD3cTf6yAQgrWLmaLcCkwfc/pq4oijgoKGOd7WBsHucfy0/7ElsBcx Sa4HQEJUSs1VwERt/qkMeH0eK4QInuaVMprNDZnOp3CPM0n9Zaa9Zdk/JZTRIf5if5q1ba3fG58hxs sKS8yn+FBV7zwjTWpbpr6pZEawiSieJ6X2yew2ur35iq0= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080733_719972_9F95D900 X-CRM114-Status: GOOD ( 15.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for changing the way we initialize the permanent ID map, update cpu_replace_ttbr1() so we can use it with the initial ID map as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 13 +++++++++---- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/suspend.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- 5 files changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 7b387c3b312a..c7ccd82db1d2 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -105,13 +105,18 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void cpu_install_idmap(void) +static inline void __cpu_install_idmap(pgd_t *idmap) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); + cpu_switch_mm(lm_alias(idmap), &init_mm); +} + +static inline void cpu_install_idmap(void) +{ + __cpu_install_idmap(idmap_pg_dir); } /* @@ -142,7 +147,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp) +static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -165,7 +170,7 @@ static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp) replace_phys = (void *)__pa_symbol(function_nocfi(idmap_cpu_replace_ttbr1)); - cpu_install_idmap(); + __cpu_install_idmap(idmap); replace_phys(ttbr1); cpu_uninstall_idmap(); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 145b43c6c651..98b48d9069a7 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3266,7 +3266,7 @@ subsys_initcall_sync(init_32bit_el0_mask); static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap) { - cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); } /* diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 2b0887e58a7c..9135fe0f3df5 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -52,7 +52,7 @@ void notrace __cpu_suspend_exit(void) /* Restore CnP bit in TTBR1_EL1 */ if (system_supports_cnp()) - cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); /* * PSTATE was not saved over suspend/resume, re-enable any detected diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index c12cd700598f..e969e68de005 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -236,7 +236,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -280,7 +280,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); } static void __init kasan_init_depth(void) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3be0e9f69406..488628f07b89 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -780,7 +780,7 @@ void __init paging_init(void) pgd_clear_fixmap(); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); init_mm.pgd = swapper_pg_dir; memblock_phys_free(__pa_symbol(init_pg_dir), From patchwork Fri Jun 24 15:06:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AC13FC43334 for ; Fri, 24 Jun 2022 15:12:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IJz3jXCTLVA5Cq3bebVbp6Gc31RIfjUpqT/PjsR9Hjk=; b=OZ5DF5F4WT0FY4 hRU6aZdSGY1rBK/VMsLOsyTT4TK63Lop8dBlgc6VM9GEzxvUJ+KW7zC0cQL1YJpqaL0y7x0a9HXiR 6+6IQLN8BC31elTeOJLLuIkBZkJ0Kh6pooLL/9UTe3IZ+zQtfNKmdQIiNf3pRxNlSZuy4kWzESIZk m57fKv3IQMElEa+dUJ98f+BVA6xFAltbMJrbYoPAgj5373thmXjophuBU3NHLErJT647onLXo5yhV xRMaESm3Ne1jwaJ7UrkFCtgoVoBSv3kdja5LiLX3M9t7jXdfyyRTUM5bF9Q+97nhU6F9hezKTzPgq xbbLi349UeNt44Z6Ziyw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kxT-002k6F-9u; Fri, 24 Jun 2022 15:10:55 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuF-002iRX-U4 for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:37 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7F91D621CF; Fri, 24 Jun 2022 15:07:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0AD7AC341C6; Fri, 24 Jun 2022 15:07:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083254; bh=ATMp2RpRB2PDdPIzVcMeparFt7O9V0QyNBWnaeS/YG8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gCvmJDKPp1jF/3fwf0BPzhWNCk85ULVd+8r0gwZpWxZSGemGn9+IytIB9qboAaU8C KBTYYAW7+KCVH/3arzL277cBnAgmd3Fp71zSuqwEgSoyCpr0dyR+xN2kI3a/qjZC5J tRV1RWcbeEiTMMuUsZ2Afe9GxqUAUeZn6iNViCyjjc8AftUjOwul85Drp2/mTi1xQM hlHnVWVaDleEWeTN/AHMeiJnT9zGG4nloFS8vdEf30tqKRIT7+m1Tkx+czK7b9HGvS 7PjZAPMEt5H8n9KVxNJMBOZz0E79KziTI7maXtZLJj/ewg+qBjma0gSPxjGJiPeHnE r/aA9kYvgbp9w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 11/21] arm64: head: add helper function to remap regions in early page tables Date: Fri, 24 Jun 2022 17:06:41 +0200 Message-Id: <20220624150651.1358849-12-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2144; h=from:subject; bh=ATMp2RpRB2PDdPIzVcMeparFt7O9V0QyNBWnaeS/YG8=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL5w39YLBMwRoAQYQhZ2j8SZ/RBgRyKcPuKtY9O g1sfmsGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS+QAKCRDDTyI5ktmPJHadDA C9YP0GUsr6OQfW4UVDp+8dvFYQXQbJ4FzkPXGvWmMDH9gYtzTKGrgYYiIju4wApKKCPLPbJwGeRfON UkTgrOs3nXT93rCtuDqk95+/lZW7D9Jm5/rkYxEVWl2YNt8OPjPzcJLRCK2EKNwxwLVciUN01ZVZje 4UATyV05Ak6OBR1xlivv58K8xBXIrEIGFi/i1Bv3hcynJn49f3rRy7i2tx73VqyP6p4xHys6J4gBlT DlDuYFNYgZOS9ns7nhWs330jKGM+n/Ve6Bijr8TZUWmS739VHIK1XheVoGO8MALvZYR7lYjWQxk7te zN3KB8uZMYLw4tffh3DWMugu5QWx83h4DmY9SwAFTeWxJHgfVd61L57TxMmcBGppJMhwa8Py3HckCk xpuotr/Y7ES0tRg55lAJDWSiZ61r425mW88GZC33y0LpU8/wATFppWYkOpqVr2cTh2GCJZa2BnB3fF crJbCtjAKlf1ZPQSVESwKcGD6VG1xqp1txiBpFfomA0ek= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080736_087876_8CF449E9 X-CRM114-Status: GOOD ( 15.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The asm macros used to create the initial ID map and kernel mappings don't support randomly remapping parts of the address space after it has been populated. What we can do, however, given that all block or page mappings are created at the final level, is take a subset of the mapped range and update its attributes or output address. This will permit us to make parts of these page tables read-only, or remap a part of it to cover the device tree. So add a helper that encapsulates this. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 ++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 70c462bbd6bf..7397555f8437 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -263,6 +263,39 @@ SYM_FUNC_END(clear_page_tables) populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp .endm +/* + * Remap a subregion created with the map_memory macro with modified attributes + * or output address. The entire remapped region must have been covered in the + * invocation of map_memory. + * + * x0: last level table address (returned in first argument to map_memory) + * x1: start VA of the existing mapping + * x2: start VA of the region to update + * x3: end VA of the region to update (exclusive) + * x4: start PA associated with the region to update + * x5: attributes to set on the updated region + * x6: order of the last level mappings + */ +SYM_FUNC_START_LOCAL(remap_region) + sub x3, x3, #1 // make end inclusive + + // Get the index offset for the start of the last level table + lsr x1, x1, x6 + bfi x1, xzr, #0, #PAGE_SHIFT - 3 + + // Derive the start and end indexes into the last level table + // associated with the provided region + lsr x2, x2, x6 + lsr x3, x3, x6 + sub x2, x2, x1 + sub x3, x3, x1 + + mov x1, #1 + lsl x6, x1, x6 // block size at this level + + populate_entries x0, x4, x2, x3, x5, x6, x7 + ret +SYM_FUNC_END(remap_region) SYM_FUNC_START_LOCAL(create_idmap) adrp x0, idmap_pg_dir From patchwork Fri Jun 24 15:06:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 43F19C433EF for ; Fri, 24 Jun 2022 15:12:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4x1eZNwaQKE7De9a4mkUYUBlBwleohw37sLaVn14BCU=; b=vf43jFxMZ0LUJT sPC12z7BKaVX2XhPgl99yl1QXLYqAGjrZ1nZpMUJ0uuY49NZMmmR1EIWmwbEGNY7jFmb5LULTe5Qn HrdggotNPQaTjeeYkSIrifyOIY/rvf1NS/I6AvzFzRHt1PM9rW/p2cSKnQ66o9YX8zggAvZYoo51C 1/9S9DIih7lqozzUV7UDEYHsVh0Es8Z64P0L+FxCKkOqgxnfDqknhvkCoiWfgKe8fAX41CudpV3bf 9u0jg9OQLYASbQ4oYUOzqfzGVakAd9ALik1jNdVfnILrUKtoJWZBvBsAO4MeAWtcNpMoei9TPsXCS +uRCljfilmXT9xWopUWA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kxj-002kEQ-GR; Fri, 24 Jun 2022 15:11:11 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuI-002iSF-6F for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C7278621CF; Fri, 24 Jun 2022 15:07:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59BAFC341CB; Fri, 24 Jun 2022 15:07:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083257; bh=3Zt2L6OVj9yTVv2u0h6fQkgIEqmCDeEV6xGHSz+CA7k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HzJ02g5AhayMr0DamwHtcb9EKEcyO5gazw/v/b5pNv0c9SZxdFRdnDP4fuYKFw/9j oWXD+MYnaY1D/RVX1JXAaW+aXYAyH24gfjfOmseH1Gh8/J8wClx/n3fnzrnJyhyuqt vXgjGK/yOKG+Cetyi64C0I8pH6d58fs2OVphpOv0vPHFDCmHO1BNFG0E3RYSWztmQr I3gwz+6hNUvnyH4dDE4GMQUPANpP/uTr0LG8gafAGcG2MbHhqmYU/yzf6SXXTEMBf3 XxxcphdZGB8VJbdIkqeaYetpK6NGJpouVLITH5qGQdcwAh0O6uwH/E4kf1ySirVSZ9 2uhH9FZ0P2mew== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 12/21] arm64: head: cover entire kernel image in initial ID map Date: Fri, 24 Jun 2022 17:06:42 +0200 Message-Id: <20220624150651.1358849-13-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8929; h=from:subject; bh=3Zt2L6OVj9yTVv2u0h6fQkgIEqmCDeEV6xGHSz+CA7k=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL73RA1xa0OylJtAhwV8+S2LlZrc0VGzCXhKJwu 91oyKeWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS+wAKCRDDTyI5ktmPJMfSC/ 9jFWJ5+vxdFgwJQtj8hGPYN6mrRHLvVEfh9M8yHhNCx1MRLlaJwPSImzuewcvNEelERaLqu6k35JTz aTacHpxrKH/DdUgLlXAjWxRp6kZcynwz3yqueVSvJ+3sBxv6aeQKre+vJThE5UP8mnt4o/Wh2+N5J1 4FumRLj+oB+vqOawQ6Mcm8SCjJWBWnpGhWTLseKy9F4wa2WV6i6hFJZ2wZJIx9xByXHa+dicDQRZvl aAiWo3D22rSmJ0opIvT04yoRPXY2GGUAFj+aFUvsn/PNvx8wGrjcYyPtp7igPTRNT99txRyp4AmBtQ gXYaOLYpkRCPd6EjuNo6Chtrqn4fV9RvWDaqX5R5kGG8LirNuYABaTHQoCSbJIK3mJ+cJc5oN8S65n IkEl2EmR6z+uxzCZVnmjDTVn9NaTQLuAF3mkL5P+//ECGOH1xtMn2UVhwDIRfV9FdkgyMz9JbO2Lkz tkOWrNWlb6fNgwgz9F/+axyd+s9AntTpFcE1j4th/CttA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080738_358066_4309FFA9 X-CRM114-Status: GOOD ( 24.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As a first step towards avoiding the need to create, tear down and recreate the kernel virtual mapping with MMU and caches disabled, start by expanding the ID map so it covers the page tables as well as all executable code. This will allow us to populate the page tables with the MMU and caches on, and call KASLR init code before setting up the virtual mapping. Since this ID map is only needed at boot, create it as a temporary set of page tables, and populate the permanent ID map after enabling the MMU and caches. While at it, switch to read-only attributes for the where possible, as writable permissions are only needed for the initial kernel page tables. Note that on 4k granule configurations, the permanent ID map will now be reduced to a single page rather than a 2M block mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 16 ++++++--- arch/arm64/kernel/head.S | 31 +++++++++++------ arch/arm64/kernel/vmlinux.lds.S | 7 ++-- arch/arm64/mm/mmu.c | 35 +++++++++++++++++++- arch/arm64/mm/proc.S | 8 +++-- 5 files changed, 76 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 96dc0f7da258..5395e5a04f35 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,10 +35,8 @@ */ #if ARM64_KERNEL_USES_PMD_MAPS #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) -#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT) - 1) #else #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) -#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT)) #endif @@ -87,7 +85,13 @@ + EARLY_PUDS((vstart), (vend)) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend))) /* each PMD needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end)) -#define IDMAP_DIR_SIZE (IDMAP_PGTABLE_LEVELS * PAGE_SIZE) + +/* the initial ID map may need two extra pages if it needs to be extended */ +#if VA_BITS < 48 +#define INIT_IDMAP_DIR_SIZE (INIT_DIR_SIZE + (2 * PAGE_SIZE)) +#else +#define INIT_IDMAP_DIR_SIZE INIT_DIR_SIZE +#endif /* Initial memory map size */ #if ARM64_KERNEL_USES_PMD_MAPS @@ -107,9 +111,11 @@ #define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) #if ARM64_KERNEL_USES_PMD_MAPS -#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) +#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) +#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) #else -#define SWAPPER_MM_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) +#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) +#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) #endif /* diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7397555f8437..93734c91a29a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -87,6 +87,7 @@ * x28 clear_page_tables() callee preserved temp register * x19/x20 __primary_switch() callee preserved temp registers * x24 __primary_switch() .. relocate_kernel() current RELR displacement + * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl preserve_boot_args @@ -298,9 +299,7 @@ SYM_FUNC_START_LOCAL(remap_region) SYM_FUNC_END(remap_region) SYM_FUNC_START_LOCAL(create_idmap) - adrp x0, idmap_pg_dir - adrp x3, __idmap_text_start // __pa(__idmap_text_start) - + mov x28, lr /* * The ID map carries a 1:1 mapping of the physical address range * covered by the loaded image, which could be anywhere in DRAM. This @@ -347,11 +346,22 @@ SYM_FUNC_START_LOCAL(create_idmap) * translation level, but the top-level table has more entries. */ #endif - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - mov x7, SWAPPER_MM_MMUFLAGS + adrp x0, init_idmap_pg_dir + adrp x3, _text + adrp x6, _end + mov x7, SWAPPER_RX_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT + /* Remap the kernel page tables r/w in the ID map */ + adrp x1, _text + adrp x2, init_pg_dir + adrp x3, init_pg_end + bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 + mov x5, SWAPPER_RW_MMUFLAGS + mov x6, #SWAPPER_BLOCK_SHIFT + bl remap_region + /* * Since the page tables have been populated with non-cacheable * accesses (MMU disabled), invalidate those tables again to @@ -359,9 +369,10 @@ SYM_FUNC_START_LOCAL(create_idmap) */ dmb sy - adrp x0, idmap_pg_dir - adrp x1, idmap_pg_end - b dcache_inval_poc // tail call + adrp x0, init_idmap_pg_dir + adrp x1, init_idmap_pg_end + bl dcache_inval_poc + ret x28 SYM_FUNC_END(create_idmap) SYM_FUNC_START_LOCAL(create_kernel_mapping) @@ -372,7 +383,7 @@ SYM_FUNC_START_LOCAL(create_kernel_mapping) adrp x3, _text // runtime __pa(_text) sub x6, x6, x3 // _end - _text add x6, x6, x5 // runtime __va(_end) - mov x7, SWAPPER_MM_MMUFLAGS + mov x7, SWAPPER_RW_MMUFLAGS map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 @@ -853,7 +864,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) #endif adrp x1, init_pg_dir - adrp x2, idmap_pg_dir + adrp x2, init_idmap_pg_dir bl __enable_mmu #ifdef CONFIG_RELOCATABLE #ifdef CONFIG_RELR diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8a078c0ee140..0ce3a7c9f8c4 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -199,8 +199,7 @@ SECTIONS } idmap_pg_dir = .; - . += IDMAP_DIR_SIZE; - idmap_pg_end = .; + . += PAGE_SIZE; #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 tramp_pg_dir = .; @@ -236,6 +235,10 @@ SECTIONS __inittext_end = .; __initdata_begin = .; + init_idmap_pg_dir = .; + . += INIT_IDMAP_DIR_SIZE; + init_idmap_pg_end = .; + .init.data : { INIT_DATA INIT_SETUP(16) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 488628f07b89..e17eacfa048c 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -769,9 +769,40 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +static void __init create_idmap(void) +{ + u64 start = __pa_symbol(__idmap_text_start); + u64 size = __pa_symbol(__idmap_text_end) - start; + pgd_t *pgd = idmap_pg_dir; + u64 pgd_phys; + + /* check if we need an additional level of translation */ + if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { + pgd_phys = early_pgtable_alloc(PAGE_SHIFT); + set_pgd(&idmap_pg_dir[start >> VA_BITS], + __pgd(pgd_phys | P4D_TYPE_TABLE)); + pgd = __va(pgd_phys); + } + __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, + early_pgtable_alloc, 0); + + if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { + extern u32 __idmap_kpti_flag; + u64 pa = __pa_symbol(&__idmap_kpti_flag); + + /* + * The KPTI G-to-nG conversion code needs a read-write mapping + * of its synchronization flag in the ID map. + */ + __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, + early_pgtable_alloc, 0); + } +} + void __init paging_init(void) { pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); + extern pgd_t init_idmap_pg_dir[]; idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); @@ -780,13 +811,15 @@ void __init paging_init(void) pgd_clear_fixmap(); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); init_mm.pgd = swapper_pg_dir; memblock_phys_free(__pa_symbol(init_pg_dir), __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); memblock_allow_resize(); + + create_idmap(); } /* diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 97cd67697212..493b8ffc9be5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -249,8 +249,10 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) * * Called exactly once from stop_machine context by each CPU found during boot. */ -__idmap_kpti_flag: - .long 1 + .pushsection ".data", "aw", %progbits +SYM_DATA(__idmap_kpti_flag, .long 1) + .popsection + SYM_FUNC_START(idmap_kpti_install_ng_mappings) cpu .req w0 temp_pte .req x0 @@ -273,7 +275,7 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 - adr flag_ptr, __idmap_kpti_flag + adr_l flag_ptr, __idmap_kpti_flag cbnz cpu, __idmap_kpti_secondary From patchwork Fri Jun 24 15:06:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83CF6CCA473 for ; Fri, 24 Jun 2022 15:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AM/JYxlOWQa5+NgdC0nhizEZDDwE+ymarypW3OxJ9ag=; b=tShA8W1lt+Jb3j jD76N5ScwMeB4oKR1dVhe5xPmynTyN2bkfEKOihHOi1azZfFU3U7V9sNbbbSauGqmjA3QAdLcwC8A xKHk7L7KmZbEKZrbiaRPMoyOtClJQbRJmmiaunR1ofd3OAd5bIvECYGxCN1QBkxfg55B25MNNShof 653VYUyFX7HI8fRFVrETjZ8eTCpUger4etO4+uhnfRbRvwAjWgh7W/p6UVROCOZjaHCnbY7swZo3S bCGoe8znZdm+qNBnHWXIBSuRdTAaIOwszKUVyFI0mjWS0aM2QFDeTR9IaDY/9P1IILF8iIsWvk9rv L7SCek2dFn2AkSEV99lg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kyR-002ka1-Ch; Fri, 24 Jun 2022 15:11:55 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuK-002iSo-MC for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:42 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2365F621DA; Fri, 24 Jun 2022 15:07:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5F96C341C0; Fri, 24 Jun 2022 15:07:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083259; bh=fkPCxLjkxw99Z25WhFdlQ1f46LkWg6cY8n9VggShdSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fXgCvolcAuirzZ/Lv/JRRtMxpAOT2HJk8+hmtSi19KTl8u8jlWUpJMVFl1GfcNDBA F86hMfR5r28n3ArH4Cf9LvzFetP76slH0EKLmNSjNahTwYMDlUb/TS/T/RCH2GjAPZ zxw+a3IJQuTvMxlkpozh5glxC059/Z+T8c08wgP9vJKCZ22h9n9Z7CbRGh/gELa47/ uxQMSgpUWFimermh0NTTFDVvyrxIIokhN1mv3ZBsbFHjRKYFq0PDmafsCk1IcWrSE/ q80gRk0BI89Olf/+OZi0j1nLZhAA/RS7d/cU3rTI21JbZbTZAfYKg2kFUOj9zBC687 zsTYxOOAkpnSQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 13/21] arm64: head: use relative references to the RELA and RELR tables Date: Fri, 24 Jun 2022 17:06:43 +0200 Message-Id: <20220624150651.1358849-14-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2627; h=from:subject; bh=fkPCxLjkxw99Z25WhFdlQ1f46LkWg6cY8n9VggShdSU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL8M8XXo9wgguFZkrO8v1yr729BvJ/6sQvNs9yr HAmdJBqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS/AAKCRDDTyI5ktmPJFzlDA CGcAdcYfDYyeoZtRWsDecz7kY5OSHqKeX0e/1q7xiojEN0RdDhVBNzMcu9GescY7akbVpZ45wrNS6i rwtvipcoejw34c17+XUJ7ITnRDiW5O31wXSBwC6J+Y1LABqnyEbtVjJ8z0q6irWRsXlszvBCUXk+BQ OO7okeNWfWaKGHRI6BZMLVQjjWRMA2c9HMMs4uASMrolmG24T01oS2TR3KWDHce2s7/Ceml7oBBjIg pfHevfihywdX/b2bCWK2FVKzFN3fC0y80U8RLetJ8iQUF+coI4IUzMPMmfNqrxVRGnks/1vWne36TA sMH+DdKLOVn6tf1jSl2+/oDSeDuiOaqQ5X54ZSdTbfmXBKppC1y70yd+yp1xfOHgz6tQ8yOAGC2eYf gvTZHxpOI6nw54NauzxFvoQXZTIQVf+YlMgZPcOnepG8mNYi/Q+tgWBVuqhPPYbap6CU8SweQWlpd8 j61TPop0rccdgkbX/3rB84R4VCN5hUYp+fuveYrYQeXic= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080740_850265_95BE261F X-CRM114-Status: GOOD ( 15.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Formerly, we had to access the RELA and RELR tables via the kernel mapping that was being relocated, and so deriving the start and end addresses using ADRP/ADD references was not possible, as the relocation code runs from the ID map. Now that we map the entire kernel image via the ID map, we can simplify this, and just load the entries via the ID map as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 13 ++++--------- arch/arm64/kernel/vmlinux.lds.S | 12 ++++-------- 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 93734c91a29a..f1497f7b4da0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -757,13 +757,10 @@ SYM_FUNC_START_LOCAL(__relocate_kernel) * Iterate over each entry in the relocation table, and apply the * relocations in place. */ - ldr w9, =__rela_offset // offset to reloc table - ldr w10, =__rela_size // size of reloc table - + adr_l x9, __rela_start + adr_l x10, __rela_end mov_q x11, KIMAGE_VADDR // default virtual offset add x11, x11, x23 // actual virtual offset - add x9, x9, x11 // __va(.rela) - add x10, x9, x10 // __va(.rela) + sizeof(.rela) 0: cmp x9, x10 b.hs 1f @@ -813,10 +810,8 @@ SYM_FUNC_START_LOCAL(__relocate_kernel) * __relocate_kernel is called twice with non-zero displacements (i.e. * if there is both a physical misalignment and a KASLR displacement). */ - ldr w9, =__relr_offset // offset to reloc table - ldr w10, =__relr_size // size of reloc table - add x9, x9, x11 // __va(.relr) - add x10, x9, x10 // __va(.relr) + sizeof(.relr) + adr_l x9, __relr_start + adr_l x10, __relr_end sub x15, x23, x24 // delta from previous offset cbz x15, 7f // nothing to do if unchanged diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 0ce3a7c9f8c4..45131e354e27 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -257,21 +257,17 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { + __rela_start = .; *(.rela .rela*) + __rela_end = .; } - __rela_offset = ABSOLUTE(ADDR(.rela.dyn) - KIMAGE_VADDR); - __rela_size = SIZEOF(.rela.dyn); - -#ifdef CONFIG_RELR .relr.dyn : ALIGN(8) { + __relr_start = .; *(.relr.dyn) + __relr_end = .; } - __relr_offset = ABSOLUTE(ADDR(.relr.dyn) - KIMAGE_VADDR); - __relr_size = SIZEOF(.relr.dyn); -#endif - . = ALIGN(SEGMENT_ALIGN); __initdata_end = .; __init_end = .; From patchwork Fri Jun 24 15:06:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62CCFC43334 for ; Fri, 24 Jun 2022 15:13:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Mi0bMt1DD5mhEvc0PI2q1brtA2xYXMhEcA6q6XvG1QE=; b=fLmCzLqTB8ZE/g 2jIWAwdDU3XPf0jdmADXKB/QlU3LmWFvfLacX6LLqsNLqnTMka4U/gv2yFHlkT0V7llCHIphfsMVQ NokwknHAez308KVjGAhfCCr2+y9keI8bywlVSzU6BmzY2iCL+i6uUU9hExF32+rPK6gBF7JJWKfhQ xeby7TXgAcTc4BoU2RDW2Vfb8zPElVi0IxP+7mVl+YPv2Hqt2F0FKjK050HxC3QB72+YQJ4/tNhYX dnFj5chLrqzNt5NOop5xwdWCO8GhdLUbPfDi/Gsg1MeJU8Vj3jffMVUtPFHAWVxM7aAc4JIAbcmHJ tAkK/2YQ3I7onBXvfKIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kyw-002kpw-2h; Fri, 24 Jun 2022 15:12:26 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuM-002iTz-QO for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:44 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6AAD5621D5; Fri, 24 Jun 2022 15:07:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1F2DC341CC; Fri, 24 Jun 2022 15:07:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083261; bh=3M3w1vJqY9mv8D4SlAcYAqDWgq2ma+xNAc+sGP2xOWc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HwhrtEfbZuqcDNyMlAaLa5GnKmTsixP3/mdPxCGDsZr3z69sYpcMr5p+ngogUAp+U VUAC1kmxPpYM17DH6i6v9k4yW0FqRqw/oOuk3iuQ6Oh4Uz3ghVf9T0R7HlbMg43DBC 6Ea9uEre39Cu3tCMz1ldKzp6r/syh4HcHLXf34GOEeIx1B839CeVtmCnqCd6j3+2VF QmqIq+3zvKqqGQWnxvRbjUwtfVfIlsURk2Kxryz+UCkIE+bBdLH3BuF4HK8Bo6DmFk q7hnqeyLkPaXm6XTo+pcjFWpCMj8ZgVLWl8U7WDyhf1Ne3In1s3aj34eHJFyToWvXt KXLT6u6jtAnoA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 14/21] arm64: head: create a temporary FDT mapping in the initial ID map Date: Fri, 24 Jun 2022 17:06:44 +0200 Message-Id: <20220624150651.1358849-15-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3280; h=from:subject; bh=3M3w1vJqY9mv8D4SlAcYAqDWgq2ma+xNAc+sGP2xOWc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdL+Ijj/1XmmfyvHFzVRuLziz1o8TEy6KFnBQyZN PWEkwFeJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXS/gAKCRDDTyI5ktmPJKYoC/ 9vOEJKzv6p1FdqZLxdBas+3Q2HsQcZ7/DXkrVHGg0alY1VLh26Sw2ZIU6pOdVGLfeeuiWwh2VmvBDL vl7R7YAJ0w4MA7WVQ+EKGwjYv0ob9Sy7cxrmMSMkXnvGnAkOPfv33dEuD78pGaeqPorVnRwJ80O/mj xxh/18Pc+kMbwXKYGIRn0QTXPLu7U0ytHGP0iH9NBIYXyXsVTeL1PMVXhmcoirzIxZdsuqr3AWj1ih uujl+C+mQUo86+CcMEAzrw44p7LNm+WA1H+UTf9o6wIHIXtq2vEMVG6Dgngj3kynbiixRYPruIPguM ArN9D0vAJ+RVf8TwEEFFXE3vhEWqLKomvqKFNFnu+plqRkvmHBsB/GJmddpqgWAtWCuRvdV5yYudkZ 8WIcXIKZgHnINsOPMx+rq/CDOHgHkc71pLZj1YFt+njRoTNnqspkJCXwEy1q553eU9VLsYedyRogHv 5IjfvolSB6nyUIEb/Zg2FmatR8DluKn271mROL5LVF8us= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080742_994385_120C203A X-CRM114-Status: GOOD ( 15.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We need to access the DT very early to get at the command line and the KASLR seed, which currently means we rely on some hacks to call into the kernel before really calling into the kernel, which is undesirable. So instead, let's create a mapping for the FDT in the initial ID map, which is feasible now that it has been extended to cover more than a single page or block, and can be updated in place to remap other output addresses. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 6 ++++-- arch/arm64/kernel/head.S | 14 +++++++++++++- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 5395e5a04f35..02e59fa8f293 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -8,6 +8,7 @@ #ifndef __ASM_KERNEL_PGTABLE_H #define __ASM_KERNEL_PGTABLE_H +#include #include #include @@ -88,10 +89,11 @@ /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 -#define INIT_IDMAP_DIR_SIZE (INIT_DIR_SIZE + (2 * PAGE_SIZE)) +#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + 2) * PAGE_SIZE) #else -#define INIT_IDMAP_DIR_SIZE INIT_DIR_SIZE +#define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) #endif +#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE) /* Initial memory map size */ #if ARM64_KERNEL_USES_PMD_MAPS diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index f1497f7b4da0..8283ff848328 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -83,6 +83,7 @@ * * Register Scope Purpose * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 + * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset * x28 clear_page_tables() callee preserved temp register * x19/x20 __primary_switch() callee preserved temp registers @@ -348,7 +349,7 @@ SYM_FUNC_START_LOCAL(create_idmap) #endif adrp x0, init_idmap_pg_dir adrp x3, _text - adrp x6, _end + adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE mov x7, SWAPPER_RX_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT @@ -362,6 +363,17 @@ SYM_FUNC_START_LOCAL(create_idmap) mov x6, #SWAPPER_BLOCK_SHIFT bl remap_region + /* Remap the FDT after the kernel image */ + adrp x1, _text + adrp x22, _end + SWAPPER_BLOCK_SIZE + bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 + bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address + add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE + bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 + mov x5, SWAPPER_RW_MMUFLAGS + mov x6, #SWAPPER_BLOCK_SHIFT + bl remap_region + /* * Since the page tables have been populated with non-cacheable * accesses (MMU disabled), invalidate those tables again to From patchwork Fri Jun 24 15:06:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 369F6C43334 for ; Fri, 24 Jun 2022 15:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iKJJvop92rZV/OiP3s8GNmuldZ+70BZ90qN+mdUlxWc=; b=mvCwwxqP24V8tQ pjt+/Q0o4fO/WhSW0hkpGIhImyh2Xc0kc15QM3p60glWjgqlHaCkCCpX9Lkarz+7IWWoNCt5hW0cI QNMqk5R8yG6aJMJUGKEj34UXmGhifbPwOzdGXqqArO2/1C+iu0eD9rF8yuQUqYYydWmlirtTtUtbB 3thvHYnEosR5V6ALup/DQEykqe+EiD5CspHHYsQETWZnul5y3cHaE3tCQD7gCvXysg2ZRec8Oz9DW sgodubpoi3ou9pdv2eTK5yf0qxPmZHwKuRXvlHRU7FVCIsuqBy2P3stJ35KGmfzgbYlumIGizCSKt SGwG00wEZAqIb4BUdbfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kzL-002l46-J6; Fri, 24 Jun 2022 15:12:52 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuO-002iVK-QY for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 57635621D1; Fri, 24 Jun 2022 15:07:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A192C341C0; Fri, 24 Jun 2022 15:07:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083264; bh=eMPSGj9Rs5On/3JVQhKqjI71n1zDlZtNVzFx/c1LtqU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lvJw32fA+Wq47+BbMgKHOVbnHDTHBqpkfwncWiu/vmawdLTy1jW0h59YPxH5hJY5x vEPCo4ZoM5ZNO8SEZE7ORA9aiCIyfeZ38gaLZ63qN8dr9dV23QnjfH68rRakLYaAy6 MxdEoYh7xnaJ2CFAeR2rc6FZa48qg42OtQNeU2eRi31rHQ56sFoOrObI4gwIwmtNIg qqF/qEuDESmOaLD/3NLTcmSOVv1Yk1foitoD3R3HwQE8YscizkX2GWjkkD5YCrmo2h LZxKmJ1X50apq0V7w7dEGqBYutEdfz9oGE/UmKDPrX0qwklmPLf/x26IHbvmB5zBMv oOLWGkwYxeBXA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 15/21] arm64: idreg-override: use early FDT mapping in ID map Date: Fri, 24 Jun 2022 17:06:45 +0200 Message-Id: <20220624150651.1358849-16-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2544; h=from:subject; bh=eMPSGj9Rs5On/3JVQhKqjI71n1zDlZtNVzFx/c1LtqU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMAAofEOujGB2k/uwZkXxISjPLZssYFhlSgxCTO ISoo9cGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTAAAKCRDDTyI5ktmPJN9zC/ 9IYWwHzP5miObH+vE7Zt1q2CRWox4TYxZKylzz5whuFZ52uMr+K18xHlqaHHmYYawq7M8bX+oGuWi9 dLJXCUt+t7uUO44/iOwjwCcoZ7P2CNf4R1sirNoTXg9O34mcd/y/8X+sG1oJ7986/bXvbfh/Hz7IeC jB06uwi1IyiyktNF+kphXrX4Wm0nF40G1/I1NiTCquk57euPt51OmczNcUeTqVO+WJEZJ6EFYsYZbe 69ZiVw4htp2ZoCVmXAR2Zax+QxQMp1g3FPHrgXA6pAtCUqq8z+xUG1fPzA+UABcFUGbB6K2fLqMZDQ XMKIPQoGVy/6YG4ZVKeMgQI/D6zioXGzL+S+rGYxwGsjp1EBZoNGiEjVVRANzga3iycHVAkNEnbG6+ WGyNMA4U6LfKy7iOJnvINETWRa+aM2Ezmtf+5hDOkwdBWO6hftrElMJgbh8H2/xnUxHBVsPOo6Jkcy KwTfsQWXtNO9e0bQCJ7DOkSJGF3epmAvNujHgJii/7Mtk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080744_987645_5BF1471F X-CRM114-Status: GOOD ( 16.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of calling into the kernel to map the FDT into the kernel page tables before even calling start_kernel(), let's switch to the initial, temporary mapping of the device tree that has been added to the ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 1 + arch/arm64/kernel/idreg-override.c | 17 ++++++----------- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 8283ff848328..64ebff634b83 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -472,6 +472,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif mov x0, x21 // pass FDT address in x0 bl early_fdt_map // Try mapping the FDT early + mov x0, x22 // pass FDT address in x0 bl init_feature_override // Parse cpu feature overrides #ifdef CONFIG_RANDOMIZE_BASE tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 8a2ceb591686..f92836e196e5 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -201,16 +201,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -222,9 +217,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -234,9 +229,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(void); +void init_feature_override(const void *fdt); -asmlinkage void __init init_feature_override(void) +asmlinkage void __init init_feature_override(const void *fdt) { int i; @@ -247,7 +242,7 @@ asmlinkage void __init init_feature_override(void) } } - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { if (regs[i]->override) From patchwork Fri Jun 24 15:06:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9417C433EF for ; Fri, 24 Jun 2022 15:14:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=inx209NZYnYDSwxFm8Cir88ldlWH8lHPZGV9+0PSPpw=; b=PO8VxDH0fOSswP 87sM5zt4HjrPpG430oNf1vRQNMDTRGaE9Tlgm0e0VpzF60gDdidRTBdQU43L4TIjPqKlVFYUkHg2B EwxBPL8D+xlYtCfDsWfW7cNIpGGvVNoV+jGyt/LrJ5I30Pwlax0dAZp06Gc7q3MZC+pIE4taEhvoa xQLFj6pUsx+h7o8Mm5gwivw8My/GvDXADDQRfb7ljikvbjXi/or42+qSQeaJ/y0Nn6+N/qo0Q5Mxd CsmslXDiCZeFPkHw5EA8swEU0mHX1XcHsYjWUviyqeLdR0+wOwbyjMLd3yvnOrXEyPHVlGSXSww7k 1pJqiIyYdE29MiEsEncw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kzq-002lNy-Se; Fri, 24 Jun 2022 15:13:23 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuR-002iVv-Fd for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:48 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0D336621D1; Fri, 24 Jun 2022 15:07:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 95CE5C36AE2; Fri, 24 Jun 2022 15:07:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083266; bh=oLenfBhi8fKcOgGl+2bnK+Q39DnuPeJE3ucoNT8wFLc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EpsT1CPikBQlsvStPKUCqYT12xKy6XrDElO0aYbm0MsecaOsTUuGpAM5G+SiGcNeY 25odBS4BA+iImjy8lMVgXdzaEWvLz7GPUV+CKmZxWi3M7rZtLPdT1FegHfrHuhcBjr 2679y21RswxEt6aH/p8Lvsbzgn3ml1ARiYED0z7XIZ/7M2lmylGDx21xpvqKpa39IE 5ESz+GktnXR1iRLCoN0a5SxPgxsb/+1vXTZiFuQKvFfRr85aoyzUOOKG7eRjz5xkuC +vEoZoHlpLcbrcf7ZBSygsHah1DV5snbWaZquBDj6MVyCWoMr9bw05DbCoiWduTLVz eoDVmMvH9+Pwg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 16/21] arm64: head: factor out TTBR1 assignment into a macro Date: Fri, 24 Jun 2022 17:06:46 +0200 Message-Id: <20220624150651.1358849-17-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1896; h=from:subject; bh=oLenfBhi8fKcOgGl+2bnK+Q39DnuPeJE3ucoNT8wFLc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMCtHm4noxpzj7kvLJx1+gjXpQDDGgcKIYdEYsq sJU+MaGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTAgAKCRDDTyI5ktmPJKYKC/ 0RKzQWUENJSiPltWWCbvP64JYHo0d5iI/dqw6maKCSgi1VhhvYvBzn1jWI6UMjt+sfbcTzJ1erdZEO UyUmi5yMYicl8Rfv6J5n1XpI2BFlZgeYHayolozJc2yv0aqlIgeq96wB2C7hMugD9dKAWaxOqmW0WK RuqtGUfSREmGdZJRlGwUjgIJhLqtXJUIK5CnxALMm9Li6QEpt6ChPU5P8HPX9Vavt3V7Maripvp4O6 aVnc0pKjkfaptSNOO6TF3dTW3cs8z5QTGKTLwkohBM4Wv8nXeD2UIJEr9zeLxVOuQQbgSgu7kVEaoo p9osl0qV9955oKV7v6v1egIT069ms6NuT4DcVVjJA7RrtQYXtqAKdDQn+y2Ldtf+zgUKAKqT7irCtE 4Mu+V9dXfJvap2ES8Wgd+2NdeXqd1pkeEi4m32ueaatiBz+6DIbbQ/DDOKccWSLlSM4ZcUV/Ix2Puy RPOzrlcboU3BlvPK7cS3fo9C+anJiIl4WwzIkQKihKZik= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080747_635592_AFC72AD7 X-CRM114-Status: GOOD ( 12.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Create a macro load_ttbr1 to avoid having to repeat the same instruction sequence 3 times in a subsequent patch. No functional change intended. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 +++++++++++++---- arch/arm64/kernel/head.S | 5 +---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 9468f45c07a6..b2584709c332 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -479,6 +479,18 @@ alternative_endif _cond_extable .Licache_op\@, \fixup .endm +/* + * load_ttbr1 - install @pgtbl as a TTBR1 page table + * pgtbl preserved + * tmp1/tmp2 clobbered, either may overlap with pgtbl + */ + .macro load_ttbr1, pgtbl, tmp1, tmp2 + phys_to_ttbr \tmp1, \pgtbl + offset_ttbr1 \tmp1, \tmp2 + msr ttbr1_el1, \tmp1 + isb + .endm + /* * To prevent the possibility of old and new partial table walks being visible * in the tlb, switch the ttbr to a zero page when we invalidate the old @@ -492,10 +504,7 @@ alternative_endif isb tlbi vmalle1 dsb nsh - phys_to_ttbr \tmp, \page_table - offset_ttbr1 \tmp, \tmp2 - msr ttbr1_el1, \tmp - isb + load_ttbr1 \page_table, \tmp, \tmp2 .endm /* diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 64ebff634b83..d704d0bd8ffc 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -722,12 +722,9 @@ SYM_FUNC_START(__enable_mmu) cmp x3, #ID_AA64MMFR0_TGRAN_SUPPORTED_MAX b.gt __no_granule_support update_early_cpu_boot_status 0, x3, x4 - phys_to_ttbr x1, x1 phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 - offset_ttbr1 x1, x3 - msr ttbr1_el1, x1 // load TTBR1 - isb + load_ttbr1 x1, x1, x3 set_sctlr_el1 x0 From patchwork Fri Jun 24 15:06:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1FA25C43334 for ; Fri, 24 Jun 2022 15:15:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CWNqazggIztN9vf3X88U1F4wlKyWmFqw7jH7j1DVc40=; b=AZTuVfguvlAmam BI5DybMgDpqSaAOUQE9znt5dHhEjW20HSJEUAxo7NpE/JcWD08z2tpAisuwQu+AuiPa878XuRYqL1 vZ6jocfmuBjOm8U3ouSY2szRuM06t/GkbOpLgDrRpzsJKZqPV/9JTzmxZLwjo2Bi0bheNb9TO8Cik Y4DrBubtCLK3wkwZzUpP7EiXf+rwiXWBouX+2FxY5T7aNS7HWLkbHKp0mhlEMAs3DjhuPmTq9Tj8W qJ6NDhfxrYkVVxlMpxiPu0AOTNwKTSOgOL39sgPkJ+n5Ot79FItDb+8Jw+zhEH5DsI7WLSC4cFyt9 3E9qkKmoOF0zol0vkORQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l0Y-002lkd-9x; Fri, 24 Jun 2022 15:14:06 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuT-002iWo-Pj for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:51 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5CCD961FE1; Fri, 24 Jun 2022 15:07:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E1C70C36AEA; Fri, 24 Jun 2022 15:07:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083268; bh=XPxRHNk/ubPTm/4oa5ivGOXgzzl0krSBgTYBVHZzn/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jiUCva+QQ+Vna37qcxIzP+wtHu5m2pa3EHAuRPl+gUUDqkUgXvTLPbYVwvWLsTnR1 z8wU5xPPUbYvApJKAhxTS2EfPZfVDdSoN6DE/ojdpm5Njn6KayFssHh6SNAZix+R2E 25MZBVrAdDH2Zo27QxGX63fFmK1JUzXfHA/ZeAyWUPEpe2J3HhHNjrachpuqI1d6u7 onU7Tw2hl8P/zInM5M4F9wmCrz5VNF4HrxIPMQkpxO6+gFWBdLPIinBw8+HDErbyuv 0LOypCgBFzrSSz7cju0Uw+1/2yWFRXUNaCp50DFcWaOsq6ptRXgtmK4CXl+pjGKyij a5IYhEMh6ZC9g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 17/21] arm64: head: populate kernel page tables with MMU and caches on Date: Fri, 24 Jun 2022 17:06:47 +0200 Message-Id: <20220624150651.1358849-18-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4295; h=from:subject; bh=XPxRHNk/ubPTm/4oa5ivGOXgzzl0krSBgTYBVHZzn/w=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMDwPEE9uMR6lvdSMLCOrfG7zwhmejFVjDKWlLy Lvmjm3WJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTAwAKCRDDTyI5ktmPJCqEDA CefRTkTMyqOeaRdyqZ5ASJCwL+e4kLONFpAOv7y2smlDhz2IY/+H17m5AEmtWnKWf3sVycS/cFv1aS +q0rlQ4b8ZEa+7BYC7HkyZEa4mCqjAAnn6S+ygMlndmk67glOucWNH5PHYKfIqtZAjoPbcDuH37s8X vPYppLqoybhynYpJbdR7w8+TI/JnMxjwID2POZhN4tpVzei9Ew84elc/2znRwVJ/nD5d3GbCpytsuB O9pSs0wFGNUpumomF3F/7EmiwwuuR9YM3Flgsg4FlUpMZTp6LLz/l3MQChnra4gRFdhVKjgVLKC+Df moJ3uvshQBgCGoNMhNV4oJhYv3uzJnG1haYe4sV90BwkGdmHBta5J7vZlNmkdCTqkzY0lzxexnp93V XRSM4oQA3kAzc70Zlrkr9Eq7/YOSyYelm5ethS2FWvFpH4dY9dLkFRI2HjHTBvzlxYcH1KxEYeAt6n 4tJDl3jd29BzKvP/UDQ71L52Npj0bgfWqWHsRoPt44hII= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080749_972248_E3522A53 X-CRM114-Status: GOOD ( 15.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we can access the entire kernel image via the ID map, we can execute the page table population code with the MMU and caches enabled. The only thing we need to ensure is that translations via TTBR1 remain disabled while we are updating the page tables the second time around, in case KASLR wants them to be randomized. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 62 +++++--------------- 1 file changed, 16 insertions(+), 46 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index d704d0bd8ffc..583cbea865e1 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -85,8 +85,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset - * x28 clear_page_tables() callee preserved temp register - * x19/x20 __primary_switch() callee preserved temp registers * x24 __primary_switch() .. relocate_kernel() current RELR displacement * x28 create_idmap() callee preserved temp register */ @@ -96,9 +94,7 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag - bl clear_page_tables bl create_idmap - bl create_kernel_mapping /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -128,32 +124,14 @@ SYM_CODE_START_LOCAL(preserve_boot_args) SYM_CODE_END(preserve_boot_args) SYM_FUNC_START_LOCAL(clear_page_tables) - mov x28, lr - - /* - * Invalidate the init page tables to avoid potential dirty cache lines - * being evicted. Other page tables are allocated in rodata as part of - * the kernel image, and thus are clean to the PoC per the boot - * protocol. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - bl dcache_inval_poc - /* * Clear the init page tables. */ adrp x0, init_pg_dir adrp x1, init_pg_end - sub x1, x1, x0 -1: stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - stp xzr, xzr, [x0], #16 - subs x1, x1, #64 - b.ne 1b - - ret x28 + sub x2, x1, x0 + mov x1, xzr + b __pi_memset // tail call SYM_FUNC_END(clear_page_tables) /* @@ -399,16 +377,8 @@ SYM_FUNC_START_LOCAL(create_kernel_mapping) map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - dmb sy - - adrp x0, init_pg_dir - adrp x1, init_pg_end - b dcache_inval_poc // tail call + dsb ishst // sync with page table walker + ret SYM_FUNC_END(create_kernel_mapping) /* @@ -863,14 +833,15 @@ SYM_FUNC_END(__relocate_kernel) #endif SYM_FUNC_START_LOCAL(__primary_switch) -#ifdef CONFIG_RANDOMIZE_BASE - mov x19, x0 // preserve new SCTLR_EL1 value - mrs x20, sctlr_el1 // preserve old SCTLR_EL1 value -#endif - - adrp x1, init_pg_dir + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + bl clear_page_tables + bl create_kernel_mapping + + adrp x1, init_pg_dir + load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE #ifdef CONFIG_RELR mov x24, #0 // no RELR displacement yet @@ -886,9 +857,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) * to take into account by discarding the current kernel mapping and * creating a new one. */ - pre_disable_mmu_workaround - msr sctlr_el1, x20 // disable the MMU - isb + adrp x1, reserved_pg_dir // Disable translations via TTBR1 + load_ttbr1 x1, x1, x2 bl clear_page_tables bl create_kernel_mapping // Recreate kernel mapping @@ -896,8 +866,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) dsb nsh isb - set_sctlr_el1 x19 // re-enable the MMU - + adrp x1, init_pg_dir // Re-enable translations via TTBR1 + load_ttbr1 x1, x1, x2 bl __relocate_kernel #endif #endif From patchwork Fri Jun 24 15:06:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39497C433EF for ; Fri, 24 Jun 2022 15:15:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cmIOCwTHggcJJSJcIMyBD7RXT8R5w57azCGxd/plSW8=; b=cfnc1alMVyRypp eOYklCoM8xHcNnBvc6x+bFhzMdwLlRMG7b0gk4eFv/aq2d1MvsdM3FKYRPkWewAv/T2+MVHtnjJ/v 5XvFP0NQ5bmoOTrUwaWdXlZ3ps0vHqDQeqAtq8dwKVLiWpknCBlGehAotJt5dp50oAPEaTIVY0ynN ISCsTt2fa/cHaZRAGP2las1fDjbNBMTyWfWoaOu4mZIO20lZwhTFC1Z3wak+XqZl/rvZ+Moaoqo87 v2YkMxzrLjTbY6VYSe14qVGvhVkbY51DekLFesElgmWKLitOi1LqYCPn7fyzObkVamq5ArNdBTj8z U2TRl6TbCF/0mcY89BrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l16-002m0c-KF; Fri, 24 Jun 2022 15:14:41 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuX-002iXy-NO for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:55 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6C039B8293D; Fri, 24 Jun 2022 15:07:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39CFFC341C6; Fri, 24 Jun 2022 15:07:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083271; bh=/E6XrsNXjjeer9NBBlG34L/h2YChbTIYfexHwIflbPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YD43GMQ8f564A7hM2PttmQ9Gy1q79FedJgPCUsYL1KK9Pm2MHlJ7MO+Iznj0e1PHV sExUR/uWMvwVnO+lRvUQoohUiruCLuNXDTg862kDzJPudYgXwCKbB7gcv6vUnfvQL3 wiOTXuAhoQ154XhcE2JI2g7zwqT9Dj4scYYtxPWxB+tf4ESAlNZgtdnLZu7oRD8Eiv BqVGrWKJhtj96T1W4d1VKZkkvXywaIvx+7US4HAH0BBoaGBVAcvGMTL3zwayh/NWvJ bzefVeBOs1PktRdTNEeBjnb5riup/qsmtxRgy25MyGn7AabgQBIDqd6le/eXn2sOoP HD6hF8ZYDFUlg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 18/21] arm64: head: record CPU boot mode after enabling the MMU Date: Fri, 24 Jun 2022 17:06:48 +0200 Message-Id: <20220624150651.1358849-19-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5885; h=from:subject; bh=/E6XrsNXjjeer9NBBlG34L/h2YChbTIYfexHwIflbPQ=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMFCBWo5bs9hbAAudtVFUyluBkT4K9TRYdc8x/W KBqyrjKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTBQAKCRDDTyI5ktmPJBQXC/ 9aNqhEzNPrQJsilJ1rguCd9n5lskGC+ufVwWkmBD+31n2vMMcjTpBXi+PCp3mDUdYXghUo/5ZnrYzn BRIyRiP10wff855Ky7LcaovtQgFqbaTKezeN2JZSdXohST5z1fOToGWigl7Vsj+vYcGx8iIKYdBaw7 30omdx+K8BQHDpGrWHknLuxYvm3K2XrE29YWbljdrm7XnD6YiFTUfaumECfcnjPncgcVCeNpKTdngG nnMqZrl7ZxEDW98JEIC1fA1RIdeVtuSdt4xFvsZ2K/Ea1JeGK3rC//67tMcMGY1vabieykApSJLfSV hCdCii533mdGI5WcSqWdJiTjq3jORR/HmONCeiblEfWOwDttvqlXoYfic0rWTmkrIc7AYPyTl3EJHf sOmpwQINrEiwI1XTkGLAbjNqzUx1MKHLt6YHhu3G1ryCAcG/20xecSprO9VHoK4hVD9rJ8L+bSH37s /98toi5bRB8UHjenIBQMTg8wTe6UEASwIx1tkkZ0dzdpE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080754_119435_A2B2196C X-CRM114-Status: GOOD ( 18.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to avoid having to touch memory with the MMU and caches disabled, and therefore having to invalidate it from the caches explicitly, just defer storing the value until after the MMU has been turned on, unless we are giving up with an error. While at it, move the associated variable definitions into C code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 50 +++++--------------- arch/arm64/kernel/hyp-stub.S | 4 +- arch/arm64/mm/mmu.c | 8 ++++ 3 files changed, 23 insertions(+), 39 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 583cbea865e1..8de346dd4470 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,6 +82,7 @@ * primary lowlevel boot path: * * Register Scope Purpose + * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset @@ -91,9 +92,9 @@ SYM_CODE_START(primary_entry) bl preserve_boot_args bl init_kernel_el // w0=cpu_boot_mode + mov x20, x0 adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 - bl set_cpu_boot_mode_flag bl create_idmap /* @@ -429,6 +430,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappings + mov x0, x20 + bl set_cpu_boot_mode_flag + // Clear BSS adr_l x0, __bss_start mov x1, xzr @@ -454,6 +458,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) ret // to __primary_switch() 0: #endif + mov x0, x20 bl switch_to_vhe // Prefer VHE if possible ldp x29, x30, [sp], #16 bl start_kernel @@ -553,52 +558,21 @@ SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) b.ne 1f add x1, x1, #4 1: str w0, [x1] // Save CPU boot mode - dmb sy - dc ivac, x1 // Invalidate potentially stale cache line ret SYM_FUNC_END(set_cpu_boot_mode_flag) -/* - * These values are written with the MMU off, but read with the MMU on. - * Writers will invalidate the corresponding address, discarding up to a - * 'Cache Writeback Granule' (CWG) worth of data. The linker script ensures - * sufficient alignment that the CWG doesn't overlap another section. - */ - .pushsection ".mmuoff.data.write", "aw" -/* - * We need to find out the CPU boot mode long after boot, so we need to - * store it in a writable variable. - * - * This is not in .bss, because we set it sufficiently early that the boot-time - * zeroing of .bss would clobber it. - */ -SYM_DATA_START(__boot_cpu_mode) - .long BOOT_CPU_MODE_EL2 - .long BOOT_CPU_MODE_EL1 -SYM_DATA_END(__boot_cpu_mode) -/* - * The booting CPU updates the failed status @__early_cpu_boot_status, - * with MMU turned off. - */ -SYM_DATA_START(__early_cpu_boot_status) - .quad 0 -SYM_DATA_END(__early_cpu_boot_status) - - .popsection - /* * This provides a "holding pen" for platforms to hold all secondary * cores are held until we're ready for them to initialise. */ SYM_FUNC_START(secondary_holding_pen) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag - mrs x0, mpidr_el1 + mrs x2, mpidr_el1 mov_q x1, MPIDR_HWID_BITMASK - and x0, x0, x1 + and x2, x2, x1 adr_l x3, secondary_holding_pen_release pen: ldr x4, [x3] - cmp x4, x0 + cmp x4, x2 b.eq secondary_startup wfe b pen @@ -610,7 +584,6 @@ SYM_FUNC_END(secondary_holding_pen) */ SYM_FUNC_START(secondary_entry) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag b secondary_startup SYM_FUNC_END(secondary_entry) @@ -618,6 +591,7 @@ SYM_FUNC_START_LOCAL(secondary_startup) /* * Common entry point for secondary CPUs. */ + mov x20, x0 // preserve boot mode bl switch_to_vhe bl __cpu_secondary_check52bitva bl __cpu_setup // initialise processor @@ -629,6 +603,9 @@ SYM_FUNC_START_LOCAL(secondary_startup) SYM_FUNC_END(secondary_startup) SYM_FUNC_START_LOCAL(__secondary_switched) + mov x0, x20 + bl set_cpu_boot_mode_flag + str_l xzr, __early_cpu_boot_status, x3 adr_l x5, vectors msr vbar_el1, x5 isb @@ -691,7 +668,6 @@ SYM_FUNC_START(__enable_mmu) b.lt __no_granule_support cmp x3, #ID_AA64MMFR0_TGRAN_SUPPORTED_MAX b.gt __no_granule_support - update_early_cpu_boot_status 0, x3, x4 phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 load_ttbr1 x1, x1, x3 diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S index 43d212618834..5bafb53fafb4 100644 --- a/arch/arm64/kernel/hyp-stub.S +++ b/arch/arm64/kernel/hyp-stub.S @@ -223,11 +223,11 @@ SYM_FUNC_END(__hyp_reset_vectors) /* * Entry point to switch to VHE if deemed capable + * + * w0: boot mode, as returned by init_kernel_el() */ SYM_FUNC_START(switch_to_vhe) // Need to have booted at EL2 - adr_l x1, __boot_cpu_mode - ldr w0, [x1] cmp w0, #BOOT_CPU_MODE_EL2 b.ne 1f diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e17eacfa048c..a723bd2cfc27 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -56,6 +56,14 @@ EXPORT_SYMBOL(kimage_vaddr); u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); +u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; + +/* + * The booting CPU updates the failed status @__early_cpu_boot_status, + * with MMU turned off. + */ +long __section(".mmuoff.data.write") __early_cpu_boot_status; + /* * Empty_zero_page is a special page that is used for zero-initialized data * and COW. From patchwork Fri Jun 24 15:06:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B05F4C43334 for ; Fri, 24 Jun 2022 15:16:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=R3x63SaHKCaSCpwNCqzjpBCHs3GKksCj0FBJytWLfhI=; b=FJOsxEicV2YRmb IHRnw0KNn0l6Ii6hpIPPsfIy80pxDBVhm4zAeIbeHOqfj60q5PAFew8znLT9LEANTDLrowuFcWa4R tTSoqZ1wSua4stVhWVoberzSRrtAHbiUgnUB8SdZMijLjKI+Cz61/w3KKikENCYAIur3RmDKGhN5j lY+bZaR7e9ACkluMezxJOnsjzfDvCTdc182yA3XUAgpXP4DVrWqMv4c6a1UnLTNut+mgrGOzPsmKk djybekbQT/ct3LqEKdsOQ7Y6BkNDHtIXP2LbchF1YWd3vfWIXfBaUfTbOP3IXz6c9N4mvh4Tzinx+ d/hohsp6f6hfOoHR9hXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l20-002mSF-UQ; Fri, 24 Jun 2022 15:15:37 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kuZ-002iZ5-UT for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:57 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A1049B826EE; Fri, 24 Jun 2022 15:07:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 862F8C385A9; Fri, 24 Jun 2022 15:07:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083273; bh=WxMd2U0vEe6WJJERzTfuNkhM0m3g5WG4XwgS4+lXDls=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jUkG4bAaraBNwuhm6LrHClD6MMoaCqmK+tGOuGFLUAajcvZ/X9CWdtZUmCdgchexG Mp+EX59/NWAT7qEZjsLCkh5A6bL2/wrqygfAj0JMTsFQzyEPQHL6p6UtfgEjxTidaP yNAfQnE40hC5DLHtcoWElynrI+8KwlP3FYPPyl95W4y751vf6AbwDQyrllHbwnsp89 2bXHVCsy5Q2FldbdJw0xNORzckknhOuTFIEuuZtS1QYbux2PauiMHLHuFdQKjP+lX3 t9f3dLXnXIQiQbQWBp6E1lt1MfBGOv2wY25vAV30ksuI6bpSoA2U1AX6fwDzg8IAxv +320bf+62+09Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 19/21] arm64: kaslr: defer initialization to initcall where permitted Date: Fri, 24 Jun 2022 17:06:49 +0200 Message-Id: <20220624150651.1358849-20-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5546; h=from:subject; bh=WxMd2U0vEe6WJJERzTfuNkhM0m3g5WG4XwgS4+lXDls=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMHzs0uEKk6T3vZzs4kDBCgJ/oR6NzhisKS1Cjx U1VSCryJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTBwAKCRDDTyI5ktmPJOmEC/ kBMoArv60ibtd47Vwy/e69SmQJ8AcW7g5z/cP+suxNC2YvLH1Zu4dgbUWPZu5ORZjoNY5ANOVqUIkV 8uvKjh1/W7AFRai2rHWyQFOMrWHKCgit0E1ZMQpXQPrlng2pquqDY0TV829cj2JtZD73X+LGOmnIiJ 7HCoOyZmQr7JG0fuhCBQ1v4FAHTuhRVWbsdLrnLtx5n9+hgShFLsby51dlSFr8dNjEyCdNSjz3tIpY cJC9W0UrtJJww8alUTLjCIk4s5fzTUctwx84L3lMMYfiYCF8QrBnMuJR2Ol/pwgEtp4HFBRgyXlpAT V4DvWkTTsnzeoDPdW0n475sEuhHfDMQfd6r5DKYP/Kf8PjRKs1rsDMrftxj+JbbgycdlJhVJU1FQuU PF2x9G0qMr674pHMV1SSNexZhjAwYk51upZO4BwyzcdBZWDqnzcVCrN9iSXAqrPmVZA7cEkLhXGW3I LI0gZDTTv78kMKfIDZ8SHaA6K4pc2lwjglHfsy5i8LJc8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080756_343526_03E622FA X-CRM114-Status: GOOD ( 23.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The early KASLR init code runs extremely early, and anything that could be deferred until later should be. So let's defer the randomization of the module region until much later - this also simplifies the arithmetic, given that we no longer have to reason about the link time vs load time placement of the core kernel explicitly. Also get rid of the global status variable, and infer the status reported by the diagnostic print from other KASLR related context. While at it, get rid of the special case for KASAN without KASAN_VMALLOC, which never occurs in practice. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/kaslr.c | 95 +++++++++----------- 1 file changed, 40 insertions(+), 55 deletions(-) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index d5542666182f..3edee81d8ea7 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -20,14 +20,6 @@ #include #include -enum kaslr_status { - KASLR_ENABLED, - KASLR_DISABLED_CMDLINE, - KASLR_DISABLED_NO_SEED, - KASLR_DISABLED_FDT_REMAP, -}; - -static enum kaslr_status __initdata kaslr_status; u64 __ro_after_init module_alloc_base; u16 __initdata memstart_offset_seed; @@ -63,15 +55,9 @@ struct arm64_ftr_override kaslr_feature_override __initdata; u64 __init kaslr_early_init(void) { void *fdt; - u64 seed, offset, mask, module_range; + u64 seed, offset, mask; unsigned long raw; - /* - * Set a reasonable default for module_alloc_base in case - * we end up running with module randomization disabled. - */ - module_alloc_base = (u64)_etext - MODULES_VSIZE; - /* * Try to map the FDT early. If this fails, we simply bail, * and proceed with KASLR disabled. We will make another @@ -79,7 +65,6 @@ u64 __init kaslr_early_init(void) */ fdt = get_early_fdt_ptr(); if (!fdt) { - kaslr_status = KASLR_DISABLED_FDT_REMAP; return 0; } @@ -93,7 +78,6 @@ u64 __init kaslr_early_init(void) * return 0 if that is the case. */ if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { - kaslr_status = KASLR_DISABLED_CMDLINE; return 0; } @@ -106,7 +90,6 @@ u64 __init kaslr_early_init(void) seed ^= raw; if (!seed) { - kaslr_status = KASLR_DISABLED_NO_SEED; return 0; } @@ -126,19 +109,43 @@ u64 __init kaslr_early_init(void) /* use the top 16 bits to randomize the linear region */ memstart_offset_seed = seed >> 48; - if (!IS_ENABLED(CONFIG_KASAN_VMALLOC) && - (IS_ENABLED(CONFIG_KASAN_GENERIC) || - IS_ENABLED(CONFIG_KASAN_SW_TAGS))) - /* - * KASAN without KASAN_VMALLOC does not expect the module region - * to intersect the vmalloc region, since shadow memory is - * allocated for each module at load time, whereas the vmalloc - * region is shadowed by KASAN zero pages. So keep modules - * out of the vmalloc region if KASAN is enabled without - * KASAN_VMALLOC, and put the kernel well within 4 GB of the - * module region. - */ - return offset % SZ_2G; + return offset; +} + +static int __init kaslr_init(void) +{ + u64 module_range; + u32 seed; + + /* + * Set a reasonable default for module_alloc_base in case + * we end up running with module randomization disabled. + */ + module_alloc_base = (u64)_etext - MODULES_VSIZE; + + if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { + pr_info("KASLR disabled on command line\n"); + return 0; + } + + if (!kaslr_offset()) { + pr_warn("KASLR disabled due to lack of seed\n"); + return 0; + } + + pr_info("KASLR enabled\n"); + + /* + * KASAN without KASAN_VMALLOC does not expect the module region to + * intersect the vmalloc region, since shadow memory is allocated for + * each module at load time, whereas the vmalloc region will already be + * shadowed by KASAN zero pages. + */ + BUILD_BUG_ON((IS_ENABLED(CONFIG_KASAN_GENERIC) || + IS_ENABLED(CONFIG_KASAN_SW_TAGS)) && + !IS_ENABLED(CONFIG_KASAN_VMALLOC)); + + seed = get_random_u32(); if (IS_ENABLED(CONFIG_RANDOMIZE_MODULE_REGION_FULL)) { /* @@ -150,8 +157,7 @@ u64 __init kaslr_early_init(void) * resolved normally.) */ module_range = SZ_2G - (u64)(_end - _stext); - module_alloc_base = max((u64)_end + offset - SZ_2G, - (u64)MODULES_VADDR); + module_alloc_base = max((u64)_end - SZ_2G, (u64)MODULES_VADDR); } else { /* * Randomize the module region by setting module_alloc_base to @@ -163,33 +169,12 @@ u64 __init kaslr_early_init(void) * when ARM64_MODULE_PLTS is enabled. */ module_range = MODULES_VSIZE - (u64)(_etext - _stext); - module_alloc_base = (u64)_etext + offset - MODULES_VSIZE; } /* use the lower 21 bits to randomize the base of the module region */ module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21; module_alloc_base &= PAGE_MASK; - return offset; -} - -static int __init kaslr_init(void) -{ - switch (kaslr_status) { - case KASLR_ENABLED: - pr_info("KASLR enabled\n"); - break; - case KASLR_DISABLED_CMDLINE: - pr_info("KASLR disabled on command line\n"); - break; - case KASLR_DISABLED_NO_SEED: - pr_warn("KASLR disabled due to lack of seed\n"); - break; - case KASLR_DISABLED_FDT_REMAP: - pr_warn("KASLR disabled due to FDT remapping failure\n"); - break; - } - return 0; } -core_initcall(kaslr_init) +subsys_initcall(kaslr_init) From patchwork Fri Jun 24 15:06:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 165E5C43334 for ; Fri, 24 Jun 2022 15:17:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rpw/BidcUYTWeogpdTZDRbk7cATdPCxAfaIGPHWPw+c=; b=WWpZJbFvIza09m LdFslWdW0POBukwW3B6FpdEdHnNp4OIUF+UUl42aYHD7qUH/h9xDajtd0YAlgvZU3LKW5ZTDJaxJO pfHEfiIdAKjItl3C0fJUZ/ZzKNF/z7E9eNV25V62JaGEGfW6OSeD/Uw3suAaavckkP9h7SelhoA9L yGQlaVC4Tz2WT5y0SMpPcZQtnCn2RAjbBgCv1tXslqe3cmqudqB82oLiIzzvexoAJelH/SvbJxxpB dkEDPAfrEz0Hk386ciH1kBASwxFKpImcUoxxL2TGydtpDPstnfe2tKRsxODiUOqmuD0dqC+yQTYyf VllKlGuyXKFXXQ2vBVpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l2u-002ms6-6u; Fri, 24 Jun 2022 15:16:33 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kub-002iZo-1F for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:07:59 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 870F262076; Fri, 24 Jun 2022 15:07:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D67BCC341CC; Fri, 24 Jun 2022 15:07:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083276; bh=2FdrIps9q7ULLuQ5FXZqiKLrVnx9Kbw7E9+8DUxxC0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ox+PgZgVgE8x0s7irH1Uuo7AysqHI5S607kA9MQOYRO7C18EAVesa0jx2SVbyNZkr g6UehJrhMJFJqI0PS6J+8s5NwWf8CJIH88Nij7pljo6oAn+qiO/gii6qe8JWKrVjs5 Rv68T3K0NwPajK6Es8tOPiWZZc3nMAFjYisgfdLiZJJbFsmiKlcDmawKs6Ltlrj2sp Km+SNdy79BhJmHC3RjsQmXlG5twHEvh6sAg6wywCKanBUiB/rskvZVnNx35xjh2mbD DvDigrEi3xEkt+OHaKAC3ZfCkQyxZwuvijCkW/91qEDGVlpH4lxAhLXnhjwnVedRXh Btrq0+M8FeYCQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 20/21] arm64: head: avoid relocating the kernel twice for KASLR Date: Fri, 24 Jun 2022 17:06:50 +0200 Message-Id: <20220624150651.1358849-21-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=15594; h=from:subject; bh=2FdrIps9q7ULLuQ5FXZqiKLrVnx9Kbw7E9+8DUxxC0A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMIkmQm9j3v98sUZVv5s1zaEGqUcm/tU+Ta2g7m FH2gQY+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTCAAKCRDDTyI5ktmPJLePC/ 4oGpVSJwdDQOiW/d0RRyVmCiodIqdmhoTDtHacpaAnd7rW0OX+Lt+goutXXlJfV208r2pO0pZk+pcY jigG0475l8+wJ6HOQIY6d5kodgTOcExk+xiAGglRw0ftswyyhrBDuJAOxPhgwkU4NvRYoEtdy3XFIl QFI5HqrxlLTYWtGExcOvWZCAZoPtJ6EY3T2XdkQ28OlpH6LO/wz8Q5sc0I5lRIk2pNLAiVNt67w/qh SMo2O9cOFqtviUE27iUF0bFsY1cvIFO2Nz6XNmVR/UKUxCTfphm5hq2ODQeqRojACdXVnnxZQBwbbt Cun6KsRAfbpCzWmbV/lwQ0OHrHL8bfr57I5eOZhYfwTd+/QKtzPR4tWxOntvk/TqpI1QCU3dBydk0O iDcANyFVx/lvOXw7cRwpTNpUeB2YOhya5A4GarzJuIagV4WpJnRALUZOSMsidqb7RUEg1bcn6hASfI N+5MXqZ9jGO84hBE9XJNpcayarO8z8QIsCK/HCHgJege4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080757_250195_0BBA93A3 X-CRM114-Status: GOOD ( 35.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, when KASLR is in effect, we set up the kernel virtual address space twice: the first time, the KASLR seed is looked up in the device tree, and the kernel virtual mapping is torn down and recreated again, after which the relocations are applied a second time. The latter step means that statically initialized global pointer variables will be reset to their initial values, and to ensure that BSS variables are not set to values based on the initial translation, they are cleared again as well. All of this is needed because we need the command line (taken from the DT) to tell us whether or not to randomize the virtual address space before entering the kernel proper. However, this code has expanded little by little and now creates global state unrelated to the virtual randomization of the kernel before the mapping is torn down and set up again, and the BSS cleared for a second time. This has created some issues in the past, and it would be better to avoid this little dance if possible. So instead, let's use the temporary mapping of the device tree, and execute the bare minimum of code to decide whether or not KASLR should be enabled, and what the seed is. Only then, create the virtual kernel mapping, clear BSS, etc and proceed as normal. This avoids the issues around inconsistent global state due to BSS being cleared twice, and is generally more maintainable, as it permits us to defer all the remaining DT parsing and KASLR initialization to a later time. This means the relocation fixup code runs only a single time as well, allowing us to simplify the RELR handling code too, which is not idempotent and was therefore required to keep track of the offset that was applied the first time around. Note that this means we have to clone a pair of FDT library objects, so that we can control how they are built - we need the stack protector and other instrumentation disabled so that the code can tolerate being called this early. Note that only the kernel page tables and the temporary stack are mapped read-write at this point, which ensures that the early code does not modify any global state inadvertently. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 2 +- arch/arm64/kernel/head.S | 73 ++++--------- arch/arm64/kernel/image-vars.h | 4 + arch/arm64/kernel/kaslr.c | 87 --------------- arch/arm64/kernel/pi/Makefile | 33 ++++++ arch/arm64/kernel/pi/kaslr_early.c | 112 ++++++++++++++++++++ 6 files changed, 171 insertions(+), 140 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index fa7981d0d917..88a96511580e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -59,7 +59,7 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 8de346dd4470..5a2ff6466b6b 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -86,15 +86,13 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset - * x24 __primary_switch() .. relocate_kernel() current RELR displacement + * x24 __primary_switch() linear map KASLR seed * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl preserve_boot_args bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 - adrp x23, __PHYS_OFFSET - and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl create_idmap /* @@ -441,6 +439,10 @@ SYM_FUNC_START_LOCAL(__primary_switched) bl __pi_memset dsb ishst // Make zero page visible to PTW +#ifdef CONFIG_RANDOMIZE_BASE + adrp x5, memstart_offset_seed // Save KASLR linear map seed + strh w24, [x5, :lo12:memstart_offset_seed] +#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -448,16 +450,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) bl early_fdt_map // Try mapping the FDT early mov x0, x22 // pass FDT address in x0 bl init_feature_override // Parse cpu feature overrides -#ifdef CONFIG_RANDOMIZE_BASE - tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? - b.ne 0f - bl kaslr_early_init // parse FDT for KASLR options - cbz x0, 0f // KASLR disabled? just proceed - orr x23, x23, x0 // record KASLR offset - ldp x29, x30, [sp], #16 // we must enable KASLR, return - ret // to __primary_switch() -0: -#endif mov x0, x20 bl switch_to_vhe // Prefer VHE if possible ldp x29, x30, [sp], #16 @@ -759,27 +751,17 @@ SYM_FUNC_START_LOCAL(__relocate_kernel) * entry in x9, the address being relocated by the current address or * bitmap entry in x13 and the address being relocated by the current * bit in x14. - * - * Because addends are stored in place in the binary, RELR relocations - * cannot be applied idempotently. We use x24 to keep track of the - * currently applied displacement so that we can correctly relocate if - * __relocate_kernel is called twice with non-zero displacements (i.e. - * if there is both a physical misalignment and a KASLR displacement). */ adr_l x9, __relr_start adr_l x10, __relr_end - sub x15, x23, x24 // delta from previous offset - cbz x15, 7f // nothing to do if unchanged - mov x24, x23 // save new offset - 2: cmp x9, x10 b.hs 7f ldr x11, [x9], #8 tbnz x11, #0, 3f // branch to handle bitmaps add x13, x11, x23 ldr x12, [x13] // relocate address entry - add x12, x12, x15 + add x12, x12, x23 str x12, [x13], #8 // adjust to start of bitmap b 2b @@ -788,7 +770,7 @@ SYM_FUNC_START_LOCAL(__relocate_kernel) cbz x11, 6f tbz x11, #0, 5f // skip bit if not set ldr x12, [x14] // relocate bit - add x12, x12, x15 + add x12, x12, x23 str x12, [x14] 5: add x14, x14, #8 // move to next bit's address @@ -812,40 +794,27 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu - +#ifdef CONFIG_RELOCATABLE + adrp x23, __PHYS_OFFSET + and x23, x23, MIN_KIMG_ALIGN - 1 +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 + adrp x1, init_pg_end + mov sp, x1 + mov x29, xzr + bl __pi_kaslr_early_init + and x24, x0, #SZ_2M - 1 // capture memstart offset seed + bic x0, x0, #SZ_2M - 1 + orr x23, x23, x0 // record kernel offset +#endif +#endif bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE -#ifdef CONFIG_RELR - mov x24, #0 // no RELR displacement yet -#endif bl __relocate_kernel -#ifdef CONFIG_RANDOMIZE_BASE - ldr x8, =__primary_switched - adrp x0, __PHYS_OFFSET - blr x8 - - /* - * If we return here, we have a KASLR displacement in x23 which we need - * to take into account by discarding the current kernel mapping and - * creating a new one. - */ - adrp x1, reserved_pg_dir // Disable translations via TTBR1 - load_ttbr1 x1, x1, x2 - bl clear_page_tables - bl create_kernel_mapping // Recreate kernel mapping - - tlbi vmalle1 // Remove any stale TLB entries - dsb nsh - isb - - adrp x1, init_pg_dir // Re-enable translations via TTBR1 - load_ttbr1 x1, x1, x2 - bl __relocate_kernel -#endif #endif ldr x8, =__primary_switched adrp x0, __PHYS_OFFSET diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 241c86b67d01..0c381a405bf0 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -41,6 +41,10 @@ __efistub_dcache_clean_poc = __pi_dcache_clean_poc; __efistub___memcpy = __pi_memcpy; __efistub___memmove = __pi_memmove; __efistub___memset = __pi_memset; + +__pi___memcpy = __pi_memcpy; +__pi___memmove = __pi_memmove; +__pi___memset = __pi_memset; #endif __efistub__text = _text; diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 3edee81d8ea7..325455d16dbc 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -23,95 +23,8 @@ u64 __ro_after_init module_alloc_base; u16 __initdata memstart_offset_seed; -static __init u64 get_kaslr_seed(void *fdt) -{ - int node, len; - fdt64_t *prop; - u64 ret; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - return 0; - - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); - if (!prop || len != sizeof(u64)) - return 0; - - ret = fdt64_to_cpu(*prop); - *prop = 0; - return ret; -} - struct arm64_ftr_override kaslr_feature_override __initdata; -/* - * This routine will be executed with the kernel mapped at its default virtual - * address, and if it returns successfully, the kernel will be remapped, and - * start_kernel() will be executed from a randomized virtual offset. The - * relocation will result in all absolute references (e.g., static variables - * containing function pointers) to be reinitialized, and zero-initialized - * .bss variables will be reset to 0. - */ -u64 __init kaslr_early_init(void) -{ - void *fdt; - u64 seed, offset, mask; - unsigned long raw; - - /* - * Try to map the FDT early. If this fails, we simply bail, - * and proceed with KASLR disabled. We will make another - * attempt at mapping the FDT in setup_machine() - */ - fdt = get_early_fdt_ptr(); - if (!fdt) { - return 0; - } - - /* - * Retrieve (and wipe) the seed from the FDT - */ - seed = get_kaslr_seed(fdt); - - /* - * Check if 'nokaslr' appears on the command line, and - * return 0 if that is the case. - */ - if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { - return 0; - } - - /* - * Mix in any entropy obtainable architecturally if enabled - * and supported. - */ - - if (arch_get_random_seed_long_early(&raw)) - seed ^= raw; - - if (!seed) { - return 0; - } - - /* - * OK, so we are proceeding with KASLR enabled. Calculate a suitable - * kernel image offset from the seed. Let's place the kernel in the - * middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of - * the lower and upper quarters to avoid colliding with other - * allocations. - * Even if we could randomize at page granularity for 16k and 64k pages, - * let's always round to 2 MB so we don't interfere with the ability to - * map using contiguous PTEs - */ - mask = ((1UL << (VA_BITS_MIN - 2)) - 1) & ~(SZ_2M - 1); - offset = BIT(VA_BITS_MIN - 3) + (seed & mask); - - /* use the top 16 bits to randomize the linear region */ - memstart_offset_seed = seed >> 48; - - return offset; -} - static int __init kaslr_init(void) { u64 module_range; diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile new file mode 100644 index 000000000000..839291430cb3 --- /dev/null +++ b/arch/arm64/kernel/pi/Makefile @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2022 Google LLC + +KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ + -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(call cc-option,-mbranch-protection=none) \ + -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ + -include $(srctree)/include/linux/hidden.h \ + -D__DISABLE_EXPORTS -ffreestanding -D__NO_FORTIFY \ + $(call cc-option,-fno-addrsig) + +# remove SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) +# disable LTO +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS)) + +GCOV_PROFILE := n +KASAN_SANITIZE := n +KCSAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ + --remove-section=.note.gnu.property \ + --prefix-alloc-sections=.init +$(obj)/%.pi.o: $(obj)/%.o FORCE + $(call if_changed,objcopy) + +$(obj)/lib-%.o: $(srctree)/lib/%.c FORCE + $(call if_changed_rule,cc_o_c) + +obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c new file mode 100644 index 000000000000..6c3855e69395 --- /dev/null +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2022 Google LLC +// Author: Ard Biesheuvel + +// NOTE: code in this file runs *very* early, and is not permitted to use +// global variables or anything that relies on absolute addressing. + +#include +#include +#include +#include +#include +#include + +#include +#include + +/* taken from lib/string.c */ +static char *__strstr(const char *s1, const char *s2) +{ + size_t l1, l2; + + l2 = strlen(s2); + if (!l2) + return (char *)s1; + l1 = strlen(s1); + while (l1 >= l2) { + l1--; + if (!memcmp(s1, s2, l2)) + return (char *)s1; + s1++; + } + return NULL; +} +static bool cmdline_contains_nokaslr(const u8 *cmdline) +{ + const u8 *str; + + str = __strstr(cmdline, "nokaslr"); + return str == cmdline || (str > cmdline && *(str - 1) == ' '); +} + +static bool is_kaslr_disabled_cmdline(void *fdt) +{ + if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { + int node; + const u8 *prop; + + node = fdt_path_offset(fdt, "/chosen"); + if (node < 0) + goto out; + + prop = fdt_getprop(fdt, node, "bootargs", NULL); + if (!prop) + goto out; + + if (cmdline_contains_nokaslr(prop)) + return true; + + if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) + goto out; + + return false; + } +out: + return cmdline_contains_nokaslr(CONFIG_CMDLINE); +} + +static u64 get_kaslr_seed(void *fdt) +{ + int node, len; + fdt64_t *prop; + u64 ret; + + node = fdt_path_offset(fdt, "/chosen"); + if (node < 0) + return 0; + + prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + if (!prop || len != sizeof(u64)) + return 0; + + ret = fdt64_to_cpu(*prop); + *prop = 0; + return ret; +} + +asmlinkage u64 kaslr_early_init(void *fdt) +{ + u64 seed; + + if (is_kaslr_disabled_cmdline(fdt)) + return 0; + + seed = get_kaslr_seed(fdt); + if (!seed) { +#ifdef CONFIG_ARCH_RANDOM + if (!__early_cpu_has_rndr() || + !__arm64_rndr((unsigned long *)&seed)) +#endif + return 0; + } + + /* + * OK, so we are proceeding with KASLR enabled. Calculate a suitable + * kernel image offset from the seed. Let's place the kernel in the + * middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of + * the lower and upper quarters to avoid colliding with other + * allocations. + */ + return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 0)); +} From patchwork Fri Jun 24 15:06:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12894761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2CDAC43334 for ; Fri, 24 Jun 2022 15:19:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SjHdbP7wfejKUWYTssPwZY37rtyo/Txu5LQN/IL9Ozk=; b=yxLjVQd3ZpsaQ8 l4hVTpCc0T8zNp1YoBbDcZgy/UnQ+PP5zM222Tz+Rh/S2x1jCdd1R0aD7a2t8AVfHs2J/O7XCHeMA XlL4jGWQDGMDGT1Lm+utrB4XyTOcudxO4dh6qlyPoJcce3JMDiRvn8SFRJ+yDMdy6rlv78JAK7fo/ JkHKJpmtRqPN1JGhv0xn4N9oDHgmNYTwwATuCMSIryEs3eSL6Mpf2yBoX3qRGXfDP/Zhrav4bWPPH HSCQtzBpF5z65XAKoouW17opUO7+Uo+y06Z8Joyby4BnLHAUtxwte6dq2G5P1ghsKHNzAhJisTkVE cLiTF68HGrdpOzh/8IHw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4l46-002nSQ-La; Fri, 24 Jun 2022 15:17:46 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o4kue-002ibx-Pv for linux-arm-kernel@lists.infradead.org; Fri, 24 Jun 2022 15:08:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7DA20B826EE; Fri, 24 Jun 2022 15:07:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62691C341C6; Fri, 24 Jun 2022 15:07:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656083278; bh=spuzcEpOzLucXw/XWBnP+9vTyaCMnPCSjSbXqwVTSXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=imkiINQ94Z8xm9fXxEStbT2tGk5FoSJfpwCELEnu2V5GtTzifm19JPTzDkUfB69gf kRFC4b9geM1kqr5N53SaSyhXJy8MAq9uhRQVYGabnBEvBlo0tCv1eltDOqXorogcZx il53yLYTY5G+4D54PHhEtEymQQqAPyTm9VS0e1kQnN1rYZ58dUzzVQsAqHMHVSrzIb Z/HDMQlYHHaZo5Op7OOqAFr28QlCZ3ZxExwMzaZeH74WLE7w95ikubZWHlmmAwiBIh r1c57As2sE5iDHqN1YDa0ZRQvJbaRNFmDPfOEoXFiVSMm7rITrP3bdD/+As/tcGPXX E4rHoKs7pgHGg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v5 21/21] arm64: setup: drop early FDT pointer helpers Date: Fri, 24 Jun 2022 17:06:51 +0200 Message-Id: <20220624150651.1358849-22-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220624150651.1358849-1-ardb@kernel.org> References: <20220624150651.1358849-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1995; h=from:subject; bh=spuzcEpOzLucXw/XWBnP+9vTyaCMnPCSjSbXqwVTSXw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBitdMK6zUbA6XTOleHxbAowfqGCZXFaEpywDdD89Xj ryghsdCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYrXTCgAKCRDDTyI5ktmPJDBkDA CWDQ/n8VPAw/MRed1TXRarmqtOJOyc5/Eh9kIslgghTfCxTgHpKcfl6eZCGmP+x+Q9p+uh1xyF11RB zHVVw1mSio3dHTs6A3pPL5p+cJqFmvDSY51s/OLVJKFE2EkT/yDtMl3XGyhdaFawJboXvudkCaMEK6 4F7HVhgaLB3IM5hwKURccXKM7rlnwhsUYwJRIyV3FLj2Iu1nRFYMA7yKojVd8OcQW2UryD+jbtw1HF ag6e+ACMqN+RtHUxC16W2k57eObPycjjXymIPkRGCgr8crTLrUZP36cDvsPr+hRO8/QObOVIDFi3og 8gS8/BIhDhMcYqtgdfHGNGitZV2RujBQyh9IxJWlGNCdEpQMf56FfMH0blPFCkahRks2fXX6MH2hsU 1zm7i5OGdQIzAIeIh22auZrripAd+m/nWldTT/qWlMcseeeMGmgxIk5je3fXQKgC0VgMfOeB/88Sq4 le+GX9PEBXfn94k18v7ihUEGCfZx0XiExucfKgl7GdY5o= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220624_080801_145318_FBF184B7 X-CRM114-Status: GOOD ( 12.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We no longer need to call into the kernel to map the FDT before calling into the kernel so let's drop the helpers we added for this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/head.S | 2 -- arch/arm64/kernel/setup.c | 15 --------------- 3 files changed, 20 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index 6437df661700..5f147a418281 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -5,9 +5,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 5a2ff6466b6b..6bf685f988f1 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -446,8 +446,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x22 // pass FDT address in x0 bl init_feature_override // Parse cpu feature overrides mov x0, x20 diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index fea3223704b6..d0e6c7a291da 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -163,21 +163,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size;