From patchwork Mon Jan 9 17:47:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13094047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63E81C54EBD for ; Mon, 9 Jan 2023 17:52:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6XhyYpZgMnezd2E7124lxfncOqTkW8rbotGllk3C9ZQ=; b=liupfeooTRgHKq qHm/uscLhkWkAPZXID2O4szr3qw/YwU3QQTE19NyMhxcSEdAtW5D4YG66wL+J5r0+KJK+UPnL6QXA X+efU0YkRSSRFJ+E6oZvDnFn/oYie5p/5PM8aYZwQgGHGyT6hJcI2x1w+IPCAtp1WjTulaLXop2mC Wppfbml31X+D+sdYRQvrb8rU6z2qSqJNe0WhI6Yuivnwb8iGPRJC7FdYOsLFTTOaV5UEjJb9pxJ8j Del1L6N3Gl70VnpE/PqE3yQQw/wwnuoRhxuMrfIVTaJIKPGmIJSQbEtqxQxcop1Rkc2d/zPYpGPL5 qT1p7pmJ3ih5GHCw+pUQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEwIk-0031ho-0G; Mon, 09 Jan 2023 17:51:15 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEwFn-00304X-DH for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 17:48:14 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D646761299; Mon, 9 Jan 2023 17:48:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B623C433D2; Mon, 9 Jan 2023 17:48:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673286490; bh=3/offehC15LZ6/BvqOb+3fbnk4F90UgfhDAUzleeXtU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YhE/JuDWg9kUepq9rqJ2F0cRA/hfZaIjFz7g6s1qHlkHnMENeIcqbgU+/b955wypZ 2ZjDHcWSuO/jfE7Qwwr8FWJ55jKWtGCvYwcZfvacOA0dmMaKx4JNz3rQz+cF+QQrUS qoGSy9J+xZkEOROkkcttLR61K5/O99i/NwmKdJyiAZSQ50I5wrgUck9iV9sSFE/PLf Kl95AOfc2oXamtG+x2DNiGyQHDkysqjvshktPGWcja455Le8BtMm1ATZwYsUPYmoDe Yzzi7tFRDAkOx2vc1u7pLB99zNLZWH0KQv2FMpS+ZwQd6j5wARhNlUBCka5Nvwf1L/ yMlhyU2ZV7Vtg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH v2 1/2] arm64: Always load shadow stack pointer directly from the task struct Date: Mon, 9 Jan 2023 18:47:59 +0100 Message-Id: <20230109174800.3286265-2-ardb@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230109174800.3286265-1-ardb@kernel.org> References: <20230109174800.3286265-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2302; i=ardb@kernel.org; h=from:subject; bh=3/offehC15LZ6/BvqOb+3fbnk4F90UgfhDAUzleeXtU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjvFNOaEusriFg80EEK+K7rfPqTe+bTnBV+M3T4Ujq nybBulqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY7xTTgAKCRDDTyI5ktmPJLR4C/ 9NM1LnJgh+iv3KCnLT9M5ZlG3ZxYu1GcvkbFmtGDuH5zCyFP1xFj+ODLStBPVNjH2Hr7ZQMe0wLbvh N/ox/N/EFjYDyJRaU2LC2Ms1OsmDdLibPlTomVxSAw4xx8z/TpdBJDfc+8dmLqUOhKahob3e93/Prp uwxscKF5RE+3mbnoeMQ6U2uNC6RIJfIB/7uNuOlJs/j4EE1OQ0w8rtEAtmacPhU5Uo5WPPH7K3ja3P tkWVVRXLvFpYe73+3ODWYRIcxCJyvoZSYgeCvAjGXri7+4tHqgjqJ8WDUKHLzUcJKPxgnnvRY+jkO3 tGeEr3+jleezqZIXg/GaZ6cTFg7GIqJjTImH2ey+iG2jwNFr+y/vo21bIGUSB7osPXEhhXRObrty+z 1U3CztJAqDyw23SbjSrwBvn9lVp4/ROvHD4zSfgJwTOMKnOJChoUwO79n3QVumwS8sr/CxIB8esnGi fDSp1jvKqmuRxwoZBbvfnAoiZorgomnAaD7zlhG17KLDc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_094812_000146_6E05F45D X-CRM114-Status: GOOD ( 12.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org All occurrences of the scs_load macro load the value of the shadow call stack pointer from the task which is current at that point. So instead of taking a task struct register argument in the scs_load macro to specify the task struct to load from, let's always reference the current task directly. This should make it much harder to exploit any instruction sequences reloading the shadow call stack pointer register from memory. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland Reviewed-by: Kees Cook --- arch/arm64/include/asm/scs.h | 7 ++++--- arch/arm64/kernel/entry.S | 4 ++-- arch/arm64/kernel/head.S | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index ff7da1268a52ab79..13df982a080805e6 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -10,15 +10,16 @@ #ifdef CONFIG_SHADOW_CALL_STACK scs_sp .req x18 - .macro scs_load tsk - ldr scs_sp, [\tsk, #TSK_TI_SCS_SP] + .macro scs_load_current + get_current_task scs_sp + ldr scs_sp, [scs_sp, #TSK_TI_SCS_SP] .endm .macro scs_save tsk str scs_sp, [\tsk, #TSK_TI_SCS_SP] .endm #else - .macro scs_load tsk + .macro scs_load_current .endm .macro scs_save tsk diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 11cb99c4d298784d..546f7773238ea45d 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -275,7 +275,7 @@ alternative_if ARM64_HAS_ADDRESS_AUTH alternative_else_nop_endif 1: - scs_load tsk + scs_load_current .else add x21, sp, #PT_REGS_SIZE get_current_task tsk @@ -848,7 +848,7 @@ SYM_FUNC_START(cpu_switch_to) msr sp_el0, x1 ptrauth_keys_install_kernel x1, x8, x9, x10 scs_save x0 - scs_load x1 + scs_load_current ret SYM_FUNC_END(cpu_switch_to) NOKPROBE(cpu_switch_to) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 952e17bd1c0b4f91..b9c1a506798ea315 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -404,7 +404,7 @@ SYM_FUNC_END(create_kernel_mapping) stp xzr, xzr, [sp, #S_STACKFRAME] add x29, sp, #S_STACKFRAME - scs_load \tsk + scs_load_current adr_l \tmp1, __per_cpu_offset ldr w\tmp2, [\tsk, #TSK_TI_CPU] From patchwork Mon Jan 9 17:48:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13094049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7DB71C5479D for ; Mon, 9 Jan 2023 17:54:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EjhEcytgZuv+iS0E1dSvW6ExF28BJM4jDPE/XwbI/hI=; b=0qe6/qM2MSHZ6Y dv5xPOFZtZBlP3YJbHib8uLHiP4MEZDMz0YtLc5zWOpe3vEjvrJgUmaG5f3MYX7ggeUy8sKzzzIDa UEzQBGaImso8Eq0N79+KBp7xpfkG0o5dvp9h3kMzuJFfLvFOgo1OejfFIZn0PXzmKVUtVv9UGvCnb hgDbG97jqX6tRM82C11UiH4vCrywPcKQYa0WaSCGLd5dyh+u/shbbIFudLarrodaixpX2wWhGan/N 8GPFpVeIkaNDmCxyzH8C0orkb0U4vv/JBnJ1hFZ27Nb4tMssnwNw0Qv3SF4U7NLf1TAsEPs1lfh1L uY1v31H4MVcOptjRcU5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEwKP-0032PI-Rq; Mon, 09 Jan 2023 17:52:58 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pEwFs-00306K-5J for linux-arm-kernel@lists.infradead.org; Mon, 09 Jan 2023 17:48:17 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3432A61295; Mon, 9 Jan 2023 17:48:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 808CFC433F0; Mon, 9 Jan 2023 17:48:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673286492; bh=wJHQcoruZ5sn0xjNYyGtLPmw6+8llRPdoaN3bIGrJjo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kLTqdVikeN6QwCa0bqpsrMyIgrNoTt/XgJT3ikqNwCWz/bHak+ciQXwu1z+Yzk8Ff 1zDfTC9FrYRZxH20zCN54E9FtUhbLLsA4xHUonJTEZXL77Gaqgbs/P9ovTlR0RxfIq /FMbwfAEtqKj2iwJ1VsUiYSdT8xW9zJZygnNaXWPRyr98WogbFASsew9btS+1abDmn JFk3RJ2Vpxn1wJRY7WA6Q7J9CpcJt3/qe5J/My7GmiLHaw09QQYnDHwjji7CW8UGkY 4+VOamC7RbAtkpFZcKZBLKoUO/kbPq68qQBmL4huc7sWzIvcv3957Dmj7/aX4eaL+X mwVPrny3gq3tg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH v2 2/2] arm64: Stash shadow stack pointer in the task struct on interrupt Date: Mon, 9 Jan 2023 18:48:00 +0100 Message-Id: <20230109174800.3286265-3-ardb@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230109174800.3286265-1-ardb@kernel.org> References: <20230109174800.3286265-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2053; i=ardb@kernel.org; h=from:subject; bh=wJHQcoruZ5sn0xjNYyGtLPmw6+8llRPdoaN3bIGrJjo=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjvFNPKWJUWx27MHGT2IueUCSIqH3xyQiym/SngNzL b1+2fweJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY7xTTwAKCRDDTyI5ktmPJGZqC/ 0YaXA4ix2I3CbwZaJCNfpmV4yOZ+MFluiU71sNlRK3Jfii2Uy0Sq9BNUHFNU2EHPTCTWR0Gxyfq6o1 sqlQIfcFo+BUU5z3Aos7n79nw17MpM9DWJMI9uCHQUQvXAiae7tRQVfBnZtATlVi0TbwXWpmJ0h65y +VsvHjj5fn23w+v17QJbbQgkTagI9tzhL13Pz3TflTNPU7y0q1s7SPT9La8zAJOhTkfsQ6OQ/x1ZjX jrfy416Lu4I2GKs6xjzSC0zwJ5Ih+9SZ9MEUmfsHNEb2WPeeNhLLNuxAA+4ctgFqt2WUv0VTzRDtzr dggwHh70WtuNxSaQcurtewp0bXBmj/IQHU/9Lk0zdqam+6fC96qcqcqZ0xps5flE1yXCTiBa7J/uvH tOdB64lmKBAB9n7tM/vmVOBiW7qS86XhEOngzl8KfGP372p+QwF8IPg8DXNL/IqdYcA0Ry4DjXG5k1 vii9feuOKXYujF5Wq/FJOBLfPuMRobVQwqWdsDqq0esgk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230109_094816_456799_F8D8D3B5 X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of reloading the shadow call stack pointer from the ordinary stack, which may be vulnerable to the kind of gadget based attacks shadow call stacks were designed to prevent, let's store a task's shadow call stack pointer in the task struct when switching to the shadow IRQ stack. Given that currently, the task_struct::scs_sp field is only used to preserve the shadow call stack pointer while a task is scheduled out or running in user space, reusing this field to preserve and restore it while running off the IRQ stack must be safe, as those occurrences are guaranteed to never overlap. (The stack switching logic only switches stacks when running from the task stack, and so the value being saved here always corresponds to the task mode shadow stack) While at it, fold a mov/add/mov sequence into a single add. Signed-off-by: Ard Biesheuvel Acked-by: Mark Rutland Reviewed-by: Kees Cook --- arch/arm64/kernel/entry.S | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 546f7773238ea45d..80d763e165fc5856 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -876,19 +876,19 @@ NOKPROBE(ret_from_fork) */ SYM_FUNC_START(call_on_irq_stack) #ifdef CONFIG_SHADOW_CALL_STACK - stp scs_sp, xzr, [sp, #-16]! + get_current_task x16 + scs_save x16 ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x17 #endif + /* Create a frame record to save our LR and SP (implicit in FP) */ stp x29, x30, [sp, #-16]! mov x29, sp ldr_this_cpu x16, irq_stack_ptr, x17 - mov x15, #IRQ_STACK_SIZE - add x16, x16, x15 /* Move to the new stack and call the function there */ - mov sp, x16 + add sp, x16, #IRQ_STACK_SIZE blr x1 /* @@ -897,9 +897,7 @@ SYM_FUNC_START(call_on_irq_stack) */ mov sp, x29 ldp x29, x30, [sp], #16 -#ifdef CONFIG_SHADOW_CALL_STACK - ldp scs_sp, xzr, [sp], #16 -#endif + scs_load_current ret SYM_FUNC_END(call_on_irq_stack) NOKPROBE(call_on_irq_stack)