From patchwork Mon Dec 5 20:03:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13065054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4417C4332F for ; Mon, 5 Dec 2022 20:04:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QvC7asorMGk5H6C/ewJxz3kCj3Qtq8z8XLdJ/6/RP/g=; b=EwYJFLZBHwwjVK mkcTjV4FGZYwUKcisxPMYS4SjMCft4K9YM88Y7u+ZTeZAGmgjDXPKnb6dkcg04s7E4BA+C4ixKWNi 9qeo6ldVNFGFWmRkf6Fw8FKJWaFQO9WhLVtgOuMsMnWAz4P2KXCV8aPKtY0OnRBz2xThidYqjM3Uz oZoYjqszEh22VI2iNuIxJfI8xoRVW5yVQ8xfDlA0yuuC1/Xz5vFRYpeEhakgXe7csML3P0QfNsJqt xuEPsFo4XXPl93/iW7Qk+c2skRz/97mNWptxiBqDslIPtfsaln2aixn6qwcMzbS2DH82jGrQoNWyz 8mNjtTuBv2OaExSv0FBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2Hh3-009t7b-Fk; Mon, 05 Dec 2022 20:04:01 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2Hgs-009sx0-8M for linux-arm-kernel@lists.infradead.org; Mon, 05 Dec 2022 20:03:51 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 44F256135A; Mon, 5 Dec 2022 20:03:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 41858C43470; Mon, 5 Dec 2022 20:03:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670270628; bh=E3ICNkcIoy9x0qZKI76uWn6D/RtU0iT2JOcqKwdg+Aw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KO2Efon2PLJ59W576ZzvVTb9IJVrXD51aV7HZKI4HAGLW4veqaIvmO36EbpxJjK0+ DVfOhjro6gxrSj0zXaS8/kboGN58kyM38s7dH/FS+qd58cNkx+jVcBB09PH06YsBgW PbqRhYUfp25Rqz+DXGMXdMFaXHaaUJZuZeyPRUAL2BHfESB2Qlerz2hMeChD/RzZ9+ 2hHBkH564YhDMpIxirbVj98oLUPkcNzZXaX5A/34rf5uGUuaOO0reJ9sDIU4cKo8K8 iaIRNn9q1m+xyKbJjPPqHEoeSJNFNAQIieo+FbVyddiQKQrz4Psdiyc047ayx0ZAQY 5hNsWkVtolCwg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH 1/2] arm64: Always load shadow stack pointer directly from the task struct Date: Mon, 5 Dec 2022 21:03:40 +0100 Message-Id: <20221205200341.463601-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221205200341.463601-1-ardb@kernel.org> References: <20221205200341.463601-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2300; i=ardb@kernel.org; h=from:subject; bh=E3ICNkcIoy9x0qZKI76uWn6D/RtU0iT2JOcqKwdg+Aw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjjk6azITPi2yRUJTPaVnS4mInKARE8lovhwg2lZAC 17VQeqiJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY45OmgAKCRDDTyI5ktmPJGgTC/ 43nhcvCDKqk3AmPnDT785gBypnS3HNsXq8te6BQHGX1rvS32HGMoyXuVQiUmIODpl4JsqGqfC0lgAr Q4gG5JB28KPrq4tNTtaSbTRn4VGBedqjGi2VpwlApFKB77EbFQ4Y97gsLWaMxO6QcXkGzmseJHroYK 1TAtjiUMVh+Y6Rhm7gFuio5OFvsmHRCjk/ptk/RA5EHOPNUG/f4cslB63rA8n5DS12BL3hkVbiNpXB NN2ShPn9cBOyFS51aDISIO6WXhSda6dqJEW16vsQDCdJQwdoadLqi5ynHYjMTRpusjk3Oj84K2ieYx 5j84qra5AUW3b53Ay1F7jxixuhtiEuMef1yt5RI89H8oSilEc51nVl9BdfqJCmErS6XBw2rtfKUbHF QC/vcoPISXW471f9PkAujkc9Fix+P9lW6igfO1fJy4Ofs0k55r/InzY7+e9UWePsZ2SLC3bdZ9xN3P MiG34bC9geAIQZUyhg220uGkk8p1EWStqyHk6/L2dyoAc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221205_120350_397211_B84A5540 X-CRM114-Status: GOOD ( 12.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org All occurrences of the scs_load macro load the value of the shadow call stack pointer from the task which is current at that point. So instead of taking a task struct register argument in the scs_load macro to specify the task struct to load from, let's always reference the current task directly. This should make it much harder to exploit any instruction sequences reloading the shadow call stack pointer register from memory. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 7 ++++--- arch/arm64/kernel/entry.S | 4 ++-- arch/arm64/kernel/head.S | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 8297bccf0784577e..5cd4d09bc69d7f6d 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -9,15 +9,16 @@ #ifdef CONFIG_SHADOW_CALL_STACK scs_sp .req x18 - .macro scs_load tsk - ldr scs_sp, [\tsk, #TSK_TI_SCS_SP] + .macro scs_load_current + get_current_task scs_sp + ldr scs_sp, [scs_sp, #TSK_TI_SCS_SP] .endm .macro scs_save tsk str scs_sp, [\tsk, #TSK_TI_SCS_SP] .endm #else - .macro scs_load tsk + .macro scs_load_current .endm .macro scs_save tsk diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index e28137d64b7688e2..20e25083eced13f5 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -272,7 +272,7 @@ alternative_if ARM64_HAS_ADDRESS_AUTH alternative_else_nop_endif 1: - scs_load tsk + scs_load_current .else add x21, sp, #PT_REGS_SIZE get_current_task tsk @@ -845,7 +845,7 @@ SYM_FUNC_START(cpu_switch_to) msr sp_el0, x1 ptrauth_keys_install_kernel x1, x8, x9, x10 scs_save x0 - scs_load x1 + scs_load_current ret SYM_FUNC_END(cpu_switch_to) NOKPROBE(cpu_switch_to) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2196aad7b55bcef0..cdbbc95eb49d025a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -404,7 +404,7 @@ SYM_FUNC_END(create_kernel_mapping) stp xzr, xzr, [sp, #S_STACKFRAME] add x29, sp, #S_STACKFRAME - scs_load \tsk + scs_load_current adr_l \tmp1, __per_cpu_offset ldr w\tmp2, [\tsk, #TSK_TI_CPU] From patchwork Mon Dec 5 20:03:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13065055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF3A8C4332F for ; Mon, 5 Dec 2022 20:05:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=z9F0QzvdRJFpHTLeUv2fGCHU9oUtRc+pPhu7r9qSZdY=; b=eCjH2L9+jyIyNA worT3o/9qJUCQ9I1W42KkHde8tw8Y1FxiC4VdIY4d79BNDEvgmUmqbbI1oI0XcgtOyuB/50J5kR09 /1Fu9U+oU6UWujGnGreo1PUkoXdy6JpKdCR5oKDBa8uF5XjkUAS3OdUum/cTyqyFsoMYvzYdp+jll mFHfLrVYCFb0kRCjpANeSMa3ZqKtYEONKKRpvIwBWgaembDeZYojVl4xLylqFXUH5ZjZAo466gV3z ZaquGuYXRtzYNGoIX7RX0Hf0L6K838xqOp0jfk1DvMjHLoT5/lXe7RQsiPZLuUiLpVwevTrvu8svG xWj7bRwF6D1hmf2vwxXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2HhC-009tCJ-Kz; Mon, 05 Dec 2022 20:04:10 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p2Hgt-009syu-Ff for linux-arm-kernel@lists.infradead.org; Mon, 05 Dec 2022 20:03:53 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 198F561369; Mon, 5 Dec 2022 20:03:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1AE80C433D7; Mon, 5 Dec 2022 20:03:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670270630; bh=seUY+O+NmEcVPaZ0Lnh9VHXWK1q8rE9XcJUhAzXTJuY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HddzadDDkE5Xy6clLIgZF+mX3Yj0s1u18xQ94ISbnHY+KfbpseVS57prM8/f6OZNQ adkaRA73+fUCkbA7ug1BZ78FFINkSi7O1Xhhq0HFiT7ch0adMQiBk48dX4HTDcEx4t 2QWRJl1PjskEn/pZaJxWvgwdbgK8RtPnMu2jIVjKVyhK8bm6a3Zzd++oSM3rdLFhRw WJ5i9KL3kFHkxphYqj1qMs/0EmktK87tFGFwS9WJh1PsHrJb3952cyrVYKgYD5NBV2 c0Jb1NcDzvFzlICtVPdBifhA2pbt6fdVKWKwAYBqeSVWur5BcrexLi46pO6ItagPjP bmFPoGsYRuW7g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, catalin.marinas@arm.com, mark.rutland@arm.com, Ard Biesheuvel , Sami Tolvanen , Kees Cook Subject: [PATCH 2/2] arm64: Stash shadow stack pointer in the task struct on interrupt Date: Mon, 5 Dec 2022 21:03:41 +0100 Message-Id: <20221205200341.463601-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221205200341.463601-1-ardb@kernel.org> References: <20221205200341.463601-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2053; i=ardb@kernel.org; h=from:subject; bh=seUY+O+NmEcVPaZ0Lnh9VHXWK1q8rE9XcJUhAzXTJuY=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjjk6coU/pTnoOSVec+JHNh/S2NoHqwzbFTmKc4yj1 lAiHLQKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY45OnAAKCRDDTyI5ktmPJJHCDA DHFiv7r3e34kCJvXPG3U5a/dMues6d9B+guvN9BzknsxiN5S+uqdKTihacNUK+3SveCF53Cc1VIEej AZsOgSdM6ePPX4b+IZKfgdEamrUVfUom1pGDxd+NbDxE0uRTb1QxxCLVuPkRTi77qk695djX2DWxAp at0izDOwHkBhlA3ew1gAPBr2wF5/W0Uv5B8DqrRQghFtAbv/Be1CTxyMy3rz8LYTrKq8OBDAHk7Vth gVgSg3FJWKQTbOiavtlcjGzhlgBAJKlHWEQWdKDpih/m0aQQTu0YAqRJLSnf99b6tNQ7Ryfuz4byzB 0af6ODTYeW733k1XI7Kfdkkim9IykqpVx4mld3uEwELl57MVXsWBPjH+R4OTc+axZkxEIlWXBQ6yLR dApoqb9mCGu6Uj6cdge4MkFtZZvVj9dw4X2hNqfRpPseGj1toAu1OyZ/9asw9brswzhSAH9ohxLTCH O8M0koU8XmLOfiipoCFRCFbyfQdLOD22aJIhC3FvM4wXY= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221205_120351_736317_F9796E33 X-CRM114-Status: GOOD ( 13.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of reloading the shadow call stack pointer from the ordinary stack, which may be vulnerable to the kind of gadget based attacks shadow call stacks were designed to prevent, let's store a task's shadow call stack pointer in the task struct when switching to the shadow IRQ stack. Given that currently, the task_struct::scs_sp field is only used to preserve the shadow call stack pointer while a task is scheduled out or running in user space, reusing this field to preserve and restore it while running off the IRQ stack must be safe, as those occurrences are guaranteed to never overlap. (The stack switching logic only switches stacks when running from the task stack, and so the value being saved here always corresponds to the task mode shadow stack) While at it, fold a mov/add/mov sequence into a single add. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/entry.S | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 20e25083eced13f5..3671d9521d4f559e 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -873,19 +873,19 @@ NOKPROBE(ret_from_fork) */ SYM_FUNC_START(call_on_irq_stack) #ifdef CONFIG_SHADOW_CALL_STACK - stp scs_sp, xzr, [sp, #-16]! + get_current_task x16 + scs_save x16 ldr_this_cpu scs_sp, irq_shadow_call_stack_ptr, x17 #endif + /* Create a frame record to save our LR and SP (implicit in FP) */ stp x29, x30, [sp, #-16]! mov x29, sp ldr_this_cpu x16, irq_stack_ptr, x17 - mov x15, #IRQ_STACK_SIZE - add x16, x16, x15 /* Move to the new stack and call the function there */ - mov sp, x16 + add sp, x16, #IRQ_STACK_SIZE blr x1 /* @@ -894,9 +894,7 @@ SYM_FUNC_START(call_on_irq_stack) */ mov sp, x29 ldp x29, x30, [sp], #16 -#ifdef CONFIG_SHADOW_CALL_STACK - ldp scs_sp, xzr, [sp], #16 -#endif + scs_load_current ret SYM_FUNC_END(call_on_irq_stack) NOKPROBE(call_on_irq_stack)