From patchwork Fri Dec 9 15:20:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13069735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5EDEC4332F for ; Fri, 9 Dec 2022 15:22:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X+EWiH5rbDrmXCZr1dqmlsn11PBk05HbwX+axH21n/c=; b=uxHvgd1UFGHaei XQGfHl04APZb4vm+qkARG5DW4sIAww/5llfSm2iQzc8H3cuBwLoN9V3D8kpjJQbd7WxhXxZJbif4i C0sRHjP7EvfcZrLYqUHF3waWvbpPegRAS2x6I5QxzbxxM3enF3DPp5ZAiKl2GtlE+v5lMicR244Ve 87b7igBhWzoCMj3mkYq69kZTTa5i3voDAPIKlqQ2jn/ZjF4ORxAyWKIaD3PvwJfp2npqnvbA0uKzW HWBGh7Wtt3sN+xy6i3kz3JAQr6WerENWvAv+Wc5jLdgiNkEMAU8va6wxhQ4ZzLNqjdvKDIFJzP+0w 1BPEThASF7iWJjYI4sIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBW-008wEl-8X; Fri, 09 Dec 2022 15:21:10 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBQ-008wCe-44 for linux-arm-kernel@lists.infradead.org; Fri, 09 Dec 2022 15:21:08 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8949FB8287F; Fri, 9 Dec 2022 15:21:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABD6FC43396; Fri, 9 Dec 2022 15:20:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670599261; bh=Mh+IoswhbapuQ7iNCDxmLomZt2cIwbv+py93QYg2OkA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Cc33JzR8XVKdslddOrkpbLI7I9A61UGHZdZSUjJteV2j6J/AufnVYkxtD+N+SM3J+ B+ndL0uiRxMA8K8Shkf/sOM8JcdD+QBA0Dwx6PK4rEhzmPwoLZguq4NpUHl4OllSJ2 W6cay0Jo9KS/i6uE1nZgc9xGCOZOJK5T9gNulEp5/Usj2MV1Jk/WIPCKH1qb5VRYPa mPGxJWpAZN8+zS4gfhrh/6QKaZFAEo//Dh55i/C4PipHoTs4ojrZlkyDnRNuj/1Kfw te0sFaZU7NI2UTKHTJJkdN+MJ8D1h4Ybj9Nok3PsmMtvUHBgYZMiWBDwDTukeXMeyl HgeLDeIODszuw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Subject: [PATCH v3 1/4] arm64: assembler: Force error on misuse of .Lframe_local_offset Date: Fri, 9 Dec 2022 16:20:45 +0100 Message-Id: <20221209152048.3517080-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221209152048.3517080-1-ardb@kernel.org> References: <20221209152048.3517080-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1032; i=ardb@kernel.org; h=from:subject; bh=Mh+IoswhbapuQ7iNCDxmLomZt2cIwbv+py93QYg2OkA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjk1JK91wgxn9PdghDRKkY+euZhZphdvjHFnK82jrQ pE4Wo1SJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5NSSgAKCRDDTyI5ktmPJHzuDA C9GIKSSPwOCBsDhtdfmYWvvcLAsrBVziDKqOXKU5v8aGuiclDVx6Kg9wzH3PiHOJ+4Ozh3sU5+Gyqi d2Oxemh2NC4PEzbAmH1WDDoaPJjxmHzuiU0K/+5LuvFHOfQkL+I0rkNfF0Ov+beKKS9OnhS+dHSdoU zVRIbAljI6r0w3YuBTiqQErIhCLcr8RjOH3VC70AKSDYR4DqkhQfbhPOoYCzXvAU1V4ndAZemPy7i9 gwi/xNadb6k9e7dV+rJgjpEXOX8P77PS8LS4d5WmveJguGTCGKyCgOktdmwAZ37b3hcNcbtTq8PXMB Palv7/iz/XjJFVAs25YJckBJ5br5/9pps59nxgXVov8bXElGwBHBfyZpXtRWuM329m92s7PJQpw1+M 8I58+qkiHX453dYfDHK8bcgYAaKpTa4JjMe2nRcif2yuASwdwoNZipbiNzxQHA/3nnGP62zY71PWY7 fi+E91yHgYIfYpKt9TC7KxdWDWSEvTsnF0HFfrsuCy+B4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221209_072104_320836_5CC95DC8 X-CRM114-Status: GOOD ( 10.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The frame_push macro sets a local symbol .Lframe_local_offset to the offset where the local variable area resides in the stack frame. However, while we take care not to nest frame_push and frame_pop sequences, .Lframe_local_offset retains its most recent value, allowing it to be referenced erroneously from outside a frame_push/frame_pop pair. So set it to an obviously wrong value that is guaranteed to trigger a link error in frame_pop. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index e5957a53be3983ac..1c04701e4fda8458 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -758,6 +758,7 @@ alternative_endif .endif ldp x29, x30, [sp], #.Lframe_local_offset + .Lframe_extra .set .Lframe_regcount, -1 + .set .Lframe_local_offset, frame_local_offset_error .endif .endm From patchwork Fri Dec 9 15:20:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13069737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E91BC4332F for ; Fri, 9 Dec 2022 15:22:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UpnmXXWwv6P7RQ+cGKl6ynh9en4aW2z0HzAEwyyPuno=; b=tf8Gh9tiHZS6B4 LvPCfSGzZGze6jRKdo6yhqVpKufmcexdUHRSdbDzQmROIq2vbDb7S2qlmqQM4do10dxr9mkhpeXvS GuxcHlZA7oQXyVghGspwHbNhxoy7SoHMprLVa81XUQHtzENmDq+FJrCmHhan9NEnqbfV1UxU9zKgp DxSObXuc36keNMTDKTdzofmY0Rci4Gu669FIMGG8zaJh7AnSqICcd4TfoDsEoywTUvzyxRkeyyyfZ uAyGKACIQDy/C1tRr6OrlgK/KpXzMEh2LEtxB+qFJxeGqi6995n+aWJf5vi2me0Xu4SaJlECmLYYg ICpGXbqK0Z9GjZ8cx03Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBs-008wKr-Vp; Fri, 09 Dec 2022 15:21:33 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBR-008wDN-SO for linux-arm-kernel@lists.infradead.org; Fri, 09 Dec 2022 15:21:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 939CDB82884; Fri, 9 Dec 2022 15:21:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4D0BC43392; Fri, 9 Dec 2022 15:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670599263; bh=L8iDZasSfhjRexNyE9qh3CLKEsm6qajvXMtUuQRsl20=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IKtbylCUO17PytA0CsEdzXyyw+XIy+ACc5gpJeqntwd2psYRh4pbN09QCuOUqqDjz C3McCHnGGh/ZT3wUU4/vN9qZxgaUFffR32xOlSwb5Gp42dycWX0tCstcS+Xb8nH936 U4DUHA2vbtNTZkbxkMpN0vCF5UwMJNLNXDJTiDP8IwGVOqHn1KOSV/yl5yn/s+9hV/ 3iaHe9Srlt9BMq0Duxqm+qxmboEJMgzkGpGRJK/e8SiNpKMIohuz0rtNL8aMWNuxdm ehx6asdf9WDWM0mZyww7q1jTp1mBGWT6MWGVZSniJ27VjKa/PZP/dJCVwOpVcces+2 jgxKs6zKCgHUA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Subject: [PATCH v3 2/4] arm64: assembler: Protect return addresses in asm routines Date: Fri, 9 Dec 2022 16:20:46 +0100 Message-Id: <20221209152048.3517080-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221209152048.3517080-1-ardb@kernel.org> References: <20221209152048.3517080-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8097; i=ardb@kernel.org; h=from:subject; bh=L8iDZasSfhjRexNyE9qh3CLKEsm6qajvXMtUuQRsl20=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjk1JLrE6C0LSkL33rucNZN2oONlqIvZkUbka9Jvnv v29/sLKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5NSSwAKCRDDTyI5ktmPJDD9DA CUdBdjWmE13EM3TXKostIS9uvsXso9kKhWsKvmBnP094ARUD3ut4SMswVHOiECqPVCfaPGA+/num6q j5T1yflz1Xh0+BaK9iMdj+aRROwgGhmsTUgNAf9S2z+LItts22BtJui2ssOkrq/CyPgtA8/Ecl9F1J 1fOfa6FPsMyEN8S1XWGzNFafClEz1MFimJE37kvvHOtez+7HDr4jxrgrZIsHCO/oHrFIc89PCEIxFM IEFJcfaD4xomkh9caEf7UPkfFnBYgIB5xehjGNRRIyS652CBxoKcuuNHzXwSxkCjf0pWcF7L2m18ZR QAKZUtpOZy+DrhH+ia3Z15CZPE9ftWhuA4RrZoQdOUQ2WXQJOS5+0n3K2wW1wgHzjCp2iWnNDul/up b6IQzcFwKngmjskQ7YohJEdTl8rfOWYgW6BSbfv3f+1SEF1HB4UxtKU0JOaj8PrCP41CGemioNJPkd tAPOM2xdNGLcjzx4laEyTsCBSZxg/g10UMt9ZsB+Mfs0s= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221209_072106_231512_F9775BF7 X-CRM114-Status: GOOD ( 29.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a set of macros that can be invoked to protect and restore the return address when it is being spilled to memory. Just like ordinary C code, the chosen method will be based on CONFIG_ARM64_PTR_AUTH_KERNEL, CONFIG_SHADOW_CALL_STACK and CONFIG_DYNAMIC_SCS, and may involve boot time patching depending on the runtime capabilities of the system (and potential command line overrides) Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 75 ++++++++++++++++++++ arch/arm64/kernel/patch-scs.c | 70 +++++++++++++----- 2 files changed, 127 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 1c04701e4fda8458..8b4afa2aaa9b0600 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -698,6 +698,79 @@ alternative_endif #endif .endm + /* + * protect_return_address - protect the return address value in + * register @reg, either by signing it using PAC and/or storing it on + * the shadow call stack. When dynamic shadow call stack is enabled, + * unwind directives are emitted so that the patching logic can find + * the instructions. + * + * These macros must not be used with reg != x30 in functions marked as + * SYM_FUNC, as in that case, each occurrence of this macro needs its + * own SYM_FUNC_CFI_START/_END section, and so these have to be emitted + * explicitly rather than via SYM_FUNC_START/_END. This is required to + * encode the return address correctly (which can only be encoded once + * per function) + */ + .macro protect_return_address, reg=x30 +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL +#ifdef CONFIG_UNWIND_TABLES + .cfi_startproc +#endif + .arch_extension pauth + .ifnc \reg, x30 +alternative_if_not ARM64_HAS_ADDRESS_AUTH + // NOP encoding with bit #22 cleared (for patching to STR) + orr xzr, xzr, xzr, lsl #0 +alternative_else + pacia \reg, sp +alternative_endif + .else + paciasp + .endif +#ifdef CONFIG_UNWIND_TABLES + .cfi_return_column \reg + .cfi_negate_ra_state + .cfi_endproc +#endif +#endif +#if defined(CONFIG_SHADOW_CALL_STACK) && !defined(CONFIG_DYNAMIC_SCS) + str \reg, [x18], #8 +#endif + .endm + + /* + * restore_return_address - restore the return address value in + * register @reg, either by authenticating it using PAC and/or + * reloading it from the shadow call stack. + */ + .macro restore_return_address, reg=x30 +#if defined(CONFIG_SHADOW_CALL_STACK) && !defined(CONFIG_DYNAMIC_SCS) + ldr \reg, [x18, #-8]! +#endif +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL +#ifdef CONFIG_UNWIND_TABLES + .cfi_startproc +#endif + .arch_extension pauth + .ifnc \reg, x30 +alternative_if_not ARM64_HAS_ADDRESS_AUTH + // NOP encoding with bit #22 set (for patching to LDR) + orr xzr, xzr, xzr, lsr #0 +alternative_else + autia \reg, sp +alternative_endif + .else + autiasp + .endif +#ifdef CONFIG_UNWIND_TABLES + .cfi_return_column \reg + .cfi_negate_ra_state + .cfi_endproc +#endif +#endif + .endm + /* * frame_push - Push @regcount callee saved registers to the stack, * starting at x19, as well as x29/x30, and set x29 to @@ -705,6 +778,7 @@ alternative_endif * for locals. */ .macro frame_push, regcount:req, extra + protect_return_address __frame st, \regcount, \extra .endm @@ -716,6 +790,7 @@ alternative_endif */ .macro frame_pop __frame ld + restore_return_address .endm .macro __frame_regs, reg1, reg2, op, num diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/patch-scs.c index 1b3da02d5b741bc3..d7319d10ca799167 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/patch-scs.c @@ -54,20 +54,33 @@ extern const u8 __eh_frame_start[], __eh_frame_end[]; enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, - SCS_PUSH = 0xf800865e, - SCS_POP = 0xf85f8e5e, + SCS_PUSH = 0xf8008640, + SCS_POP = 0xf85f8e40, + + // Special NOP encodings to identify locations where a register other + // than x30 is being used to carry the return address + NOP_PUSH = 0xaa1f03ff, // orr xzr, xzr, xzr, lsl #0 + NOP_POP = 0xaa5f03ff, // orr xzr, xzr, xzr, lsr #0 }; -static void __always_inline scs_patch_loc(u64 loc) +static void __always_inline scs_patch_loc(u64 loc, int ra_reg) { u32 insn = le32_to_cpup((void *)loc); switch (insn) { + case NOP_PUSH: + if (WARN_ON(ra_reg == 30)) + break; + fallthrough; case PACIASP: - *(u32 *)loc = cpu_to_le32(SCS_PUSH); + *(u32 *)loc = cpu_to_le32(SCS_PUSH | ra_reg); break; + case NOP_POP: + if (WARN_ON(ra_reg == 30)) + break; + fallthrough; case AUTIASP: - *(u32 *)loc = cpu_to_le32(SCS_POP); + *(u32 *)loc = cpu_to_le32(SCS_POP | ra_reg); break; default: /* @@ -76,9 +89,12 @@ static void __always_inline scs_patch_loc(u64 loc) * also appear after a DW_CFA_restore_state directive that * restores a state that is only partially accurate, and is * followed by DW_CFA_negate_ra_state directive to toggle the - * PAC bit again. So we permit other instructions here, and ignore - * them. + * PAC bit again. So we permit other instructions here, and + * ignore them (unless they appear in handwritten assembly + * using a different return address register, where this should + * never happen). */ + WARN_ON(ra_reg != 30); return; } dcache_clean_pou(loc, loc + sizeof(u32)); @@ -130,7 +146,8 @@ struct eh_frame { static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, bool fde_has_augmentation_data, - int code_alignment_factor) + int code_alignment_factor, + int ra_reg) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -184,7 +201,7 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; case DW_CFA_negate_ra_state: - scs_patch_loc(loc - 4); + scs_patch_loc(loc - 4, ra_reg); break; case 0x40 ... 0x7f: @@ -206,6 +223,7 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, int noinstr scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; + int ra_reg = 30; while (size > 4) { const struct eh_frame *frame = (const void *)p; @@ -219,23 +237,39 @@ int noinstr scs_patch(const u8 eh_frame[], int size) break; if (frame->cie_id_or_pointer == 0) { - const u8 *p = frame->augmentation_string; + const u8 *as = frame->augmentation_string; /* a 'z' in the augmentation string must come first */ - fde_has_augmentation_data = *p == 'z'; + fde_has_augmentation_data = *as == 'z'; + as += strlen(as) + 1; + + /* check for at least 3 more bytes in the frame */ + if (as - (u8 *)&frame->cie_id_or_pointer + 3 > frame->size) + return -ENOEXEC; /* - * The code alignment factor is a uleb128 encoded field - * but given that the only sensible values are 1 or 4, - * there is no point in decoding the whole thing. + * The code and data alignment factors are uleb128 + * encoded fields but given that the only sensible + * values are 1 or 4, there is no point in decoding + * them entirely. The return address register number is + * a single byte in version 1 and a uleb128 in newer + * versions. */ - p += strlen(p) + 1; - if (!WARN_ON(*p & BIT(7))) - code_alignment_factor = *p; + if (WARN_ON(as[0] & BIT(7) || as[1] & BIT(7) || + (as[2] & BIT(7)) && frame->version > 1)) + return -ENOEXEC; + + code_alignment_factor = as[0]; + + // Grab the return address register + ra_reg = as[2]; + if (WARN_ON(ra_reg > 30)) + return -ENOEXEC; } else { ret = scs_handle_fde_frame(frame, fde_has_augmentation_data, - code_alignment_factor); + code_alignment_factor, + ra_reg); if (ret) return ret; } From patchwork Fri Dec 9 15:20:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13069738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A644CC4332F for ; Fri, 9 Dec 2022 15:22:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LUKGX801dlmIcN0h0YssWP3m/68I1tWZmNz7sa6YgLk=; b=JhT6dQwB+U3hW1 PZEAKxBfSbY+UEWO50mTvrtZi0DsqgLVGa9PIOeW3J9JBBGXYIuZ2YMuJzkIbrWq4qF02pcVaWGUu DxP9rD9db/OGczK2IdfSzDFNU5Y5/Wunx5V04VMho6SBeU0r5lAPo6LudIRoCwom6WrosET/WBQLX gvm7KCZo6gpV/uNGlsyThtNRqvRa4sbb2R8aBUn91GbW3SZftqTWvLzDviWhsL5lY7cxa2j5bng5l drgp9WebYDzi1oXniKHb8ZOLRRi8GtRQTovsJYkJeb7sasS5Br2J33Sl7MoOVnIK9H8cRoghIwr7j ekrCnq1K/QlpOMDtmKPg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fC1-008wNe-RW; Fri, 09 Dec 2022 15:21:41 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBT-008wDy-UI for linux-arm-kernel@lists.infradead.org; Fri, 09 Dec 2022 15:21:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A3E1BB82886; Fri, 9 Dec 2022 15:21:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCF92C43398; Fri, 9 Dec 2022 15:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670599265; bh=MuAqbMt06XF1iJceDCzI425FTyqaSZ5kb9rVjh9KUm8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e4LxfsOfk8FSqnuh4b759Q6i/C0qShCFfnUanEWyr9C79Hg9bJ1jFcWkxP0Yma/EX Vqu1tMRg3HGiBcBkfBjSO/2oHJKzOKtuiGbmJacpzHrQOpZi35SG6GFJHrtemOPytO 2G6SujFiAjitGYhbBMkHosN6K3xAaFkJ8wDa4sfGEGcRQqiWvdqFMC+w6FPyKTKUET QQXZrVTZ7zOnfMlEFXER4V7h9FzsMJ3lVJ16G6l0mz6O6IRW+CmZrBi7ep3YX9d2GT Woz2taYAhqF+kInSCFRQk75HFnUu0p63YH/Q3z6IqCg10PvpyfMuw6lpMJHCbtWd6X 2E6KWiojZ66bg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Subject: [PATCH v3 3/4] arm64: ftrace: Preserve original link register value in ftrace_regs Date: Fri, 9 Dec 2022 16:20:47 +0100 Message-Id: <20221209152048.3517080-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221209152048.3517080-1-ardb@kernel.org> References: <20221209152048.3517080-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2800; i=ardb@kernel.org; h=from:subject; bh=MuAqbMt06XF1iJceDCzI425FTyqaSZ5kb9rVjh9KUm8=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjk1JN0ae0fdChGD/DbFO22B9sFpJ8ohHbw0F8aKnU 8c+9g8CJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5NSTQAKCRDDTyI5ktmPJB6YC/ 9r6frmWjotANI+43Ki/rxBLsKwHZv2ZWgassGBw/IPSQUuXSQturUcmAnsfQvAPJOpd7eIkhhQZT8E CNCGJH4BqMDWpUKZF5OzxSdqnhQubfhGUdmGiRArXcfxcXp15/E3kFJOCjxgB03kcey+NKlob1WFmJ L0JAw89gPkL5aJ599fYSeMm/Vn2X+iQP2Bnq12WXJE5BonCG2zmW5DIKF0Go+p+GSq+ZMZPmmFy8ae Lj4p5/Jeq7Eti91prP6yWr7uLEXb4g/KIWR7GR+oFpc5oinOSiSTXCyst3PqX4HUCUDLWE9r1ej2Aw YJ1PCiHwaDmRLHL+ZcMBtFL4Ao0RLE/XSFSTnUxQD97cmOhyy9am0nvVXTO4UOXz+dJFrhGzNXtPHP BjFZdO7NCKsaaEPYV99Xg/PIsSq1FNkv0yUTGQmyRGJgggRPaTuGht9TzSl0XenovlEMgvKhS83Qej P1NRBzFfsmVuFRxi5m+fiFS48IAUjPyrpUy1tEsTcjjNc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221209_072108_284540_CAEB8163 X-CRM114-Status: GOOD ( 16.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to be able to add pointer authentication and/or shadow call stack support to the ftrace asm routines, it will need to reason about whether or not the callsite's return address was updated to point to return_to_handler(), as in this case, we want the authentication to occur there and not before returning to the call site. To make this a bit easier, preserve the value of register X9, which carries the callsite's LR value upon entry to ftrace_caller, so in a later patch, we can compare it to the callsite's effective LR upon return, and omit the authentication if the caller will be returning via return_to_handler(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/ftrace.h | 2 +- arch/arm64/kernel/entry-ftrace.S | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 5664729800ae1c13..b07501645a74031a 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -86,7 +86,7 @@ struct ftrace_ops; struct ftrace_regs { /* x0 - x8 */ unsigned long regs[9]; - unsigned long __unused; + unsigned long orig_lr; // must follow ®s[8] unsigned long fp; unsigned long lr; diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 30cc2a9d1757a6a7..bccd525241ab615d 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -42,12 +42,12 @@ SYM_CODE_START(ftrace_caller) /* Make room for ftrace regs, plus two frame records */ sub sp, sp, #(FREGS_SIZE + 32) - /* Save function arguments */ + /* Save function arguments and original callsite LR */ stp x0, x1, [sp, #FREGS_X0] stp x2, x3, [sp, #FREGS_X2] stp x4, x5, [sp, #FREGS_X4] stp x6, x7, [sp, #FREGS_X6] - str x8, [sp, #FREGS_X8] + stp x8, x9, [sp, #FREGS_X8] /* Save the callsite's FP, LR, SP */ str x29, [sp, #FREGS_FP] @@ -78,22 +78,22 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) * x19-x29 per the AAPCS, and we created frame records upon entry, so we need * to restore x0-x8, x29, and x30. */ - /* Restore function arguments */ + /* Restore function arguments and original callsite LR */ ldp x0, x1, [sp, #FREGS_X0] ldp x2, x3, [sp, #FREGS_X2] ldp x4, x5, [sp, #FREGS_X4] ldp x6, x7, [sp, #FREGS_X6] - ldr x8, [sp, #FREGS_X8] + ldp x8, x9, [sp, #FREGS_X8] /* Restore the callsite's FP, LR, PC */ ldr x29, [sp, #FREGS_FP] ldr x30, [sp, #FREGS_LR] - ldr x9, [sp, #FREGS_PC] + ldr x10, [sp, #FREGS_PC] /* Restore the callsite's SP */ add sp, sp, #FREGS_SIZE + 32 - ret x9 + ret x10 SYM_CODE_END(ftrace_caller) #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ From patchwork Fri Dec 9 15:20:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13069739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E24FEC4332F for ; Fri, 9 Dec 2022 15:22:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VULxvmgKwE6WMd8cu5XYAnmw9jm/ai6jiS3+luA8Hvc=; b=nCdauOFV5vmSfu ilLFyfM4GphS+aZ4zWCmXhvs+RbdjetXLCcKJVuZhYjYdanKdcBbc6PLKVWeokm4mX6N9i1xUvBsm lM9vSnR+hqUUF+Jimdzoc1Es4tH3thh/GqO9teFwCMBpJhZkS3ZzJsf4btlBCG8v992rnW9PIst2E nwKvsz4xPbUZjXNMYvBJ9ZPorWNV2d8fBazY0VmQ1EBrwxWWvHcPwqVXvAZBalfVa5obAXeLsoXR7 mziklY/zcaxZtjLuxgshYEz3nmapnPM+uAlYMiTpBcbX5m969ccNpEENHVqmH4VJXyVsD3VVKz/nD cszhk4xd3mEx/SjW9sDw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fCE-008wRx-3M; Fri, 09 Dec 2022 15:21:54 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBW-008wEd-39 for linux-arm-kernel@lists.infradead.org; Fri, 09 Dec 2022 15:21:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C9F95B82887; Fri, 9 Dec 2022 15:21:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C619FC43396; Fri, 9 Dec 2022 15:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670599267; bh=MqT2F3MLZTaJHBNFjcvmSzJ/Q5zR07h4UcC/7suziHk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bZmY44djnv+67qd7HuP08y7uJgcIZRXUAXskzSF/lxll8y92BVk0XZ6kPkcPZ7Wvm XDjl4Pdc9UcfWUN7ee7oBXudBtTgYVmNDNBflyCQq0O2cCNrDPI8bFMNbzoCKYswnJ CNYDNN+Sl8vZZAfQ1djoyF3Oy82EbwN/wuZvfn/H1czLfR2B7tlHVMQ2h2Tw2fxxh/ PqSpPb6eBGvzAZ/+k0ytn1FFFc873IFTq2p80ntbi9zcyTbTJf4XuPERIWDbbmySJw DdveUckNzLHfLUdabYrptD85ZQiTDHb7LTUFeea8aJKhG+6gtkh6HLRA0Eo4dKcd4S eEzbW0of4cvaA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Subject: [PATCH v3 4/4] arm64: ftrace: Add return address protection Date: Fri, 9 Dec 2022 16:20:48 +0100 Message-Id: <20221209152048.3517080-5-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221209152048.3517080-1-ardb@kernel.org> References: <20221209152048.3517080-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2660; i=ardb@kernel.org; h=from:subject; bh=MqT2F3MLZTaJHBNFjcvmSzJ/Q5zR07h4UcC/7suziHk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjk1JPubSdmkIXyFkN5IX9YAlnXJvDRKqy4Q41y0k+ C/K8E4CJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5NSTwAKCRDDTyI5ktmPJN5FC/ 0ecKxBXTUetdMfWxGaGjfSyNRG4ERcKriULQRMe2mNskxKenepj1erbLGHRZvkzil84aveI8K4eoH8 2lIxEVZXtpzINRyql4ZCB0f0IuSglC7fwqLulml6/Lkeo9YShh1RazEkwPIRiA0/aHP/OudNJNSFo3 U+wW9nmViArbMdn80lVL23/zXVS+7n6fGgmldIz0MvyqwIBBtlV80GNfg6rLyil76T533u6coHApW5 yN+3Ebj+oiZXbdb4tcpv4BdKTJP9ANjWPVFm1Lmg1XfE9vpbduQC6FquSnf5ANeiu6hTF98vHpOAhh 40bAdZzOz/K2uWyq44dAil2DK+NrYg7s1WGcRkRCD+Q1SayDyZhJAgi+Hh3tJ54Yu0Nimcnf7QCbvR 8rEHSwXd4dQaIfdYbvTFoVFbki1Y+d0BQQ4WZGa3+e8lmFkzvQeWfpk8yRMxsCHvjY/eROs4/UvT5A oy0Wcn4NigpAGXJmh3FgBpwIqwPa4uvZ6rzOQKRZ0+zFM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221209_072110_449955_5C9474B7 X-CRM114-Status: GOOD ( 16.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the newly added asm macros to protect and restore the return address in the ftrace call wrappers, based on whichever method is active (PAC and/or shadow call stack). If the graph tracer is in use, this covers both the return address *to* the ftrace call site as well as the return address *at* the call site, and the latter will either be restored in return_to_handler(), or before returning to the call site. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/entry-ftrace.S | 17 ++++++++++++++++- arch/arm64/kernel/stacktrace.c | 1 + 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index bccd525241ab615d..4acfe12ac594da58 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -33,9 +33,13 @@ * record, its caller is missing from the LR and existing chain of frame * records. */ + SYM_CODE_START(ftrace_caller) bti c + protect_return_address x9 + protect_return_address x30 + /* Save original SP */ mov x10, sp @@ -65,6 +69,9 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE + alternative_insn nop, "xpaci x30", ARM64_HAS_ADDRESS_AUTH, \ + IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) + sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) ldr_l x2, function_trace_op // op @@ -93,7 +100,14 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) /* Restore the callsite's SP */ add sp, sp, #FREGS_SIZE + 32 - ret x10 + restore_return_address x10 +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + // Check whether the callsite's LR has been overridden + cmp x9, x30 + b.ne 0f +#endif + restore_return_address x30 +0: ret x10 SYM_CODE_END(ftrace_caller) #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ @@ -265,6 +279,7 @@ SYM_CODE_START(return_to_handler) ldp x6, x7, [sp, #48] add sp, sp, #64 + restore_return_address x30 ret SYM_CODE_END(return_to_handler) #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index 634279b3b03d1b07..e323a8ac50168261 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -102,6 +102,7 @@ static int notrace unwind_next(struct unwind_state *state) */ orig_pc = ftrace_graph_ret_addr(tsk, NULL, state->pc, (void *)state->fp); + orig_pc = ptrauth_strip_insn_pac(orig_pc); if (WARN_ON_ONCE(state->pc == orig_pc)) return -EINVAL; state->pc = orig_pc;