From patchwork Fri Dec 9 15:20:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13069737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E91BC4332F for ; Fri, 9 Dec 2022 15:22:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UpnmXXWwv6P7RQ+cGKl6ynh9en4aW2z0HzAEwyyPuno=; b=tf8Gh9tiHZS6B4 LvPCfSGzZGze6jRKdo6yhqVpKufmcexdUHRSdbDzQmROIq2vbDb7S2qlmqQM4do10dxr9mkhpeXvS GuxcHlZA7oQXyVghGspwHbNhxoy7SoHMprLVa81XUQHtzENmDq+FJrCmHhan9NEnqbfV1UxU9zKgp DxSObXuc36keNMTDKTdzofmY0Rci4Gu669FIMGG8zaJh7AnSqICcd4TfoDsEoywTUvzyxRkeyyyfZ uAyGKACIQDy/C1tRr6OrlgK/KpXzMEh2LEtxB+qFJxeGqi6995n+aWJf5vi2me0Xu4SaJlECmLYYg ICpGXbqK0Z9GjZ8cx03Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBs-008wKr-Vp; Fri, 09 Dec 2022 15:21:33 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p3fBR-008wDN-SO for linux-arm-kernel@lists.infradead.org; Fri, 09 Dec 2022 15:21:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 939CDB82884; Fri, 9 Dec 2022 15:21:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4D0BC43392; Fri, 9 Dec 2022 15:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670599263; bh=L8iDZasSfhjRexNyE9qh3CLKEsm6qajvXMtUuQRsl20=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IKtbylCUO17PytA0CsEdzXyyw+XIy+ACc5gpJeqntwd2psYRh4pbN09QCuOUqqDjz C3McCHnGGh/ZT3wUU4/vN9qZxgaUFffR32xOlSwb5Gp42dycWX0tCstcS+Xb8nH936 U4DUHA2vbtNTZkbxkMpN0vCF5UwMJNLNXDJTiDP8IwGVOqHn1KOSV/yl5yn/s+9hV/ 3iaHe9Srlt9BMq0Duxqm+qxmboEJMgzkGpGRJK/e8SiNpKMIohuz0rtNL8aMWNuxdm ehx6asdf9WDWM0mZyww7q1jTp1mBGWT6MWGVZSniJ27VjKa/PZP/dJCVwOpVcces+2 jgxKs6zKCgHUA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown Subject: [PATCH v3 2/4] arm64: assembler: Protect return addresses in asm routines Date: Fri, 9 Dec 2022 16:20:46 +0100 Message-Id: <20221209152048.3517080-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221209152048.3517080-1-ardb@kernel.org> References: <20221209152048.3517080-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8097; i=ardb@kernel.org; h=from:subject; bh=L8iDZasSfhjRexNyE9qh3CLKEsm6qajvXMtUuQRsl20=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjk1JLrE6C0LSkL33rucNZN2oONlqIvZkUbka9Jvnv v29/sLKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5NSSwAKCRDDTyI5ktmPJDD9DA CUdBdjWmE13EM3TXKostIS9uvsXso9kKhWsKvmBnP094ARUD3ut4SMswVHOiECqPVCfaPGA+/num6q j5T1yflz1Xh0+BaK9iMdj+aRROwgGhmsTUgNAf9S2z+LItts22BtJui2ssOkrq/CyPgtA8/Ecl9F1J 1fOfa6FPsMyEN8S1XWGzNFafClEz1MFimJE37kvvHOtez+7HDr4jxrgrZIsHCO/oHrFIc89PCEIxFM IEFJcfaD4xomkh9caEf7UPkfFnBYgIB5xehjGNRRIyS652CBxoKcuuNHzXwSxkCjf0pWcF7L2m18ZR QAKZUtpOZy+DrhH+ia3Z15CZPE9ftWhuA4RrZoQdOUQ2WXQJOS5+0n3K2wW1wgHzjCp2iWnNDul/up b6IQzcFwKngmjskQ7YohJEdTl8rfOWYgW6BSbfv3f+1SEF1HB4UxtKU0JOaj8PrCP41CGemioNJPkd tAPOM2xdNGLcjzx4laEyTsCBSZxg/g10UMt9ZsB+Mfs0s= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221209_072106_231512_F9775BF7 X-CRM114-Status: GOOD ( 29.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a set of macros that can be invoked to protect and restore the return address when it is being spilled to memory. Just like ordinary C code, the chosen method will be based on CONFIG_ARM64_PTR_AUTH_KERNEL, CONFIG_SHADOW_CALL_STACK and CONFIG_DYNAMIC_SCS, and may involve boot time patching depending on the runtime capabilities of the system (and potential command line overrides) Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 75 ++++++++++++++++++++ arch/arm64/kernel/patch-scs.c | 70 +++++++++++++----- 2 files changed, 127 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 1c04701e4fda8458..8b4afa2aaa9b0600 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -698,6 +698,79 @@ alternative_endif #endif .endm + /* + * protect_return_address - protect the return address value in + * register @reg, either by signing it using PAC and/or storing it on + * the shadow call stack. When dynamic shadow call stack is enabled, + * unwind directives are emitted so that the patching logic can find + * the instructions. + * + * These macros must not be used with reg != x30 in functions marked as + * SYM_FUNC, as in that case, each occurrence of this macro needs its + * own SYM_FUNC_CFI_START/_END section, and so these have to be emitted + * explicitly rather than via SYM_FUNC_START/_END. This is required to + * encode the return address correctly (which can only be encoded once + * per function) + */ + .macro protect_return_address, reg=x30 +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL +#ifdef CONFIG_UNWIND_TABLES + .cfi_startproc +#endif + .arch_extension pauth + .ifnc \reg, x30 +alternative_if_not ARM64_HAS_ADDRESS_AUTH + // NOP encoding with bit #22 cleared (for patching to STR) + orr xzr, xzr, xzr, lsl #0 +alternative_else + pacia \reg, sp +alternative_endif + .else + paciasp + .endif +#ifdef CONFIG_UNWIND_TABLES + .cfi_return_column \reg + .cfi_negate_ra_state + .cfi_endproc +#endif +#endif +#if defined(CONFIG_SHADOW_CALL_STACK) && !defined(CONFIG_DYNAMIC_SCS) + str \reg, [x18], #8 +#endif + .endm + + /* + * restore_return_address - restore the return address value in + * register @reg, either by authenticating it using PAC and/or + * reloading it from the shadow call stack. + */ + .macro restore_return_address, reg=x30 +#if defined(CONFIG_SHADOW_CALL_STACK) && !defined(CONFIG_DYNAMIC_SCS) + ldr \reg, [x18, #-8]! +#endif +#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL +#ifdef CONFIG_UNWIND_TABLES + .cfi_startproc +#endif + .arch_extension pauth + .ifnc \reg, x30 +alternative_if_not ARM64_HAS_ADDRESS_AUTH + // NOP encoding with bit #22 set (for patching to LDR) + orr xzr, xzr, xzr, lsr #0 +alternative_else + autia \reg, sp +alternative_endif + .else + autiasp + .endif +#ifdef CONFIG_UNWIND_TABLES + .cfi_return_column \reg + .cfi_negate_ra_state + .cfi_endproc +#endif +#endif + .endm + /* * frame_push - Push @regcount callee saved registers to the stack, * starting at x19, as well as x29/x30, and set x29 to @@ -705,6 +778,7 @@ alternative_endif * for locals. */ .macro frame_push, regcount:req, extra + protect_return_address __frame st, \regcount, \extra .endm @@ -716,6 +790,7 @@ alternative_endif */ .macro frame_pop __frame ld + restore_return_address .endm .macro __frame_regs, reg1, reg2, op, num diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/patch-scs.c index 1b3da02d5b741bc3..d7319d10ca799167 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/patch-scs.c @@ -54,20 +54,33 @@ extern const u8 __eh_frame_start[], __eh_frame_end[]; enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, - SCS_PUSH = 0xf800865e, - SCS_POP = 0xf85f8e5e, + SCS_PUSH = 0xf8008640, + SCS_POP = 0xf85f8e40, + + // Special NOP encodings to identify locations where a register other + // than x30 is being used to carry the return address + NOP_PUSH = 0xaa1f03ff, // orr xzr, xzr, xzr, lsl #0 + NOP_POP = 0xaa5f03ff, // orr xzr, xzr, xzr, lsr #0 }; -static void __always_inline scs_patch_loc(u64 loc) +static void __always_inline scs_patch_loc(u64 loc, int ra_reg) { u32 insn = le32_to_cpup((void *)loc); switch (insn) { + case NOP_PUSH: + if (WARN_ON(ra_reg == 30)) + break; + fallthrough; case PACIASP: - *(u32 *)loc = cpu_to_le32(SCS_PUSH); + *(u32 *)loc = cpu_to_le32(SCS_PUSH | ra_reg); break; + case NOP_POP: + if (WARN_ON(ra_reg == 30)) + break; + fallthrough; case AUTIASP: - *(u32 *)loc = cpu_to_le32(SCS_POP); + *(u32 *)loc = cpu_to_le32(SCS_POP | ra_reg); break; default: /* @@ -76,9 +89,12 @@ static void __always_inline scs_patch_loc(u64 loc) * also appear after a DW_CFA_restore_state directive that * restores a state that is only partially accurate, and is * followed by DW_CFA_negate_ra_state directive to toggle the - * PAC bit again. So we permit other instructions here, and ignore - * them. + * PAC bit again. So we permit other instructions here, and + * ignore them (unless they appear in handwritten assembly + * using a different return address register, where this should + * never happen). */ + WARN_ON(ra_reg != 30); return; } dcache_clean_pou(loc, loc + sizeof(u32)); @@ -130,7 +146,8 @@ struct eh_frame { static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, bool fde_has_augmentation_data, - int code_alignment_factor) + int code_alignment_factor, + int ra_reg) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -184,7 +201,7 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; case DW_CFA_negate_ra_state: - scs_patch_loc(loc - 4); + scs_patch_loc(loc - 4, ra_reg); break; case 0x40 ... 0x7f: @@ -206,6 +223,7 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, int noinstr scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; + int ra_reg = 30; while (size > 4) { const struct eh_frame *frame = (const void *)p; @@ -219,23 +237,39 @@ int noinstr scs_patch(const u8 eh_frame[], int size) break; if (frame->cie_id_or_pointer == 0) { - const u8 *p = frame->augmentation_string; + const u8 *as = frame->augmentation_string; /* a 'z' in the augmentation string must come first */ - fde_has_augmentation_data = *p == 'z'; + fde_has_augmentation_data = *as == 'z'; + as += strlen(as) + 1; + + /* check for at least 3 more bytes in the frame */ + if (as - (u8 *)&frame->cie_id_or_pointer + 3 > frame->size) + return -ENOEXEC; /* - * The code alignment factor is a uleb128 encoded field - * but given that the only sensible values are 1 or 4, - * there is no point in decoding the whole thing. + * The code and data alignment factors are uleb128 + * encoded fields but given that the only sensible + * values are 1 or 4, there is no point in decoding + * them entirely. The return address register number is + * a single byte in version 1 and a uleb128 in newer + * versions. */ - p += strlen(p) + 1; - if (!WARN_ON(*p & BIT(7))) - code_alignment_factor = *p; + if (WARN_ON(as[0] & BIT(7) || as[1] & BIT(7) || + (as[2] & BIT(7)) && frame->version > 1)) + return -ENOEXEC; + + code_alignment_factor = as[0]; + + // Grab the return address register + ra_reg = as[2]; + if (WARN_ON(ra_reg > 30)) + return -ENOEXEC; } else { ret = scs_handle_fde_frame(frame, fde_has_augmentation_data, - code_alignment_factor); + code_alignment_factor, + ra_reg); if (ret) return ret; }