From patchwork Thu Oct 27 15:59:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13022341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00EE0FA3740 for ; Thu, 27 Oct 2022 16:00:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iKvtk3sRT607j96pAkXdBNHNJE5M0IKuGtk+5xH9LS8=; b=vDNjmLV87uGFkY Is3dd56oSHtZuTP785n70vRoi+wJXyTc7H16TWQE+SYij6YIzUohRWs83UURzSTz9ZGnT9xa+nRMf EgSuhxfjt8WR8mN+G0tCI/PV8NpF59rlUNdQeox59V8KlFW2zn//x2DgOw+0+Kcq5OGobWik+AP/L prxWqSrVJGzxA+ApAhk+M4oW804pr+NUaeE5XJplxLJhbWbmiYpxeZaI/WzIp+nsDlsi8qU5n4+yg sKRYBPj+tmsGodDh/7sNguzPyxQywXcJTxzIWiYnAb8sKHqZE7th3TCEVPOpl7Y4yeiQ1LLISDkOU krFSYztso6dLkMm4dGwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5I3-00E908-MQ; Thu, 27 Oct 2022 15:59:31 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5Hs-00E8xj-4Y for linux-arm-kernel@lists.infradead.org; Thu, 27 Oct 2022 15:59:21 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BD8CE623C1; Thu, 27 Oct 2022 15:59:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 25656C433C1; Thu, 27 Oct 2022 15:59:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666886359; bh=EB2b3J2UJ9v9GfC0liu3TjsAdiXjoVoRmp2AaTLPRws=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KUEe/fAQdij5f01IeBWtculNgm5HqzSfhN9tEprTdA4L77j5gecZ4WoeRTCctCo9J qVPKlmOYKxbtXvPGio3xbY3zWJOzfjWymHaNEta3Aqs8T28DPEqFqiSlwrgfshA3L0 mEGhEc8C5EGW7+ITZVg90lxQRZ55+NvuBo/DzQiA8KDwXEAW11HA+83luiDlmL93Yv xbGAk/DJIjf3OfDgnGaHbI44TZ0j+aWlgLS306rlMBpYzWAifwXOkbXjk2PgEPXeoh xjTR1jcIalkgvIwCl6fWSt5cKl+zbEXUzumUjeOIPsULQq9hjDCe70Y7lxEMFBEmNI ahpJAdO2RU/IA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland , Mark Brown , Sami Tolvanen , Nick Desaulniers , Kees Cook Subject: [PATCH v6 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules Date: Thu, 27 Oct 2022 17:59:06 +0200 Message-Id: <20221027155908.1940624-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221027155908.1940624-1-ardb@kernel.org> References: <20221027155908.1940624-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6806; i=ardb@kernel.org; h=from:subject; bh=EB2b3J2UJ9v9GfC0liu3TjsAdiXjoVoRmp2AaTLPRws=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjWqrIwepmv5M6iAXYlb8TV7bUlLz/qMByna63trGu 5QoZstqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY1qqyAAKCRDDTyI5ktmPJFq3DA COlqBO7vNC/JKT8dFu2+Mv2W0RqUrP9h1zNCTPRCae0YYCPQ5UDx7Xd8owRyYERqvQq1bB0rsDZRNw ckpls3R6fz93n4Gms8RePSu5Hyq6Yi+VSrVyWUUGplB8HjIZuRWzsQpdPqriCHAc07GGewDznLCzzi IWLrHzGS2K44az05W51vHnd/0LQ2CoRonOJ7kAg/jfFoAy9EsV3X/CqdEX5D23B48eTwz+aaktAsHE Tkv4sZ6ZhP+LNutIsmtM1MYrEwUfMHhi0ll7UrOWWXS9nEqg0Gv2nUwNuUEmbvAkZ87MQREbT6Daog JQghYeGDJmWw5+OMRX2lom/WI/+JDf0jkW6sv9AwoxT2Fwq3HfPVX/7LmwEJURkV7JVBO6DMute7pd MqJiuZLRWSTHVmhsPgrseiW85uUNuUBJk2sYi4NA90q1FoszMKaKtaYgLiDul+CuI9HNPIoIYyKiRD EcHh1LrKZwd0ncuAOnIx3+PA6jsZqDrXMiht7ezSh0F8I= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221027_085920_271729_BEA5007E X-CRM114-Status: GOOD ( 20.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time. This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions. Signed-off-by: Ard Biesheuvel Reviewed-by: Nick Desaulniers Reviewed-by: Sami Tolvanen Tested-by: Sami Tolvanen --- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + include/asm-generic/vmlinux.lds.h | 9 +++++++-- scripts/module.lds.S | 6 ++++++ 9 files changed, 45 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 505c8a1ccbe0..7e3a9cf2193d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -370,6 +370,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 5e56d26a2239..7868a176993f 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h index 094701ec5500..dbba4b7559aa 100644 --- a/arch/arm64/include/asm/module.lds.h +++ b/arch/arm64/include/asm/module.lds.h @@ -17,4 +17,12 @@ SECTIONS { */ .text.hot : { *(.text.hot) } #endif + +#ifdef CONFIG_UNWIND_TABLES + /* + * Currently, we only use unwind info at module load time, so we can + * put it into the .init allocation. + */ + .init.eh_frame : { *(.eh_frame) } +#endif } diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 839291430cb3..4c0ea3cd4ea4 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -7,6 +7,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \ -D__DISABLE_EXPORTS -ffreestanding -D__NO_FORTIFY \ + -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) # remove SCS flags from all objects in this directory diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 45131e354e27..4c13dafc98b8 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -121,6 +121,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif +#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif + /* * The size of the PE/COFF section that covers the kernel image, which * runs from _stext to _edata, must be a round multiple of the PE/COFF @@ -231,6 +242,8 @@ SECTIONS __alt_instructions_end = .; } + UNWIND_DATA_SECTIONS + . = ALIGN(SEGMENT_ALIGN); __inittext_end = .; __initdata_begin = .; diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index b5c5119c7396..f566aad90875 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -89,6 +89,7 @@ quiet_cmd_hypcopy = HYPCOPY $@ # Remove ftrace, Shadow Call Stack, and CFI CFLAGS. # This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations. KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS)) +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables # KVM nVHE code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index b1601aad7e1a..1016f0b5311d 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -20,6 +20,7 @@ cflags-$(CONFIG_X86) += -m$(BITS) -D__KERNEL__ \ # disable the stackleak plugin cflags-$(CONFIG_ARM64) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fpie $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-unwind-tables -fno-asynchronous-unwind-tables \ $(call cc-option,-mbranch-protection=none) cflags-$(CONFIG_ARM) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fno-builtin -fpic \ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index c15de165ec8f..9ce7476696f8 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -1021,14 +1021,19 @@ * keep any .init_array.* sections. * https://bugs.llvm.org/show_bug.cgi?id=46478 */ +#ifdef CONFIG_UNWIND_TABLES +#define DISCARD_EH_FRAME +#else +#define DISCARD_EH_FRAME *(.eh_frame) +#endif #if defined(CONFIG_GCOV_KERNEL) || defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KCSAN) # ifdef CONFIG_CONSTRUCTORS # define SANITIZER_DISCARDS \ - *(.eh_frame) + DISCARD_EH_FRAME # else # define SANITIZER_DISCARDS \ *(.init_array) *(.init_array.*) \ - *(.eh_frame) + DISCARD_EH_FRAME # endif #else # define SANITIZER_DISCARDS diff --git a/scripts/module.lds.S b/scripts/module.lds.S index da4bddd26171..bf5bcf2836d8 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -3,6 +3,12 @@ * Archs are free to supply their own linker scripts. ld will * combine them automatically. */ +#ifdef CONFIG_UNWIND_TABLES +#define DISCARD_EH_FRAME +#else +#define DISCARD_EH_FRAME *(.eh_frame) +#endif + SECTIONS { /DISCARD/ : { *(.discard) From patchwork Thu Oct 27 15:59:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13022343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01CC5FA3740 for ; Thu, 27 Oct 2022 16:00:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Pjwy6R8wpMNIr3TsN/AlRFThFTISQ79dEwOTvCkOSis=; b=KkSfofX/8dayVQ 8AgbCSd5JNNdTOb3PXUh8kojxdUcOmBlvuwv4Jx13QCCx+1goxHexnTa1O9FcpupdEIsaHRKFTzQo 9S4lXZNuSLANJQeuj5v4zfQqywRLIrWjx7lg58QmoKen+ZvZ53S+zRq52vnWt3FLgLPK9KkYMZfGZ lARHTUkwB8u3VPKDrXY/80p/rj3WZrIi6omVO0+FMUrFHAKZLTKsg5yIPWIpIZ5SSUvwGTkOoYNUp 2ao0OoMcb12YwYQpEws15VvI2EpUYA8s9WdQosQ17nT4w2CjQv8Htb7o+K+CIpiKp+sI5zmErKd0r lhoUMtMHTJtnwYPuOD1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5IC-00E91K-4x; Thu, 27 Oct 2022 15:59:40 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5Hw-00E8yZ-L3 for linux-arm-kernel@lists.infradead.org; Thu, 27 Oct 2022 15:59:26 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 11969B826FB; Thu, 27 Oct 2022 15:59:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C565C433D7; Thu, 27 Oct 2022 15:59:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666886361; bh=wT+tw5Q4vteWoPSx5XbApVgN2IINrJNuZDv+NurMYYY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TVpPMmK4WpF0zaR0fCv86aQs5SfQMi0BTP1klu8ikUulvt5W99m3giLv9VQm5is0L BKXVk4qgsKpR/IsCNAhkA2PORml7tSowTXKXDZVT8FrZss7gkIaaL++RqA55Zdiw5w cCuUVT71aslD01xWo2bRTB0V/gFb5BgYG1cO45PbTO9NoZB5nZUmu8cITKW90ZKRts uWmWtJJc231NVckRuofytVq0cS74OfNi6ANAE12etu3x3Clw/JvRR+YXFg16ajZpSA Z79RWcZtEVCEox7pLsxf4Br5B0jfM7HOF1UEVK5fwnDiOmWAlXLndG1do3IPSHA+ej IZN9Pl44IhANg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland , Mark Brown , Sami Tolvanen , Nick Desaulniers , Kees Cook Subject: [PATCH v6 2/3] scs: add support for dynamic shadow call stacks Date: Thu, 27 Oct 2022 17:59:07 +0200 Message-Id: <20221027155908.1940624-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221027155908.1940624-1-ardb@kernel.org> References: <20221027155908.1940624-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4239; i=ardb@kernel.org; h=from:subject; bh=wT+tw5Q4vteWoPSx5XbApVgN2IINrJNuZDv+NurMYYY=; b=owEB7AET/pANAwAKAcNPIjmS2Y8kAcsmYgBjWqrJg/b8LGuiygg0nqQScmXE0DYGLgVKBHdsjo9n zR525yKJAbIEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY1qqyQAKCRDDTyI5ktmPJOTfC/ Yi45DkMOU0dlUpPrZ8LEbpnYKujsukxwL1NBNc+dC67ymdzwIt2mTj3W+GoldOmND15dMWYSbKHhXl 9P5AU9Ysu0dsY93UgU+RR+tSlDPAtWrDRj0A6ZjMfO29VD3gAtS6RrT+uHgW0lI1rTky3hIk24kD53 hR+axl4WcmfvyVOJdoJnQElf4CdPcgSI+NrzFALaK/2q95o5qpdpjzNeyuD9GUYvvPFzUo3S7ccxwJ a1FMR/me+JovKhXtIHHG0d0AR/gY0VR+Gukhu9ujNrcZYdOCqKzowfxMwhXjfQjW5D1BlcprSoB7wV LDcOXTLOfDJWt2bT8HFeAoXxKLm87HBuvzSNv9gTyiY7lZYJnX9vFLQ0TFB9TD7TyQ+YyJZKOkeWsU os53Vn4wmpwO3px2JBktyFzQLLoUy0e5HAWkuYwuna3XbFU0vsAghdJqdvS/78YAYVde5Z1lM4jY1D CtWr6WBag2ffus//Sl1usm+jMKZMCyn2tQajFj6UpN8g== X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221027_085925_001966_005AA8C6 X-CRM114-Status: GOOD ( 21.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a Kconfig symbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted. Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime. Signed-off-by: Ard Biesheuvel Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen Tested-by: Sami Tolvanen --- Makefile | 2 ++ arch/Kconfig | 7 +++++++ include/linux/scs.h | 18 ++++++++++++++++++ kernel/scs.c | 14 ++++++++++++-- 4 files changed, 39 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index f41ec8c8426b..53f52cf3ea3a 100644 --- a/Makefile +++ b/Makefile @@ -966,8 +966,10 @@ LDFLAGS_vmlinux += --gc-sections endif ifdef CONFIG_SHADOW_CALL_STACK +ifndef CONFIG_DYNAMIC_SCS CC_FLAGS_SCS := -fsanitize=shadow-call-stack KBUILD_CFLAGS += $(CC_FLAGS_SCS) +endif export CC_FLAGS_SCS endif diff --git a/arch/Kconfig b/arch/Kconfig index 8f138e580d1a..072a1b39e3af 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -651,6 +651,13 @@ config SHADOW_CALL_STACK reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying the stacks. +config DYNAMIC_SCS + bool + help + Set by the arch code if it relies on code patching to insert the + shadow call stack push and pop instructions rather than on the + compiler. + config LTO bool help diff --git a/include/linux/scs.h b/include/linux/scs.h index 18122d9e17ff..4ab5bdc898cf 100644 --- a/include/linux/scs.h +++ b/include/linux/scs.h @@ -53,6 +53,22 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk) return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC; } +DECLARE_STATIC_KEY_FALSE(dynamic_scs_enabled); + +static inline bool scs_is_dynamic(void) +{ + if (!IS_ENABLED(CONFIG_DYNAMIC_SCS)) + return false; + return static_branch_likely(&dynamic_scs_enabled); +} + +static inline bool scs_is_enabled(void) +{ + if (!IS_ENABLED(CONFIG_DYNAMIC_SCS)) + return true; + return scs_is_dynamic(); +} + #else /* CONFIG_SHADOW_CALL_STACK */ static inline void *scs_alloc(int node) { return NULL; } @@ -62,6 +78,8 @@ static inline void scs_task_reset(struct task_struct *tsk) {} static inline int scs_prepare(struct task_struct *tsk, int node) { return 0; } static inline void scs_release(struct task_struct *tsk) {} static inline bool task_scs_end_corrupted(struct task_struct *tsk) { return false; } +static inline bool scs_is_enabled(void) { return false; } +static inline bool scs_is_dynamic(void) { return false; } #endif /* CONFIG_SHADOW_CALL_STACK */ diff --git a/kernel/scs.c b/kernel/scs.c index b7e1b096d906..d7809affe740 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -12,6 +12,10 @@ #include #include +#ifdef CONFIG_DYNAMIC_SCS +DEFINE_STATIC_KEY_FALSE(dynamic_scs_enabled); +#endif + static void __scs_account(void *s, int account) { struct page *scs_page = vmalloc_to_page(s); @@ -101,14 +105,20 @@ static int scs_cleanup(unsigned int cpu) void __init scs_init(void) { + if (!scs_is_enabled()) + return; cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "scs:scs_cache", NULL, scs_cleanup); } int scs_prepare(struct task_struct *tsk, int node) { - void *s = scs_alloc(node); + void *s; + if (!scs_is_enabled()) + return 0; + + s = scs_alloc(node); if (!s) return -ENOMEM; @@ -148,7 +158,7 @@ void scs_release(struct task_struct *tsk) { void *s = task_scs(tsk); - if (!s) + if (!scs_is_enabled() || !s) return; WARN(task_scs_end_corrupted(tsk), From patchwork Thu Oct 27 15:59:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13022344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1850FA3740 for ; Thu, 27 Oct 2022 16:01:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0jaLjZ63JYd8KKJJ3NJj9ij4lGQU3AA0QnsU8Ob7sx0=; b=gHZtObwnHk8gMj PS7i7pAIBCshp9dgwJ61tF81j/7TNuJWpc+DLz2wlYYpQyZaR1QmDug/J6VuVX26Z2yXBahIroIVR 5NS77kejMcPJqitW1idQUzGQVWIcZLD6JaZfQCKWthe3wREMFhfXJsYlooexl613PgDmJp/u7ElFw m/LL8iv0lBnfjv6k0Zzgj743REx5F525MickZjpgXW8FPEWJke9AMJYxMh4Yzwh4scdBHvBDMNvHG qwFwmsHodgKgeBBMNSETnTmYCwleP+d3fIxBcq0tndARb2p+vy0KZZg+cmflAPRed+D8R0o3NQejT qEF/L9ypg4ZiujL3w/jg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5IS-00E943-3h; Thu, 27 Oct 2022 15:59:56 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oo5Hx-00E8z5-Uv for linux-arm-kernel@lists.infradead.org; Thu, 27 Oct 2022 15:59:28 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E01A0623C3; Thu, 27 Oct 2022 15:59:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F4B6C433B5; Thu, 27 Oct 2022 15:59:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666886364; bh=1+1wHkPMlSl+MNfD75pXnVioPL7aFG83bOsOSD7TbQs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gm7da8rdsWNIQyK/1sSUvmqUGfZMS45MGkN0pTigZgmHSpVp+UwAR972l5zjZQ7n8 BOX4Eef4gh3bRGV4ATi3QMFsTHn57+9YVi7SbnWeCrtrau9GmtqaILZhlBevBZowG/ 460UsocIiNt7Ci1rq0lVrPYobiNQhK5KEdPV3/pC4gqdJ3Il0ohxoO2eneXuDm2WYp BqazGursRRJJLwDY+Hmv10SO6X0pe229ZIIetGE0rMqaVvE9/UzxDEVc688Aj1NePC B8XmCNvHIZmJHTNxheuen2Q6riHA8/WLxqEB1f/6ZWnOqPMi+qY4+/gEbMZzaaWju9 B0Tm+JVBzF/SQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Catalin Marinas , Marc Zyngier , Mark Rutland , Mark Brown , Sami Tolvanen , Nick Desaulniers , Kees Cook Subject: [PATCH v6 3/3] arm64: implement dynamic shadow call stack for Clang Date: Thu, 27 Oct 2022 17:59:08 +0200 Message-Id: <20221027155908.1940624-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221027155908.1940624-1-ardb@kernel.org> References: <20221027155908.1940624-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=15945; i=ardb@kernel.org; h=from:subject; bh=1+1wHkPMlSl+MNfD75pXnVioPL7aFG83bOsOSD7TbQs=; b=owEB7AET/pANAwAKAcNPIjmS2Y8kAcsmYgBjWqrLHCdPVSFbvk9VmFanbrGP08E69tIjwm4vvV8+ q9NDcoKJAbIEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY1qqywAKCRDDTyI5ktmPJK75C/ itR0GZ/1OXT7lceHbeSun1A56kFzuQNPDG9qFcZN/MKn1BShh2YlVq/T1S9tyYl+HIOclufjncfpIC Mxy/rJZN2aQv5zfYIpKlbng3dvt56Y0wYl1ICJN3z0X1Pq31c9wPOLmHaeluxpfrtHKL7FeNC+Ui3K pkMvRjIuSgXCowYZTsGLoH0yGUAExCGFAcgeVD7P7v9JFeyjLU54sII9DIRKoZaNLKBrSPZakjh65z aqR8IxuwodeMNkyZRvnrDlQY/joA/lYdP5vc2RLEr7Jt3RDSFj4yiqKMn6BFIMNXKaQLcGRSjHMGQP 4eLkrCmJN4vewMlFISs8U7Un76n83TgJ6dWkir3ub8o6ZwHLx6x0YZKXvS8Rz0/HbYUAPaITA7eHDb M2Cp4D0YsunmCXgCqGSMBZWi/2vQh2R68aZk9OL4OMHU/DKAbRX+9mupWe/zQJSRv+9Ef1i0ZTGGmY EuA8oaOIu+xutRgvguAunUAuGJsP0W/m3K+3irg+SUJg== X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221027_085926_110297_DAB7C094 X-CRM114-Status: GOOD ( 31.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively. This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior. Signed-off-by: Ard Biesheuvel Reviewed-by: Sami Tolvanen Tested-by: Sami Tolvanen --- arch/arm64/Kconfig | 9 + arch/arm64/Makefile | 10 +- arch/arm64/include/asm/scs.h | 49 ++++ arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 8 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 4 + 10 files changed, 342 insertions(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7e3a9cf2193d..170832f31eff 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2160,6 +2160,15 @@ config ARCH_NR_GPIO If unsure, leave the default value. +config UNWIND_PATCH_PAC_INTO_SCS + bool "Enable shadow call stack dynamically using code patching" + # needs Clang with https://reviews.llvm.org/D111780 incorporated + depends on CC_IS_CLANG && CLANG_VERSION >= 150000 + depends on ARM64_PTR_AUTH_KERNEL && CC_HAS_BRANCH_PROT_PAC_RET + depends on SHADOW_CALL_STACK + select UNWIND_TABLES + select DYNAMIC_SCS + endmenu # "Kernel Features" menu "Boot options" diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 7868a176993f..4d272ad1df1f 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -77,10 +77,16 @@ branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address= # We enable additional protection for leaf functions as there is some # narrow potential for ROP protection benefits and no substantial # performance impact has been observed. +PACRET-y := pac-ret+leaf + +# Using a shadow call stack in leaf functions is too costly, so avoid PAC there +# as well when we may be patching PAC into SCS +PACRET-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) := pac-ret + ifeq ($(CONFIG_ARM64_BTI_KERNEL),y) -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=pac-ret+leaf+bti +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=$(PACRET-y)+bti else -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=pac-ret+leaf +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=$(PACRET-y) endif # -march=armv8.3-a enables the non-nops instructions for PAC, to avoid the # compiler to generate them and consequently to break the single image contract diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 8297bccf0784..ff7da1268a52 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -5,6 +5,7 @@ #ifdef __ASSEMBLY__ #include +#include #ifdef CONFIG_SHADOW_CALL_STACK scs_sp .req x18 @@ -24,6 +25,54 @@ .endm #endif /* CONFIG_SHADOW_CALL_STACK */ + +#else + +#include +#include + +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS +static inline bool should_patch_pac_into_scs(void) +{ + u64 reg; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); + if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | + SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) + return false; + + reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); + if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) + return false; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { + reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); + if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) + return false; + } + return true; +} + +static inline void dynamic_scs_init(void) +{ + if (should_patch_pac_into_scs()) { + pr_info("Enabling dynamic shadow call stack\n"); + static_branch_enable(&dynamic_scs_enabled); + } +} +#else +static inline void dynamic_scs_init(void) {} +#endif + +int scs_patch(const u8 eh_frame[], int size); + #endif /* __ASSEMBLY __ */ #endif /* _ASM_SCS_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 2f361a883d8c..8dd925f4a4c6 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -80,6 +80,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o +CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2196aad7b55b..952e17bd1c0b 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -462,6 +462,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status bl init_feature_override // Parse cpu feature overrides +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + bl scs_patch_vmlinux +#endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible ldp x29, x30, [sp], #16 diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 38dbd3828f13..9d8eaab742a4 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -41,7 +41,7 @@ static void init_irq_scs(void) { int cpu; - if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK)) + if (!scs_is_enabled()) return; for_each_possible_cpu(cpu) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 76b41e4ca9fa..fa7b3228944b 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -15,9 +15,11 @@ #include #include #include +#include #include #include #include +#include #include void *module_alloc(unsigned long size) @@ -514,5 +516,11 @@ int module_finalize(const Elf_Ehdr *hdr, if (s) apply_alternatives_module((void *)s->sh_addr, s->sh_size); + if (scs_is_dynamic()) { + s = find_section(hdr, sechdrs, ".init.eh_frame"); + if (s) + scs_patch((void *)s->sh_addr, s->sh_size); + } + return module_init_ftrace_plt(hdr, sechdrs, me); } diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/patch-scs.c new file mode 100644 index 000000000000..1b3da02d5b74 --- /dev/null +++ b/arch/arm64/kernel/patch-scs.c @@ -0,0 +1,257 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +// +// This minimal DWARF CFI parser is partially based on the code in +// arch/arc/kernel/unwind.c, and on the document below: +// https://refspecs.linuxbase.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html +// + +#define DW_CFA_nop 0x00 +#define DW_CFA_set_loc 0x01 +#define DW_CFA_advance_loc1 0x02 +#define DW_CFA_advance_loc2 0x03 +#define DW_CFA_advance_loc4 0x04 +#define DW_CFA_offset_extended 0x05 +#define DW_CFA_restore_extended 0x06 +#define DW_CFA_undefined 0x07 +#define DW_CFA_same_value 0x08 +#define DW_CFA_register 0x09 +#define DW_CFA_remember_state 0x0a +#define DW_CFA_restore_state 0x0b +#define DW_CFA_def_cfa 0x0c +#define DW_CFA_def_cfa_register 0x0d +#define DW_CFA_def_cfa_offset 0x0e +#define DW_CFA_def_cfa_expression 0x0f +#define DW_CFA_expression 0x10 +#define DW_CFA_offset_extended_sf 0x11 +#define DW_CFA_def_cfa_sf 0x12 +#define DW_CFA_def_cfa_offset_sf 0x13 +#define DW_CFA_val_offset 0x14 +#define DW_CFA_val_offset_sf 0x15 +#define DW_CFA_val_expression 0x16 +#define DW_CFA_lo_user 0x1c +#define DW_CFA_negate_ra_state 0x2d +#define DW_CFA_GNU_args_size 0x2e +#define DW_CFA_GNU_negative_offset_extended 0x2f +#define DW_CFA_hi_user 0x3f + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +enum { + PACIASP = 0xd503233f, + AUTIASP = 0xd50323bf, + SCS_PUSH = 0xf800865e, + SCS_POP = 0xf85f8e5e, +}; + +static void __always_inline scs_patch_loc(u64 loc) +{ + u32 insn = le32_to_cpup((void *)loc); + + switch (insn) { + case PACIASP: + *(u32 *)loc = cpu_to_le32(SCS_PUSH); + break; + case AUTIASP: + *(u32 *)loc = cpu_to_le32(SCS_POP); + break; + default: + /* + * While the DW_CFA_negate_ra_state directive is guaranteed to + * appear right after a PACIASP/AUTIASP instruction, it may + * also appear after a DW_CFA_restore_state directive that + * restores a state that is only partially accurate, and is + * followed by DW_CFA_negate_ra_state directive to toggle the + * PAC bit again. So we permit other instructions here, and ignore + * them. + */ + return; + } + dcache_clean_pou(loc, loc + sizeof(u32)); +} + +/* + * Skip one uleb128/sleb128 encoded quantity from the opcode stream. All bytes + * except the last one have bit #7 set. + */ +static int __always_inline skip_xleb128(const u8 **opcode, int size) +{ + u8 c; + + do { + c = *(*opcode)++; + size--; + } while (c & BIT(7)); + + return size; +} + +struct eh_frame { + /* + * The size of this frame if 0 < size < U32_MAX, 0 terminates the list. + */ + u32 size; + + /* + * The first frame is a Common Information Entry (CIE) frame, followed + * by one or more Frame Description Entry (FDE) frames. In the former + * case, this field is 0, otherwise it is the negated offset relative + * to the associated CIE frame. + */ + u32 cie_id_or_pointer; + + union { + struct { // CIE + u8 version; + u8 augmentation_string[]; + }; + + struct { // FDE + s32 initial_loc; + s32 range; + u8 opcodes[]; + }; + }; +}; + +static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor) +{ + int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; + u64 loc = (u64)offset_to_ptr(&frame->initial_loc); + const u8 *opcode = frame->opcodes; + + if (fde_has_augmentation_data) { + int l; + + // assume single byte uleb128_t + if (WARN_ON(*opcode & BIT(7))) + return -ENOEXEC; + + l = *opcode++; + opcode += l; + size -= l + 1; + } + + /* + * Starting from 'loc', apply the CFA opcodes that advance the location + * pointer, and identify the locations of the PAC instructions. + */ + while (size-- > 0) { + switch (*opcode++) { + case DW_CFA_nop: + case DW_CFA_remember_state: + case DW_CFA_restore_state: + break; + + case DW_CFA_advance_loc1: + loc += *opcode++ * code_alignment_factor; + size--; + break; + + case DW_CFA_advance_loc2: + loc += *opcode++ * code_alignment_factor; + loc += (*opcode++ << 8) * code_alignment_factor; + size -= 2; + break; + + case DW_CFA_def_cfa: + case DW_CFA_offset_extended: + size = skip_xleb128(&opcode, size); + fallthrough; + case DW_CFA_def_cfa_offset: + case DW_CFA_def_cfa_offset_sf: + case DW_CFA_def_cfa_register: + case DW_CFA_same_value: + case DW_CFA_restore_extended: + case 0x80 ... 0xbf: + size = skip_xleb128(&opcode, size); + break; + + case DW_CFA_negate_ra_state: + scs_patch_loc(loc - 4); + break; + + case 0x40 ... 0x7f: + // advance loc + loc += (opcode[-1] & 0x3f) * code_alignment_factor; + break; + + case 0xc0 ... 0xff: + break; + + default: + pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); + return -ENOEXEC; + } + } + return 0; +} + +int noinstr scs_patch(const u8 eh_frame[], int size) +{ + const u8 *p = eh_frame; + + while (size > 4) { + const struct eh_frame *frame = (const void *)p; + bool fde_has_augmentation_data = true; + int code_alignment_factor = 1; + int ret; + + if (frame->size == 0 || + frame->size == U32_MAX || + frame->size > size) + break; + + if (frame->cie_id_or_pointer == 0) { + const u8 *p = frame->augmentation_string; + + /* a 'z' in the augmentation string must come first */ + fde_has_augmentation_data = *p == 'z'; + + /* + * The code alignment factor is a uleb128 encoded field + * but given that the only sensible values are 1 or 4, + * there is no point in decoding the whole thing. + */ + p += strlen(p) + 1; + if (!WARN_ON(*p & BIT(7))) + code_alignment_factor = *p; + } else { + ret = scs_handle_fde_frame(frame, + fde_has_augmentation_data, + code_alignment_factor); + if (ret) + return ret; + } + + p += sizeof(frame->size) + frame->size; + size -= sizeof(frame->size) + frame->size; + } + return 0; +} + +asmlinkage void __init scs_patch_vmlinux(void) +{ + if (!should_patch_pac_into_scs()) + return; + + WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); + icache_inval_all_pou(); + isb(); +} diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index d56e170e1ca7..830be01af32d 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -144,7 +144,7 @@ static int init_sdei_scs(void) int cpu; int err = 0; - if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK)) + if (!scs_is_enabled()) return 0; for_each_possible_cpu(cpu) { diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index fea3223704b6..12cfe9d0d3fa 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -312,6 +314,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) jump_label_init(); parse_early_param(); + dynamic_scs_init(); + /* * Unmask asynchronous aborts and fiq after bringing up possible * earlycon. (Report possible System Errors once we can report this