From patchwork Thu May 5 16:10:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12839762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 857D3C433EF for ; Thu, 5 May 2022 16:11:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=x02EYO4nFeOPHsckcTbsFVvQNdZVUREOGakZoGYF48A=; b=D1Btqz8+LrvIod 0MsIq9crJ/6uHHcDFWupQfGLGDZIJ6o6bgu9vVVb6vfu83dF288f1aJXKbp8FIRqLAIiG8P6WUFiU urJXUJ7d046REg4zGsd3WnPzbOu203FR4VGBt2gmQ43g1YgQ4wmR/neVKB0UOCn0OAEyi1BrkcCHy Ty4aWSvLKZV1kSHbYEeRMoVELeNZd2Me0LwHskevEpBYzIeS+Y/GJGIfV+9lzCoxrHbojawvRG4Tr ThZchDZPGqlzW19jg7NeipcW2hykYw+pTwIRaF1XkC7XUNxVf08MzXMNNhCs3arRlvxw2kpn0agRm LYHYwA+f/yNvEat8Hl+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme3u-00Gn3B-PT; Thu, 05 May 2022 16:10:42 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme3k-00Gn09-JT for linux-arm-kernel@lists.infradead.org; Thu, 05 May 2022 16:10:34 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E400061E26; Thu, 5 May 2022 16:10:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 190C1C385B1; Thu, 5 May 2022 16:10:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651767031; bh=QDVLzpeJgRUO8ekEYXYRCe3+zpE+zhaLUf11RtkqhFc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BoCZCZp7HfqVbJ2HGFg+X5VGoC+qtXtIV+wLW7o8EGu5riV4dihz0jVWv2wUp5hBU Jo/CAbmrYp2riaLztEF82S6hwBNbZN4/BATxACnc+BbTwM2Knc6KREVyH4oe1S+NUW idSit+rF/xTwg5VizgEY6kkLEAeyJoV7uhb1ZgjAT4KcC9LsIb8M4J3wvO6LcsFzwE 9McQeeUIO1rELr06ihOUVU2g6carbJBlEXXMOYiKbu+FGI5j2Wi3HirI4gWZlnIh6u VLv4ICPP2gdg1YVpHbR+1zwpXbM3i9UHrtvT2QEhxngjiskUJJSBTP812GwK4nrkjb 9PjDkp8HvQxbg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, maz@kernel.org, Ard Biesheuvel , Kees Cook , Sami Tolvanen , Fangrui Song , Nick Desaulniers , Dan Li Subject: [RFC PATCH v2 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules Date: Thu, 5 May 2022 18:10:09 +0200 Message-Id: <20220505161011.1801596-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220505161011.1801596-1-ardb@kernel.org> References: <20220505161011.1801596-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4048; h=from:subject; bh=QDVLzpeJgRUO8ekEYXYRCe3+zpE+zhaLUf11RtkqhFc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBic/bgDpiZPmvpifUJRS/J/CS+mpUuj8FzYXrnF/g5 +CuIQKeJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYnP24AAKCRDDTyI5ktmPJMo5DA CDtRbJ0ujhl/8Z65lQUeJ8XPhUGSS/JwcSr2rfyPRJ9YX5eLLj/bPi+Wt4MmVn5yuFGjV3iA3eCkEi PW4EVL92NKijvPk3TAxLmZvxRf77ho66SgmSsnGO96rPQbPDLBsEQ9CKnlInRi49iIUiwfdj/sOhCJ EJJ6uo3fUglZ8bx3uUNT1M0QZa1NpMPjkOpMvk7TZNctW0UR2lV6TBdolxASLW0i3c+ZOoFL6ifoaw fibjmmB+qEiDRXuNcHLX6MygTZIBQbNyZXkTHP2G1ZipTYBqFGaAbiUAbNWK48eKJcTxW6pA2HUNJs +DeWG1PzmnpWQkO4dai0SIFuDAhgm09UPKW74dvX7wGaNs4clbl/Kfz4a+nRPgH5DAqDo0jcsqcmVX sPL/+ON1LxcSbKGS5om6kILWt5jyTXzcSqzdOLfrvJVeu3D+2vx+CzVUg+CaSCrl7jm1ILbBe+kJWT FHsQXhd8SIROYnxuTac/N2qaxO/yd4Qmf7YW3Y8dBCYY4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220505_091032_756335_DC516E5B X-CRM114-Status: GOOD ( 18.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time. This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions. Signed-off-by: Ard Biesheuvel Reviewed-by: Nick Desaulniers --- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 16 ++++++++++++++++ drivers/firmware/efi/libstub/Makefile | 1 + 5 files changed, 33 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 57c4c995965f..b6302f7cd73f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -354,6 +354,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 2f1de88651e6..a4c6807ecbaf 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h index 094701ec5500..dbba4b7559aa 100644 --- a/arch/arm64/include/asm/module.lds.h +++ b/arch/arm64/include/asm/module.lds.h @@ -17,4 +17,12 @@ SECTIONS { */ .text.hot : { *(.text.hot) } #endif + +#ifdef CONFIG_UNWIND_TABLES + /* + * Currently, we only use unwind info at module load time, so we can + * put it into the .init allocation. + */ + .init.eh_frame : { *(.eh_frame) } +#endif } diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index edaf0faf766f..2f4908c8d152 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -122,6 +122,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif +#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif + /* * The size of the PE/COFF section that covers the kernel image, which * runs from _stext to _edata, must be a round multiple of the PE/COFF @@ -150,6 +161,9 @@ SECTIONS /DISCARD/ : { *(.interp .dynamic) *(.dynsym .dynstr .hash .gnu.hash) +#ifndef CONFIG_UNWIND_TABLES + *(.eh_frame) +#endif } . = KIMAGE_VADDR; @@ -228,6 +242,8 @@ SECTIONS __alt_instructions_end = .; } + UNWIND_DATA_SECTIONS + . = ALIGN(SEGMENT_ALIGN); __inittext_end = .; __initdata_begin = .; diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index d0537573501e..78c46638707a 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -20,6 +20,7 @@ cflags-$(CONFIG_X86) += -m$(BITS) -D__KERNEL__ \ # disable the stackleak plugin cflags-$(CONFIG_ARM64) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fpie $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-unwind-tables -fno-asynchronous-unwind-tables \ $(call cc-option,-mbranch-protection=none) cflags-$(CONFIG_ARM) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fno-builtin -fpic \ From patchwork Thu May 5 16:10:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12839763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60690C433EF for ; Thu, 5 May 2022 16:12:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=t8JuulrmBWlfHxP6omloRLTa6TlSBkFN+qDSC0L3Y/c=; b=wRkGfKwdArm7BU F59jxdZBpZ3B9D169c6Z/BCVpUOZ9gsUiwmqqk7YWuS4yTflufBSP5Oljm2drJpyYQo2uPYyMEnJK jEpH6UGaeMys/k7HXsjpX15Qink2r8K2e2y2GNpN9+IBYGiTDrgO0AY1E+jaHOoOB9LVBl6zCGeA5 4MC6ZfH3eGhVMPOiiDS2TESggMK1FBG63GmIhAH87K0pCv4bpFIGOja2t9qEiGa34zjqGkXmXUNqP 8448RJ4PVz6v3HgbtWQrJnbCmjoaPFR+D/EQ/TWsh/ZZGQ6Vb4uSYbFRZOr1wxAQbsSEe21FsKlP7 +AhV7Eg0zdqI94x1R2RA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme4E-00GnAB-SF; Thu, 05 May 2022 16:11:03 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme3p-00Gn1Q-2i for linux-arm-kernel@lists.infradead.org; Thu, 05 May 2022 16:10:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6A4F4B82DF4; Thu, 5 May 2022 16:10:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0574C385A4; Thu, 5 May 2022 16:10:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651767034; bh=Clcqu/Ezs830DckhrTOlJvTLynFQaPtrk4ln1+Vrkjw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F0V3de2zt1aBnajXGAOxT+pooeY4eJElF/h14JMSfWcbjJExxAjJHoro4332KjENe N/4jrAA+510O8FuVq7QDf+k6lJbd+xy2oCNmtr8rC3ROFGXBX6g/HSFA9lNoYlYRo3 IHiku7QHarFnCt1UGTLeEpVrKkqTsdaaQ4Rj2RzWH4RhBi7xzbDei0OgWNzCryqSjx 42TpZV9fsINDyFU0nF7jIHpPcOovWpxxT+B5RwvW9UXfwr36MI+we/7yBY9UTN1STF 22tJEpkg2382UkfxlPlV0yUpRuEQgNsDpNbjFQZuZEZfV2VWWm/woy9z1/1QGlpn8y /f5lU49citx8g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, maz@kernel.org, Ard Biesheuvel , Kees Cook , Sami Tolvanen , Fangrui Song , Nick Desaulniers , Dan Li Subject: [RFC PATCH v2 2/3] scs: add support for dynamic shadow call stacks Date: Thu, 5 May 2022 18:10:10 +0200 Message-Id: <20220505161011.1801596-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220505161011.1801596-1-ardb@kernel.org> References: <20220505161011.1801596-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3819; h=from:subject; bh=Clcqu/Ezs830DckhrTOlJvTLynFQaPtrk4ln1+Vrkjw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBic/bhmx8IcuNnb+l+fT9BVoswvmVT3K0ESJpl6iVv NE73DF2JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYnP24QAKCRDDTyI5ktmPJHfHC/ 93vk8iwRxhBKQG0oavOnrTrk6lthUVcHDFBrb7Dzxyj/WN0zIobXin99IOynHqvaOifmcEEn77COzF x9MkJro/uL9/KUup64+W0vh4gOWicQ0ej6VoJGhft8TFpzJw3T0rT7SQMcpgdpq3sG3Gs/1kPdqICT lmaLUSptKhkdaB0lnb6F1mzU1iXE8xMLXy8kIHMKUYWhyjb5ncrz3M8/GqMJ4Fkoh3LPrJGoAsDQJp mK119DW/9/hgnC8S6iCbtEuK6epoM3CUbMwe5A+A/yHo9ZHuw3FarWIrFkV1xtZ3jG8R9fJ2i+9s9s M4/XubLMLaaWu51uboPHGY8wYvYP207s+vVtLinllL5yFsDbH9vj1m16lBVD1pPFybJSkuAaoVvIi/ ViecxUrC1x2aMJnhIjjef/JhI0pKKU3aBOgYwnQjJbrzXF/Tjs8tNjfitE7B8jxhyefB0pvJnv5+me pMiu8MxLEjUwZyxYoJ4E7yTorYjzXXW8Cj9fUj8KCcjZs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220505_091037_470798_5D3490C8 X-CRM114-Status: GOOD ( 21.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a Kconfig symbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted. Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime. Signed-off-by: Ard Biesheuvel Reviewed-by: Nick Desaulniers Reviewed-by: Kees Cook --- Makefile | 2 ++ arch/Kconfig | 7 +++++++ include/linux/scs.h | 10 ++++++++++ kernel/scs.c | 14 ++++++++++++-- 4 files changed, 31 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index fa5112a0ec1b..a578fffc0337 100644 --- a/Makefile +++ b/Makefile @@ -882,8 +882,10 @@ LDFLAGS_vmlinux += --gc-sections endif ifdef CONFIG_SHADOW_CALL_STACK +ifndef CONFIG_DYNAMIC_SCS CC_FLAGS_SCS := -fsanitize=shadow-call-stack KBUILD_CFLAGS += $(CC_FLAGS_SCS) +endif export CC_FLAGS_SCS endif diff --git a/arch/Kconfig b/arch/Kconfig index 29b0167c088b..126caa75969a 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -627,6 +627,13 @@ config SHADOW_CALL_STACK reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying the stacks. +config DYNAMIC_SCS + bool + help + Set by the arch code if it relies on code patching to insert the + shadow call stack push and pop instructions rather than on the + compiler. + config LTO bool help diff --git a/include/linux/scs.h b/include/linux/scs.h index 18122d9e17ff..4cc01f21b17a 100644 --- a/include/linux/scs.h +++ b/include/linux/scs.h @@ -53,6 +53,15 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk) return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC; } +DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled); + +static inline bool scs_is_enabled(void) +{ + if (!IS_ENABLED(CONFIG_DYNAMIC_SCS)) + return true; + return static_branch_likely(&dynamic_scs_enabled); +} + #else /* CONFIG_SHADOW_CALL_STACK */ static inline void *scs_alloc(int node) { return NULL; } @@ -62,6 +71,7 @@ static inline void scs_task_reset(struct task_struct *tsk) {} static inline int scs_prepare(struct task_struct *tsk, int node) { return 0; } static inline void scs_release(struct task_struct *tsk) {} static inline bool task_scs_end_corrupted(struct task_struct *tsk) { return false; } +static inline bool scs_is_enabled(void) { return false; } #endif /* CONFIG_SHADOW_CALL_STACK */ diff --git a/kernel/scs.c b/kernel/scs.c index b7e1b096d906..8826794d2645 100644 --- a/kernel/scs.c +++ b/kernel/scs.c @@ -12,6 +12,10 @@ #include #include +#ifdef CONFIG_DYNAMIC_SCS +DEFINE_STATIC_KEY_TRUE(dynamic_scs_enabled); +#endif + static void __scs_account(void *s, int account) { struct page *scs_page = vmalloc_to_page(s); @@ -101,14 +105,20 @@ static int scs_cleanup(unsigned int cpu) void __init scs_init(void) { + if (!scs_is_enabled()) + return; cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "scs:scs_cache", NULL, scs_cleanup); } int scs_prepare(struct task_struct *tsk, int node) { - void *s = scs_alloc(node); + void *s; + if (!scs_is_enabled()) + return 0; + + s = scs_alloc(node); if (!s) return -ENOMEM; @@ -148,7 +158,7 @@ void scs_release(struct task_struct *tsk) { void *s = task_scs(tsk); - if (!s) + if (!scs_is_enabled() || !s) return; WARN(task_scs_end_corrupted(tsk), From patchwork Thu May 5 16:10:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12839764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1D0BEC433EF for ; Thu, 5 May 2022 16:12:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4H2X4gI2oxlCZfHoRQzQDb1uhcg+vIDV3Vk622mwPLs=; b=RwwdBUQoWPsHGo VediQ4dZ9YFeXTTo8PrRmuCqi13/3o0RzcCrPMl1DSPGN5e4SZhGWhJIatjqatbUsDZR1ssJjpk8W 3XKS3EK18xDOZVKc2BlTw3BJscmPC7gI8BmlbWRYObdcSYsGSthO19JYJrpxYMg9irLs9j19lOZF1 y/QOn3zsQf43ihioct9tct43ryI0XAuRG1or9dr9UNs4DNztUMT6FQhxECI9AeqPqEyjOb7/0uGm6 Qz5zuDwT0k+CdU4ycLcnYVqAsJ8GbjK//Mg352Muagqkr6+xHkndnUIFyI1w5hXe6P9+XTXARDlFG Ain+Vxi6JTYK6R3rbpQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme4Q-00GnF5-3p; Thu, 05 May 2022 16:11:14 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nme3p-00Gn26-OF for linux-arm-kernel@lists.infradead.org; Thu, 05 May 2022 16:10:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4AD4161E36; Thu, 5 May 2022 16:10:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 76531C385B2; Thu, 5 May 2022 16:10:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651767036; bh=IgZCWXR0/fFFaIpY8uXmImi8ocdmjxfEYE3vF2X6jis=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mLnno54xO/QZ147JyxJ/v65wYxSN+6oLkWXkL9qGoG2Fp+xnsq4NsVFQAPLVM9qIa 30D0LZfX2g5XgWVXcnbe0O+LX3fZLbUaKLTiGtn89or0PtgLVUTq7gzkZ+jB9ce6mK 7VOiwFcsKOnR7jjeha6wEBcKMUCCVw8jtSP+BM9jIFmyY1wLVFcSXjoHkY3wXZTwMx XUgT8xlaxHVToZjRDcBhbx6zXIcZGoOSkrVJscYpd/MlOzxDdxQrvzqoXCcS6cI3xF yLiZxKbbEPZ/X0LQxGFzyX0OqTWH5WwUvBsxhfG5D/ZS4BEND7DB2jPXmC7EkV4S2y 8BREkrDdMKASA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: will@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, maz@kernel.org, Ard Biesheuvel , Kees Cook , Sami Tolvanen , Fangrui Song , Nick Desaulniers , Dan Li Subject: [RFC PATCH v2 3/3] arm64: implement dynamic shadow call stack for Clang Date: Thu, 5 May 2022 18:10:11 +0200 Message-Id: <20220505161011.1801596-4-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220505161011.1801596-1-ardb@kernel.org> References: <20220505161011.1801596-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=13201; h=from:subject; bh=IgZCWXR0/fFFaIpY8uXmImi8ocdmjxfEYE3vF2X6jis=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBic/bju3zMlYkeNWF9tZ4TrpVZH/HF11N0n7f8xsoc crl8PpaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYnP24wAKCRDDTyI5ktmPJKyhC/ 4iPfrgvHdaRRBrFGRkX3xNTInWLuYkUSwEXbNe2Ywa96TkXDkOsL7TN1UQT7CsxkBFMLYykZxWJc96 SyWmgQp6bTwqAOtpCH6jV22jTnml4mYIRobgqcdhDNYBzbofxEt+oa9MsxJfBp5hLAADIjSqxS0znO FxxoXyfztWB8QqprMOYx1gRtJRWjFhhRmx5Zz51AFevmP9einSu3WZcX6Dor3EJuvCqSsOJyk5z/6K M/k6198nHXdmkAI2b6HVj9oBLYeG3zeUdBzGngBgw33jDERM5ZEMNg5XnXvf8yxGwO5VxlIxkvJjkE xN6olYlklkDar65lY2APdFmvnboe4YtarO8rxa1ofhiWMBsmYy4iZR8Dn19x27fE++eDUZ4/xqy7Nk pxoJYtIwymwqZz16cNtAUsn3wAImHDp2sRniM3Xz4hcVK6QwxlkgErsLWgg3u63vu3Li7UKKoClCZA +19UaLrtDoJZe/p3JnQsIeAVPY2dwPMC6dq+S2KMciCj4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220505_091037_932515_BCC4DF12 X-CRM114-Status: GOOD ( 32.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively. This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the expected behavior. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 8 + arch/arm64/include/asm/scs.h | 12 + arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 10 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 5 + 9 files changed, 299 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b6302f7cd73f..df7a7aff456a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -357,6 +357,14 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool +config UNWIND_PATCH_PAC_INTO_SCS + def_bool y + depends on CC_IS_CLANG && CLANG_VERSION >= 150000 + depends on SHADOW_CALL_STACK + depends on ARM64_PTR_AUTH_KERNEL + select UNWIND_TABLES + select DYNAMIC_SCS + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 8297bccf0784..09aed251e695 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -24,6 +24,18 @@ .endm #endif /* CONFIG_SHADOW_CALL_STACK */ + +#else + + +#ifdef CONFIG_UNWIND_PATCH_PAC_TO_SCS +extern bool should_disable_dynamic_scs; +#else +#define should_disable_dynamic_scs (false) +#endif + +int scs_patch(const u8 eh_frame[], int size); + #endif /* __ASSEMBLY __ */ #endif /* _ASM_SCS_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 986837d7ec82..d7074470031a 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -74,6 +74,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o +CFLAGS_patch-scs.o += -mbranch-protection=none obj-y += probes/ head-y := head.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 6a98f1a38c29..e9601c8a1bcd 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -453,6 +453,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x21 // pass FDT address in x0 bl early_fdt_map // Try mapping the FDT early bl init_feature_override // Parse cpu feature overrides +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + bl scs_patch_vmlinux +#endif #ifdef CONFIG_RANDOMIZE_BASE tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? b.ne 0f diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index bda49430c9ea..c284ec35c27c 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -39,7 +39,7 @@ static void init_irq_scs(void) { int cpu; - if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK)) + if (!scs_is_enabled()) return; for_each_possible_cpu(cpu) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index f2d4bb14bfab..7e9e63600d28 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -18,6 +18,7 @@ #include #include #include +#include #include void *module_alloc(unsigned long size) @@ -529,5 +530,14 @@ int module_finalize(const Elf_Ehdr *hdr, if (s) apply_alternatives_module((void *)s->sh_addr, s->sh_size); + if (IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) && + !system_supports_address_auth() && + (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) || !system_supports_bti())) { + + s = find_section(hdr, sechdrs, ".init.eh_frame"); + if (s) + scs_patch((void *)s->sh_addr, s->sh_size); + } + return module_init_ftrace_plt(hdr, sechdrs, me); } diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/patch-scs.c new file mode 100644 index 000000000000..8c534630c2a1 --- /dev/null +++ b/arch/arm64/kernel/patch-scs.c @@ -0,0 +1,257 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include + +#include + +#define DW_CFA_nop 0x00 +#define DW_CFA_set_loc 0x01 +#define DW_CFA_advance_loc1 0x02 +#define DW_CFA_advance_loc2 0x03 +#define DW_CFA_advance_loc4 0x04 +#define DW_CFA_offset_extended 0x05 +#define DW_CFA_restore_extended 0x06 +#define DW_CFA_undefined 0x07 +#define DW_CFA_same_value 0x08 +#define DW_CFA_register 0x09 +#define DW_CFA_remember_state 0x0a +#define DW_CFA_restore_state 0x0b +#define DW_CFA_def_cfa 0x0c +#define DW_CFA_def_cfa_register 0x0d +#define DW_CFA_def_cfa_offset 0x0e +#define DW_CFA_def_cfa_expression 0x0f +#define DW_CFA_expression 0x10 +#define DW_CFA_offset_extended_sf 0x11 +#define DW_CFA_def_cfa_sf 0x12 +#define DW_CFA_def_cfa_offset_sf 0x13 +#define DW_CFA_val_offset 0x14 +#define DW_CFA_val_offset_sf 0x15 +#define DW_CFA_val_expression 0x16 +#define DW_CFA_lo_user 0x1c +#define DW_CFA_negate_ra_state 0x2d +#define DW_CFA_GNU_args_size 0x2e +#define DW_CFA_GNU_negative_offset_extended 0x2f +#define DW_CFA_hi_user 0x3f + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +struct fde_frame { + s32 initial_loc; + s32 range; +}; + +enum { + PACIASP = 0xd503233f, + AUTIASP = 0xd50323bf, + SCS_PUSH = 0xf800865e, + SCS_POP = 0xf85f8e5e, +}; + +static void __always_inline scs_patch_loc(u64 loc) +{ + u32 insn = le32_to_cpup((void *)loc); + + switch (insn) { + case PACIASP: + *(u32 *)loc = cpu_to_le32(SCS_PUSH); + break; + case AUTIASP: + *(u32 *)loc = cpu_to_le32(SCS_POP); + break; + default: + /* + * While the DW_CFA_negate_ra_state directive is guaranteed to + * appear right after a PACIASP/AUTIASP instruction, it may + * also appear after a DW_CFA_restore_state directive that + * restores a state that is only partially accurate, and is + * followed by DW_CFA_negate_ra_state directive to toggle the + * PAC bit again. So we permit other instructions here, and ignore + * them. + */ + break; + } +} + +/* + * Skip one uleb128/sleb128 encoded quantity from the opcode stream. All bytes + * except the last one have bit #7 set. + */ +static int __always_inline skip_xleb128(const u8 **opcode, int size) +{ + u8 c; + + do { + c = *(*opcode)++; + size--; + } while (c & 0x80); + + return size; +} + +static int noinstr scs_handle_frame(const u8 eh_frame[], u32 size) +{ + const struct fde_frame *fde; + const u8 *opcode; + u64 loc; + + /* + * For patching PAC opcodes, we only care about the FDE records, and + * not the CIE, which carries the initial CFA directives but they only + * pertain to which register is the stack pointer. + * TODO this is not 100% true - we need the augmentation string and the + * encoding but they are always the same in practice. + */ + if (*(u32 *)eh_frame == 0) + return 0; + + fde = (const struct fde_frame *)(eh_frame + 4); + loc = (u64)offset_to_ptr(&fde->initial_loc); + opcode = (const u8 *)(fde + 1); + + // TODO check augmentation data + WARN_ON(*opcode++); + size -= sizeof(u32) + sizeof(*fde) + 1; + + /* + * Starting from 'loc', apply the CFA opcodes that advance the location + * pointer, and identify the locations of the PAC instructions. + */ + do { + switch (*opcode & 0xC0) { + case 0: + // handle DW_CFA_xxx opcodes + switch (*opcode) { + case DW_CFA_nop: + case DW_CFA_remember_state: + case DW_CFA_restore_state: + break; + + case DW_CFA_advance_loc1: + loc += *++opcode; + size--; + break; + + case DW_CFA_advance_loc2: + loc += *++opcode; + loc += *++opcode << 8; + size -= 2; + break; + + case DW_CFA_def_cfa: + opcode++; + size = skip_xleb128(&opcode, --size); + size = skip_xleb128(&opcode, size); + continue; + case DW_CFA_def_cfa_offset: + case DW_CFA_def_cfa_offset_sf: + case DW_CFA_def_cfa_register: + case DW_CFA_same_value: + opcode++; + size = skip_xleb128(&opcode, --size); + continue; + + case DW_CFA_negate_ra_state: + scs_patch_loc(loc - 4); + break; + + default: + pr_err("unhandled opcode: %02x in FDE frame %lx\n", *opcode, (uintptr_t)eh_frame); + return -ENOEXEC; + } + opcode++; + size--; + break; + + case 0x40: + // advance loc + loc += *opcode++ & 0x3f; + size--; + break; + + case 0x80: + opcode++; + size = skip_xleb128(&opcode, --size); + continue; + + default: + // ignore + opcode++; + size--; + break; + } + } while (size > 0); + + return 0; +} + +int noinstr scs_patch(const u8 eh_frame[], int size) +{ + const u8 *p = eh_frame; + + while (size > 4) { + const u32 *frame_size = (const u32 *)p; + int ret; + + if (*frame_size != -1 && *frame_size <= size) { + ret = scs_handle_frame(p + 4, *frame_size); + if (ret) + return ret; + p += 4 + *frame_size; + size -= 4 + *frame_size; + } + } + return 0; +} + +extern struct arm64_ftr_override id_aa64isar1_override; +extern struct arm64_ftr_override id_aa64isar2_override; +extern struct arm64_ftr_override id_aa64pfr1_override; + +bool __initdata should_disable_dynamic_scs = true; + +asmlinkage void __init scs_patch_vmlinux(void) +{ + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS roughly + * provide the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + u64 reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); + + reg &= ~id_aa64isar1_override.mask; + reg |= id_aa64isar1_override.val; + + if (reg & ((0xf << ID_AA64ISAR1_APA_SHIFT) | + (0xf << ID_AA64ISAR1_API_SHIFT))) + return; + + reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); + reg &= ~id_aa64isar2_override.mask; + reg |= id_aa64isar2_override.val; + + if (reg & (0xf << ID_AA64ISAR2_APA3_SHIFT)) + return; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { + reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); + reg &= ~id_aa64pfr1_override.mask; + reg |= id_aa64pfr1_override.val; + + if (reg & (0xf << ID_AA64PFR1_BT_SHIFT)) + return; + } + + WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); + should_disable_dynamic_scs = false; +} diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c index d20620a1c51a..30f3c7563694 100644 --- a/arch/arm64/kernel/sdei.c +++ b/arch/arm64/kernel/sdei.c @@ -144,7 +144,7 @@ static int init_sdei_scs(void) int cpu; int err = 0; - if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK)) + if (!scs_is_enabled()) return 0; for_each_possible_cpu(cpu) { diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 3505789cf4bd..17fad5749f4a 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -323,6 +325,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) jump_label_init(); parse_early_param(); + if (should_disable_dynamic_scs) + static_branch_disable(&dynamic_scs_enabled); + /* * Unmask asynchronous aborts and fiq after bringing up possible * earlycon. (Report possible System Errors once we can report this