From patchwork Fri Nov 11 17:11:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5661C433FE for ; Fri, 11 Nov 2022 17:29:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HoJD/+UFSv7jSMA6dUhw0rQVI2QOaAPRw/9QmpVXqiw=; b=N/962YP3+40l3y 6Q85UfhhMiCds2+nSNy9iFtLb/eRBwA3/pV+DR7BsSixxLWnB4356k0tbmXTQGDQq0ihAoF/DWx38 gYKUntv7K1EImNpbIXAaugu6HjcIEFU5gZ/4oxcGxyH54jC/MDutoNlPiTegvEeDRl7/bNZg7jpqs 7cgaTt4apCa8sNwN2fE+PY6i0KZlhgmg7neYgnNlJ6VbgsdSTZ45WFSMIm5QO2DDhfiOJ6knkLj5E bjdAlgaZlZagBNWUbvrF/VvYv8u3shG49vA7o52J7WTNoO3kmD5r5dY+ar2l2DJKtWdNhgGDYlIsm kNmFRrA0sNI9WnEiBgbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXou-00HFos-AC; Fri, 11 Nov 2022 17:28:01 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaZ-00H7wv-Jc for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:14 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4E663B82680; Fri, 11 Nov 2022 17:13:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 226FFC433D6; Fri, 11 Nov 2022 17:13:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186789; bh=79UT6hrkKbxqKInsN7QsQMwompw/Ew3GtJHjIs1jLXU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=robjUWvcvm2msA2fwxvQGQuv8ztBuVy0B4Fh03OD81lhhFl64GtSFvkpV+u8zQua1 4R2JrM66Rnmg8M0WMeJ7Cwd17REfEo1xaq6T7Ga41jVc7+R5B6MjPdBiDT0gC5dW5K P3gMBmPhIwO876M63jAB+iVt/RWRddskpBXDlAtzzdkaK72FKe1mOV/CFZ+GL3zGUJ 4jhAFq4bhtoWbSF6a3tnzoyZk1BxZ6Dp7erGnQbMBtdIbZid8mmfX+VC9XSWypwd6z xiB+T7eHoaWgIQiynawTAyzZ1xl6vQsomihuFc1W4/W8MyXTa4HzQMqxLjYa56A3eY uA+BUUBWqGfqQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 22/33] arm64: head: move dynamic shadow call stack patching into early C runtime Date: Fri, 11 Nov 2022 18:11:50 +0100 Message-Id: <20221111171201.2088501-23-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6196; i=ardb@kernel.org; h=from:subject; bh=79UT6hrkKbxqKInsN7QsQMwompw/Ew3GtJHjIs1jLXU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJN9O5RJSizh1MbM4rjnjacaPyD2X6qu3TAZnBu pqUNAZCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CTQAKCRDDTyI5ktmPJNT4DA CCp5fSQ7DvQq1OdZetq/pegK+l3TJm25nV4Ddmfl9qSu1q8DvvCcpzMyrrOuXsl8DTobP4Ie82o2r6 RWbPY9LyzAeM5c8S7Ypg94DIB/fg6VdDRJvugTqPOyFXEBc9ePq/wnMRQJwyIJIHjIZ3dIlX8HNTjL FSfskBEvF6z7WeERcQZQPGiMSFiRyVrMugwxyO/GGrPoAZ4qPZYKaXfsKaa76dwYDW2vqrWhwJJuW9 x1UwMvY7D3Ts+VXAsdpjfw9ltSbvwCxovAn9zTq77GthbDa/q/OWb9zC+NKzQ8cwoxD++RvGFS6J4o C7Kap/Lv8D7x2JrpavDzT5pjYYsNFuf1PDVdqhqAw0d6o3QHS0qlvKCHYy8qMb/iVRRQnaZvy7WSpf bYMcyckLYp3JfN0ev/5dS8LICfhJXuiKhk3RVBXpGEMiG7LRhwZb8MFWLA4KE2Ieh0dbixKXoDKil6 fn4/fXx0F4QUlQAE7GkRTVT6mOlbWkKfhZzmjDriyIBUI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091311_988697_4B92749F X-CRM114-Status: GOOD ( 18.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 2 +- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 ++++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 ++++---- arch/arm64/kernel/{ => pi}/patch-scs.c | 24 ++++++++++---------- 6 files changed, 25 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index ff7da1268a52ab79..d20dcb2a1a5284ce 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -71,7 +71,7 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); +int __pi_scs_patch(const u8 eh_frame[], int size); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 2b003834c320c20e..35354365f5f39f3f 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -79,8 +79,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 9ea7f4e355ef5849..5f1476c0f3a33d75 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -438,9 +438,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -720,6 +717,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index fa7b3228944b325e..ea505022737d531d 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -519,7 +519,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 47d3ffcff3ac9a03..293a04a5dc3ef516 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index 1b3da02d5b741bc3..d15833df10d3d4c6 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,9 +129,9 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -196,14 +197,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -246,12 +246,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); }