From patchwork Tue Sep 12 14:16:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9077CA0EED for ; Tue, 12 Sep 2023 14:19:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=g2Dt+nWaKxl+Bi2TqB2OzuFK9dTH4Q9NoEC+bizmy8Q=; b=xm6RpmM/cFkpWpzg2oeF+d8PD8 mxBgDZdSv3YPHkqNL+EH1LYbfeBWj/WZf9fjbeNjeSVeyXL29R+nVAQFBgQnsGgpMrIORWylfbdwU K00P55oH6xyzyFlWq8AswhJPV60Xlv7tbtUgVn7JzzybB4iTPf2cDC7s6He3PrEZpcvyuEpHpkctw F6qFMOorhRlYrWQ25L7LzynRYKZelN80p1jQB7+naT8Rh1PDCGWY0o9ReT4mHJPlc303zSyx9LtGp ictrLyTDVEAEADeX8pA0XX9Uavmu+RPLFdGPGCTSP+Y8giwmvYF5D6G3MzUDarxjK1JvxIK+jMsmG QYTrWwSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E9-003X0f-1c; Tue, 12 Sep 2023 14:18:53 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DZ-003WV2-0v for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:30 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d80db590b1cso3198565276.0 for ; Tue, 12 Sep 2023 07:18:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528295; x=1695133095; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MJYd3lTCvsjOLSBORSzk2v4/+nDoBR/U9Sn4XhI4spM=; b=AgFj6uIQocpUNc+vBi/NrCzkdBYfqOIQaFxRlSMRlo78sO6xsf+L5sIE1LGJJyFUZ9 N0oWWuQlnw7l2eYULF/1ciPGlKgpJ9dOctSItJopoeuii6UC+PHxEwShSIVvhErFYYrF 5M1ZBlifBPkQXIBHpoppdi8/KQuI4bcR28X+Sx7kb33rMHIQjKMp5B1w5APYy5C4FTvo 9um7U0PqKTF1I1T4U3tpuSScDSTd3D1az1+8oP132BqbIIwwVcQPM/4MdJBmm9ZojJ94 v7JHJLnMiOzK2azv8yXjGQknLTllunxtMnwETTFhlYk4FxYyFUjRUaz1qEo+fny81vrx 2wFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528295; x=1695133095; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MJYd3lTCvsjOLSBORSzk2v4/+nDoBR/U9Sn4XhI4spM=; b=c2yMKd4ZNmOWU8oqsqePB5oUGpnkxVLUO76sHH8iwHFCL+lAN3+CyAdRlVdTjFH8b+ kKz24sB8IWNoGEUU3JvzSFOhHaUYgu2fZWlfvJf4riHg0tKOhpmGv3pdVBpUuFlsJhan iwOlNv6ooWs/kOePfUZkt3I1UO9GASM07Yp9R6WTPn14i3dv3QbvJfV7hsy6L7kHoHbu vD1St8VPjds6QDeKUKi23XNNc7YnsAJf0V777kOW/2gNd6VfXTY383XlemBkrsax1rdi QDNtRxZ0kF7UP2c4myxPRecLCJjcH43hnD7ZnwJvX0agDvaKU7M5BCJlV2tI7WpOLJYw 413Q== X-Gm-Message-State: AOJu0YzsTXl9lNU5c04TKchRtFOnx6yyRmaocoMJPg9QSiCJRxhex+b5 PTX+SMSLfxYxVxNw+8nmtVLOnrHps7T+ZHqS5qI7TLxG4wZIEEy2D9dKhOF56GMBmV6WueOAwDH /y2Ulwt9xtmzV7KPOZSFAI0APYmTznECvAJMfEelnc5WFXqKWRqpmENS3MVe32xj671k0Ga7gql A= X-Google-Smtp-Source: AGHT+IFJM+NP3uZBHi/VU0uc2xx/SdEc5SUci3GVYb3SZgPn/X+1aPrQKoYOMNBMiCFUOCMVibIkPZ8/ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:fb08:0:b0:d77:f789:f9da with SMTP id j8-20020a25fb08000000b00d77f789f9damr78300ybe.6.1694528295365; Tue, 12 Sep 2023 07:18:15 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:15 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6303; i=ardb@kernel.org; h=from:subject; bh=g5GBUcsqZAcIXTHO1ibxUVqxqf2uvMSyG9Or42q3JwA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaINswsLeQ3tWntzCNv//YubG9DP9hzVvFpWH/wi0X bBWbE9ERykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhI9G2G/27HqrV2aPdMPFHD dm7nPbWiIwlebB6z/0z/vkRk2vstitlAFVG73Pj5ljfM6cxTmSuu5R6/WI9R31pDPtzP8ct3n7U cAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-88-ardb@google.com> Subject: [PATCH v4 25/61] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071817_366136_789BB4E3 X-CRM114-Status: GOOD ( 19.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index dc85dc2ee4ed..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); }