From patchwork Fri Nov 24 10:19:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A0C6EC61DF4 for ; Fri, 24 Nov 2023 11:31:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=wDqd/gGDvMGtMO1/KD7n1W8okVzZ73kGrZ0PzlAcs8Q=; b=lhy0mHhaYNDlFeK1XoR/a0YFTd jvs40MKElZZ3wp7zUP3JJmYjadh9LMWSz0jS9c5hJwizs/Js574y+kzNCQ3S9eBJF7rlAp00qCzes xcADMeAsMf1QunWNgWOzmExtZQHRkTOvqS6kUQfL2f30Vaky/jgBy3bOxqWOajlbBb7SVFQF7cUXZ SGkenoE8sQtL5gOi50jwDkW3vGZBBhyhGR8ug7yz4nKOhuZoNC4aDOYU2R0pXPfX2XZoOiTkPAum0 KPOYKyK77x8PUOSqKzMqiUnYpJsYFhzK2xqp+fNJGJUZWmOSp3843ETNNzTSMB1iKwBRjg2bYAjpE T8je/m8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPa-006yY0-1O; Fri, 24 Nov 2023 11:31:54 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJA-006oyU-2T for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:26 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ae5b12227fso22942637b3.0 for ; Fri, 24 Nov 2023 02:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821271; x=1701426071; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=o6Yv2Ggi3pJQrrqFJo8tK8/jQXxtTFym4bIQ4XZUjZ1g38GuZlrZNp1TYWmQwceVYA jtKfvzLTmqlJk/sSQaN7xyVUTPtn8H1ABBJbmynekoYTpIrGR/QtlOdjRfFbCUAOQNyf z6tzDToGuIX8F481djhmUYEDHNetuXHN6MDcck4q4pVzZwhDbn+ir12bHlAIor7BstpB Q6HpqKMv8OFvt2TEw7T2FvkNC23/gZ7v77CO19/Dq2YevHMubXkZvOGKy6LFEu71yC5V fJdi8JI0u1qxWSTU8srBIHrGDWSDNNkfmhhzYq4gAqEshNQX/mNzploXuwJ+bJH4Ue9q IdtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821271; x=1701426071; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=Ggoqse3o8TXCkPsKUZ7bhj4AE/0ocxgGoNXp9YV/cNw5VEhjDZWiNzPEWVJmxc3rRT gEBti8GuGPNsMFR5NiguNuIVdcLwfnTH8MKzFUdAsRfDlCMhkSyqmO9j/xBQF+K091Mg 02ler9Vn1k4aL1Bv0wAsQiuU/07AKlStnW2fZPH4X1R/UnOUF50oxcbS6qVYyFEK2DwU B/8BZVtoKGQ9jc/4YF/h3eDjUaivCr7oZEUr5azXj2CuMdeCB7snvsi5l1GZJo6VcitG l+geEYjrRBBvbYX1D8kBubmFk0Nw5xKgZ8v/ZmCSS/cFtuPOFN2OBrIJ1UkICC4pTCD2 JTVg== X-Gm-Message-State: AOJu0YwVvg3K97P+vtXM1kmrPs4UpO1OGtjn8iey/nPo8fWxC0SRppQ1 qnk3u7FGXIYQ3ZfSSA9cbhMO8ChyFztG5tLPXhEmPns+VAFLDZlggpcAK64xA/laLEVzzxe0lh9 VK08Y1Cfn42TDYYhFD/kJ/RMqXrLRFTcNmValfWZMOtL3aHuDKLRcMn3MHyTGxTp3uqJV+6JN9D w= X-Google-Smtp-Source: AGHT+IEoLpNDf0BTqWnPT+rYGd7qhqdp5E2zt1lZFNdHKH7/lL+JBG/dk2nMXQUTWWTPTEGBI9hz9QhN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:3603:b0:5ca:e7e5:e8b9 with SMTP id ft3-20020a05690c360300b005cae7e5e8b9mr79755ywb.9.1700821270986; Fri, 24 Nov 2023 02:21:10 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:05 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6307; i=ardb@kernel.org; h=from:subject; bh=+CAandW2REwobvpWTcepqeb8aChD0Vxr0cE2KhPU7EE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYons496u55PFf7k8tMx51LO26k/wDOa8/WcEVSbl8 O7XvirUUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYiw8fIMOsIQ8vynw35xkwX /9jlvb3zu/jLj7ypFYyt4btufd/YJMTI8HwaQ1fGzqPJ1TMeqPlNOXvJQEfgbfLKqj7b52nrUid +5gEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-66-ardb@google.com> Subject: [PATCH v5 25/39] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022112_851481_C827DD58 X-CRM114-Status: GOOD ( 18.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index dc85dc2ee4ed..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); }