From patchwork Fri Nov 5 14:59:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12604841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C6A2C433F5 for ; Fri, 5 Nov 2021 15:01:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C80F16120E for ; Fri, 5 Nov 2021 15:01:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C80F16120E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=c2w5BMyxdDwNWr7w59hzvU2lgrFqajKAIWx2rdS1ul4=; b=2e17UOjDZckD1m nB/qx60slNkFJRow3R3dRMeX8J1s+CPkmQ81L8eyHCkGLb635lBtF+1hJpN8Wh8/5x2FNiye7aswV Ah3fk/odOzT72T7iABQK8MtzYp9MEVSdjOVn5zdI2P5qwhd96PWVkNtTbxZlFa0p4gDZS3h/fjZor HJr642mx9ESveAhs5dxgSDNui0stXm7j/Vf2K7/K6BIuhYy+wXF2AFRD+GxD0TkFVcuamAZ4bP+RG SPAQwmfHMwSwE1/odf3PvoT8TODGzwhrsoKDaQSKIASOb5cyrcMvnKl63ADo5TeNVaTLLUrVvBL8F U+UJIwUfA1XD+3HiQmeQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mj0h4-00BdMd-43; Fri, 05 Nov 2021 14:59:50 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mj0gl-00BdIB-Qa for linux-arm-kernel@lists.infradead.org; Fri, 05 Nov 2021 14:59:33 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 65A7861252; Fri, 5 Nov 2021 14:59:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1636124371; bh=B1qFtiscq4n7antXqmGoMknyo6BxXwdLVygSWI2bsPA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s2i0kvGJwMJI50mg04XRW/RZ2q+d7nTMRFXtP0YDkyL8fk5umOuCqDN/R8YfAXfXe RRK4AFgigwgNA9m5HavDbb8OxMrkkpsKSeIj1JTXkfC7GX4dXkiTfx0rynRSWfVjaR 0qTWBFivlP/8B14R28K3hGYsbR6CleRXgNUTfM+SwRMeqItCDycOsMP4FAZKGRFgSu NKuGXleCe40BvlyiWM/Gsd4V+d0BHJ5HMYfxn/T42s7Sd0J7lg4zuAq/aDnvniG9Jw mrX2FAyAIBnN2A+lsCRuZMilDKOvk+xEK5vUSZ9Mit426PuAdmZKt3AxitUIInyw2w 91/WejVowEngQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Mark Rutland , Quentin Perret , Catalin Marinas , James Morse , Will Deacon , Frederic Weisbecker , Peter Zijlstra , Kees Cook , Sami Tolvanen , Andy Lutomirski , Josh Poimboeuf , Steven Rostedt Subject: [PATCH v6 2/2] arm64: implement support for static call trampolines Date: Fri, 5 Nov 2021 15:59:17 +0100 Message-Id: <20211105145917.2828911-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211105145917.2828911-1-ardb@kernel.org> References: <20211105145917.2828911-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7392; h=from:subject; bh=B1qFtiscq4n7antXqmGoMknyo6BxXwdLVygSWI2bsPA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhhUbEdLzHjvYTKAjHfvxW2Q8AWCuqtPm00YUzg9Xc M0InLJ6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYYVGxAAKCRDDTyI5ktmPJL9qC/ 9hfM5WTFRR7SIU8NED63TvgTH5X6a3gdqfir8dnDBIuqOkZEywbvzTx6X0BzmmPFBVUMIup5uyciJP Ap+bPTBYYhDrBlar9kbpBVHCeDbCVoFihQRdl1sLA6XamVuIAksSQkZkwoMA1nYepQx2WzxOt1YfoU Dkn3R09Nnurpsw50cl5peElR0JBBUyIE1rt/hI/6Sl7kHxTXG00CiZ6rDYOOQfvg7DD+sSb9ADDhwt fdNvPHZscFCLgtuC6zC56PoOmwMGS8X21M1ZQzGEc36FqSZ9e+7wfRlMc81EpasJ6YBgEROzLUQtwl +1fFsCOlbW72S6GiUQavidl+czV9EuhbQkIKW/IwLseLBmhAUjyBl8J7DsrrbI5ITYB4avZnGwFsBO u15o063EPjYdDt0JLZylc+36L03XkYDcoH4w523Ocp1MraqeqYsO+cW9eQqsqfYdPjRp2N6Umsh8L+ AUDWVkglZJbmJ4aPQqtmVAbzDDu1JApKxKAYlg7waVbBU= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211105_075931_971029_1552E42F X-CRM114-Status: GOOD ( 29.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function. It is expected that the direct branch instruction will be able to cover the common case. However, given that static call targets may be located in modules loaded out of direct branching range, we need a fallback path that loads the address into R16 and uses a branch-to-register (BR) instruction to perform an indirect call. Unlike on x86, there is no pressing need on arm64 to avoid indirect calls at all cost, but hiding it from the compiler as is done here does have some benefits: - the literal is located in .text, which gives us the same robustness advantage that code patching does; - no performance hit on CFI enabled Clang builds that decorate compiler emitted indirect calls with branch target validity checks. Acked-by: Peter Zijlstra Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 2 + arch/arm64/include/asm/static_call.h | 40 ++++++++++ arch/arm64/kernel/patching.c | 77 +++++++++++++++++++- arch/arm64/kernel/vmlinux.lds.S | 1 + 4 files changed, 117 insertions(+), 3 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 176d6fddc4f2..ccc33b85769c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -193,6 +193,8 @@ config ARM64 select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_POSIX_CPU_TIMERS_TASK_WORK + # https://github.com/ClangBuiltLinux/linux/issues/1354 + select HAVE_STATIC_CALL if !LTO_CLANG_THIN || CLANG_VERSION >= 130000 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUTEX_CMPXCHG if FUTEX select MMU_GATHER_RCU_TABLE_FREE diff --git a/arch/arm64/include/asm/static_call.h b/arch/arm64/include/asm/static_call.h new file mode 100644 index 000000000000..6ee918991510 --- /dev/null +++ b/arch/arm64/include/asm/static_call.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_STATIC_CALL_H +#define _ASM_STATIC_CALL_H + +/* + * The sequence below is laid out in a way that guarantees that the literal and + * the instruction are always covered by the same cacheline, and can be updated + * using a single store-pair instruction (provided that we rewrite the BTI C + * instruction as well). This means the literal and the instruction are always + * in sync when observed via the D-side. + * + * However, this does not guarantee that the I-side will catch up immediately + * as well: until the I-cache maintenance completes, CPUs may branch to the old + * target, or execute a stale NOP or RET. We deal with this by writing the + * literal unconditionally, even if it is 0x0 or the branch is in range. That + * way, a stale NOP will fall through and call the new target via an indirect + * call. Stale RETs or Bs will be taken as before, and branch to the old + * target. + */ +#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insn) \ + asm(" .pushsection .static_call.text, \"ax\" \n" \ + " .align 4 \n" \ + " .globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + "0: .quad 0x0 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " hint 34 /* BTI C */ \n" \ + insn " \n" \ + " ldr x16, 0b \n" \ + " cbz x16, 1f \n" \ + " br x16 \n" \ + "1: ret \n" \ + " .popsection \n") + +#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "b " #func) + +#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret") + +#endif /* _ASM_STATIC_CALL_H */ diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c index 771f543464e0..a265a87d4d9e 100644 --- a/arch/arm64/kernel/patching.c +++ b/arch/arm64/kernel/patching.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -66,7 +67,7 @@ int __kprobes aarch64_insn_read(void *addr, u32 *insnp) return ret; } -static int __kprobes __aarch64_insn_write(void *addr, __le32 insn) +static int __kprobes __aarch64_insn_write(void *addr, void *insn, int size) { void *waddr = addr; unsigned long flags = 0; @@ -75,7 +76,7 @@ static int __kprobes __aarch64_insn_write(void *addr, __le32 insn) raw_spin_lock_irqsave(&patch_lock, flags); waddr = patch_map(addr, FIX_TEXT_POKE0); - ret = copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE); + ret = copy_to_kernel_nofault(waddr, insn, size); patch_unmap(FIX_TEXT_POKE0); raw_spin_unlock_irqrestore(&patch_lock, flags); @@ -85,7 +86,77 @@ static int __kprobes __aarch64_insn_write(void *addr, __le32 insn) int __kprobes aarch64_insn_write(void *addr, u32 insn) { - return __aarch64_insn_write(addr, cpu_to_le32(insn)); + __le32 i = cpu_to_le32(insn); + + return __aarch64_insn_write(addr, &i, AARCH64_INSN_SIZE); +} + +static void *strip_cfi_jt(void *addr) +{ + if (IS_ENABLED(CONFIG_CFI_CLANG)) { + void *p = addr; + u32 insn; + + /* + * Taking the address of a function produces the address of the + * jump table entry when Clang CFI is enabled. Such entries are + * ordinary jump instructions, preceded by a BTI C instruction + * if BTI is enabled for the kernel. + */ + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) + p += 4; + + insn = le32_to_cpup(p); + if (aarch64_insn_is_b(insn)) + return p + aarch64_get_branch_offset(insn); + + WARN_ON(1); + } + return addr; +} + +void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) +{ + /* + * -0x8 + * 0x0 bti c <--- trampoline entry point + * 0x4 + * 0x8 ldr x16, + * 0xc cbz x16, 20 + * 0x10 br x16 + * 0x14 ret + */ + struct { + u64 literal; + __le32 insn[2]; + } insns; + u32 insn; + int ret; + + insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_BTIC); + insns.literal = (u64)func; + insns.insn[0] = cpu_to_le32(insn); + + if (!func) { + insn = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_LR, + AARCH64_INSN_BRANCH_RETURN); + } else { + insn = aarch64_insn_gen_branch_imm((u64)tramp + 4, + (u64)strip_cfi_jt(func), + AARCH64_INSN_BRANCH_NOLINK); + + /* + * Use a NOP if the branch target is out of range, and rely on + * the indirect call instead. + */ + if (insn == AARCH64_BREAK_FAULT) + insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_NOP); + } + insns.insn[1] = cpu_to_le32(insn); + + ret = __aarch64_insn_write(tramp - 8, &insns, sizeof(insns)); + if (!WARN_ON(ret)) + caches_clean_inval_pou((u64)tramp - 8, sizeof(insns)); } int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 50bab186c49b..e16860a14eaf 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -173,6 +173,7 @@ SECTIONS HIBERNATE_TEXT KEXEC_TEXT TRAMP_TEXT + STATIC_CALL_TEXT *(.gnu.warning) . = ALIGN(16); *(.got) /* Global offset table */