From patchwork Mon Oct 19 14:12:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 11844453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE1CEC433E7 for ; Mon, 19 Oct 2020 14:14:32 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 64D362177B for ; Mon, 19 Oct 2020 14:14:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hDMXFfQJ"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="iqQ7FRVA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 64D362177B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Owner; bh=BR4ha+D19oGYF9kL0m3DMeeRFvjlVwcpBn6YuRhWBCk=; b=hDMXFfQJTcP+d9zEzfr+gaqzfX pM7a/7uxx2gdqsf7pwfpVjanK46AL/CFLYV5dgMg9oiqs5zNF01gW3ajs6FhH80PX/g7zJf616Hy5 X3MF/ek6VbmFPgHYNDddmedIFFLSlDXtHOQgEjwmruFBICCVHXeCgKGHP22LFimhjU/q5ENfKzqMS B0JtbWEfibWHRfsDYaEE5Jw37SMCIxmVxzsLQlVqPTvhZQU7ePmdqcA4s0zZu5Evbe6dsCnjM84Gs FYcsnTOxfE5mEiwez9opYNUi9StI8+av4RBqWR2kpVwnes+DZuy4vRqPMXM+ekh9zz6Pir/3ehubn B9bfckow==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUVuN-0006em-Az; Mon, 19 Oct 2020 14:13:07 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUVuH-0006d9-Hp for linux-arm-kernel@lists.infradead.org; Mon, 19 Oct 2020 14:13:03 +0000 Received: from e123331-lin.nice.arm.com (lfbn-nic-1-188-42.w2-15.abo.wanadoo.fr [2.15.37.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 66B262177B; Mon, 19 Oct 2020 14:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603116780; bh=MNvtudC6Kc6ts/aHIIW1OwAOppP5jOH+6blOwtz2sVc=; h=From:To:Cc:Subject:Date:From; b=iqQ7FRVAab2G2UIqPPMDsoIuUPNKCJilAGX2VhBAnwaPpOIoZgBDkqe+EkwqlK7ai FAej8MZgzv8Z3F+5cB24FKTbU/JCTeEMNotcSOIpNYC2pwqDz/OwMl060/kp0a5zP2 VPm54lUglV0jNcCP23ZCl+p5pokiZJEW/USXEtvs= From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Subject: [PATCH] arm64: implement support for static call trampolines Date: Mon, 19 Oct 2020 16:12:47 +0200 Message-Id: <20201019141247.25122-1-ardb@kernel.org> X-Mailer: git-send-email 2.17.1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201019_101301_770789_165CC0DC X-CRM114-Status: GOOD ( 26.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, Peter Zijlstra , catalin.marinas@arm.com, broonie@kernel.org, will@kernel.org, Ard Biesheuvel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function. Since static call targets may be located in modules loaded out of direct branching range, we need to be able to fall back to issuing a ADRP/ADD pair to load the branch target into R16 and use a BR instruction. As this involves patching more than a single B or NOP instruction (for which the architecture makes special provisions in terms of the synchronization needed), we should take care to only use aarch64_insn_patch_text_nosync() if the subsequent instruction is still a 'RET' (which guarantees that the one being patched is a B or a NOP) Cc: Peter Zijlstra (Intel) Signed-off-by: Ard Biesheuvel --- Note that I needed the changes after the --8<-- below to enable the selftest, since it assumes HAVE_STATIC_CALL_INLINE arch/arm64/Kconfig | 1 + arch/arm64/include/asm/static_call.h | 26 ++++++++++ arch/arm64/kernel/Makefile | 2 +- arch/arm64/kernel/static_call.c | 52 ++++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 1 + 5 files changed, 81 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 96633b174e39..7841026694e8 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -172,6 +172,7 @@ config ARM64 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_STATIC_CALL select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUTEX_CMPXCHG if FUTEX select MMU_GATHER_RCU_TABLE_FREE diff --git a/arch/arm64/include/asm/static_call.h b/arch/arm64/include/asm/static_call.h new file mode 100644 index 000000000000..d70e6495768a --- /dev/null +++ b/arch/arm64/include/asm/static_call.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_STATIC_CALL_H +#define _ASM_STATIC_CALL_H + +#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insns) \ + asm(".pushsection .static_call.text, \"ax\" \n" \ + ".align 2 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + "hint 34; /* BTI C */ \n" \ + insns " \n" \ + ".popsection \n") + +/* + * We have to account for the possibility that the static call site may + * be updated to refer to a target that is out of range for an ordinary + * 'B' branch instruction, and so we need to pre-allocate some space for + * a ADRP/ADD/BR sequence. + */ +#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "b " #func "; ret; br x16") + +#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "nop; ret; br x16") + +#endif /* _ASM_STATIC_CALL_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index bbaf0bc4ad60..f579800eb860 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -17,7 +17,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o + syscall.o proton-pack.o static_call.o targets += efi-entry.o diff --git a/arch/arm64/kernel/static_call.c b/arch/arm64/kernel/static_call.c new file mode 100644 index 000000000000..e6e42b52aea4 --- /dev/null +++ b/arch/arm64/kernel/static_call.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +/* + * The static call trampoline consists of one of the following sequences: + * + * (A) (B) (C) + * 0x0: BTI C BTI C BTI C + * 0x4: B NOP ADRP X16, + * 0x8: RET RET ADD X16, X16, :lo12: + * 0xc: BR X16 BR X16 BR X16 + * + * The architecture permits us to patch B instructions into NOPs or vice versa + * directly, but patching a multi-instruction sequence requires careful + * synchronization. + */ +void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) +{ + unsigned long pc = (unsigned long)tramp + 4; + unsigned long dst = (unsigned long)func; + void *addrs[] = { (void *)pc, (void *)(pc + 4) }; + u32 ret = le32_to_cpup(addrs[1]); + u32 insn[2]; + + insn[0] = func ? aarch64_insn_gen_branch_imm(pc, dst, + AARCH64_INSN_BRANCH_NOLINK) + : AARCH64_INSN_HINT_NOP; + insn[1] = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_30, + AARCH64_INSN_BRANCH_RETURN); + + if (ret == insn[1] && insn[0] != AARCH64_BREAK_FAULT) { + aarch64_insn_patch_text_nosync(addrs[0], insn[0]); + return; + } + + if (insn[0] == AARCH64_BREAK_FAULT) { + insn[0] = aarch64_insn_gen_adr(pc, dst, AARCH64_INSN_REG_16, + AARCH64_INSN_ADR_TYPE_ADRP); + insn[1] = aarch64_insn_gen_add_sub_imm(AARCH64_INSN_REG_16, + AARCH64_INSN_REG_16, + dst % SZ_4K, + AARCH64_INSN_VARIANT_64BIT, + AARCH64_INSN_ADSB_ADD); + + BUG_ON(insn[0] == AARCH64_BREAK_FAULT); + } + aarch64_insn_patch_text(addrs, insn, ARRAY_SIZE(insn)); +} +EXPORT_SYMBOL_GPL(arch_static_call_transform); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 5ca957e656ab..77f9f23934db 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -123,6 +123,7 @@ SECTIONS IDMAP_TEXT HIBERNATE_TEXT TRAMP_TEXT + STATIC_CALL_TEXT *(.fixup) *(.gnu.warning) . = ALIGN(16); -- 2.17.1 ---------------8<--------------------- diff --git a/kernel/Makefile b/kernel/Makefile index b74820d8b264..3d7fec961b40 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -111,7 +111,7 @@ obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ obj-$(CONFIG_KCSAN) += kcsan/ obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o -obj-$(CONFIG_HAVE_STATIC_CALL_INLINE) += static_call.o +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/static_call.c b/kernel/static_call.c index 84565c2a41b8..efee9f72dc06 100644 --- a/kernel/static_call.c +++ b/kernel/static_call.c @@ -10,6 +10,8 @@ #include #include +#ifdef CONFIG_STATIC_CALL_INLINE + extern struct static_call_site __start_static_call_sites[], __stop_static_call_sites[]; @@ -437,6 +439,7 @@ int __init static_call_init(void) return 0; } early_initcall(static_call_init); +#endif #ifdef CONFIG_STATIC_CALL_SELFTEST