From patchwork Tue Feb 27 15:11:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13574018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1106DC5478C for ; Tue, 27 Feb 2024 15:12:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wLV3L9vo8KBQ+Q6WHhidn80kqRVHF44hzc1H2Av2vuI=; b=XzTyiRQeRKuvtR YpOjotQzDV/y7YPXBdMLut+PIMRyxIKJ6A166Q4nJHh4HXh2ty2tc7yn9YhwJ8JV1fybxlhU/VY0h 0LlPn7uuMq+XEWNiqei5/P8wmhXRSqLm0BrPlEZ7k/fEPC9PvgIDtsrnpkuZzrA7wQyWliCZ7rfGH sAI2XyZ1DowoupqEzXi0hH3pjJxPnIR3SxAsC9VtH05+EyYlfOXhH9EkobbMPZGsYLlzVoWZ9gOH4 l1Q/UujZIBO5SVpDStEjtxrpxMocmAxYdtQtzggoKoHGnL4xIaze2+EL1bgfrhs4IT9bIBnrNjYYw +T1Gj33LCFlSJfu3rQRg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rez7c-00000005jKg-1xyi; Tue, 27 Feb 2024 15:11:56 +0000 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rez7T-00000005jG1-1jUo for linux-arm-kernel@lists.infradead.org; Tue, 27 Feb 2024 15:11:50 +0000 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-3392b045e0aso3627011f8f.2 for ; Tue, 27 Feb 2024 07:11:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709046705; x=1709651505; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YgUZYYEewWOr0eZgCM/D+yRszpCU2Zn6RVKpSWiScZI=; b=KdfOc7hh7tW6Emy2PoJ5fhAkcfzptvRDGQoGYorkyEfr/YxySCPGpEMJr6g/EhcUSq XtJEfWZ5hUx0Tuql2NZtbg9PRiKN8BI9sMgU/5dc32twUVLGOpgm1Ngk78S9G2JjKX1F 500Um7v34MtIguzjLYLHACJywQGC41c8HKukSruQHTXlLSHkS3v0UWlzebecwrFt9uGT aViWVburqwlW1WgzP0sysLxjPPyK74y+fuhco8LbZ9PZkI5TXg0joZCD5I3u548NKuGt UfOSTEjbwgB2ciuuAwmC5j9qggABDTNpV4ZaimozP9JkMsArB4J94yLRu4u/TM5ZqL2F 0NBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709046705; x=1709651505; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YgUZYYEewWOr0eZgCM/D+yRszpCU2Zn6RVKpSWiScZI=; b=UP2OK2nDh+MY0EeBJjWSvDON94Uzv/XSPMonRG1/Q9qgCkVGvJOdp8+YjI8kAiRz+A a2Y6fD0Q1SoGIfavlRCwzQJwMyJiDXeG4Iw64RC8TYl7pvGzOrP5n7iNc4k23O0mCqJG SVvr17lUSlUu9ikezep6teBXAbktzsSO1Ilpz8XWYFkcHq0avSki0tcB8Acrcdhrkpcx vCF63e3FEWnjCSzt5jUZS83PNQZYblwmmRJ3psm8X3dcqDXIAMUjOTBE72jFqY7u2OIU hcclzcSUnt2GFe9NrOdraUkBR09tH//72HktV6OY8J4wBQ17N5faGLF76o5xcOkmvtov 60jA== X-Forwarded-Encrypted: i=1; AJvYcCV04P+wwWhJSx0CAjNTXLzu5QWwExwC8Ig6dEYXKA78fUV+xnlSK53lAI7g1Tz/f6OpwiI416jTasfoM4iRarDIhuUfl5DwvGnNOnUwryJM/zfx9pw= X-Gm-Message-State: AOJu0YwBy0m/RJFjnSdKj17SL+N3I4YDKyvBHbKeIGUGeXZ8+UDbukxo vZUTjoIMSTDvLNUJSSs/7NtK0D9YUsx0WaYjk/cTSZiGZ6cACUwV X-Google-Smtp-Source: AGHT+IGLirqfah9d4iGWTMCqewDLhnmBg8hi2Iy2nBqKNPhbA3EjAa6SiYGrHFnWv/yqvMG7UwDjxA== X-Received: by 2002:a5d:58cd:0:b0:33d:b872:1c1b with SMTP id o13-20020a5d58cd000000b0033db8721c1bmr7385349wrf.23.1709046704756; Tue, 27 Feb 2024 07:11:44 -0800 (PST) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id bq7-20020a5d5a07000000b0033cddadde6esm11917259wrb.80.2024.02.27.07.11.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Feb 2024 07:11:44 -0800 (PST) From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Mark Rutland , Suzuki K Poulose , Mark Brown , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), Josh Poimboeuf Cc: puranjay12@gmail.com Subject: [PATCH bpf-next 1/1] arm64/cfi,bpf: Support kCFI + BPF on arm64 Date: Tue, 27 Feb 2024 15:11:15 +0000 Message-Id: <20240227151115.4623-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240227151115.4623-1-puranjay12@gmail.com> References: <20240227151115.4623-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240227_071147_507822_3359B58F X-CRM114-Status: GOOD ( 26.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, bpf_dispatcher_*_func() is marked with `__nocfi` therefore calling BPF programs from this interface doesn't cause CFI warnings. When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated. Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared. Signed-off-by: Puranjay Mohan --- arch/arm64/include/asm/cfi.h | 23 ++++++++++++++ arch/arm64/kernel/alternative.c | 54 +++++++++++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 26 ++++++++++++---- 3 files changed, 97 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h diff --git a/arch/arm64/include/asm/cfi.h b/arch/arm64/include/asm/cfi.h new file mode 100644 index 000000000000..670e191f8628 --- /dev/null +++ b/arch/arm64/include/asm/cfi.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#ifdef CONFIG_CFI_CLANG +#define __bpfcall +static inline int cfi_get_offset(void) +{ + return 4; +} +#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; +extern u32 cfi_get_func_hash(void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash(void *func) +{ + return 0; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */ diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 8ff6610af496..350057a28abe 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -298,3 +299,56 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, updptr[i] = cpu_to_le32(aarch64_insn_gen_nop()); } EXPORT_SYMBOL(alt_cb_patch_nops); + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX(const void *ctx, + const struct bpf_insn *insn); + +/* + * Force a reference to the external symbol so the compiler generates + * __kcfi_typid. + */ +__ADDRESSABLE(__bpf_prog_runX); + +/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_hash,@object \n" +" .globl cfi_bpf_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_hash: \n" +" .long __kcfi_typeid___bpf_prog_runX \n" +" .size cfi_bpf_hash, 4 \n" +" .popsection \n" +); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); + +__ADDRESSABLE(__bpf_callback_fn); + +/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_subprog_hash,@object \n" +" .globl cfi_bpf_subprog_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_subprog_hash: \n" +" .word __kcfi_typeid___bpf_callback_fn \n" +" .size cfi_bpf_subprog_hash, 4 \n" +" .popsection \n" +); + +u32 cfi_get_func_hash(void *func) +{ + u32 hash; + + if (get_kernel_nofault(hash, func - cfi_get_offset())) + return 0; + + return hash; +} +#endif diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index cfd5434de483..fb02862e1a3a 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx) emit(insn, ctx); } +static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx) +{ + if (IS_ENABLED(CONFIG_CFI_CLANG)) + emit(hash, ctx); +} + /* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address @@ -285,7 +292,7 @@ static bool is_lsi_offset(int offset, int scale) /* Tail call offset to jump into */ #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) -static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) +static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_subprog) { const struct bpf_prog *prog = ctx->prog; const bool is_main_prog = !bpf_is_subprog(prog); @@ -296,7 +303,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) const u8 fp = bpf2a64[BPF_REG_FP]; const u8 tcc = bpf2a64[TCALL_CNT]; const u8 fpb = bpf2a64[FP_BOTTOM]; - const int idx0 = ctx->idx; int cur_offset; /* @@ -322,6 +328,8 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) * */ + emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx); + const int idx0 = ctx->idx; /* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump @@ -1575,7 +1583,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * BPF line info needs ctx->offset[i] to be the offset of * instruction[i] in jited image, so build prologue first. */ - if (build_prologue(&ctx, was_classic)) { + if (build_prologue(&ctx, was_classic, bpf_is_subprog(prog))) { prog = orig_prog; goto out_off; } @@ -1614,7 +1622,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.idx = 0; ctx.exentry_idx = 0; - build_prologue(&ctx, was_classic); + build_prologue(&ctx, was_classic, bpf_is_subprog(prog)); if (build_body(&ctx, extra_pass)) { bpf_jit_binary_free(header); @@ -1654,9 +1662,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->image = image_ptr; jit_data->header = header; } - prog->bpf_func = (void *)ctx.image; + prog->bpf_func = (void *)ctx.image + cfi_get_offset(); prog->jited = 1; - prog->jited_len = prog_size; + prog->jited_len = prog_size - cfi_get_offset(); if (!prog->is_func || extra_pass) { int i; @@ -1905,6 +1913,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, /* return address locates above FP */ retaddr_off = stack_size + 8; + if (flags & BPF_TRAMP_F_INDIRECT) { + /* + * Indirect call for bpf_struct_ops + */ + emit_kcfi(cfi_get_func_hash(func_addr), ctx); + } /* bpf trampoline may be invoked by 3 instruction types: * 1. bl, attached to bpf prog or kernel function via short jump * 2. br, attached to bpf prog or kernel function via long jump