From patchwork Sun Nov 10 15:51:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13869974 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8186081741; Sun, 10 Nov 2024 15:51:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731253896; cv=none; b=Azf9uNYnL72bc5Uda/XjbKUiWrkrN07uzBu+rLZMuW/ojtYOmpjhpaQ58SYprcjIPzxpGzlEK/96/mRG59XmY5RIWtjIIWD/HLwrY+oxh5sjB1JzKcJEW3CLqkxvJCyadRWr9oJcFXqIgrE+X3i0ka9oLEfwe4HSftH3hoaOkFg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731253896; c=relaxed/simple; bh=/HkSeJVyvgLEhQ5zekLtpHg9nCl4G6foTq1nkaeg7FY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Zd0vKluxAdXMHFboCzKBzyxO1ISzKsDAzsE112ITUaS0uJpleItQ3Aedb9LmvDtIiTpw0csZ/n1E9JCcGrMfj0EkOy4Q99E+MH+SJ1lypE8+fxnqT6a/Zq9ia6gQfnybNItfw1kTa/26koZh74pjjhGvnYrRyabgbGv2qHtLhCU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lNB+45qC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lNB+45qC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77318C4CECD; Sun, 10 Nov 2024 15:51:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1731253894; bh=/HkSeJVyvgLEhQ5zekLtpHg9nCl4G6foTq1nkaeg7FY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lNB+45qC6uZp/1TFNdePkucUf4gFtCe4yhhJA2u9XEsPbaAamdcH5YbGzmruhiNHn lTcaWBVhGajyRjMPAm9OLK5FFY5PDPUQ2se9ivOK3mmjyXOXTIYC8njq/0qyWIXd91 4/vuZJGR6Cy8jxAz2yf7k0NRce1m+yTL5jCDOAHT9tt5KlvrNaPsiLext6QK1Hkplk i3SYqXEvmQfG9W693fFV9ks2nfGzDYIuKRllJYKHa7XNqaLrk/MUnyCKEN2OLd5kjm icOsNzxT3FmZ3rOd8lUfzisKW3tJncyzwbCAPO0IZV9HtnbYhB2pbBrdE07F1tW072 FRnhKXIJlAW+A== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Alexei Starovoitov , Jiri Olsa , Alan Maguire , Mark Rutland , linux-arch@vger.kernel.org, Catalin Marinas , Will Deacon , Huacai Chen , WANG Xuerui , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Arnd Bergmann , Masami Hiramatsu , Mathieu Desnoyers Subject: [PATCH v19 13/19] fprobe: Add fprobe_header encoding feature Date: Mon, 11 Nov 2024 00:51:25 +0900 Message-ID: <173125388510.172790.1161831132316963172.stgit@devnote2> X-Mailer: git-send-email 2.43.0 In-Reply-To: <173125372214.172790.6929368952404083802.stgit@devnote2> References: <173125372214.172790.6929368952404083802.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Fprobe store its data structure address and size on the fgraph return stack by __fprobe_header. But most 64bit architecture can combine those to one unsigned long value because 4 MSB in the kernel address are the same. With this encoding, fprobe can consume less space on ret_stack. This introduces asm/fprobe.h to define arch dependent encode/decode macros. Note that since fprobe depends on CONFIG_HAVE_FUNCTION_GRAPH_FREGS, currently only arm64, loongarch, riscv, s390 and x86 are supported. Signed-off-by: Masami Hiramatsu (Google) Cc: Catalin Marinas Cc: Will Deacon Cc: Huacai Chen Cc: WANG Xuerui Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Cc: "H. Peter Anvin" Cc: Arnd Bergmann Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Acked-by: Heiko Carstens # s390 --- Changes in v19: - Fix to use Kbuild for generic arm/fprobe.h - Check CONFIG_64BIT in asm-generic/fprobe.h. --- arch/arm64/include/asm/Kbuild | 1 + arch/loongarch/include/asm/fprobe.h | 12 +++++++++ arch/riscv/include/asm/Kbuild | 1 + arch/s390/include/asm/fprobe.h | 10 ++++++++ arch/x86/include/asm/Kbuild | 1 + include/asm-generic/fprobe.h | 46 +++++++++++++++++++++++++++++++++++ kernel/trace/fprobe.c | 29 ++++++++++++++++++++++ 7 files changed, 100 insertions(+) create mode 100644 arch/loongarch/include/asm/fprobe.h create mode 100644 arch/s390/include/asm/fprobe.h create mode 100644 include/asm-generic/fprobe.h diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 4e350df9a02d..d2ff8f6c3231 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -8,6 +8,7 @@ syscall-y += unistd_32.h syscall-y += unistd_compat_32.h generic-y += early_ioremap.h +generic-y += fprobe.h generic-y += mcs_spinlock.h generic-y += mmzone.h generic-y += qrwlock.h diff --git a/arch/loongarch/include/asm/fprobe.h b/arch/loongarch/include/asm/fprobe.h new file mode 100644 index 000000000000..7af3b3126caf --- /dev/null +++ b/arch/loongarch/include/asm/fprobe.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_LOONGARCH_FPROBE_H +#define _ASM_LOONGARCH_FPROBE_H + +/* + * Explicitly undef ARCH_DEFINE_ENCODE_FPROBE_HEADER, because loongarch does not + * have enough number of fixed MSBs of the address of kernel objects for + * encoding the size of data in fprobe_header. Use 2-entries encoding instead. + */ +#undef ARCH_DEFINE_ENCODE_FPROBE_HEADER + +#endif /* _ASM_LOONGARCH_FPROBE_H */ diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 1461af12da6e..e4c8790d728a 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -4,6 +4,7 @@ syscall-y += syscall_table_64.h generic-y += early_ioremap.h generic-y += flat.h +generic-y += fprobe.h generic-y += kvm_para.h generic-y += mmzone.h generic-y += parport.h diff --git a/arch/s390/include/asm/fprobe.h b/arch/s390/include/asm/fprobe.h new file mode 100644 index 000000000000..5ef600b372f4 --- /dev/null +++ b/arch/s390/include/asm/fprobe.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_S390_FPROBE_H +#define _ASM_S390_FPROBE_H + +#include + +#undef FPROBE_HEADER_MSB_PATTERN +#define FPROBE_HEADER_MSB_PATTERN 0 + +#endif /* _ASM_S390_FPROBE_H */ diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild index 6c23d1661b17..58f4ddecc5fa 100644 --- a/arch/x86/include/asm/Kbuild +++ b/arch/x86/include/asm/Kbuild @@ -10,5 +10,6 @@ generated-y += unistd_64_x32.h generated-y += xen-hypercalls.h generic-y += early_ioremap.h +generic-y += fprobe.h generic-y += mcs_spinlock.h generic-y += mmzone.h diff --git a/include/asm-generic/fprobe.h b/include/asm-generic/fprobe.h new file mode 100644 index 000000000000..8659a4dc6eb6 --- /dev/null +++ b/include/asm-generic/fprobe.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Generic arch dependent fprobe macros. + */ +#ifndef __ASM_GENERIC_FPROBE_H__ +#define __ASM_GENERIC_FPROBE_H__ + +#include + +#ifdef CONFIG_64BIT +/* + * Encoding the size and the address of fprobe into one 64bit entry. + * The 32bit architectures should use 2 entries to store those info. + */ + +#define ARCH_DEFINE_ENCODE_FPROBE_HEADER + +#define FPROBE_HEADER_MSB_SIZE_SHIFT (BITS_PER_LONG - FPROBE_DATA_SIZE_BITS) +#define FPROBE_HEADER_MSB_MASK \ + GENMASK(FPROBE_HEADER_MSB_SIZE_SHIFT - 1, 0) + +/* + * By default, this expects the MSBs in the address of kprobe is 0xf. + * If any arch needs another fixed pattern (e.g. s390 is zero filled), + * override this. + */ +#define FPROBE_HEADER_MSB_PATTERN \ + GENMASK(BITS_PER_LONG - 1, FPROBE_HEADER_MSB_SIZE_SHIFT) + +#define arch_fprobe_header_encodable(fp) \ + (((unsigned long)(fp) & ~FPROBE_HEADER_MSB_MASK) == \ + FPROBE_HEADER_MSB_PATTERN) + +#define arch_encode_fprobe_header(fp, size) \ + (((unsigned long)(fp) & FPROBE_HEADER_MSB_MASK) | \ + ((unsigned long)(size) << FPROBE_HEADER_MSB_SIZE_SHIFT)) + +#define arch_decode_fprobe_header_size(val) \ + ((unsigned long)(val) >> FPROBE_HEADER_MSB_SIZE_SHIFT) + +#define arch_decode_fprobe_header_fp(val) \ + ((struct fprobe *)(((unsigned long)(val) & FPROBE_HEADER_MSB_MASK) | \ + FPROBE_HEADER_MSB_PATTERN)) +#endif /* CONFIG_64BIT */ + +#endif /* __ASM_GENERIC_FPROBE_H__ */ diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index ed9c1d79426a..2560b312ad57 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -13,6 +13,8 @@ #include #include +#include + #include "trace.h" #define FPROBE_IP_HASH_BITS 8 @@ -143,6 +145,31 @@ static int del_fprobe_hash(struct fprobe *fp) return 0; } +#ifdef ARCH_DEFINE_ENCODE_FPROBE_HEADER + +/* The arch should encode fprobe_header info into one unsigned long */ +#define FPROBE_HEADER_SIZE_IN_LONG 1 + +static inline bool write_fprobe_header(unsigned long *stack, + struct fprobe *fp, unsigned int size_words) +{ + if (WARN_ON_ONCE(size_words > MAX_FPROBE_DATA_SIZE_WORD || + !arch_fprobe_header_encodable(fp))) + return false; + + *stack = arch_encode_fprobe_header(fp, size_words); + return true; +} + +static inline void read_fprobe_header(unsigned long *stack, + struct fprobe **fp, unsigned int *size_words) +{ + *fp = arch_decode_fprobe_header_fp(*stack); + *size_words = arch_decode_fprobe_header_size(*stack); +} + +#else + /* Generic fprobe_header */ struct __fprobe_header { struct fprobe *fp; @@ -173,6 +200,8 @@ static inline void read_fprobe_header(unsigned long *stack, *size_words = fph->size_words; } +#endif + /* * fprobe shadow stack management: * Since fprobe shares a single fgraph_ops, it needs to share the stack entry