From patchwork Sun Sep 24 13:35:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396965 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F549C8CC; Sun, 24 Sep 2023 13:36:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1710AC433C8; Sun, 24 Sep 2023 13:36:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562565; bh=LhYnyuDASlXkoXmwCgKG/bRCaq518SQW1Kv1bqMglGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oqIRRymLElY+P25kTdNjEVyJB1A9ldCYTxgORJ2fY34/Up+1xmNu/blxLvgvO5d9v YiLVnE8f+DqJ9TRtFd+qISOHrQMIzP3no7VHIYYfltA8O88TF9oWSzN1loDL1H8+Av y2pog5Q99aZxCYiO+eGs+EcZEK5QCFRKxQ1H0BoMXnKzdM7ZmuvT++GsX3E5aZ6zkP RcNxCFEkV5UQYuJsW4OtyYkLV7KQiGlqwJLrJvlkTB1UERoEbXgseOI6XFhvKBFitR O+Q57mxrGf5S2tiOPkc0LYv4MVrdrsoBejfM774lHvH+n9ZtLBCoy5+6jnf/Ar9sx+ WlJQquCgwUxMg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 01/12] riscv: ftrace: Fix to pass correct ftrace_regs to ftrace_func_t functions Date: Sun, 24 Sep 2023 22:35:59 +0900 Message-Id: <169556255915.146934.16583650430641758878.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Since ftrace_func_t requires to pass 'struct ftrace_regs *' as the 4th argument even if FTRACE_OPS_FL_SAVE_REGS is not set, ftrace_caller must pass 'struct ftrace_regs *', which is a partial pt_regs, on the stack to the ftrace_func_t functions, so that the ftrace_func_t functions can access some partial registers. Fix to allocate 'struct ftrace_regs' (which has the same size of 'struct pt_regs') on the stack and save partial (argument) registers on it instead of reduced size custom data structure. Fixes: afc76b8b8011 ("riscv: Using PATCHABLE_FUNCTION_ENTRY instead of MCOUNT") Signed-off-by: Masami Hiramatsu (Google) --- arch/riscv/kernel/mcount-dyn.S | 65 +++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 37 deletions(-) diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 669b8697aa38..84963680eff4 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -14,46 +14,37 @@ .text #define FENTRY_RA_OFFSET 8 -#define ABI_SIZE_ON_STACK 80 -#define ABI_A0 0 -#define ABI_A1 8 -#define ABI_A2 16 -#define ABI_A3 24 -#define ABI_A4 32 -#define ABI_A5 40 -#define ABI_A6 48 -#define ABI_A7 56 -#define ABI_T0 64 -#define ABI_RA 72 .macro SAVE_ABI - addi sp, sp, -ABI_SIZE_ON_STACK - - REG_S a0, ABI_A0(sp) - REG_S a1, ABI_A1(sp) - REG_S a2, ABI_A2(sp) - REG_S a3, ABI_A3(sp) - REG_S a4, ABI_A4(sp) - REG_S a5, ABI_A5(sp) - REG_S a6, ABI_A6(sp) - REG_S a7, ABI_A7(sp) - REG_S t0, ABI_T0(sp) - REG_S ra, ABI_RA(sp) + addi sp, sp, -PT_SIZE_ON_STACK + + /* Save t0 as epc for ftrace_regs_get_instruction_pointer() */ + REG_S t0, PT_EPC(sp) + REG_S a0, PT_A0(sp) + REG_S a1, PT_A1(sp) + REG_S a2, PT_A2(sp) + REG_S a3, PT_A3(sp) + REG_S a4, PT_A4(sp) + REG_S a5, PT_A5(sp) + REG_S a6, PT_A6(sp) + REG_S a7, PT_A7(sp) + REG_S t0, PT_T0(sp) + REG_S ra, PT_RA(sp) .endm .macro RESTORE_ABI - REG_L a0, ABI_A0(sp) - REG_L a1, ABI_A1(sp) - REG_L a2, ABI_A2(sp) - REG_L a3, ABI_A3(sp) - REG_L a4, ABI_A4(sp) - REG_L a5, ABI_A5(sp) - REG_L a6, ABI_A6(sp) - REG_L a7, ABI_A7(sp) - REG_L t0, ABI_T0(sp) - REG_L ra, ABI_RA(sp) - - addi sp, sp, ABI_SIZE_ON_STACK + REG_L a0, PT_A0(sp) + REG_L a1, PT_A1(sp) + REG_L a2, PT_A2(sp) + REG_L a3, PT_A3(sp) + REG_L a4, PT_A4(sp) + REG_L a5, PT_A5(sp) + REG_L a6, PT_A6(sp) + REG_L a7, PT_A7(sp) + REG_L t0, PT_T0(sp) + REG_L ra, PT_RA(sp) + + addi sp, sp, PT_SIZE_ON_STACK .endm #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS @@ -96,8 +87,8 @@ ftrace_call: call ftrace_stub #ifdef CONFIG_FUNCTION_GRAPH_TRACER - addi a0, sp, ABI_RA - REG_L a1, ABI_T0(sp) + addi a0, sp, PT_RA + REG_L a1, PT_T0(sp) addi a1, a1, -FENTRY_RA_OFFSET #ifdef HAVE_FUNCTION_GRAPH_FP_TEST mv a2, s0 From patchwork Sun Sep 24 13:36:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396966 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50A5B6122; Sun, 24 Sep 2023 13:36:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48650C433C8; Sun, 24 Sep 2023 13:36:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562578; bh=+UQOaXyf7wYEVeoHCkzfXpWdvixzMC9xY2Nvd7e4ziM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LmvhuVJMBb4WW8ut2rEFn6UqOgvJDGGewzA/jDlUL48jPr1c/bNobdkx2NKnlo7Jh lB9eKDPuxclzehBqTqfcTwjLmnEf9wRDc2SmVrkIecey2MC6t/4LCd1sjEkTRPeNBX bwHYgiQagiGiYzkqeV1iTB8DaunYCY6ephxg16OGbHls7bo9Ng0r5CcWbHScWgp0eW h/0EMkw1VNjh7Gsh4nlyJglbvbbtB0QKLVHb/B5iX9tGvJ+nnEodkvqM6AKs1B3S0y 0/FECAJYwyA9GXlPDxNsMBaHSxvg0rhn12xppztVkaGfeU70D8ksb+2iRyocUnNab6 VbbnhtgTPd/Eg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 02/12] Documentation: probes: Add a new ret_ip callback parameter Date: Sun, 24 Sep 2023 22:36:11 +0900 Message-Id: <169556257133.146934.13560704846459957726.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Add a new ret_ip callback parameter description. Fixes: cb16330d1274 ("fprobe: Pass return address to the handlers") Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes in v4: - Update ret_ip description (Thanks Florent!) --- Documentation/trace/fprobe.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst index 7a895514b537..196f52386aaa 100644 --- a/Documentation/trace/fprobe.rst +++ b/Documentation/trace/fprobe.rst @@ -91,9 +91,9 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); Note that the @entry_ip is saved at function entry and passed to exit handler. If the entry callback function returns !0, the corresponding exit callback will be cancelled. @@ -108,6 +108,10 @@ If the entry callback function returns !0, the corresponding exit callback will Note that this may not be the actual entry address of the function but the address where the ftrace is instrumented. +@ret_ip + This is the return address that the traced function will return to, + somewhere in the caller. This can be used at both entry and exit. + @regs This is the `pt_regs` data structure at the entry and exit. Note that the instruction pointer of @regs may be different from the @entry_ip From patchwork Sun Sep 24 13:36:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396967 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C57B0CA4C; Sun, 24 Sep 2023 13:36:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 720EFC433C9; Sun, 24 Sep 2023 13:36:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562590; bh=tVeWRc6NY/P1SVA880iIBWSSfIP1iQe2tBoTWU/xgMg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hjlxvWFQgaGpBMlqFwlnIZ9VGGXN02c/MIxxrj0jkjoXq5Dkp5EVjKLTiQCIZ+qQi t69ATsL1NxjpLZ/BSsGg23XpeCctFsJn6LLHvDohosO1KgMJ6u+R2PRBbG7L+Zaxgx y6+QFBBfA0LjX6LTPp5lOsuNH8U89yONdRb5wIrRKo7ESmLsGg742zPxiF2hCsAyA1 jYMJ6u5zAyz3Zsp/OsY2JWun3v1LFrGm7TIkVPFmKVtxfg03TrJKX5ursI4EgZ+w4w J6bWYTBctWkELztWXwOD2byUYJswu6XGjzXsV8QRIL2Wu6TuAKKbm8K53sFgeXY1Rn +euG0vig0MbyQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 03/12] tracing: Add a comment about the requirements of the ftrace_regs Date: Sun, 24 Sep 2023 22:36:23 +0900 Message-Id: <169556258347.146934.12248055282843332442.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Add a comment about the requirements of the ftrace_regs if it is implemented on the arch-dependent code with CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS. Signed-off-by: Masami Hiramatsu (Google) --- include/linux/ftrace.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index e8921871ef9a..5da70f238645 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -118,6 +118,14 @@ extern int ftrace_enabled; #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/* + * The ftrace_regs will be just a wrapper of the pt_regs if + * CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS is not set. If it is set on an + * architecture, it has to define the ftrace_regs data structure. + * The ftrace_regs is expected to save the registers for the function + * arguments, the registers for stack dump (e.g. stack pointer and the + * frame pointer) and the instruction pointer for reference. + */ struct ftrace_regs { struct pt_regs regs; }; From patchwork Sun Sep 24 13:36:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396968 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7E376FAF; Sun, 24 Sep 2023 13:36:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B826CC433C9; Sun, 24 Sep 2023 13:36:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562602; bh=NDtbtS1XdwGmnj+/dKk4f5xEF7uHU3gFzH2zR7S98C0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=us+FCS94DuPWfSBumyQflpoUc47AIea40ziGxK4E5nmvTG9V5xx15+nWZCkP1vNnm hOIZR4LM6AKXVShhIoVoSzyGlfH/RSpm/aJBsHpuPrb7K+LrC9sHUvgWozLWhNZ21+ qG+fWpX5/wrgMLHsQ73qffbjfBahY9t3Cp373Ypxlxk+F2E9LpOz7azoqJSOY421zN xyg6hVSemumvLcUYzb5hohhh0rp1g+B9BO33lOKQiTNankN2tf3c62WAXg52A0cR0Y owm54DkSy/a6F4jR0CRa7rtowHbMXj2Idpg6gc/0YokvjA6I55tYLYBwqwSFf224gT 5hlkjYsKN9CUw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 04/12] fprobe: Use ftrace_regs in fprobe entry handler Date: Sun, 24 Sep 2023 22:36:36 +0900 Message-Id: <169556259571.146934.4558592076420704031.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) This allows fprobes to be available with CONFIG_DYNAMIC_FTRACE_WITH_ARGS instead of CONFIG_DYNAMIC_FTRACE_WITH_REGS, then we can enable fprobe on arm64. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes in v3: - Use FTRACE_OPS_FL_SAVE_ARGS instead of FTRACE_OPS_FL_SAVE_REGS. Changes in v4: - Keep CONFIG_FPROBE check for multi-kprobe because this depends on FPROBE API. Changes in v5: - Just fix the typo in title. --- include/linux/fprobe.h | 2 +- kernel/trace/Kconfig | 3 ++- kernel/trace/bpf_trace.c | 10 +++++++--- kernel/trace/fprobe.c | 4 ++-- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 4 ++-- samples/fprobe/fprobe_example.c | 2 +- 7 files changed, 20 insertions(+), 11 deletions(-) diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 3e03758151f4..36c0595f7b93 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -35,7 +35,7 @@ struct fprobe { int nr_maxactive; int (*entry_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 61c541c36596..976fd594b446 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -287,7 +287,7 @@ config DYNAMIC_FTRACE_WITH_ARGS config FPROBE bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_RETHOOK select RETHOOK default n @@ -672,6 +672,7 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS + depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index a7264b2c17ad..efb3c265cad8 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2501,7 +2501,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#ifdef CONFIG_FPROBE +#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2729,10 +2729,14 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, static int kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); @@ -2988,7 +2992,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE */ +#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 3b21f4063258..07deb52df44a 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -46,7 +46,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, } if (fp->entry_handler) - ret = fp->entry_handler(fp, ip, parent_ip, ftrace_get_regs(fregs), entry_data); + ret = fp->entry_handler(fp, ip, parent_ip, fregs, entry_data); /* If entry_handler returns !0, nmissed is not counted. */ if (rh) { @@ -182,7 +182,7 @@ static void fprobe_init(struct fprobe *fp) fp->ops.func = fprobe_kprobe_handler; else fp->ops.func = fprobe_handler; - fp->ops.flags |= FTRACE_OPS_FL_SAVE_REGS; + fp->ops.flags |= FTRACE_OPS_FL_SAVE_ARGS; } static int fprobe_init_rethook(struct fprobe *fp, int num) diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 8bfe23af9c73..71bf38d698f1 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -320,12 +320,16 @@ NOKPROBE_SYMBOL(fexit_perf_func); #endif /* CONFIG_PERF_EVENTS */ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; + if (!regs) + return 0; + if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fentry_trace_func(tf, entry_ip, regs); #ifdef CONFIG_PERF_EVENTS diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index 24de0e5ff859..ff607babba18 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -40,7 +40,7 @@ static noinline u32 fprobe_selftest_nest_target(u32 value, u32 (*nest)(u32)) static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); /* This can be called on the fprobe_selftest_target and the fprobe_selftest_target2 */ @@ -81,7 +81,7 @@ static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); return 0; diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 64e715e7ed11..1545a1aac616 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -50,7 +50,7 @@ static void show_backtrace(void) static int sample_entry_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { if (use_trace) /* From patchwork Sun Sep 24 13:36:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396969 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F0F36ADE; Sun, 24 Sep 2023 13:36:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 27DC8C433C7; Sun, 24 Sep 2023 13:36:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562614; bh=jmHWZoJzNc7w2WNK0aSXMKcROawBX8r9FSb7vj/g9k4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AQRxpZ9URRHAkZC5HdJ8I8W4mys6wynoEVlftisU26BNvb9oiXK0Ro0PXYd9qLB2l UL3C0Oy+xI57Izrd/yxuWzsvG4CXpOxOGRD78wJxDt02xtJijhUFSWR7QcUorUwFdx O5hf0wkh9BjAh382ei6TnpnBPotUpzDvkt+JUKJwJnCdDi+nKlNZVymAvIk54keZYY BW4+MqOrdmXmTAfaZXd8nd359ake5zpaQynZsXrSQENdGQz9VryIndwLRoYQVqp8+n R/DUZZZ+mgL/oGSxsVqdK92HPyXr6YtPOR6dFKfl+Ty7MWFdnNNnzqBRbH37eg1mOI eDmk2HLeM2faQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 05/12] tracing: Expose ftrace_regs regardless of CONFIG_FUNCTION_TRACER Date: Sun, 24 Sep 2023 22:36:48 +0900 Message-Id: <169556260811.146934.12334081780807613682.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) In order to be able to use ftrace_regs even from features unrelated to function tracer (e.g. kretprobe), expose ftrace_regs structures and APIs even if the CONFIG_FUNCTION_TRACER=n. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes in v3: - arch/s390/include/asm/ftrace.h: hide ftrace_regs parts if CONFIG_FUNCTION_TRACER=n to avoid conflict. - Fixed typo. - Define accessor macros only if HAVE_REGS_AND_STACK_ACCESS_API=y --- arch/s390/include/asm/ftrace.h | 4 +++ include/linux/ftrace.h | 51 +++++++++++++++++++++++----------------- 2 files changed, 33 insertions(+), 22 deletions(-) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 5a82b08f03cd..6de3bf4d0092 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -41,6 +41,8 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) return addr; } +#ifdef CONFIG_FUNCTION_TRACER + struct ftrace_regs { struct pt_regs regs; }; @@ -97,6 +99,8 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#endif /* CONFIG_FUNCTION_TRACER */ + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS /* * When an ftrace registered caller is tracing a function that is diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 5da70f238645..15f4865a4083 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -112,11 +112,11 @@ static inline int ftrace_mod_get_kallsym(unsigned int symnum, unsigned long *val } #endif -#ifdef CONFIG_FUNCTION_TRACER - -extern int ftrace_enabled; - -#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +/* + * If the architecture doesn't support FTRACE_WITH_ARGS or disables function + * tracer, define the default(pt_regs compatible) ftrace_regs. + */ +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) || !defined(CONFIG_FUNCTION_TRACER) /* * The ftrace_regs will be just a wrapper of the pt_regs if @@ -131,13 +131,33 @@ struct ftrace_regs { }; #define arch_ftrace_get_regs(fregs) (&(fregs)->regs) +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API + /* * ftrace_regs_set_instruction_pointer() is to be defined by the architecture * if to allow setting of the instruction pointer from the ftrace_regs when * HAVE_DYNAMIC_FTRACE_WITH_ARGS is set and it supports live kernel patching. */ #define ftrace_regs_set_instruction_pointer(fregs, ip) do { } while (0) -#endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +#define ftrace_regs_get_instruction_pointer(fregs) \ + instruction_pointer(ftrace_get_regs(fregs)) +#define ftrace_regs_get_argument(fregs, n) \ + regs_get_kernel_argument(ftrace_get_regs(fregs), n) +#define ftrace_regs_get_stack_pointer(fregs) \ + kernel_stack_pointer(ftrace_get_regs(fregs)) +#define ftrace_regs_return_value(fregs) \ + regs_return_value(ftrace_get_regs(fregs)) +#define ftrace_regs_set_return_value(fregs, ret) \ + regs_set_return_value(ftrace_get_regs(fregs), ret) +#define ftrace_override_function_with_return(fregs) \ + override_function_with_return(ftrace_get_regs(fregs)) +#define ftrace_regs_query_register_offset(name) \ + regs_query_register_offset(name) + +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || !CONFIG_FUNCTION_TRACER */ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { @@ -159,22 +179,9 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) return ftrace_get_regs(fregs) != NULL; } -#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS -#define ftrace_regs_get_instruction_pointer(fregs) \ - instruction_pointer(ftrace_get_regs(fregs)) -#define ftrace_regs_get_argument(fregs, n) \ - regs_get_kernel_argument(ftrace_get_regs(fregs), n) -#define ftrace_regs_get_stack_pointer(fregs) \ - kernel_stack_pointer(ftrace_get_regs(fregs)) -#define ftrace_regs_return_value(fregs) \ - regs_return_value(ftrace_get_regs(fregs)) -#define ftrace_regs_set_return_value(fregs, ret) \ - regs_set_return_value(ftrace_get_regs(fregs), ret) -#define ftrace_override_function_with_return(fregs) \ - override_function_with_return(ftrace_get_regs(fregs)) -#define ftrace_regs_query_register_offset(name) \ - regs_query_register_offset(name) -#endif +#ifdef CONFIG_FUNCTION_TRACER + +extern int ftrace_enabled; typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); From patchwork Sun Sep 24 13:37:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60D856ADE; Sun, 24 Sep 2023 13:37:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 56F92C433C8; Sun, 24 Sep 2023 13:37:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562627; bh=F+SQMPQuFwuNvF2/gAvfM3FFfKmhMFYc8ubpZC9kP40=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UOSw5wFF1cOxb0v3+RYbN3I6PSGUuE2ntY/dy6Iwkp0YZhWizfjusW6NbHsvgCQ/b tfvjknFeR5VfLV7gE0TfgTBj9E0rVhj2D6YY7xO9JiZSyWbpECwSuWNsVELn0CdIQl 8slXKQhlZnNIX7c63f1ZZ0mM9nMAtck4WiBFv/kl2cRHcDkEClM1BCGkvStq481CZJ 17bWdAyMdmZtl4UERvk5l+aHYStgYS8i2GLsuGgzGoIfPJCS6uvmrmoZXhTk/nsrIK lRv2ESr1lLGq2m3v6dUpjWrPn9T0ICaLilPwUb5vwlPUUxIZUFdzeAhOYZL4PmiUJw sXYzDLUl/ThTA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 06/12] fprobe: rethook: Use ftrace_regs in fprobe exit handler and rethook Date: Sun, 24 Sep 2023 22:37:00 +0900 Message-Id: <169556262030.146934.16624533747935252102.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Change the fprobe exit handler and rethook to use ftrace_regs structure instead of pt_regs. This also introduce HAVE_PT_REGS_TO_FTRACE_REGS_CAST which means the ftrace_regs's memory layout is equal to the pt_regs so that those are able to cast. Only if it is enabled, kretprobe will use rethook since kretprobe requires pt_regs for backward compatibility. This means the archs which currently implement rethook for kretprobes needs to set that flag and it must ensure struct ftrace_regs is same as pt_regs. If not, it must be either disabling kretprobe or implementing kretprobe trampoline separately from rethook trampoline. Signed-off-by: Masami Hiramatsu (Google) Signed-off-by: Sven Schnelle Acked-by: Florent Revest --- Changes in v3: - Config rename to HAVE_PT_REGS_TO_FTRACE_REGS_CAST - Use ftrace_regs_* APIs instead of ftrace_get_regs(). Changes in v4: - Add static_assert() to ensure at least the size of pt_regs and ftrace_regs are same if HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y. Changes in v5: - Add s390 rethook update by Sven Schnelle. - Add loongarch rethook update. - Add riscv rethook update (with saving regs.s0 in ftrace) --- arch/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/loongarch/kernel/rethook.c | 10 +++++----- arch/loongarch/kernel/rethook.h | 4 ++-- arch/riscv/kernel/mcount-dyn.S | 2 ++ arch/riscv/kernel/probes/rethook.c | 12 ++++++------ arch/riscv/kernel/probes/rethook.h | 6 ++++-- arch/s390/Kconfig | 1 + arch/s390/kernel/rethook.c | 10 ++++++---- arch/s390/kernel/rethook.h | 2 +- arch/x86/Kconfig | 1 + arch/x86/kernel/rethook.c | 13 +++++++------ include/linux/fprobe.h | 2 +- include/linux/ftrace.h | 6 ++++++ include/linux/rethook.h | 11 ++++++----- kernel/kprobes.c | 10 ++++++++-- kernel/trace/Kconfig | 7 +++++++ kernel/trace/bpf_trace.c | 6 +++++- kernel/trace/fprobe.c | 6 +++--- kernel/trace/rethook.c | 16 ++++++++-------- kernel/trace/trace_fprobe.c | 6 +++++- lib/test_fprobe.c | 6 +++--- samples/fprobe/fprobe_example.c | 2 +- 23 files changed, 90 insertions(+), 51 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 12d51495caec..300d76c2ad77 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -191,6 +191,7 @@ config KRETPROBE_ON_RETHOOK def_bool y depends on HAVE_RETHOOK depends on KRETPROBES + depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST || !HAVE_DYNAMIC_FTRACE_WITH_ARGS select RETHOOK config USER_RETURN_NOTIFIER diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e14396a2ddcb..258e9bee1503 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -108,6 +108,7 @@ config LOONGARCH select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT diff --git a/arch/loongarch/kernel/rethook.c b/arch/loongarch/kernel/rethook.c index db1c5f5024fd..d718327d1e88 100644 --- a/arch/loongarch/kernel/rethook.c +++ b/arch/loongarch/kernel/rethook.c @@ -8,19 +8,19 @@ #include "rethook.h" /* This is called from arch_rethook_trampoline() */ -unsigned long __used arch_rethook_trampoline_callback(struct pt_regs *regs) +unsigned long __used arch_rethook_trampoline_callback(struct ftrace_regs *fregs) { - return rethook_trampoline_handler(regs, 0); + return rethook_trampoline_handler(fregs, 0); } NOKPROBE_SYMBOL(arch_rethook_trampoline_callback); -void arch_rethook_prepare(struct rethook_node *rhn, struct pt_regs *regs, bool mcount) +void arch_rethook_prepare(struct rethook_node *rhn, struct ftrace_regs *fregs, bool mcount) { rhn->frame = 0; - rhn->ret_addr = regs->regs[1]; + rhn->ret_addr = fregs->regs.regs[1]; /* replace return addr with trampoline */ - regs->regs[1] = (unsigned long)arch_rethook_trampoline; + fregs->regs.regs[1] = (unsigned long)arch_rethook_trampoline; } NOKPROBE_SYMBOL(arch_rethook_prepare); diff --git a/arch/loongarch/kernel/rethook.h b/arch/loongarch/kernel/rethook.h index 3f1c1edf0d0b..0643a8d6a8dd 100644 --- a/arch/loongarch/kernel/rethook.h +++ b/arch/loongarch/kernel/rethook.h @@ -2,7 +2,7 @@ #ifndef __LOONGARCH_RETHOOK_H #define __LOONGARCH_RETHOOK_H -unsigned long arch_rethook_trampoline_callback(struct pt_regs *regs); -void arch_rethook_prepare(struct rethook_node *rhn, struct pt_regs *regs, bool mcount); +unsigned long arch_rethook_trampoline_callback(struct ftrace_regs *fregs); +void arch_rethook_prepare(struct rethook_node *rhn, struct ftrace_regs *fregs, bool mcount); #endif diff --git a/arch/riscv/kernel/mcount-dyn.S b/arch/riscv/kernel/mcount-dyn.S index 84963680eff4..184f76394ab8 100644 --- a/arch/riscv/kernel/mcount-dyn.S +++ b/arch/riscv/kernel/mcount-dyn.S @@ -20,6 +20,8 @@ /* Save t0 as epc for ftrace_regs_get_instruction_pointer() */ REG_S t0, PT_EPC(sp) + /* Save s0 for reading frame pointer (read only) */ + REG_S s0, PT_S0(sp) REG_S a0, PT_A0(sp) REG_S a1, PT_A1(sp) REG_S a2, PT_A2(sp) diff --git a/arch/riscv/kernel/probes/rethook.c b/arch/riscv/kernel/probes/rethook.c index 5c27c1f50989..052fa1a363e6 100644 --- a/arch/riscv/kernel/probes/rethook.c +++ b/arch/riscv/kernel/probes/rethook.c @@ -8,20 +8,20 @@ #include "rethook.h" /* This is called from arch_rethook_trampoline() */ -unsigned long __used arch_rethook_trampoline_callback(struct pt_regs *regs) +unsigned long __used arch_rethook_trampoline_callback(struct ftrace_regs *fregs) { - return rethook_trampoline_handler(regs, regs->s0); + return rethook_trampoline_handler(fregs, fregs->regs.s0); } NOKPROBE_SYMBOL(arch_rethook_trampoline_callback); -void arch_rethook_prepare(struct rethook_node *rhn, struct pt_regs *regs, bool mcount) +void arch_rethook_prepare(struct rethook_node *rhn, struct ftrace_regs *fregs, bool mcount) { - rhn->ret_addr = regs->ra; - rhn->frame = regs->s0; + rhn->ret_addr = fregs->regs.ra; + rhn->frame = fregs->regs.s0; /* replace return addr with trampoline */ - regs->ra = (unsigned long)arch_rethook_trampoline; + fregs->regs.ra = (unsigned long)arch_rethook_trampoline; } NOKPROBE_SYMBOL(arch_rethook_prepare); diff --git a/arch/riscv/kernel/probes/rethook.h b/arch/riscv/kernel/probes/rethook.h index 4758f7e3ce88..f4ce353d2008 100644 --- a/arch/riscv/kernel/probes/rethook.h +++ b/arch/riscv/kernel/probes/rethook.h @@ -2,7 +2,9 @@ #ifndef __RISCV_RETHOOK_H #define __RISCV_RETHOOK_H -unsigned long arch_rethook_trampoline_callback(struct pt_regs *regs); -void arch_rethook_prepare(struct rethook_node *rhn, struct pt_regs *regs, bool mcount); +#include + +unsigned long arch_rethook_trampoline_callback(struct ftrace_regs *fregs); +void arch_rethook_prepare(struct rethook_node *rhn, struct ftrace_regs *fregs, bool mcount); #endif diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ae29e4392664..5aedb4320e7c 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -167,6 +167,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT if HAVE_MARCH_Z196_FEATURES diff --git a/arch/s390/kernel/rethook.c b/arch/s390/kernel/rethook.c index af10e6bdd34e..4e86c0a1a064 100644 --- a/arch/s390/kernel/rethook.c +++ b/arch/s390/kernel/rethook.c @@ -3,8 +3,9 @@ #include #include "rethook.h" -void arch_rethook_prepare(struct rethook_node *rh, struct pt_regs *regs, bool mcount) +void arch_rethook_prepare(struct rethook_node *rh, struct ftrace_regs *fregs, bool mcount) { + struct pt_regs *regs = (struct pt_regs *)fregs; rh->ret_addr = regs->gprs[14]; rh->frame = regs->gprs[15]; @@ -13,10 +14,11 @@ void arch_rethook_prepare(struct rethook_node *rh, struct pt_regs *regs, bool mc } NOKPROBE_SYMBOL(arch_rethook_prepare); -void arch_rethook_fixup_return(struct pt_regs *regs, +void arch_rethook_fixup_return(struct ftrace_regs *fregs, unsigned long correct_ret_addr) { /* Replace fake return address with real one. */ + struct pt_regs *regs = (struct pt_regs *)fregs; regs->gprs[14] = correct_ret_addr; } NOKPROBE_SYMBOL(arch_rethook_fixup_return); @@ -24,9 +26,9 @@ NOKPROBE_SYMBOL(arch_rethook_fixup_return); /* * Called from arch_rethook_trampoline */ -unsigned long arch_rethook_trampoline_callback(struct pt_regs *regs) +unsigned long arch_rethook_trampoline_callback(struct ftrace_regs *fregs) { - return rethook_trampoline_handler(regs, regs->gprs[15]); + return rethook_trampoline_handler(fregs, fregs->regs.gprs[15]); } NOKPROBE_SYMBOL(arch_rethook_trampoline_callback); diff --git a/arch/s390/kernel/rethook.h b/arch/s390/kernel/rethook.h index 32f069eed3f3..0fe62424fc78 100644 --- a/arch/s390/kernel/rethook.h +++ b/arch/s390/kernel/rethook.h @@ -2,6 +2,6 @@ #ifndef __S390_RETHOOK_H #define __S390_RETHOOK_H -unsigned long arch_rethook_trampoline_callback(struct pt_regs *regs); +unsigned long arch_rethook_trampoline_callback(struct ftrace_regs *fregs); #endif diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 66bfabae8814..daca05e10956 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -209,6 +209,7 @@ config X86 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64 + select HAVE_PT_REGS_TO_FTRACE_REGS_CAST if X86_64 select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_SAMPLE_FTRACE_DIRECT if X86_64 select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64 diff --git a/arch/x86/kernel/rethook.c b/arch/x86/kernel/rethook.c index 8a1c0111ae79..d714d0276c93 100644 --- a/arch/x86/kernel/rethook.c +++ b/arch/x86/kernel/rethook.c @@ -83,7 +83,8 @@ __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs) * arch_rethook_fixup_return() which called from this * rethook_trampoline_handler(). */ - rethook_trampoline_handler(regs, (unsigned long)frame_pointer); + rethook_trampoline_handler((struct ftrace_regs *)regs, + (unsigned long)frame_pointer); /* * Copy FLAGS to 'pt_regs::ss' so that arch_rethook_trapmoline() @@ -104,22 +105,22 @@ NOKPROBE_SYMBOL(arch_rethook_trampoline_callback); STACK_FRAME_NON_STANDARD_FP(arch_rethook_trampoline); /* This is called from rethook_trampoline_handler(). */ -void arch_rethook_fixup_return(struct pt_regs *regs, +void arch_rethook_fixup_return(struct ftrace_regs *fregs, unsigned long correct_ret_addr) { - unsigned long *frame_pointer = (void *)(regs + 1); + unsigned long *frame_pointer = (void *)(fregs + 1); /* Replace fake return address with real one. */ *frame_pointer = correct_ret_addr; } NOKPROBE_SYMBOL(arch_rethook_fixup_return); -void arch_rethook_prepare(struct rethook_node *rh, struct pt_regs *regs, bool mcount) +void arch_rethook_prepare(struct rethook_node *rh, struct ftrace_regs *fregs, bool mcount) { - unsigned long *stack = (unsigned long *)regs->sp; + unsigned long *stack = (unsigned long *)ftrace_regs_get_stack_pointer(fregs); rh->ret_addr = stack[0]; - rh->frame = regs->sp; + rh->frame = (unsigned long)stack; /* Replace the return addr with trampoline addr */ stack[0] = (unsigned long) arch_rethook_trampoline; diff --git a/include/linux/fprobe.h b/include/linux/fprobe.h index 36c0595f7b93..b9c0c216dedb 100644 --- a/include/linux/fprobe.h +++ b/include/linux/fprobe.h @@ -38,7 +38,7 @@ struct fprobe { unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); void (*exit_handler)(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *entry_data); }; diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 15f4865a4083..39a765c71f7e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -159,6 +159,12 @@ struct ftrace_regs { #endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || !CONFIG_FUNCTION_TRACER */ +#ifdef CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST + +static_assert(sizeof(struct pt_regs) == sizeof(struct ftrace_regs)); + +#endif /* CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs) { if (!fregs) diff --git a/include/linux/rethook.h b/include/linux/rethook.h index 26b6f3c81a76..138d64c8b67b 100644 --- a/include/linux/rethook.h +++ b/include/linux/rethook.h @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -14,7 +15,7 @@ struct rethook_node; -typedef void (*rethook_handler_t) (struct rethook_node *, void *, unsigned long, struct pt_regs *); +typedef void (*rethook_handler_t) (struct rethook_node *, void *, unsigned long, struct ftrace_regs *); /** * struct rethook - The rethook management data structure. @@ -64,12 +65,12 @@ void rethook_free(struct rethook *rh); void rethook_add_node(struct rethook *rh, struct rethook_node *node); struct rethook_node *rethook_try_get(struct rethook *rh); void rethook_recycle(struct rethook_node *node); -void rethook_hook(struct rethook_node *node, struct pt_regs *regs, bool mcount); +void rethook_hook(struct rethook_node *node, struct ftrace_regs *regs, bool mcount); unsigned long rethook_find_ret_addr(struct task_struct *tsk, unsigned long frame, struct llist_node **cur); /* Arch dependent code must implement arch_* and trampoline code */ -void arch_rethook_prepare(struct rethook_node *node, struct pt_regs *regs, bool mcount); +void arch_rethook_prepare(struct rethook_node *node, struct ftrace_regs *regs, bool mcount); void arch_rethook_trampoline(void); /** @@ -84,11 +85,11 @@ static inline bool is_rethook_trampoline(unsigned long addr) } /* If the architecture needs to fixup the return address, implement it. */ -void arch_rethook_fixup_return(struct pt_regs *regs, +void arch_rethook_fixup_return(struct ftrace_regs *regs, unsigned long correct_ret_addr); /* Generic trampoline handler, arch code must prepare asm stub */ -unsigned long rethook_trampoline_handler(struct pt_regs *regs, +unsigned long rethook_trampoline_handler(struct ftrace_regs *regs, unsigned long frame); #ifdef CONFIG_RETHOOK diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 0c6185aefaef..821dff656149 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -2132,7 +2132,11 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) if (rp->entry_handler && rp->entry_handler(ri, regs)) rethook_recycle(rhn); else - rethook_hook(rhn, regs, kprobe_ftrace(p)); + /* + * We can cast pt_regs to ftrace_regs because this depends on + * HAVE_PT_REGS_TO_FTRACE_REGS_CAST. + */ + rethook_hook(rhn, (struct ftrace_regs *)regs, kprobe_ftrace(p)); return 0; } @@ -2140,9 +2144,11 @@ NOKPROBE_SYMBOL(pre_handler_kretprobe); static void kretprobe_rethook_handler(struct rethook_node *rh, void *data, unsigned long ret_addr, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct kretprobe *rp = (struct kretprobe *)data; + /* Ditto, this depends on HAVE_PT_REGS_TO_FTRACE_REGS_CAST. */ + struct pt_regs *regs = (struct pt_regs *)fregs; struct kretprobe_instance *ri; struct kprobe_ctlblk *kcb; diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 976fd594b446..d56304276318 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -57,6 +57,13 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of ftrace_regs_get_argument() and ftrace_regs_get_stack_pointer(). +config HAVE_PT_REGS_TO_FTRACE_REGS_CAST + bool + help + If this is set, the memory layout of the ftrace_regs data structure + is the same as the pt_regs. So the pt_regs is possible to be casted + to ftrace_regs. + config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE bool help diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index efb3c265cad8..8bb003ce7bb2 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2745,10 +2745,14 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, static void kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *data) { struct bpf_kprobe_multi_link *link; + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); diff --git a/kernel/trace/fprobe.c b/kernel/trace/fprobe.c index 07deb52df44a..dfddc7e8424e 100644 --- a/kernel/trace/fprobe.c +++ b/kernel/trace/fprobe.c @@ -53,7 +53,7 @@ static inline void __fprobe_handler(unsigned long ip, unsigned long parent_ip, if (ret) rethook_recycle(rh); else - rethook_hook(rh, ftrace_get_regs(fregs), true); + rethook_hook(rh, fregs, true); } } @@ -120,7 +120,7 @@ static void fprobe_kprobe_handler(unsigned long ip, unsigned long parent_ip, } static void fprobe_exit_handler(struct rethook_node *rh, void *data, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct fprobe *fp = (struct fprobe *)data; struct fprobe_rethook_node *fpr; @@ -141,7 +141,7 @@ static void fprobe_exit_handler(struct rethook_node *rh, void *data, return; } - fp->exit_handler(fp, fpr->entry_ip, ret_ip, regs, + fp->exit_handler(fp, fpr->entry_ip, ret_ip, fregs, fp->entry_data_size ? (void *)fpr->data : NULL); ftrace_test_recursion_unlock(bit); } diff --git a/kernel/trace/rethook.c b/kernel/trace/rethook.c index 5eb9b598f4e9..7c5cf9d5910c 100644 --- a/kernel/trace/rethook.c +++ b/kernel/trace/rethook.c @@ -189,7 +189,7 @@ NOKPROBE_SYMBOL(rethook_try_get); /** * rethook_hook() - Hook the current function return. * @node: The struct rethook node to hook the function return. - * @regs: The struct pt_regs for the function entry. + * @fregs: The struct ftrace_regs for the function entry. * @mcount: True if this is called from mcount(ftrace) context. * * Hook the current running function return. This must be called when the @@ -199,9 +199,9 @@ NOKPROBE_SYMBOL(rethook_try_get); * from the real function entry (e.g. kprobes) @mcount must be set false. * This is because the way to hook the function return depends on the context. */ -void rethook_hook(struct rethook_node *node, struct pt_regs *regs, bool mcount) +void rethook_hook(struct rethook_node *node, struct ftrace_regs *fregs, bool mcount) { - arch_rethook_prepare(node, regs, mcount); + arch_rethook_prepare(node, fregs, mcount); __llist_add(&node->llist, ¤t->rethooks); } NOKPROBE_SYMBOL(rethook_hook); @@ -269,7 +269,7 @@ unsigned long rethook_find_ret_addr(struct task_struct *tsk, unsigned long frame } NOKPROBE_SYMBOL(rethook_find_ret_addr); -void __weak arch_rethook_fixup_return(struct pt_regs *regs, +void __weak arch_rethook_fixup_return(struct ftrace_regs *fregs, unsigned long correct_ret_addr) { /* @@ -281,7 +281,7 @@ void __weak arch_rethook_fixup_return(struct pt_regs *regs, } /* This function will be called from each arch-defined trampoline. */ -unsigned long rethook_trampoline_handler(struct pt_regs *regs, +unsigned long rethook_trampoline_handler(struct ftrace_regs *fregs, unsigned long frame) { struct llist_node *first, *node = NULL; @@ -295,7 +295,7 @@ unsigned long rethook_trampoline_handler(struct pt_regs *regs, BUG_ON(1); } - instruction_pointer_set(regs, correct_ret_addr); + ftrace_regs_set_instruction_pointer(fregs, correct_ret_addr); /* * These loops must be protected from rethook_free_rcu() because those @@ -315,7 +315,7 @@ unsigned long rethook_trampoline_handler(struct pt_regs *regs, handler = READ_ONCE(rhn->rethook->handler); if (handler) handler(rhn, rhn->rethook->data, - correct_ret_addr, regs); + correct_ret_addr, fregs); if (first == node) break; @@ -323,7 +323,7 @@ unsigned long rethook_trampoline_handler(struct pt_regs *regs, } /* Fixup registers for returning to correct address. */ - arch_rethook_fixup_return(regs, correct_ret_addr); + arch_rethook_fixup_return(fregs, correct_ret_addr); /* Unlink used shadow stack */ first = current->rethooks.first; diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index 71bf38d698f1..c60d0d9f1a95 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -341,10 +341,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, NOKPROBE_SYMBOL(fentry_dispatcher); static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); + struct pt_regs *regs = ftrace_get_regs(fregs); + + if (!regs) + return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) fexit_trace_func(tf, entry_ip, ret_ip, regs); diff --git a/lib/test_fprobe.c b/lib/test_fprobe.c index ff607babba18..d1e80653bf0c 100644 --- a/lib/test_fprobe.c +++ b/lib/test_fprobe.c @@ -59,9 +59,9 @@ static notrace int fp_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void fp_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { - unsigned long ret = regs_return_value(regs); + unsigned long ret = ftrace_regs_return_value(fregs); KUNIT_EXPECT_FALSE(current_test, preemptible()); if (ip != target_ip) { @@ -89,7 +89,7 @@ static notrace int nest_entry_handler(struct fprobe *fp, unsigned long ip, static notrace void nest_exit_handler(struct fprobe *fp, unsigned long ip, unsigned long ret_ip, - struct pt_regs *regs, void *data) + struct ftrace_regs *fregs, void *data) { KUNIT_EXPECT_FALSE(current_test, preemptible()); KUNIT_EXPECT_EQ(current_test, ip, target_nest_ip); diff --git a/samples/fprobe/fprobe_example.c b/samples/fprobe/fprobe_example.c index 1545a1aac616..d476d1f07538 100644 --- a/samples/fprobe/fprobe_example.c +++ b/samples/fprobe/fprobe_example.c @@ -67,7 +67,7 @@ static int sample_entry_handler(struct fprobe *fp, unsigned long ip, } static void sample_exit_handler(struct fprobe *fp, unsigned long ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *regs, void *data) { unsigned long rip = ret_ip; From patchwork Sun Sep 24 13:37:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396971 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8777D6ADE; Sun, 24 Sep 2023 13:37:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 869FAC433C7; Sun, 24 Sep 2023 13:37:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562639; bh=z3ZqUJHGicIifDXnSOHi+t2Ua5M0QIoavgEAxehdPoM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HE5ucNqgLQ6mEyv5bXIhJHQFfj8vqMflwi8sAXOd+r/H9A2fcYyvgNzrF5aLrM0rq Gnn88OCM45M4cgm1ZQ7zAPV25NOTja+1iT+JWKYWyKkcUojtYaudI5f3L6clmtSE2g 5/0HrCCah3Igi9JDGgSKUvLSXSBoBDnMLVT1OrHDPHU87Lfq5uptwOeto28FJB537Z zxt6cdDCq3A2bd1RT0qIBVQvH7qatk/eQ1LYEgUFEnlTL41DhpWvKisjmh3BwP8B8S 1XV7tvJl15gzEi6hOEyKgFti4ZpfKx5bkECyuOBmtplpl3StNTBjyVOOUNk94eizZY GcddpbvM7kI9w== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 07/12] tracing: Add ftrace_partial_regs() for converting ftrace_regs to pt_regs Date: Sun, 24 Sep 2023 22:37:12 +0900 Message-Id: <169556263259.146934.4055143510535198098.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Add ftrace_partial_regs() which converts the ftrace_regs to pt_regs. If the architecture defines its own ftrace_regs, this copies partial registers to pt_regs and returns it. If not, ftrace_regs is the same as pt_regs and ftrace_partial_regs() will return ftrace_regs::regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Changes in v3: - Fix to use pt_regs::regs instead of x. - Return ftrace_regs::regs forcibly if HAVE_PT_REGS_COMPAT_FTRACE_REGS=y. - Fix typo. - Fix to copy correct registers to the pt_regs on arm64. Changes in v4: - Change the patch order in the series so that fprobe event can use this. Changes in v5: - Fix tag in the title (ftrace: -> tracing:) --- arch/arm64/include/asm/ftrace.h | 11 +++++++++++ include/linux/ftrace.h | 17 +++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index ab158196480c..5ad24f315d52 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -137,6 +137,17 @@ ftrace_override_function_with_return(struct ftrace_regs *fregs) fregs->pc = fregs->lr; } +static __always_inline struct pt_regs * +ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) +{ + memcpy(regs->regs, fregs->regs, sizeof(u64) * 9); + regs->sp = fregs->sp; + regs->pc = fregs->pc; + regs->regs[29] = fregs->fp; + regs->regs[30] = fregs->lr; + return regs; +} + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 39a765c71f7e..819511ab75b7 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -173,6 +173,23 @@ static __always_inline struct pt_regs *ftrace_get_regs(struct ftrace_regs *fregs return arch_ftrace_get_regs(fregs); } +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) || \ + defined(CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST) + +static __always_inline struct pt_regs * +ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + /* + * If CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST=y, ftrace_regs memory + * layout is the same as pt_regs. So always returns that address. + * Since arch_ftrace_get_regs() will check some members and may return + * NULL, we can not use it. + */ + return &fregs->regs; +} + +#endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Sun Sep 24 13:37:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396972 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32FBC6ADE; Sun, 24 Sep 2023 13:37:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB2F9C433B9; Sun, 24 Sep 2023 13:37:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562651; bh=zU6UylpWMhPcntmE4WUBkG80m4hiVOExDKzxHQrqwhQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZqPcYsbqMLiLbpcfHCwh07roWj6RSCGhOTxB51sOEG8nlGLfdVQXdMrr6blomR6QZ SuCGpfrfjSRb+y93ncmw10Oz978PpXIvEtJlrWM6pRoWORsgv2wLCjRYHvAU0p31C6 GVhnp3XTxLRED9uh32ukhfRHXrh4+k1+gnP5HOoYfFFJyAZu4otNhFToVm9y2HieYl iAmTpbX+fG+UeQWRUj9ZVp7rbVHma8Hssi8Qcp7BXSfVxuf9mEtpw//tNLW1U6Dcem 9hRCT9cfL3R9u0K359sTpg1ZGlQVnHNpIi3m5CSX5z35uqzD5UNbUiim43xlAU4PGF 1eLs7YeqE8Hxg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 08/12] tracing: Add ftrace_fill_perf_regs() for perf event Date: Sun, 24 Sep 2023 22:37:25 +0900 Message-Id: <169556264478.146934.11293821252078942426.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Add ftrace_fill_perf_regs() which should be compatible with the perf_fetch_caller_regs(). In other words, the pt_regs returned from the ftrace_fill_perf_regs() must satisfy 'user_mode(regs) == false' and can be used for stack tracing. Signed-off-by: Masami Hiramatsu (Google) --- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/powerpc/include/asm/ftrace.h | 7 +++++++ arch/s390/include/asm/ftrace.h | 5 +++++ arch/x86/include/asm/ftrace.h | 7 +++++++ include/linux/ftrace.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 57 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 5ad24f315d52..5fbe8fe3e8a5 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -148,6 +148,13 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs) return regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->pc = (fregs)->pc; \ + (_regs)->regs[29] = (fregs)->fp; \ + (_regs)->sp = (fregs)->sp; \ + (_regs)->pstate = PSR_MODE_EL1h; \ + } while (0) + int ftrace_regs_query_register_offset(const char *name); int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 9e5a39b6a311..53fa39ef476b 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -52,6 +52,13 @@ static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs * return fregs->regs.msr ? &fregs->regs : NULL; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->result = 0; \ + (_regs)->nip = (fregs)->regs.nip; \ + (_regs)->gpr[1] = (fregs)->regs.gpr[1]; \ + asm volatile("mfmsr %0" : "=r" ((_regs)->msr)); \ + } while (0) + static __always_inline void ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, unsigned long ip) diff --git a/arch/s390/include/asm/ftrace.h b/arch/s390/include/asm/ftrace.h index 6de3bf4d0092..94ceb9564d44 100644 --- a/arch/s390/include/asm/ftrace.h +++ b/arch/s390/include/asm/ftrace.h @@ -99,6 +99,11 @@ ftrace_regs_set_instruction_pointer(struct ftrace_regs *fregs, #define ftrace_regs_query_register_offset(name) \ regs_query_register_offset(name) +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->psw.addr = (fregs)->regs.psw.addr; \ + (_regs)->gprs[15] = (fregs)->regs.gprs[15]; \ + } while (0) + #endif /* CONFIG_FUNCTION_TRACER */ #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 897cf02c20b1..800c2af8af01 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -48,6 +48,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) return &fregs->regs; } +#define arch_ftrace_fill_perf_regs(fregs, _regs) do { \ + (_regs)->ip = (fregs)->regs.ip; \ + (_regs)->sp = (fregs)->regs.sp; \ + (_regs)->cs = __KERNEL_CS; \ + (_regs)->flags = 0; \ + } while (0) + #define ftrace_regs_set_instruction_pointer(fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 819511ab75b7..962c3e6c5e3b 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -190,6 +190,37 @@ ftrace_partial_regs(struct ftrace_regs *fregs, struct pt_regs *regs) #endif /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS || CONFIG_HAVE_PT_REGS_TO_FTRACE_REGS_CAST */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + +/* + * Please define arch dependent pt_regs which compatible to the + * perf_arch_fetch_caller_regs() but based on ftrace_regs. + * This requires + * - user_mode(_regs) returns false (always kernel mode). + * - able to use the _regs for stack trace. + */ +#ifndef arch_ftrace_fill_perf_regs +/* As same as perf_arch_fetch_caller_regs(), do nothing by default */ +#define arch_ftrace_fill_perf_regs(fregs, _regs) do {} while (0) +#endif + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + arch_ftrace_fill_perf_regs(fregs, regs); + return regs; +} + +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ + +static __always_inline struct pt_regs * +ftrace_fill_perf_regs(struct ftrace_regs *fregs, struct pt_regs *regs) +{ + return &fregs->regs; +} + +#endif + /* * When true, the ftrace_regs_{get,set}_*() functions may be used on fregs. * Note: this can be true even when ftrace_get_regs() cannot provide a pt_regs. From patchwork Sun Sep 24 13:37:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396973 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 305CFCA53; Sun, 24 Sep 2023 13:37:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 053E5C433C7; Sun, 24 Sep 2023 13:37:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562664; bh=o6W3Qq5RhVZ0SuyRUhPk+n1TEye2b+9YcatTCtUd+IE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u9qqRKmNlNkqrouEsh1ocPk0NicfkGh62kAgsy+mtih9E1MzdbehuGZolMFwcuJjS CfiZ151TcUVYnFH7dsSFXU7wdzQguWvbA5l8Nbrkv646kf8G8EyNk43J7+J9+F6p0k HC8FwQVtaWn+2mlZkE5xwVEbxnJu0jzWXcRYNk7wdVg9U/rPZPGxBbnIo239mwNq+Q Y7kAkYYNb6VKDY5wRO40erURZSzZeGF0GMFUiQ49TsjtV0sL/AxzdlMxmewOCZtmmY RcaT2tP0ox1RFjFDBv7X/HIkPNxhdK/BugvoRcaSsf20dfv7ema73z8PYCViZmKXT8 ClbBtYg6fqtNg== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 09/12] tracing/fprobe: Enable fprobe events with CONFIG_DYNAMIC_FTRACE_WITH_ARGS Date: Sun, 24 Sep 2023 22:37:37 +0900 Message-Id: <169556265707.146934.9759320469271410104.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Allow fprobe events to be enabled with CONFIG_DYNAMIC_FTRACE_WITH_ARGS. With this change, fprobe events mostly use ftrace_regs instead of pt_regs. Note that if the arch doesn't enable HAVE_PT_REGS_COMPAT_FTRACE_REGS, fprobe events will not be able to be used from perf. Signed-off-by: Masami Hiramatsu (Google) --- Changes in v3: - introduce ftrace_regs_get_kernel_stack_nth(). - fix typo. Changes in v4: - Use per-cpu pt_regs stack and ftrace_partial_regs() for perf event. Changes in v5: - Use perf's per-cpu pt_regs. - Use new ftrace_fill_perf_regs() to fill perf required regs. --- include/linux/ftrace.h | 15 ++++++++ kernel/trace/Kconfig | 1 - kernel/trace/trace_fprobe.c | 74 ++++++++++++++++++++------------------- kernel/trace/trace_probe_tmpl.h | 2 + 4 files changed, 53 insertions(+), 39 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 962c3e6c5e3b..3748c8100924 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -233,6 +233,21 @@ static __always_inline bool ftrace_regs_has_args(struct ftrace_regs *fregs) return ftrace_get_regs(fregs) != NULL; } +#ifdef CONFIG_HAVE_REGS_AND_STACK_ACCESS_API +static __always_inline unsigned long +ftrace_regs_get_kernel_stack_nth(struct ftrace_regs *fregs, unsigned int nth) +{ + unsigned long *stackp; + + stackp = (unsigned long *)ftrace_regs_get_stack_pointer(fregs); + if (((unsigned long)(stackp + nth) & ~(THREAD_SIZE - 1)) == + ((unsigned long)stackp & ~(THREAD_SIZE - 1))) + return *(stackp + nth); + + return 0; +} +#endif /* CONFIG_HAVE_REGS_AND_STACK_ACCESS_API */ + #ifdef CONFIG_FUNCTION_TRACER extern int ftrace_enabled; diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index d56304276318..6fb4ecf8767d 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -679,7 +679,6 @@ config FPROBE_EVENTS select TRACING select PROBE_EVENTS select DYNAMIC_EVENTS - depends on DYNAMIC_FTRACE_WITH_REGS default y help This allows user to add tracing events on the function entry and diff --git a/kernel/trace/trace_fprobe.c b/kernel/trace/trace_fprobe.c index c60d0d9f1a95..882c220333cc 100644 --- a/kernel/trace/trace_fprobe.c +++ b/kernel/trace/trace_fprobe.c @@ -132,7 +132,7 @@ static int process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, void *base) { - struct pt_regs *regs = rec; + struct ftrace_regs *fregs = rec; unsigned long val; int ret; @@ -140,17 +140,17 @@ process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, /* 1st stage: get value from context */ switch (code->op) { case FETCH_OP_STACK: - val = regs_get_kernel_stack_nth(regs, code->param); + val = ftrace_regs_get_kernel_stack_nth(fregs, code->param); break; case FETCH_OP_STACKP: - val = kernel_stack_pointer(regs); + val = ftrace_regs_get_stack_pointer(fregs); break; case FETCH_OP_RETVAL: - val = regs_return_value(regs); + val = ftrace_regs_return_value(fregs); break; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API case FETCH_OP_ARG: - val = regs_get_kernel_argument(regs, code->param); + val = ftrace_regs_get_argument(fregs, code->param); break; #endif case FETCH_NOP_SYMBOL: /* Ignore a place holder */ @@ -170,7 +170,7 @@ NOKPROBE_SYMBOL(process_fetch_insn) /* function entry handler */ static nokprobe_inline void __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs, + struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fentry_trace_entry_head *entry; @@ -184,36 +184,36 @@ __fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->ip = entry_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fentry_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fentry_trace_func(tf, entry_ip, regs, link->file); + __fentry_trace_func(tf, entry_ip, fregs, link->file); } NOKPROBE_SYMBOL(fentry_trace_func); /* Kretprobe handler */ static nokprobe_inline void __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs, + unsigned long ret_ip, struct ftrace_regs *fregs, struct trace_event_file *trace_file) { struct fexit_trace_entry_head *entry; @@ -227,60 +227,63 @@ __fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, if (trace_trigger_soft_disabled(trace_file)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); entry = trace_event_buffer_reserve(&fbuffer, trace_file, sizeof(*entry) + tf->tp.size + dsize); if (!entry) return; - fbuffer.regs = regs; + fbuffer.regs = ftrace_get_regs(fregs); entry = fbuffer.entry = ring_buffer_event_data(fbuffer.event); entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); trace_event_buffer_commit(&fbuffer); } static void fexit_trace_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct event_file_link *link; trace_probe_for_each_link_rcu(link, &tf->tp) - __fexit_trace_func(tf, entry_ip, ret_ip, regs, link->file); + __fexit_trace_func(tf, entry_ip, ret_ip, fregs, link->file); } NOKPROBE_SYMBOL(fexit_trace_func); #ifdef CONFIG_PERF_EVENTS static int fentry_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - struct pt_regs *regs) + struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fentry_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return 0; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return 0; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->ip = entry_ip; memset(&entry[1], 0, dsize); - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); return 0; @@ -289,30 +292,33 @@ NOKPROBE_SYMBOL(fentry_perf_func); static void fexit_perf_func(struct trace_fprobe *tf, unsigned long entry_ip, - unsigned long ret_ip, struct pt_regs *regs) + unsigned long ret_ip, struct ftrace_regs *fregs) { struct trace_event_call *call = trace_probe_event_call(&tf->tp); struct fexit_trace_entry_head *entry; struct hlist_head *head; int size, __size, dsize; + struct pt_regs *regs; int rctx; head = this_cpu_ptr(call->perf_events); if (hlist_empty(head)) return; - dsize = __get_data_size(&tf->tp, regs); + dsize = __get_data_size(&tf->tp, fregs); __size = sizeof(*entry) + tf->tp.size + dsize; size = ALIGN(__size + sizeof(u32), sizeof(u64)); size -= sizeof(u32); - entry = perf_trace_buf_alloc(size, NULL, &rctx); + entry = perf_trace_buf_alloc(size, ®s, &rctx); if (!entry) return; + regs = ftrace_fill_perf_regs(fregs, regs); + entry->func = entry_ip; entry->ret_ip = ret_ip; - store_trace_args(&entry[1], &tf->tp, regs, sizeof(*entry), dsize); + store_trace_args(&entry[1], &tf->tp, fregs, sizeof(*entry), dsize); perf_trace_buf_submit(entry, size, rctx, call->event.type, 1, regs, head, NULL); } @@ -324,17 +330,14 @@ static int fentry_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); int ret = 0; - if (!regs) - return 0; - if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fentry_trace_func(tf, entry_ip, regs); + fentry_trace_func(tf, entry_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - ret = fentry_perf_func(tf, entry_ip, regs); + ret = fentry_perf_func(tf, entry_ip, fregs); #endif return ret; } @@ -345,16 +348,13 @@ static void fexit_dispatcher(struct fprobe *fp, unsigned long entry_ip, void *entry_data) { struct trace_fprobe *tf = container_of(fp, struct trace_fprobe, fp); - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; if (trace_probe_test_flag(&tf->tp, TP_FLAG_TRACE)) - fexit_trace_func(tf, entry_ip, ret_ip, regs); + fexit_trace_func(tf, entry_ip, ret_ip, fregs); + #ifdef CONFIG_PERF_EVENTS if (trace_probe_test_flag(&tf->tp, TP_FLAG_PROFILE)) - fexit_perf_func(tf, entry_ip, ret_ip, regs); + fexit_perf_func(tf, entry_ip, ret_ip, fregs); #endif } NOKPROBE_SYMBOL(fexit_dispatcher); diff --git a/kernel/trace/trace_probe_tmpl.h b/kernel/trace/trace_probe_tmpl.h index 3935b347f874..05445a745a07 100644 --- a/kernel/trace/trace_probe_tmpl.h +++ b/kernel/trace/trace_probe_tmpl.h @@ -232,7 +232,7 @@ process_fetch_insn_bottom(struct fetch_insn *code, unsigned long val, /* Sum up total data length for dynamic arrays (strings) */ static nokprobe_inline int -__get_data_size(struct trace_probe *tp, struct pt_regs *regs) +__get_data_size(struct trace_probe *tp, void *regs) { struct probe_arg *arg; int i, len, ret = 0; From patchwork Sun Sep 24 13:37:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396974 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 445D5CA53; Sun, 24 Sep 2023 13:37:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5909EC433C8; Sun, 24 Sep 2023 13:37:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562676; bh=FEfGDz6clxZ9PLKtBAXOYmli6LSrFixQDspF7u/Z4KA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NUhxY4N8h3ShSDm2qYxveksvNl9zrO2XDzm2UaaVojXVnQrGNh+H6DYGuxn/NNlqj btDQMa8Lp6z5LAEwgsx1ooN6yll4I1549gag/udsMHgDxg8jVgLUELsvsRiwwaD8SZ d39JuIuxXZKH2zhdTwqlI7KRzuTedHAYT/fr9E5NsPCCHlDWTK01If8Dy6tW3UvD9b ecguWWebaF78uUGf3swAMjGj3Tm/7gKa4qZqCs6hPskHFYoJdF1UC/YekLBEUPUpr5 AadOSeUAGI673e+JmebPT3pq1IXStV6K/Jl8n4n38MvkEddiFZQKSX6geivcghXsbJ ntOcH7s15m+Gw== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 10/12] bpf: Enable kprobe_multi feature if CONFIG_FPROBE is enabled Date: Sun, 24 Sep 2023 22:37:49 +0900 Message-Id: <169556266942.146934.6508273802845934438.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Enable kprobe_multi feature if CONFIG_FPROBE is enabled. The pt_regs is converted from ftrace_regs by ftrace_partial_regs(), thus some registers may always returns 0. But it should be enough for function entry (access arguments) and exit (access return value). Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- kernel/trace/bpf_trace.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 8bb003ce7bb2..808acd56e432 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2501,7 +2501,7 @@ static int __init bpf_event_init(void) fs_initcall(bpf_event_init); #endif /* CONFIG_MODULES */ -#if defined(CONFIG_FPROBE) && defined(CONFIG_DYNAMIC_FTRACE_WITH_REGS) +#ifdef CONFIG_FPROBE struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; @@ -2524,6 +2524,8 @@ struct user_syms { char *buf; }; +static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs); + static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt) { unsigned long __user usymbol; @@ -2700,13 +2702,14 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx) static int kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, - unsigned long entry_ip, struct pt_regs *regs) + unsigned long entry_ip, struct ftrace_regs *fregs) { struct bpf_kprobe_multi_run_ctx run_ctx = { .link = link, .entry_ip = entry_ip, }; struct bpf_run_ctx *old_run_ctx; + struct pt_regs *regs; int err; if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) { @@ -2716,6 +2719,7 @@ kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link, migrate_disable(); rcu_read_lock(); + regs = ftrace_partial_regs(fregs, this_cpu_ptr(&bpf_kprobe_multi_pt_regs)); old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); err = bpf_prog_run(link->link.prog, regs); bpf_reset_run_ctx(old_run_ctx); @@ -2733,13 +2737,9 @@ kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return 0; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); return 0; } @@ -2749,13 +2749,9 @@ kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip, void *data) { struct bpf_kprobe_multi_link *link; - struct pt_regs *regs = ftrace_get_regs(fregs); - - if (!regs) - return; link = container_of(fp, struct bpf_kprobe_multi_link, fp); - kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), regs); + kprobe_multi_link_prog_run(link, get_entry_ip(fentry_ip), fregs); } static int symbols_cmp_r(const void *a, const void *b, const void *priv) @@ -2996,7 +2992,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr kvfree(cookies); return err; } -#else /* !CONFIG_FPROBE || !CONFIG_DYNAMIC_FTRACE_WITH_REGS */ +#else /* !CONFIG_FPROBE */ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) { return -EOPNOTSUPP; From patchwork Sun Sep 24 13:38:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396975 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF416CA4C; Sun, 24 Sep 2023 13:38:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80C1CC433C9; Sun, 24 Sep 2023 13:38:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562688; bh=CLUahLECFFMThaMlg6kBR2ihukQ2SmAsdwcA4AdH8rk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pV93df4UU5q1m05W20xMqKCAie39MI7vyF8y2HIqDTWOsxFBvDzv2eP9xz/EzI0oF e0r4Iuz1rqZhPm2sX88w62J6AMOUShFxf9hVU3p8esqhejBVuWBiqvGfDkIsyv8cdz V02/uAiHnmVFhuk0yj086D+g54+ycpHy3bAB3VryqWXWYY/mxSa66sxl400CIEnKNH tHNbC2rokwrO5AowIFkrUaC4WenWq9wlV/3DYfIR5XkbfMCb0g0IcWm9n45dsLd5Mb 4bF04iwvGb3sWZPighPuFFn7s7wBq6B5nrMjhIYKf1I0uJ+RR6vuiW51e7xhy4OvoC ghBfS8cf5aNOA== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 11/12] Documentation: probes: Update fprobe document to use ftrace_regs Date: Sun, 24 Sep 2023 22:38:01 +0900 Message-Id: <169556268150.146934.2968626447452628416.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Update fprobe document so that the entry/exit handler uses ftrace_regs instead of pt_regs. Signed-off-by: Masami Hiramatsu (Google) Acked-by: Florent Revest --- Documentation/trace/fprobe.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst index 196f52386aaa..64ef522f7a64 100644 --- a/Documentation/trace/fprobe.rst +++ b/Documentation/trace/fprobe.rst @@ -91,9 +91,9 @@ The prototype of the entry/exit callback function are as follows: .. code-block:: c - int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + int entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); - void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct pt_regs *regs, void *entry_data); + void exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long ret_ip, struct ftrace_regs *fregs, void *entry_data); Note that the @entry_ip is saved at function entry and passed to exit handler. If the entry callback function returns !0, the corresponding exit callback will be cancelled. @@ -112,12 +112,10 @@ If the entry callback function returns !0, the corresponding exit callback will This is the return address that the traced function will return to, somewhere in the caller. This can be used at both entry and exit. -@regs - This is the `pt_regs` data structure at the entry and exit. Note that - the instruction pointer of @regs may be different from the @entry_ip - in the entry_handler. If you need traced instruction pointer, you need - to use @entry_ip. On the other hand, in the exit_handler, the instruction - pointer of @regs is set to the current return address. +@fregs + This is the `ftrace_regs` data structure at the entry and exit. Note that + the instruction pointer of @fregs may be incorrect in entry handler and + exit handler, so you have to use @entry_ip and @ret_ip instead. @entry_data This is a local storage to share the data between entry and exit handlers. From patchwork Sun Sep 24 13:38:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 13396976 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3C4ECA4C; Sun, 24 Sep 2023 13:38:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AE0B3C433C7; Sun, 24 Sep 2023 13:38:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695562700; bh=Ylc3TSXa5ifgz7bjH+Ib7u/kImYYyrGh5B2vMyFRLTg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Noce4mwmQNAoIZUogYsYCHEBjWbN2VvgLO6lbPFKdhnc1rWxsAAm5qcB/hlWV7MzL 6M8uaClcpljpII0+yuMjMcfoXR2E4R03fEuYY1TBanEt2HyxGqBltgcZGu2xlwMcw1 GT1nBUt9ctylc2zKowKw8YXTloz3jhxqUqutXd+I1UwjeJr3Fz+fgwqCp1jn6obYvX EbnAwCmrhQ7uQFXszvaCznoy6tg4E3WMRuPOSYmv8+iowSy3I/m+5sghxqx/z/F2qm QL6KhyoirapL2JQ/JB64iPfjQKFoZV2VO2aGFRDutjDqBmqcL1+dhLeHtBXi9ik/4B gmhXMtk+X90zQ== From: "Masami Hiramatsu (Google)" To: Alexei Starovoitov , Steven Rostedt , Florent Revest Cc: linux-trace-kernel@vger.kernel.org, LKML , Martin KaFai Lau , bpf , Sven Schnelle , Alexei Starovoitov , Jiri Olsa , Arnaldo Carvalho de Melo , Daniel Borkmann , Alan Maguire , Mark Rutland , Peter Zijlstra , Thomas Gleixner , Guo Ren Subject: [PATCH v5 12/12] Documentation: tracing: Add a note about argument and retval access Date: Sun, 24 Sep 2023 22:38:14 +0900 Message-Id: <169556269377.146934.14829235476649685954.stgit@devnote2> X-Mailer: git-send-email 2.34.1 In-Reply-To: <169556254640.146934.5654329452696494756.stgit@devnote2> References: <169556254640.146934.5654329452696494756.stgit@devnote2> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Masami Hiramatsu (Google) Add a note about the argument and return value accecss will be best effort. Depending on the type, it will be passed via stack or a pair of the registers, but $argN and $retval only support the single register access. Suggested-by: Alexei Starovoitov Signed-off-by: Masami Hiramatsu (Google) --- Documentation/trace/fprobetrace.rst | 8 ++++++-- Documentation/trace/kprobetrace.rst | 8 ++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/Documentation/trace/fprobetrace.rst b/Documentation/trace/fprobetrace.rst index 8e9bebcf0a2e..e35e6b18df40 100644 --- a/Documentation/trace/fprobetrace.rst +++ b/Documentation/trace/fprobetrace.rst @@ -59,8 +59,12 @@ Synopsis of fprobe-events and bitfield are supported. (\*1) This is available only when BTF is enabled. - (\*2) only for the probe on function entry (offs == 0). - (\*3) only for return probe. + (\*2) only for the probe on function entry (offs == 0). Note, this argument access + is best effort, because depending on the argument type, it may be passed on + the stack. But this only support the arguments via registers. + (\*3) only for return probe. Note that this is also best effort. Depending on the + return value type, it might be passed via a pair of registers. But this only + accesses one register. (\*4) this is useful for fetching a field of data structures. (\*5) "u" means user-space dereference. diff --git a/Documentation/trace/kprobetrace.rst b/Documentation/trace/kprobetrace.rst index 8a2dfee38145..bf9cecb69fc9 100644 --- a/Documentation/trace/kprobetrace.rst +++ b/Documentation/trace/kprobetrace.rst @@ -61,8 +61,12 @@ Synopsis of kprobe_events (x8/x16/x32/x64), "char", "string", "ustring", "symbol", "symstr" and bitfield are supported. - (\*1) only for the probe on function entry (offs == 0). - (\*2) only for return probe. + (\*1) only for the probe on function entry (offs == 0). Note, this argument access + is best effort, because depending on the argument type, it may be passed on + the stack. But this only support the arguments via registers. + (\*2) only for return probe. Note that this is also best effort. Depending on the + return value type, it might be passed via a pair of registers. But this only + accesses one register. (\*3) this is useful for fetching a field of data structures. (\*4) "u" means user-space dereference. See :ref:`user_mem_access`.