From patchwork Wed Feb 1 16:34:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124718 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88F23C636D7 for ; Wed, 1 Feb 2023 16:35:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232303AbjBAQff (ORCPT ); Wed, 1 Feb 2023 11:35:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232127AbjBAQfd (ORCPT ); Wed, 1 Feb 2023 11:35:33 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6575B744BA for ; Wed, 1 Feb 2023 08:35:25 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id n13so6113824wmr.4 for ; Wed, 01 Feb 2023 08:35:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mWFrI/ZwPkf24GvAi4zS93lAjO18wCo9ugFdCRD9vug=; b=R/wJRycxVlXr98cXymYAxYn59y2w9nV7JoCbhkjevWZR1Cd6qPhPtO6JgzG3VxSI9Q GKjV0BjUpT2ySQyMbsatvRUEOMMnI83NorJqYQruLmLn9Dd7v4V8prfj6ZAjCD1ul2/+ TWRYbF22VUKSRARcJ2XvXOHcRQs/Rg82E+MiE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mWFrI/ZwPkf24GvAi4zS93lAjO18wCo9ugFdCRD9vug=; b=BOw4609MZYun9NwGxmMz1fK4YhCsARLUjociryHdNDfsFIg/B8tNvYnaGHAko3+/cM cvrZhg62txuzysyvVWQ3nDFJ7PMjvE63bJU/m/aZd7D50He+g1zmbCgHIYhlnNOsM9Ex xAipDxtj1reAvaZbg7l1Wy+CLJhmPZUfRo00rwMCAkZpVjDL/VCwvYOA06AUpLjHsR8r fHtp+xJ8g+JJlT0+Ckrxmq4ke2px330Z9ypN89B5WGJvSBtsDbaRa3OzYIan1CmJQgsX Tfcago5tD+LMT9E6D55fA/SgWRjgBPA+yUGToxpJ+Pqd+GXvC0BORHvVGiOYqbNLdOP3 dKRQ== X-Gm-Message-State: AO0yUKUxI/GfDPatO8xpFxrqNTHeZ5Zoo1eGGrU4+q0ZzntI6qSqSeyo tPP1xFh4C0oRG0iEuZtlR4pTog== X-Google-Smtp-Source: AK7set+/RTqNE3i6RzBFCPFMdJxcxKMsNBsU7or9P4oRwy2TN0ZjlO+eOnMfvdrBDV3haIG0hgP9Kw== X-Received: by 2002:a05:600c:3544:b0:3dd:1c46:b92 with SMTP id i4-20020a05600c354400b003dd1c460b92mr2882937wmq.16.1675269323861; Wed, 01 Feb 2023 08:35:23 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:23 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 1/8] ftrace: Replace uses of _ftrace_direct APIs with _ftrace_direct_multi Date: Wed, 1 Feb 2023 17:34:13 +0100 Message-Id: <20230201163420.1579014-2-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org The _multi API requires that users keep their own ops and can enforce that an op is only associated to one direct call. Signed-off-by: Florent Revest Acked-by: Mark Rutland Tested-by: Mark Rutland --- kernel/trace/trace_selftest.c | 9 ++++++--- samples/ftrace/ftrace-direct-modify.c | 11 +++++++---- samples/ftrace/ftrace-direct-too.c | 12 +++++++----- samples/ftrace/ftrace-direct.c | 12 +++++++----- 4 files changed, 27 insertions(+), 17 deletions(-) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index ff0536cea968..9b7f10cbc51d 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -806,6 +806,9 @@ trace_selftest_startup_function_graph(struct tracer *trace, int ret; unsigned long count; char *func_name __maybe_unused; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + struct ftrace_ops direct = {}; +#endif #ifdef CONFIG_DYNAMIC_FTRACE if (ftrace_filter_param) { @@ -870,8 +873,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, * Register direct function together with graph tracer * and make sure we get graph trace. */ - ret = register_ftrace_direct((unsigned long) DYN_FTRACE_TEST_NAME, - (unsigned long) trace_direct_tramp); + ftrace_set_filter_ip(&direct, (unsigned long)DYN_FTRACE_TEST_NAME, 0, 0); + ret = register_ftrace_direct_multi(&direct, (unsigned long)trace_direct_tramp); if (ret) goto out; @@ -891,7 +894,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, unregister_ftrace_graph(&fgraph_ops); - ret = unregister_ftrace_direct((unsigned long) DYN_FTRACE_TEST_NAME, + ret = unregister_ftrace_direct_multi(&direct, (unsigned long) trace_direct_tramp); if (ret) goto out; diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c index de5a0f67f320..e1e6c286970c 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -96,6 +96,8 @@ asm ( #endif /* CONFIG_S390 */ +static struct ftrace_ops direct; + static unsigned long my_tramp = (unsigned long)my_tramp1; static unsigned long tramps[2] = { (unsigned long)my_tramp1, @@ -114,7 +116,7 @@ static int simple_thread(void *arg) if (ret) continue; t ^= 1; - ret = modify_ftrace_direct(my_ip, my_tramp, tramps[t]); + ret = modify_ftrace_direct_multi(&direct, tramps[t]); if (!ret) my_tramp = tramps[t]; WARN_ON_ONCE(ret); @@ -129,7 +131,8 @@ static int __init ftrace_direct_init(void) { int ret; - ret = register_ftrace_direct(my_ip, my_tramp); + ftrace_set_filter_ip(&direct, (unsigned long) my_ip, 0, 0); + ret = register_ftrace_direct_multi(&direct, my_tramp); if (!ret) simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); return ret; @@ -138,12 +141,12 @@ static int __init ftrace_direct_init(void) static void __exit ftrace_direct_exit(void) { kthread_stop(simple_tsk); - unregister_ftrace_direct(my_ip, my_tramp); + unregister_ftrace_direct_multi(&direct, my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct()"); +MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct_multi()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c index e13fb59a2b47..0e907092e2c0 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -70,21 +70,23 @@ asm ( #endif /* CONFIG_S390 */ +static struct ftrace_ops direct; + static int __init ftrace_direct_init(void) { - return register_ftrace_direct((unsigned long)handle_mm_fault, - (unsigned long)my_tramp); + ftrace_set_filter_ip(&direct, (unsigned long) handle_mm_fault, 0, 0); + + return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); } static void __exit ftrace_direct_exit(void) { - unregister_ftrace_direct((unsigned long)handle_mm_fault, - (unsigned long)my_tramp); + unregister_ftrace_direct_multi(&direct, (unsigned long)my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Another example use case of using register_ftrace_direct()"); +MODULE_DESCRIPTION("Another example use case of using register_ftrace_direct_multi()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 1f769d0db20f..e446c38f6b58 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -63,21 +63,23 @@ asm ( #endif /* CONFIG_S390 */ +static struct ftrace_ops direct; + static int __init ftrace_direct_init(void) { - return register_ftrace_direct((unsigned long)wake_up_process, - (unsigned long)my_tramp); + ftrace_set_filter_ip(&direct, (unsigned long) wake_up_process, 0, 0); + + return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); } static void __exit ftrace_direct_exit(void) { - unregister_ftrace_direct((unsigned long)wake_up_process, - (unsigned long)my_tramp); + unregister_ftrace_direct_multi(&direct, (unsigned long)my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Example use case of using register_ftrace_direct()"); +MODULE_DESCRIPTION("Example use case of using register_ftrace_direct_multi()"); MODULE_LICENSE("GPL"); From patchwork Wed Feb 1 16:34:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124719 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5209DC38142 for ; Wed, 1 Feb 2023 16:35:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232367AbjBAQfi (ORCPT ); Wed, 1 Feb 2023 11:35:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232272AbjBAQfh (ORCPT ); Wed, 1 Feb 2023 11:35:37 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2A7879F27 for ; Wed, 1 Feb 2023 08:35:26 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id bg26so7276999wmb.0 for ; Wed, 01 Feb 2023 08:35:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X+Q37Omt/MAOZB85dyYH0VxCAqGyfL1V0CNFe50alWw=; b=PwWcZyLy+Z19hGnMU/rvK3uCeqYMJcoOHacKNeZ2E5eLBXT6GqsH1Mt31hw0qtXIxl aWTFapycW8vcGH3EpuYLmCKLHN5VfiMVOtmfowc1hqteDstL5qRq9q7/8c1pJFIGvahS gvTvMiculTkSVln7xHYS3kSRkQyOfyWWvdZI0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X+Q37Omt/MAOZB85dyYH0VxCAqGyfL1V0CNFe50alWw=; b=FCH5wnkIhMOq4o88lhkTMnzNxwSIfHv90Tx1b3ZaYpYbZJwdgwlenw7ErI/M3beOR9 m9Pre3axHn00Ljtxkv1VcW+akpi8Gc6EGxiIFgfh2T8TQQ35ZTiU0bbycO8ehD3jYycT jEyWth4gdje/FXSxErGdzhc089Psq/FmFE6j3zipPMXCJJ/PuJg/NQeW9oEri1DYRVZp k3oREVSfW6l4FDAhkfcVIKaUIXehWWPobGNWh1EhsaTS20Ii0AjudD2gsHyMObXpVyZU w8qCtfhRJsPqoPJGXlykBoRROjziilqB3cNt5zhBmdlqJNN6XJFdLV6GjCfVP1y+jXqh GBrQ== X-Gm-Message-State: AO0yUKUoziSp1U0FJs+OJyfTz2Y08Ak23gMtC8kcbbx3jjEkezidxHkk 1v12ioFN9r2Yrl7AI4gKcGx5rQ== X-Google-Smtp-Source: AK7set9awnPFkuzEz1Qn8JL87YeR7SuUxoCuq0lSCj5ljB3NM5/2b/wlbhBsLnwXxTbEcLRQcu8xnw== X-Received: by 2002:a7b:c85a:0:b0:3d2:392e:905f with SMTP id c26-20020a7bc85a000000b003d2392e905fmr2646774wml.24.1675269325301; Wed, 01 Feb 2023 08:35:25 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:24 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 2/8] ftrace: Remove the legacy _ftrace_direct API Date: Wed, 1 Feb 2023 17:34:14 +0100 Message-Id: <20230201163420.1579014-3-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This API relies on a single global ops, used for all direct calls registered with it. However, to implement arm64 direct calls, we need each ops to point to a single direct call trampoline. Signed-off-by: Florent Revest Acked-by: Mark Rutland Tested-by: Mark Rutland --- include/linux/ftrace.h | 32 ---- kernel/trace/ftrace.c | 393 ----------------------------------------- 2 files changed, 425 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 366c730beaa3..2d7c85e47c2d 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -397,14 +397,6 @@ struct ftrace_func_entry { #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS extern int ftrace_direct_func_count; -int register_ftrace_direct(unsigned long ip, unsigned long addr); -int unregister_ftrace_direct(unsigned long ip, unsigned long addr); -int modify_ftrace_direct(unsigned long ip, unsigned long old_addr, unsigned long new_addr); -struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr); -int ftrace_modify_direct_caller(struct ftrace_func_entry *entry, - struct dyn_ftrace *rec, - unsigned long old_addr, - unsigned long new_addr); unsigned long ftrace_find_rec_direct(unsigned long ip); int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); @@ -414,30 +406,6 @@ int modify_ftrace_direct_multi_nolock(struct ftrace_ops *ops, unsigned long addr #else struct ftrace_ops; # define ftrace_direct_func_count 0 -static inline int register_ftrace_direct(unsigned long ip, unsigned long addr) -{ - return -ENOTSUPP; -} -static inline int unregister_ftrace_direct(unsigned long ip, unsigned long addr) -{ - return -ENOTSUPP; -} -static inline int modify_ftrace_direct(unsigned long ip, - unsigned long old_addr, unsigned long new_addr) -{ - return -ENOTSUPP; -} -static inline struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr) -{ - return NULL; -} -static inline int ftrace_modify_direct_caller(struct ftrace_func_entry *entry, - struct dyn_ftrace *rec, - unsigned long old_addr, - unsigned long new_addr) -{ - return -ENODEV; -} static inline unsigned long ftrace_find_rec_direct(unsigned long ip) { return 0; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index e634b80f49d1..5efe201428fa 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2585,20 +2585,6 @@ static void call_direct_funcs(unsigned long ip, unsigned long pip, arch_ftrace_set_direct_caller(fregs, addr); } - -struct ftrace_ops direct_ops = { - .func = call_direct_funcs, - .flags = FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_REGS - | FTRACE_OPS_FL_PERMANENT, - /* - * By declaring the main trampoline as this trampoline - * it will never have one allocated for it. Allocated - * trampolines should not call direct functions. - * The direct_ops should only be called by the builtin - * ftrace_regs_caller trampoline. - */ - .trampoline = FTRACE_REGS_ADDR, -}; #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ /** @@ -5295,387 +5281,8 @@ struct ftrace_direct_func { static LIST_HEAD(ftrace_direct_funcs); -/** - * ftrace_find_direct_func - test an address if it is a registered direct caller - * @addr: The address of a registered direct caller - * - * This searches to see if a ftrace direct caller has been registered - * at a specific address, and if so, it returns a descriptor for it. - * - * This can be used by architecture code to see if an address is - * a direct caller (trampoline) attached to a fentry/mcount location. - * This is useful for the function_graph tracer, as it may need to - * do adjustments if it traced a location that also has a direct - * trampoline attached to it. - */ -struct ftrace_direct_func *ftrace_find_direct_func(unsigned long addr) -{ - struct ftrace_direct_func *entry; - bool found = false; - - /* May be called by fgraph trampoline (protected by rcu tasks) */ - list_for_each_entry_rcu(entry, &ftrace_direct_funcs, next) { - if (entry->addr == addr) { - found = true; - break; - } - } - if (found) - return entry; - - return NULL; -} - -static struct ftrace_direct_func *ftrace_alloc_direct_func(unsigned long addr) -{ - struct ftrace_direct_func *direct; - - direct = kmalloc(sizeof(*direct), GFP_KERNEL); - if (!direct) - return NULL; - direct->addr = addr; - direct->count = 0; - list_add_rcu(&direct->next, &ftrace_direct_funcs); - ftrace_direct_func_count++; - return direct; -} - static int register_ftrace_function_nolock(struct ftrace_ops *ops); -/** - * register_ftrace_direct - Call a custom trampoline directly - * @ip: The address of the nop at the beginning of a function - * @addr: The address of the trampoline to call at @ip - * - * This is used to connect a direct call from the nop location (@ip) - * at the start of ftrace traced functions. The location that it calls - * (@addr) must be able to handle a direct call, and save the parameters - * of the function being traced, and restore them (or inject new ones - * if needed), before returning. - * - * Returns: - * 0 on success - * -EBUSY - Another direct function is already attached (there can be only one) - * -ENODEV - @ip does not point to a ftrace nop location (or not supported) - * -ENOMEM - There was an allocation failure. - */ -int register_ftrace_direct(unsigned long ip, unsigned long addr) -{ - struct ftrace_direct_func *direct; - struct ftrace_func_entry *entry; - struct ftrace_hash *free_hash = NULL; - struct dyn_ftrace *rec; - int ret = -ENODEV; - - mutex_lock(&direct_mutex); - - ip = ftrace_location(ip); - if (!ip) - goto out_unlock; - - /* See if there's a direct function at @ip already */ - ret = -EBUSY; - if (ftrace_find_rec_direct(ip)) - goto out_unlock; - - ret = -ENODEV; - rec = lookup_rec(ip, ip); - if (!rec) - goto out_unlock; - - /* - * Check if the rec says it has a direct call but we didn't - * find one earlier? - */ - if (WARN_ON(rec->flags & FTRACE_FL_DIRECT)) - goto out_unlock; - - /* Make sure the ip points to the exact record */ - if (ip != rec->ip) { - ip = rec->ip; - /* Need to check this ip for a direct. */ - if (ftrace_find_rec_direct(ip)) - goto out_unlock; - } - - ret = -ENOMEM; - direct = ftrace_find_direct_func(addr); - if (!direct) { - direct = ftrace_alloc_direct_func(addr); - if (!direct) - goto out_unlock; - } - - entry = ftrace_add_rec_direct(ip, addr, &free_hash); - if (!entry) - goto out_unlock; - - ret = ftrace_set_filter_ip(&direct_ops, ip, 0, 0); - - if (!ret && !(direct_ops.flags & FTRACE_OPS_FL_ENABLED)) { - ret = register_ftrace_function_nolock(&direct_ops); - if (ret) - ftrace_set_filter_ip(&direct_ops, ip, 1, 0); - } - - if (ret) { - remove_hash_entry(direct_functions, entry); - kfree(entry); - if (!direct->count) { - list_del_rcu(&direct->next); - synchronize_rcu_tasks(); - kfree(direct); - if (free_hash) - free_ftrace_hash(free_hash); - free_hash = NULL; - ftrace_direct_func_count--; - } - } else { - direct->count++; - } - out_unlock: - mutex_unlock(&direct_mutex); - - if (free_hash) { - synchronize_rcu_tasks(); - free_ftrace_hash(free_hash); - } - - return ret; -} -EXPORT_SYMBOL_GPL(register_ftrace_direct); - -static struct ftrace_func_entry *find_direct_entry(unsigned long *ip, - struct dyn_ftrace **recp) -{ - struct ftrace_func_entry *entry; - struct dyn_ftrace *rec; - - rec = lookup_rec(*ip, *ip); - if (!rec) - return NULL; - - entry = __ftrace_lookup_ip(direct_functions, rec->ip); - if (!entry) { - WARN_ON(rec->flags & FTRACE_FL_DIRECT); - return NULL; - } - - WARN_ON(!(rec->flags & FTRACE_FL_DIRECT)); - - /* Passed in ip just needs to be on the call site */ - *ip = rec->ip; - - if (recp) - *recp = rec; - - return entry; -} - -int unregister_ftrace_direct(unsigned long ip, unsigned long addr) -{ - struct ftrace_direct_func *direct; - struct ftrace_func_entry *entry; - struct ftrace_hash *hash; - int ret = -ENODEV; - - mutex_lock(&direct_mutex); - - ip = ftrace_location(ip); - if (!ip) - goto out_unlock; - - entry = find_direct_entry(&ip, NULL); - if (!entry) - goto out_unlock; - - hash = direct_ops.func_hash->filter_hash; - if (hash->count == 1) - unregister_ftrace_function(&direct_ops); - - ret = ftrace_set_filter_ip(&direct_ops, ip, 1, 0); - - WARN_ON(ret); - - remove_hash_entry(direct_functions, entry); - - direct = ftrace_find_direct_func(addr); - if (!WARN_ON(!direct)) { - /* This is the good path (see the ! before WARN) */ - direct->count--; - WARN_ON(direct->count < 0); - if (!direct->count) { - list_del_rcu(&direct->next); - synchronize_rcu_tasks(); - kfree(direct); - kfree(entry); - ftrace_direct_func_count--; - } - } - out_unlock: - mutex_unlock(&direct_mutex); - - return ret; -} -EXPORT_SYMBOL_GPL(unregister_ftrace_direct); - -static struct ftrace_ops stub_ops = { - .func = ftrace_stub, -}; - -/** - * ftrace_modify_direct_caller - modify ftrace nop directly - * @entry: The ftrace hash entry of the direct helper for @rec - * @rec: The record representing the function site to patch - * @old_addr: The location that the site at @rec->ip currently calls - * @new_addr: The location that the site at @rec->ip should call - * - * An architecture may overwrite this function to optimize the - * changing of the direct callback on an ftrace nop location. - * This is called with the ftrace_lock mutex held, and no other - * ftrace callbacks are on the associated record (@rec). Thus, - * it is safe to modify the ftrace record, where it should be - * currently calling @old_addr directly, to call @new_addr. - * - * This is called with direct_mutex locked. - * - * Safety checks should be made to make sure that the code at - * @rec->ip is currently calling @old_addr. And this must - * also update entry->direct to @new_addr. - */ -int __weak ftrace_modify_direct_caller(struct ftrace_func_entry *entry, - struct dyn_ftrace *rec, - unsigned long old_addr, - unsigned long new_addr) -{ - unsigned long ip = rec->ip; - int ret; - - lockdep_assert_held(&direct_mutex); - - /* - * The ftrace_lock was used to determine if the record - * had more than one registered user to it. If it did, - * we needed to prevent that from changing to do the quick - * switch. But if it did not (only a direct caller was attached) - * then this function is called. But this function can deal - * with attached callers to the rec that we care about, and - * since this function uses standard ftrace calls that take - * the ftrace_lock mutex, we need to release it. - */ - mutex_unlock(&ftrace_lock); - - /* - * By setting a stub function at the same address, we force - * the code to call the iterator and the direct_ops helper. - * This means that @ip does not call the direct call, and - * we can simply modify it. - */ - ret = ftrace_set_filter_ip(&stub_ops, ip, 0, 0); - if (ret) - goto out_lock; - - ret = register_ftrace_function_nolock(&stub_ops); - if (ret) { - ftrace_set_filter_ip(&stub_ops, ip, 1, 0); - goto out_lock; - } - - entry->direct = new_addr; - - /* - * By removing the stub, we put back the direct call, calling - * the @new_addr. - */ - unregister_ftrace_function(&stub_ops); - ftrace_set_filter_ip(&stub_ops, ip, 1, 0); - - out_lock: - mutex_lock(&ftrace_lock); - - return ret; -} - -/** - * modify_ftrace_direct - Modify an existing direct call to call something else - * @ip: The instruction pointer to modify - * @old_addr: The address that the current @ip calls directly - * @new_addr: The address that the @ip should call - * - * This modifies a ftrace direct caller at an instruction pointer without - * having to disable it first. The direct call will switch over to the - * @new_addr without missing anything. - * - * Returns: zero on success. Non zero on error, which includes: - * -ENODEV : the @ip given has no direct caller attached - * -EINVAL : the @old_addr does not match the current direct caller - */ -int modify_ftrace_direct(unsigned long ip, - unsigned long old_addr, unsigned long new_addr) -{ - struct ftrace_direct_func *direct, *new_direct = NULL; - struct ftrace_func_entry *entry; - struct dyn_ftrace *rec; - int ret = -ENODEV; - - mutex_lock(&direct_mutex); - - mutex_lock(&ftrace_lock); - - ip = ftrace_location(ip); - if (!ip) - goto out_unlock; - - entry = find_direct_entry(&ip, &rec); - if (!entry) - goto out_unlock; - - ret = -EINVAL; - if (entry->direct != old_addr) - goto out_unlock; - - direct = ftrace_find_direct_func(old_addr); - if (WARN_ON(!direct)) - goto out_unlock; - if (direct->count > 1) { - ret = -ENOMEM; - new_direct = ftrace_alloc_direct_func(new_addr); - if (!new_direct) - goto out_unlock; - direct->count--; - new_direct->count++; - } else { - direct->addr = new_addr; - } - - /* - * If there's no other ftrace callback on the rec->ip location, - * then it can be changed directly by the architecture. - * If there is another caller, then we just need to change the - * direct caller helper to point to @new_addr. - */ - if (ftrace_rec_count(rec) == 1) { - ret = ftrace_modify_direct_caller(entry, rec, old_addr, new_addr); - } else { - entry->direct = new_addr; - ret = 0; - } - - if (unlikely(ret && new_direct)) { - direct->count++; - list_del_rcu(&new_direct->next); - synchronize_rcu_tasks(); - kfree(new_direct); - ftrace_direct_func_count--; - } - - out_unlock: - mutex_unlock(&ftrace_lock); - mutex_unlock(&direct_mutex); - return ret; -} -EXPORT_SYMBOL_GPL(modify_ftrace_direct); - #define MULTI_FLAGS (FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_REGS) static int check_direct_multi(struct ftrace_ops *ops) From patchwork Wed Feb 1 16:34:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124720 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF0D4C636CD for ; Wed, 1 Feb 2023 16:35:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232495AbjBAQfr (ORCPT ); Wed, 1 Feb 2023 11:35:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232356AbjBAQfh (ORCPT ); Wed, 1 Feb 2023 11:35:37 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B8E641B77 for ; Wed, 1 Feb 2023 08:35:28 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id n13so6113941wmr.4 for ; Wed, 01 Feb 2023 08:35:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xjgzv+CaWDCCmc9m2PtN7hh7GiZmUlZ8epcrKuwyKlk=; b=fB2BygETm73mrQ4E+jxUvmjud7LCPGfH0ctmgJAjk8dhSSEtRFb5FRPeDawZvISnAX SMWlkq1RxsqY0n9ATCYVe4TmRwqBeWUWhE9DKB1oj8V5RTWazQWdsb6E5fUjUwRH4ayK MiA1JFPW7zxvbGfJUF+AszD+OWqU9CTiC5tOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xjgzv+CaWDCCmc9m2PtN7hh7GiZmUlZ8epcrKuwyKlk=; b=IgcdeG/mSs28XV1/s347V0StPoJcsUg2hfpMQ3KTctl2wTp67O4qxBjxaVbxTGTTG7 zRFh1MM8LrX+NhMg/72isnVR/IBk2bGo37fCDQ2yFoI6wjPeemEl1y2JSLKF5P17JIYb q1W13NYWQyX6BZ+bleVJlj5djGR5sMTOz9/w0R/jQaU2jmy62ejh/ZJXR0D2lpOp9llm 9Y7SI5uaYEaz9AJEsV2QtYsVDnscZ5jNAAswc20ZHN/PKXiPrFOWB1YQ7ECe9MhGL4ya ubYZhXWH/E+CyY78PpHsefW2M3gPoQggVatly6t6bXNh7rb7vPID06Fi9Le13VlxQjny xDbQ== X-Gm-Message-State: AO0yUKV+R51+BVMywGf3UY1P1ibHXnkdmAqbYs7gqe+DK/ZJ8RIxfUI9 j2OohU5JKQcogjkqESc3WuujTA== X-Google-Smtp-Source: AK7set8XqLfrpqZpOrgpsr7TxJ7QDVnSIcMeu3VwRNabuM38cqVSAX6unAWbUntH0DbhPAD9m3AdRg== X-Received: by 2002:a05:600c:468e:b0:3db:1434:c51a with SMTP id p14-20020a05600c468e00b003db1434c51amr2813766wmo.40.1675269326776; Wed, 01 Feb 2023 08:35:26 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:26 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 3/8] ftrace: Rename _ftrace_direct_multi APIs to _ftrace_direct APIs Date: Wed, 1 Feb 2023 17:34:15 +0100 Message-Id: <20230201163420.1579014-4-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Now that the original _ftrace_direct APIs are gone, the "_multi" suffixes only add confusion. Signed-off-by: Florent Revest Acked-by: Mark Rutland Tested-by: Mark Rutland --- include/linux/ftrace.h | 16 +++++------ kernel/bpf/trampoline.c | 14 ++++----- kernel/trace/ftrace.c | 32 ++++++++++----------- kernel/trace/trace_selftest.c | 7 +++-- samples/Kconfig | 2 +- samples/ftrace/ftrace-direct-modify.c | 8 +++--- samples/ftrace/ftrace-direct-multi-modify.c | 8 +++--- samples/ftrace/ftrace-direct-multi.c | 6 ++-- samples/ftrace/ftrace-direct-too.c | 6 ++-- samples/ftrace/ftrace-direct.c | 6 ++-- 10 files changed, 53 insertions(+), 52 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 2d7c85e47c2d..a7dbd307c3a4 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -398,10 +398,10 @@ struct ftrace_func_entry { #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS extern int ftrace_direct_func_count; unsigned long ftrace_find_rec_direct(unsigned long ip); -int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); -int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); -int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); -int modify_ftrace_direct_multi_nolock(struct ftrace_ops *ops, unsigned long addr); +int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); +int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); +int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr); +int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr); #else struct ftrace_ops; @@ -410,19 +410,19 @@ static inline unsigned long ftrace_find_rec_direct(unsigned long ip) { return 0; } -static inline int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { return -ENODEV; } -static inline int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { return -ENODEV; } -static inline int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { return -ENODEV; } -static inline int modify_ftrace_direct_multi_nolock(struct ftrace_ops *ops, unsigned long addr) +static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr) { return -ENODEV; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d0ed7d6f5eec..150b53316df2 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -39,14 +39,14 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_ops_cmd int ret = 0; if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { - /* This is called inside register_ftrace_direct_multi(), so + /* This is called inside register_ftrace_direct(), so * tr->mutex is already locked. */ lockdep_assert_held_once(&tr->mutex); /* Instead of updating the trampoline here, we propagate - * -EAGAIN to register_ftrace_direct_multi(). Then we can - * retry register_ftrace_direct_multi() after updating the + * -EAGAIN to register_ftrace_direct(). Then we can + * retry register_ftrace_direct() after updating the * trampoline. */ if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) && @@ -198,7 +198,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) int ret; if (tr->func.ftrace_managed) - ret = unregister_ftrace_direct_multi(tr->fops, (long)old_addr); + ret = unregister_ftrace_direct(tr->fops, (long)old_addr); else ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL); @@ -215,9 +215,9 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad if (tr->func.ftrace_managed) { if (lock_direct_mutex) - ret = modify_ftrace_direct_multi(tr->fops, (long)new_addr); + ret = modify_ftrace_direct(tr->fops, (long)new_addr); else - ret = modify_ftrace_direct_multi_nolock(tr->fops, (long)new_addr); + ret = modify_ftrace_direct_nolock(tr->fops, (long)new_addr); } else { ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr); } @@ -243,7 +243,7 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) if (tr->func.ftrace_managed) { ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1); - ret = register_ftrace_direct_multi(tr->fops, (long)new_addr); + ret = register_ftrace_direct(tr->fops, (long)new_addr); } else { ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr); } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 5efe201428fa..cb77a0a208c7 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5312,7 +5312,7 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long } /** - * register_ftrace_direct_multi - Call a custom trampoline directly + * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops * @ops: The address of the struct ftrace_ops object * @addr: The address of the trampoline to call at @ops functions @@ -5333,7 +5333,7 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long * -ENODEV - @ip does not point to a ftrace nop location (or not supported) * -ENOMEM - There was an allocation failure. */ -int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { struct ftrace_hash *hash, *free_hash = NULL; struct ftrace_func_entry *entry, *new; @@ -5391,11 +5391,11 @@ int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } return err; } -EXPORT_SYMBOL_GPL(register_ftrace_direct_multi); +EXPORT_SYMBOL_GPL(register_ftrace_direct); /** - * unregister_ftrace_direct_multi - Remove calls to custom trampoline - * previously registered by register_ftrace_direct_multi for @ops object. + * unregister_ftrace_direct - Remove calls to custom trampoline + * previously registered by register_ftrace_direct for @ops object. * @ops: The address of the struct ftrace_ops object * * This is used to remove a direct calls to @addr from the nop locations @@ -5406,7 +5406,7 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct_multi); * 0 on success * -EINVAL - The @ops object was not properly registered. */ -int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { struct ftrace_hash *hash = ops->func_hash->filter_hash; int err; @@ -5426,10 +5426,10 @@ int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) ops->trampoline = 0; return err; } -EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi); +EXPORT_SYMBOL_GPL(unregister_ftrace_direct); static int -__modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +__modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { struct ftrace_hash *hash; struct ftrace_func_entry *entry, *iter; @@ -5476,7 +5476,7 @@ __modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } /** - * modify_ftrace_direct_multi_nolock - Modify an existing direct 'multi' call + * modify_ftrace_direct_nolock - Modify an existing direct 'multi' call * to call something else * @ops: The address of the struct ftrace_ops object * @addr: The address of the new trampoline to call at @ops functions @@ -5493,19 +5493,19 @@ __modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) * Returns: zero on success. Non zero on error, which includes: * -EINVAL - The @ops object was not properly registered. */ -int modify_ftrace_direct_multi_nolock(struct ftrace_ops *ops, unsigned long addr) +int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr) { if (check_direct_multi(ops)) return -EINVAL; if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) return -EINVAL; - return __modify_ftrace_direct_multi(ops, addr); + return __modify_ftrace_direct(ops, addr); } -EXPORT_SYMBOL_GPL(modify_ftrace_direct_multi_nolock); +EXPORT_SYMBOL_GPL(modify_ftrace_direct_nolock); /** - * modify_ftrace_direct_multi - Modify an existing direct 'multi' call + * modify_ftrace_direct - Modify an existing direct 'multi' call * to call something else * @ops: The address of the struct ftrace_ops object * @addr: The address of the new trampoline to call at @ops functions @@ -5519,7 +5519,7 @@ EXPORT_SYMBOL_GPL(modify_ftrace_direct_multi_nolock); * Returns: zero on success. Non zero on error, which includes: * -EINVAL - The @ops object was not properly registered. */ -int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) { int err; @@ -5529,11 +5529,11 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) return -EINVAL; mutex_lock(&direct_mutex); - err = __modify_ftrace_direct_multi(ops, addr); + err = __modify_ftrace_direct(ops, addr); mutex_unlock(&direct_mutex); return err; } -EXPORT_SYMBOL_GPL(modify_ftrace_direct_multi); +EXPORT_SYMBOL_GPL(modify_ftrace_direct); #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ /** diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 9b7f10cbc51d..06218fc9374b 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -874,7 +874,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, * and make sure we get graph trace. */ ftrace_set_filter_ip(&direct, (unsigned long)DYN_FTRACE_TEST_NAME, 0, 0); - ret = register_ftrace_direct_multi(&direct, (unsigned long)trace_direct_tramp); + ret = register_ftrace_direct(&direct, + (unsigned long)trace_direct_tramp); if (ret) goto out; @@ -894,8 +895,8 @@ trace_selftest_startup_function_graph(struct tracer *trace, unregister_ftrace_graph(&fgraph_ops); - ret = unregister_ftrace_direct_multi(&direct, - (unsigned long) trace_direct_tramp); + ret = unregister_ftrace_direct(&direct, + (unsigned long)trace_direct_tramp); if (ret) goto out; diff --git a/samples/Kconfig b/samples/Kconfig index 0d81c00289ee..e85998ca354d 100644 --- a/samples/Kconfig +++ b/samples/Kconfig @@ -38,7 +38,7 @@ config SAMPLE_FTRACE_DIRECT that hooks to wake_up_process and prints the parameters. config SAMPLE_FTRACE_DIRECT_MULTI - tristate "Build register_ftrace_direct_multi() example" + tristate "Build register_ftrace_direct() on multiple ips example" depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS && m depends on HAVE_SAMPLE_FTRACE_DIRECT_MULTI help diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c index e1e6c286970c..0f1b8859e1e1 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -116,7 +116,7 @@ static int simple_thread(void *arg) if (ret) continue; t ^= 1; - ret = modify_ftrace_direct_multi(&direct, tramps[t]); + ret = modify_ftrace_direct(&direct, tramps[t]); if (!ret) my_tramp = tramps[t]; WARN_ON_ONCE(ret); @@ -132,7 +132,7 @@ static int __init ftrace_direct_init(void) int ret; ftrace_set_filter_ip(&direct, (unsigned long) my_ip, 0, 0); - ret = register_ftrace_direct_multi(&direct, my_tramp); + ret = register_ftrace_direct(&direct, my_tramp); if (!ret) simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); return ret; @@ -141,12 +141,12 @@ static int __init ftrace_direct_init(void) static void __exit ftrace_direct_exit(void) { kthread_stop(simple_tsk); - unregister_ftrace_direct_multi(&direct, my_tramp); + unregister_ftrace_direct(&direct, my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct_multi()"); +MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c index d52370cad0b6..407c56325e65 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -123,7 +123,7 @@ static int simple_thread(void *arg) if (ret) continue; t ^= 1; - ret = modify_ftrace_direct_multi(&direct, tramps[t]); + ret = modify_ftrace_direct(&direct, tramps[t]); if (!ret) my_tramp = tramps[t]; WARN_ON_ONCE(ret); @@ -141,7 +141,7 @@ static int __init ftrace_direct_multi_init(void) ftrace_set_filter_ip(&direct, (unsigned long) wake_up_process, 0, 0); ftrace_set_filter_ip(&direct, (unsigned long) schedule, 0, 0); - ret = register_ftrace_direct_multi(&direct, my_tramp); + ret = register_ftrace_direct(&direct, my_tramp); if (!ret) simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); @@ -151,12 +151,12 @@ static int __init ftrace_direct_multi_init(void) static void __exit ftrace_direct_multi_exit(void) { kthread_stop(simple_tsk); - unregister_ftrace_direct_multi(&direct, my_tramp); + unregister_ftrace_direct(&direct, my_tramp); } module_init(ftrace_direct_multi_init); module_exit(ftrace_direct_multi_exit); MODULE_AUTHOR("Jiri Olsa"); -MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct_multi()"); +MODULE_DESCRIPTION("Example use case of using modify_ftrace_direct()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c index ec1088922517..42a94806a446 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -73,17 +73,17 @@ static int __init ftrace_direct_multi_init(void) ftrace_set_filter_ip(&direct, (unsigned long) wake_up_process, 0, 0); ftrace_set_filter_ip(&direct, (unsigned long) schedule, 0, 0); - return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); + return register_ftrace_direct(&direct, (unsigned long) my_tramp); } static void __exit ftrace_direct_multi_exit(void) { - unregister_ftrace_direct_multi(&direct, (unsigned long) my_tramp); + unregister_ftrace_direct(&direct, (unsigned long) my_tramp); } module_init(ftrace_direct_multi_init); module_exit(ftrace_direct_multi_exit); MODULE_AUTHOR("Jiri Olsa"); -MODULE_DESCRIPTION("Example use case of using register_ftrace_direct_multi()"); +MODULE_DESCRIPTION("Example use case of using register_ftrace_direct()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c index 0e907092e2c0..7ee5dd3cc61d 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -76,17 +76,17 @@ static int __init ftrace_direct_init(void) { ftrace_set_filter_ip(&direct, (unsigned long) handle_mm_fault, 0, 0); - return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); + return register_ftrace_direct(&direct, (unsigned long) my_tramp); } static void __exit ftrace_direct_exit(void) { - unregister_ftrace_direct_multi(&direct, (unsigned long)my_tramp); + unregister_ftrace_direct(&direct, (unsigned long)my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Another example use case of using register_ftrace_direct_multi()"); +MODULE_DESCRIPTION("Another example use case of using register_ftrace_direct()"); MODULE_LICENSE("GPL"); diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index e446c38f6b58..5ffce87fa83e 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -69,17 +69,17 @@ static int __init ftrace_direct_init(void) { ftrace_set_filter_ip(&direct, (unsigned long) wake_up_process, 0, 0); - return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); + return register_ftrace_direct(&direct, (unsigned long) my_tramp); } static void __exit ftrace_direct_exit(void) { - unregister_ftrace_direct_multi(&direct, (unsigned long)my_tramp); + unregister_ftrace_direct(&direct, (unsigned long)my_tramp); } module_init(ftrace_direct_init); module_exit(ftrace_direct_exit); MODULE_AUTHOR("Steven Rostedt"); -MODULE_DESCRIPTION("Example use case of using register_ftrace_direct_multi()"); +MODULE_DESCRIPTION("Example use case of using register_ftrace_direct()"); MODULE_LICENSE("GPL"); From patchwork Wed Feb 1 16:34:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124721 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9803DC636D7 for ; Wed, 1 Feb 2023 16:35:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232556AbjBAQft (ORCPT ); Wed, 1 Feb 2023 11:35:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232453AbjBAQfq (ORCPT ); Wed, 1 Feb 2023 11:35:46 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A393F79F15 for ; Wed, 1 Feb 2023 08:35:29 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id m16-20020a05600c3b1000b003dc4050c94aso1921068wms.4 for ; Wed, 01 Feb 2023 08:35:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p0B1nR0feNzt6iYs1fcCSNwhqATraAQZbyN0DagmRgs=; b=jcn6cTs4+XY1hUWmIDmmwPG44YIXWbCtGh35J88AOeps79NC6Aub1gKAPPv419PyFm HTPegDj2V+wQXQ7DcQvVkjHumtHWHz7verUSkgH7IJrJQQz1clTf1SwWQ3717K7f1cqs BiPYS75krdieSyLXCxa9QTp4cnoHXv5nKh5HI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p0B1nR0feNzt6iYs1fcCSNwhqATraAQZbyN0DagmRgs=; b=8FyGY6hUOc9FvTTnhkl9x0Z42j+FpqDdBZIMNpLzuTACnt/d0hARdpjcQVj4ire3P/ 3a4zRun4dueIPey3e+JEau0GbN2PF66chicO4tJBxVVzKqduhKYesToVUw+KqCH1gP83 k6zP7gZ5iCyt0fsFWTglvwskgrS5ojHM1xT3ibwYqPtEUgMMOmICB0WV3wPXMHYBcI8j pS1Bkbk4XZb6bQs8oqj2hHX7hST38fZCqGKXTkgllHiHtcHlw6+q1FOeZbkb3+LoiiPT Oz/qBAQFs8U6RNBz8zAQxiA+a/rCduQ6PJsoTmKFUh4AhpSE/MxJP4JNrZ14Oppf+Tk+ 4tNg== X-Gm-Message-State: AO0yUKU5wOmjztxeQoOXNgtVLsqaQgGZ8rZyrlQ0N0+VAOl7Dz4n+S9Y a8rgbwOQtbU0Jmy2mXHsonl5XzwVLWBb7XmlznM= X-Google-Smtp-Source: AK7set9sCy9RhmFzbGR11y6SAdcOcsShJEwbhl2h5tKcFqmeH9CrjXQjVjAY0i+Q+yzsszXf1HIeKQ== X-Received: by 2002:a05:600c:491c:b0:3d9:ee01:60a4 with SMTP id f28-20020a05600c491c00b003d9ee0160a4mr2827768wmp.1.1675269328196; Wed, 01 Feb 2023 08:35:28 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:27 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 4/8] ftrace: Store direct called addresses in their ops Date: Wed, 1 Feb 2023 17:34:16 +0100 Message-Id: <20230201163420.1579014-5-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org All direct calls are now registered using the register_ftrace_direct API so each ops can jump to only one direct-called trampoline. By storing the direct called trampoline address directly in the ops we can save one hashmap lookup in the direct call ops and implement arm64 direct calls on top of call ops. Signed-off-by: Florent Revest --- include/linux/ftrace.h | 3 +++ kernel/trace/ftrace.c | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a7dbd307c3a4..84f717f8959e 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -321,6 +321,9 @@ struct ftrace_ops { unsigned long trampoline_size; struct list_head list; ftrace_ops_func_t ops_func; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + unsigned long direct_call; +#endif #endif }; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index cb77a0a208c7..b0426de11c45 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2577,9 +2577,8 @@ ftrace_add_rec_direct(unsigned long ip, unsigned long addr, static void call_direct_funcs(unsigned long ip, unsigned long pip, struct ftrace_ops *ops, struct ftrace_regs *fregs) { - unsigned long addr; + unsigned long addr = ops->direct_call; - addr = ftrace_find_rec_direct(ip); if (!addr) return; @@ -5375,6 +5374,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) ops->func = call_direct_funcs; ops->flags = MULTI_FLAGS; ops->trampoline = FTRACE_REGS_ADDR; + ops->direct_call = addr; err = register_ftrace_function_nolock(ops); @@ -5445,6 +5445,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash = ops->func_hash; + tmp_ops.direct_call = addr; err = register_ftrace_function_nolock(&tmp_ops); if (err) @@ -5466,6 +5467,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) entry->direct = addr; } } + ops->direct_call = addr; mutex_unlock(&ftrace_lock); From patchwork Wed Feb 1 16:34:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124722 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E792FC63797 for ; Wed, 1 Feb 2023 16:35:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232599AbjBAQfu (ORCPT ); Wed, 1 Feb 2023 11:35:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232477AbjBAQfr (ORCPT ); Wed, 1 Feb 2023 11:35:47 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 255EA62276 for ; Wed, 1 Feb 2023 08:35:31 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id n28-20020a05600c3b9c00b003ddca7a2bcbso1922044wms.3 for ; Wed, 01 Feb 2023 08:35:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V920tXDgaE7z8bCeiKLUkBejFiJJs4xj6S06IJdrTDo=; b=DYuGfqui4JmugKay/2snlgInuN8NoSZVoexURJTLb8Kn8f4JIyvQgTexs4woxHNsnz wO8Z6dafnQ/XbyCE2eZZeApXulDjM44aq8GV5zcg7oNuhHSxUTyM5itzgz5nZpMAO8jf QGTjtSgg4ehtaz3yn7QunU2caI+VvbmJ2wS54= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V920tXDgaE7z8bCeiKLUkBejFiJJs4xj6S06IJdrTDo=; b=Wh3v4+YwngqQZ4d0tnKE5qGRSNL45PztV0aStxh3cuX/A+/mgx+pIURAmFXTKgrcaC DkY1w6p7yUrNCgIjGtDcqX33ScPh3/EI6oN/YJ1m6BEPggR43HweZkaYWDCSE533CoNT lLBPRCZHcNtIfXbHIEBqjdQixFJCCoTplPGUqNkA3R9uzwPmfGV2VDx1V50Jp2C3xzQH 4aIzM5nEP0j90hyLTZo4dZ1qCn/TsMljPSJ5Rtm3qu+0o8gJ3iHofa+GVqEZI/AFKgjg s4u9zzgcUp0jChat4w3W7iCFOJWF1NMTK/qmDiDxEBPl80UCjF+rQd/Wp0lWSKNzhAKe aD4g== X-Gm-Message-State: AO0yUKXpDCXxj0t3hkcLXDXvZ85ODP3wgXD3JS/pQr0xAwIQZ0tM4Zwt l05A3tA55qkkkj7dU901+pvh1Q== X-Google-Smtp-Source: AK7set/w1a9aVC2lSJ1aNaktqbPxU7ogYk690xUJ/cBVHkXalBaadY++SDNwkAXdZxgSJMASgFFFGQ== X-Received: by 2002:a05:600c:2e06:b0:3dd:dd46:1274 with SMTP id o6-20020a05600c2e0600b003dddd461274mr2822832wmf.4.1675269329591; Wed, 01 Feb 2023 08:35:29 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:29 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 5/8] ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS Date: Wed, 1 Feb 2023 17:34:17 +0100 Message-Id: <20230201163420.1579014-6-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Direct called trampolines can be called in two ways: - either from the ftrace callsite. In this case, they do not access any struct ftrace_regs nor pt_regs - Or, if a ftrace ops is also attached, from the end of a ftrace trampoline. In this case, the call_direct_funcs ops is in charge of setting the direct call trampoline's address in a struct ftrace_regs Since "ftrace: pass fregs to arch_ftrace_set_direct_caller()", the later case no longer requires a full pt_regs. It only needs a struct ftrace_regs so DIRECT_CALLS can work with both WITH_ARGS or WITH_REGS. With architectures like arm64 already abandoning WITH_REGS in favor of WITH_ARGS, it's important to have DIRECT_CALLS work WITH_ARGS only. Signed-off-by: Florent Revest --- kernel/trace/Kconfig | 2 +- kernel/trace/ftrace.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 5df427a2321d..4496a7c69810 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -257,7 +257,7 @@ config DYNAMIC_FTRACE_WITH_REGS config DYNAMIC_FTRACE_WITH_DIRECT_CALLS def_bool y - depends on DYNAMIC_FTRACE_WITH_REGS + depends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS depends on HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS config DYNAMIC_FTRACE_WITH_CALL_OPS diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index b0426de11c45..73b6f6489ba1 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5282,7 +5282,7 @@ static LIST_HEAD(ftrace_direct_funcs); static int register_ftrace_function_nolock(struct ftrace_ops *ops); -#define MULTI_FLAGS (FTRACE_OPS_FL_DIRECT | FTRACE_OPS_FL_SAVE_REGS) +#define MULTI_FLAGS (FTRACE_OPS_FL_DIRECT) static int check_direct_multi(struct ftrace_ops *ops) { From patchwork Wed Feb 1 16:34:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124723 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CF06C64EC3 for ; Wed, 1 Feb 2023 16:35:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232631AbjBAQfv (ORCPT ); Wed, 1 Feb 2023 11:35:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232487AbjBAQfr (ORCPT ); Wed, 1 Feb 2023 11:35:47 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC9FA60C9D for ; Wed, 1 Feb 2023 08:35:32 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id f47-20020a05600c492f00b003dc584a7b7eso1827681wmp.3 for ; Wed, 01 Feb 2023 08:35:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4h7L/6sUB2KnxbssfPmMIi6qgi9N82FF7O8XsVF6WqE=; b=eO7ef0lbntzUy8m51n14OyG3x4x9Mi3sefAioFTJImzU5ZFg5Vs78dtPVoYekbT3pk Cg3uxLoGda0UZ7Xy5jaLG8tClEL1RJYhO9a8twyYmoUmUPMXXAoMJjtxTKVjATmXRm3i jc39euXLJjlYSDkRk5RHgqVAXCCYmDq7Le2k8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4h7L/6sUB2KnxbssfPmMIi6qgi9N82FF7O8XsVF6WqE=; b=maudI6WAqWBYJYpT5Fd5Y0naaoim1ph73/N56sBI4bheWz6LAgMTS0usu1+bRmClq/ 2bI+Zvjvl1pHbkWSnyzkhpgobZyHaEqPAqLnguJ9iZc+li8I2Dx3G25agUZco69vbtY9 MQRvHzDSnagYCx1Whz2C4097kg6Ynx6Qgzux7gbRTvTwZMq1CRorbbJy+3P3g2xw+2Fi LuZMlErcSK7rIVzOKOgwIDMWrZW06lBQNhn9L+IxPSFve3+nzZ1XhCoB0PUddXeWCHTh UzNUjPwlFToSOFCFpVcA1KCx/5MKjZNrAPWJ3GJ1Z1iKPb1F+KHZhVqVzoV5jJ1hUrqq s2Zw== X-Gm-Message-State: AO0yUKVnvTZpnUUw+YK3D76ayZQit1QU6gGqemr6uwK2iipO0C+baNcJ BvkMWCt6X3lAGIrLBt/mIafmvQ== X-Google-Smtp-Source: AK7set/s4PL//844ITpXCCvWBv/H9BGkGF1snDf6k8ign1+tNi1Gp86Z60+eLs40rYOD38/iM9jfmg== X-Received: by 2002:a05:600c:4f12:b0:3de:271:7b24 with SMTP id l18-20020a05600c4f1200b003de02717b24mr2802649wmq.16.1675269331250; Wed, 01 Feb 2023 08:35:31 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:30 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Xu Kuohai , Li Huafei , Florent Revest Subject: [PATCH 6/8] ftrace: Fix dead loop caused by direct call in ftrace selftest Date: Wed, 1 Feb 2023 17:34:18 +0100 Message-Id: <20230201163420.1579014-7-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Xu Kuohai After direct call is enabled for arm64, ftrace selftest enters a dead loop: : 00 bti c 01 mov x9, x30 : 02 bl ----------> ret | lr/x30 is 03, return to 03 | 03 mov w0, #0x0 <-----------------------------| | | | dead loop! | | | 04 ret ---- lr/x30 is still 03, go back to 03 ----| The reason is that when the direct caller trace_direct_tramp() returns to the patched function trace_selftest_dynamic_test_func(), lr is still the address after the instrumented instruction in the patched function, so when the patched function exits, it returns to itself! To fix this issue, we need to restore lr before trace_direct_tramp() exits, so use a dedicated trace_direct_tramp() for arm64. Reported-by: Li Huafei Signed-off-by: Xu Kuohai Acked-by: Steven Rostedt (Google) Signed-off-by: Florent Revest Signed-off-by: Mark Rutland --- arch/arm64/include/asm/ftrace.h | 7 +++++++ arch/arm64/kernel/entry-ftrace.S | 10 ++++++++++ kernel/trace/trace_selftest.c | 2 ++ 3 files changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 1c2672bbbf37..cf6d9c42ff36 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -168,6 +168,13 @@ static inline bool arch_syscall_match_sym_name(const char *sym, */ return !strcmp(sym + 8, name); } + +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ + defined(CONFIG_FTRACE_SELFTEST) +extern void ftrace_dummy_tramp(void); +#define trace_direct_tramp ftrace_dummy_tramp +#endif + #endif /* ifndef __ASSEMBLY__ */ #endif /* __ASM_FTRACE_H */ diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 350ed81324ac..9869debd22fb 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -118,6 +118,16 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ret x9 SYM_CODE_END(ftrace_caller) +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ + defined(CONFIG_FTRACE_SELFTEST) +SYM_CODE_START(ftrace_dummy_tramp) + bti c + mov x10, x30 + mov x30, x9 + ret x10 +SYM_CODE_END(ftrace_dummy_tramp) +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ + #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ /* diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index 06218fc9374b..f9f5d4e8ab50 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -789,11 +789,13 @@ static struct fgraph_ops fgraph_ops __initdata = { #define CALL_DEPTH_ACCOUNT "" #endif +#ifndef trace_direct_tramp noinline __noclone static void trace_direct_tramp(void) { asm(CALL_DEPTH_ACCOUNT); } #endif +#endif /* * Pretty much the same than for the function tracer from which the selftest From patchwork Wed Feb 1 16:34:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124724 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B3C2C636CD for ; Wed, 1 Feb 2023 16:35:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232419AbjBAQfz (ORCPT ); Wed, 1 Feb 2023 11:35:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232503AbjBAQfr (ORCPT ); Wed, 1 Feb 2023 11:35:47 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38C96677A6 for ; Wed, 1 Feb 2023 08:35:34 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id n13so6114202wmr.4 for ; Wed, 01 Feb 2023 08:35:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MbHY+Gmtwa2HHtiXzFaehEcw/QkGNqN5J6j1VumLZ5w=; b=B5Kk5qpWjq/NWChUoJdLAZ15WLm2v1L//Jj1sE7X8MZC3AVQVVYby77BqsgXYZk/qt b8j//orYYdKgUmjIVviXcRXUnAipPwUDBe7ZWqHGBPP1TTVK+p6qMOTArnZw+wO7ZigT 3gGskC/gE5tLjsWGLwCN3/IVdjIhSxpQLR9bo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MbHY+Gmtwa2HHtiXzFaehEcw/QkGNqN5J6j1VumLZ5w=; b=e/rpWLblYvbMoF8an9UU8X85kbgR70O2UFDl68dT55IHG965Ij6BTj9f+InaFBauS3 GalY/hkv5608RBdHLqJwq7ApJZipW6pMSXPA+mwrHqCDlIsUfdO3cxhc6ODF7LdW5yhh pBryY4tAFc6NGJGTBSxdGPyWiN4jLAzygAlwwZ22hRgbZU8W+BvgSeh1kSnAs9VgdsBb 73p9WaXh2i2+8nH3nT9ySa4WJ+G1X+7Q6Fmy+vGqjhYu36qIV2QG8u2MZvz65lyBIucz TXvABzl13tUno2691NIUIAvlahEWmUbv1S/biV49R4rkv0ofcFnDgErqAnS8WlKglkxU 3iJQ== X-Gm-Message-State: AO0yUKXIqh5JdSNcSU6nwt0jpSi6ImXmTX7tx0bfMSdNDkfS9/QHOv/D w8R+p53y+oWpZL79rQcGc6sS4JJkGN7o9UhHDSQ= X-Google-Smtp-Source: AK7set9M34kub5/FQQZ2rEhTdT78awUB8bbp7/omx+W0n7FyqEelSYMdNpH/uDndWUi0UrY9espvyQ== X-Received: by 2002:a7b:c5ca:0:b0:3dd:e38b:a12f with SMTP id n10-20020a7bc5ca000000b003dde38ba12fmr5384458wmk.40.1675269332734; Wed, 01 Feb 2023 08:35:32 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:32 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 7/8] arm64: ftrace: Add direct call support Date: Wed, 1 Feb 2023 17:34:19 +0100 Message-Id: <20230201163420.1579014-8-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This builds up on the CALL_OPS work which extends the ftrace patchsite on arm64 with an ops pointer usable by the ftrace trampoline. This ops pointer is valid at all time. Indeed, it is either pointing to ftrace_list_ops or to the single ops which should be called from that patchsite. There are a few cases to distinguish: - If a direct call ops is the only one tracing a function: - If the direct called trampoline is within the reach of a BL instruction -> the ftrace patchsite jumps to the trampoline - Else -> the ftrace patchsite jumps to the ftrace_caller trampoline which reads the ops pointer in the patchsite and jumps to the direct call address stored in the ops - Else -> the ftrace patchsite jumps to the ftrace_caller trampoline and its ops literal points to ftrace_list_ops so it iterates over all registered ftrace ops, including the direct call ops and calls its call_direct_funcs handler which stores the direct called trampoline's address in the ftrace_regs and the ftrace_caller trampoline will return to that address instead of returning to the traced function Signed-off-by: Florent Revest Suggested-by: Mark Rutland --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/ftrace.h | 17 +++++++++ arch/arm64/kernel/asm-offsets.c | 6 ++++ arch/arm64/kernel/entry-ftrace.S | 60 +++++++++++++++++++++++--------- arch/arm64/kernel/ftrace.c | 36 ++++++++++++++++--- 5 files changed, 100 insertions(+), 21 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6f6f37161cf6..7deafd653c42 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -188,6 +188,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_ARGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ + if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \ if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG) select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \ diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index cf6d9c42ff36..9fb95966b6d5 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -80,6 +80,10 @@ struct ftrace_regs { unsigned long sp; unsigned long pc; + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + unsigned long custom_tramp; +#endif }; static __always_inline unsigned long @@ -136,6 +140,19 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs); #define ftrace_graph_func ftrace_graph_func + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS +static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, + unsigned long addr) +{ + /* + * The ftrace trampoline will return to this address instead of the + * instrumented function. + */ + fregs->custom_tramp = addr; +} +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ + #endif #define ftrace_return_address(n) return_address(n) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index ae345b06e9f7..c67f0373b88c 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -93,6 +93,9 @@ int main(void) DEFINE(FREGS_LR, offsetof(struct ftrace_regs, lr)); DEFINE(FREGS_SP, offsetof(struct ftrace_regs, sp)); DEFINE(FREGS_PC, offsetof(struct ftrace_regs, pc)); +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + DEFINE(FREGS_CUSTOM_TRAMP, offsetof(struct ftrace_regs, custom_tramp)); +#endif DEFINE(FREGS_SIZE, sizeof(struct ftrace_regs)); BLANK(); #endif @@ -197,6 +200,9 @@ int main(void) #endif #ifdef CONFIG_FUNCTION_TRACER DEFINE(FTRACE_OPS_FUNC, offsetof(struct ftrace_ops, func)); +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + DEFINE(FTRACE_OPS_DIRECT_CALL, offsetof(struct ftrace_ops, direct_call)); +#endif #endif return 0; } diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 9869debd22fb..0576f38e6362 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -36,6 +36,30 @@ SYM_CODE_START(ftrace_caller) bti c +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS + /* + * The literal pointer to the ops is at an 8-byte aligned boundary + * which is either 12 or 16 bytes before the BL instruction in the call + * site. See ftrace_call_adjust() for details. + * + * Therefore here the LR points at `literal + 16` or `literal + 20`, + * and we can find the address of the literal in either case by + * aligning to an 8-byte boundary and subtracting 16. We do the + * alignment first as this allows us to fold the subtraction into the + * LDR. + */ + bic x11, x30, 0x7 + ldr x11, [x11, #-(4 * AARCH64_INSN_SIZE)] // op + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* If the op has a direct call address, return to it */ + ldr x12, [x11, #FTRACE_OPS_DIRECT_CALL] // op->direct_call + cbz x12, 1f + ret x12 +1: +#endif +#endif + /* Save original SP */ mov x10, sp @@ -57,6 +81,11 @@ SYM_CODE_START(ftrace_caller) /* Save the PC after the ftrace callsite */ str x30, [sp, #FREGS_PC] +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* Set custom_tramp to zero */ + str xzr, [sp, #FREGS_CUSTOM_TRAMP] +#endif + /* Create a frame record for the callsite above the ftrace regs */ stp x29, x9, [sp, #FREGS_SIZE + 16] add x29, sp, #FREGS_SIZE + 16 @@ -71,20 +100,7 @@ SYM_CODE_START(ftrace_caller) mov x3, sp // regs #ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS - /* - * The literal pointer to the ops is at an 8-byte aligned boundary - * which is either 12 or 16 bytes before the BL instruction in the call - * site. See ftrace_call_adjust() for details. - * - * Therefore here the LR points at `literal + 16` or `literal + 20`, - * and we can find the address of the literal in either case by - * aligning to an 8-byte boundary and subtracting 16. We do the - * alignment first as this allows us to fold the subtraction into the - * LDR. - */ - bic x2, x30, 0x7 - ldr x2, [x2, #-16] // op - + mov x2, x11 // op ldr x4, [x2, #FTRACE_OPS_FUNC] // op->func blr x4 // op->func(ip, parent_ip, op, regs) @@ -110,12 +126,24 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) /* Restore the callsite's FP, LR, PC */ ldr x29, [sp, #FREGS_FP] ldr x30, [sp, #FREGS_LR] - ldr x9, [sp, #FREGS_PC] + ldr x10, [sp, #FREGS_PC] + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + ldr x11, [sp, #FREGS_CUSTOM_TRAMP] + cbz x11, 1f + /* Set x9 to parent ip before jump to custom trampoline */ + mov x9, x30 + /* Set lr to self ip */ + ldr x30, [sp, #FREGS_PC] + /* Set x10 (used for return address) to custom trampoline */ + mov x10, x11 +1: +#endif /* Restore the callsite's SP */ add sp, sp, #FREGS_SIZE + 32 - ret x9 + ret x10 SYM_CODE_END(ftrace_caller) #if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index 5545fe1a9012..758436727fba 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -206,6 +206,13 @@ static struct plt_entry *get_ftrace_plt(struct module *mod, unsigned long addr) return NULL; } +static bool reachable_by_bl(unsigned long addr, unsigned long pc) +{ + long offset = (long)addr - (long)pc; + + return offset >= -SZ_128M && offset < SZ_128M; +} + /* * Find the address the callsite must branch to in order to reach '*addr'. * @@ -220,14 +227,21 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec, unsigned long *addr) { unsigned long pc = rec->ip; - long offset = (long)*addr - (long)pc; struct plt_entry *plt; + /* + * If a custom trampoline is unreachable, rely on the ftrace_caller + * trampoline which knows how to indirectly reach that trampoline + * through ops->direct_call. + */ + if (*addr != FTRACE_ADDR && !reachable_by_bl(*addr, pc)) + *addr = FTRACE_ADDR; + /* * When the target is within range of the 'BL' instruction, use 'addr' * as-is and branch to that directly. */ - if (offset >= -SZ_128M && offset < SZ_128M) + if (reachable_by_bl(*addr, pc)) return true; /* @@ -330,12 +344,24 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) { - if (WARN_ON_ONCE(old_addr != (unsigned long)ftrace_caller)) + unsigned long pc = rec->ip; + u32 old, new; + int ret; + + ret = ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec)); + if (ret) + return ret; + + if (!ftrace_find_callable_addr(rec, NULL, &old_addr)) return -EINVAL; - if (WARN_ON_ONCE(addr != (unsigned long)ftrace_caller)) + if (!ftrace_find_callable_addr(rec, NULL, &addr)) return -EINVAL; - return ftrace_rec_update_ops(rec); + old = aarch64_insn_gen_branch_imm(pc, old_addr, + AARCH64_INSN_BRANCH_LINK); + new = aarch64_insn_gen_branch_imm(pc, addr, AARCH64_INSN_BRANCH_LINK); + + return ftrace_modify_code(pc, old, new, true); } #endif From patchwork Wed Feb 1 16:34:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florent Revest X-Patchwork-Id: 13124725 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 539BFC05027 for ; Wed, 1 Feb 2023 16:36:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232796AbjBAQgB (ORCPT ); Wed, 1 Feb 2023 11:36:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232528AbjBAQfs (ORCPT ); Wed, 1 Feb 2023 11:35:48 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0ECC79F33 for ; Wed, 1 Feb 2023 08:35:35 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id k8-20020a05600c1c8800b003dc57ea0dfeso1856611wms.0 for ; Wed, 01 Feb 2023 08:35:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dDqx7ldh5sGOgDsQIvKZBz9tRbOKRXiGSQRucIDUJVg=; b=J54wQ4LdJPNiFkmWlYnjOwU/5oAaLycrib7NWTRHHuK6LMI3Oh32ersiqtFKxfcDbf bB6OiooeswDyNYnjN3aY6RBtQR6dJmPIK0aKZ+9s0YM5RuF6lyeC+gyvairavzp9M7YA N5I7wrCbCmBvjNwadTlPK7Y0aM+0Hxi4Dk35E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dDqx7ldh5sGOgDsQIvKZBz9tRbOKRXiGSQRucIDUJVg=; b=QvGTKgV1NEqr5O0Lvu1HyctaoMTg1JDBcDWlFWGAgtdG+keMPsdm6OwQBKoiWVgqTq 4mm9VJviobjnFGUEOPJ5D4sGfvd/KMiUFAJvEVSK7peVKdLGVsaceyhpBNpjNG3W+73T 3SczZDQg0mf+7/BpVDHRF8W0wA+KW5z2JzGB2Qhp08xw5NViiQDEz+T9+0tlUWJ6vv+S 16WM+hMMcgc9mhz6FtRRPGtHW6AsnZ4yX8G9JCrLgwjqR8bQftIk1Z76KUhtanlYMmoJ lzxOwTYevlMF0sqDa2U1Q3X8U32+IWj5GLNA+PJBObeRpflZtp4Dsrim9sgWIGSVRBxv QPXA== X-Gm-Message-State: AO0yUKVpYFe/459s58Z4xVTmS0fwkFfBfxI/s1QczC0kd4SzzGA8hPHV okRluyVZPVyt+WaVr1g029583A== X-Google-Smtp-Source: AK7set9psF+XRxldeHUztWWScErtmaNpX1AHOLYMxaxlE9Y/phclTgMu2MScoA737D6nu3Ph+lOKWg== X-Received: by 2002:a05:600c:4fd2:b0:3dc:523c:22b4 with SMTP id o18-20020a05600c4fd200b003dc523c22b4mr2671866wmq.27.1675269334144; Wed, 01 Feb 2023 08:35:34 -0800 (PST) Received: from revest.zrh.corp.google.com ([2a00:79e0:9d:6:4399:89a1:4a86:9630]) by smtp.gmail.com with ESMTPSA id r38-20020a05600c322600b003dd7edcc960sm2058522wmp.45.2023.02.01.08.35.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 08:35:33 -0800 (PST) From: Florent Revest To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, rostedt@goodmis.org, mhiramat@kernel.org, mark.rutland@arm.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kpsingh@kernel.org, jolsa@kernel.org, xukuohai@huaweicloud.com, Florent Revest Subject: [PATCH 8/8] arm64: ftrace: Add direct called trampoline samples support Date: Wed, 1 Feb 2023 17:34:20 +0100 Message-Id: <20230201163420.1579014-9-revest@chromium.org> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog In-Reply-To: <20230201163420.1579014-1-revest@chromium.org> References: <20230201163420.1579014-1-revest@chromium.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org The ftrace samples need per-architecture trampoline implementations to save and restore argument registers around the calls to my_direct_func* and to restore polluted registers (eg: x30). These samples also include which does not exist on arm64 and which, on arm64, is not necessary and redefines previously defined macros (resulting in warnings) so these includes are guarded by !CONFIG_ARM64. Signed-off-by: Florent Revest --- arch/arm64/Kconfig | 2 ++ samples/ftrace/ftrace-direct-modify.c | 32 ++++++++++++++++++ samples/ftrace/ftrace-direct-multi-modify.c | 36 +++++++++++++++++++++ samples/ftrace/ftrace-direct-multi.c | 22 +++++++++++++ samples/ftrace/ftrace-direct-too.c | 25 ++++++++++++++ samples/ftrace/ftrace-direct.c | 23 +++++++++++++ 6 files changed, 140 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7deafd653c42..5480ef8eaa2a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -194,6 +194,8 @@ config ARM64 if (DYNAMIC_FTRACE_WITH_ARGS && !CFI_CLANG) select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \ if DYNAMIC_FTRACE_WITH_ARGS + select HAVE_SAMPLE_FTRACE_DIRECT + select HAVE_SAMPLE_FTRACE_DIRECT_MULTI select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c index 0f1b8859e1e1..944b327e130c 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -2,8 +2,10 @@ #include #include #include +#ifndef CONFIG_ARM64 #include #include +#endif extern void my_direct_func1(void); extern void my_direct_func2(void); @@ -96,6 +98,36 @@ asm ( #endif /* CONFIG_S390 */ +#ifdef CONFIG_ARM64 + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp1, @function\n" +" .globl my_tramp1\n" +" my_tramp1:" +" sub sp, sp, #16\n" +" stp x9, x30, [sp]\n" +" bl my_direct_func1\n" +" ldp x30, x9, [sp]\n" +" add sp, sp, #16\n" +" ret x9\n" +" .size my_tramp1, .-my_tramp1\n" + +" .type my_tramp2, @function\n" +" .globl my_tramp2\n" +" my_tramp2:" +" sub sp, sp, #16\n" +" stp x9, x30, [sp]\n" +" bl my_direct_func2\n" +" ldp x30, x9, [sp]\n" +" add sp, sp, #16\n" +" ret x9\n" +" .size my_tramp2, .-my_tramp2\n" +" .popsection\n" +); + +#endif /* CONFIG_ARM64 */ + static struct ftrace_ops direct; static unsigned long my_tramp = (unsigned long)my_tramp1; diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c index 407c56325e65..ae1f97271d1a 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -2,8 +2,10 @@ #include #include #include +#ifndef CONFIG_ARM64 #include #include +#endif extern void my_direct_func1(unsigned long ip); extern void my_direct_func2(unsigned long ip); @@ -103,6 +105,40 @@ asm ( #endif /* CONFIG_S390 */ +#ifdef CONFIG_ARM64 + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp1, @function\n" +" .globl my_tramp1\n" +" my_tramp1:" +" sub sp, sp, #32\n" +" stp x9, x30, [sp]\n" +" str x0, [sp, #16]\n" +" bl my_direct_func1\n" +" ldp x30, x9, [sp]\n" +" ldr x0, [sp, #16]\n" +" add sp, sp, #32\n" +" ret x9\n" +" .size my_tramp1, .-my_tramp1\n" + +" .type my_tramp2, @function\n" +" .globl my_tramp2\n" +" my_tramp2:" +" sub sp, sp, #32\n" +" stp x9, x30, [sp]\n" +" str x0, [sp, #16]\n" +" bl my_direct_func2\n" +" ldp x30, x9, [sp]\n" +" ldr x0, [sp, #16]\n" +" add sp, sp, #32\n" +" ret x9\n" +" .size my_tramp2, .-my_tramp2\n" +" .popsection\n" +); + +#endif /* CONFIG_ARM64 */ + static unsigned long my_tramp = (unsigned long)my_tramp1; static unsigned long tramps[2] = { (unsigned long)my_tramp1, diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c index 42a94806a446..84713fe28b74 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -4,8 +4,10 @@ #include /* for handle_mm_fault() */ #include #include +#ifndef CONFIG_ARM64 #include #include +#endif extern void my_direct_func(unsigned long ip); @@ -66,6 +68,26 @@ asm ( #endif /* CONFIG_S390 */ +#ifdef CONFIG_ARM64 + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp, @function\n" +" .globl my_tramp\n" +" my_tramp:" +" sub sp, sp, #32\n" +" stp x9, x30, [sp]\n" +" str x0, [sp, #16]\n" +" bl my_direct_func\n" +" ldp x30, x9, [sp]\n" +" ldr x0, [sp, #16]\n" +" add sp, sp, #32\n" +" ret x9\n" +" .size my_tramp, .-my_tramp\n" +" .popsection\n" +); + +#endif /* CONFIG_ARM64 */ static struct ftrace_ops direct; static int __init ftrace_direct_multi_init(void) diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c index 7ee5dd3cc61d..f46ee08caa2b 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -3,8 +3,10 @@ #include /* for handle_mm_fault() */ #include +#ifndef CONFIG_ARM64 #include #include +#endif extern void my_direct_func(struct vm_area_struct *vma, unsigned long address, unsigned int flags); @@ -70,6 +72,29 @@ asm ( #endif /* CONFIG_S390 */ +#ifdef CONFIG_ARM64 + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp, @function\n" +" .globl my_tramp\n" +" my_tramp:" +" sub sp, sp, #48\n" +" stp x9, x30, [sp]\n" +" stp x0, x1, [sp, #16]\n" +" str x2, [sp, #32]\n" +" bl my_direct_func\n" +" ldp x30, x9, [sp]\n" +" ldp x0, x1, [sp, #16]\n" +" ldr x2, [sp, #32]\n" +" add sp, sp, #48\n" +" ret x9\n" +" .size my_tramp, .-my_tramp\n" +" .popsection\n" +); + +#endif /* CONFIG_ARM64 */ + static struct ftrace_ops direct; static int __init ftrace_direct_init(void) diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 5ffce87fa83e..e37e8d9e855c 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -3,8 +3,10 @@ #include /* for wake_up_process() */ #include +#ifndef CONFIG_ARM64 #include #include +#endif extern void my_direct_func(struct task_struct *p); @@ -63,6 +65,27 @@ asm ( #endif /* CONFIG_S390 */ +#ifdef CONFIG_ARM64 + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp, @function\n" +" .globl my_tramp\n" +" my_tramp:" +" sub sp, sp, #32\n" +" stp x9, x30, [sp]\n" +" str x0, [sp, #16]\n" +" bl my_direct_func\n" +" ldp x30, x9, [sp]\n" +" ldr x0, [sp, #16]\n" +" add sp, sp, #32\n" +" ret x9\n" +" .size my_tramp, .-my_tramp\n" +" .popsection\n" +); + +#endif /* CONFIG_ARM64 */ + static struct ftrace_ops direct; static int __init ftrace_direct_init(void)