| Message ID | 20240326134616.7691-1-jarkko@kernel.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [v7,1/2] kprobes: Implement trampoline memory allocator for tracing | expand |
Le 26/03/2024 à 14:46, Jarkko Sakkinen a écrit : > Tracing with kprobes while running a monolithic kernel is currently > impossible because CONFIG_KPROBES depends on CONFIG_MODULES. > > Introduce alloc_execmem() and free_execmem() for allocating executable > memory. If an arch implements these functions, it can mark this up with > the HAVE_ALLOC_EXECMEM kconfig flag. > > The second new kconfig flag is ALLOC_EXECMEM, which can be selected if > either MODULES is selected or HAVE_ALLOC_EXECMEM is support by the arch. If > HAVE_ALLOC_EXECMEM is not supported by an arch, module_alloc() and > module_memfree() are used as a fallback, thus retaining backwards > compatibility to earlier kernel versions. > > This will allow architecture to enable kprobes traces without requiring > to enable module. > > The support can be implemented with four easy steps: > > 1. Implement alloc_execmem(). > 2. Implement free_execmem(). > 3. Edit arch/<arch>/Makefile. > 4. Set HAVE_ALLOC_EXECMEM in arch/<arch>/Kconfig. > > Link: https://lore.kernel.org/all/20240325115632.04e37297491cadfbbf382767@kernel.org/ > Suggested-by: Masami Hiramatsu <mhiramat@kernel.org> > Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org> > --- > v7: > - Use "depends on" for ALLOC_EXECMEM instead of "select" > - Reduced and narrowed CONFIG_MODULES checks further in kprobes.c. > v6: > - Use null pointer for notifiers and register the module notifier only if > IS_ENABLED(CONFIG_MODULES) is set. > - Fixed typo in the commit message and wrote more verbose description > of the feature. > v5: > - alloc_execmem() was missing GFP_KERNEL parameter. The patch set did > compile because 2/2 had the fixup (leaked there when rebasing the > patch set). > v4: > - Squashed a couple of unrequired CONFIG_MODULES checks. > - See https://lore.kernel.org/all/D034M18D63EC.2Y11D954YSZYK@kernel.org/ > v3: > - A new patch added. > - For IS_DEFINED() I need advice as I could not really find that many > locations where it would be applicable. > --- > arch/Kconfig | 17 +++++++++++- > include/linux/execmem.h | 13 +++++++++ > kernel/kprobes.c | 53 ++++++++++++++++++++++--------------- > kernel/trace/trace_kprobe.c | 15 +++++++++-- > 4 files changed, 73 insertions(+), 25 deletions(-) > create mode 100644 include/linux/execmem.h > > diff --git a/arch/Kconfig b/arch/Kconfig > index a5af0edd3eb8..5e9735f60f3c 100644 > --- a/arch/Kconfig > +++ b/arch/Kconfig > @@ -52,8 +52,8 @@ config GENERIC_ENTRY > > config KPROBES > bool "Kprobes" > - depends on MODULES > depends on HAVE_KPROBES > + depends on ALLOC_EXECMEM > select KALLSYMS > select TASKS_RCU if PREEMPTION > help > @@ -215,6 +215,21 @@ config HAVE_OPTPROBES > config HAVE_KPROBES_ON_FTRACE > bool > > +config HAVE_ALLOC_EXECMEM > + bool > + help > + Architectures that select this option are capable of allocating trampoline > + executable memory for tracing subsystems, indepedently of the kernel module > + subsystem. > + > +config ALLOC_EXECMEM > + bool "Executable (trampoline) memory allocation" Why make it user selectable ? Previously I was able to select KPROBES as soon as MODULES was selected. Now I will have to first select ALLOC_EXECMEM in addition ? What is the added value of allowing the user to disable it ? > + default y > + depends on MODULES || HAVE_ALLOC_EXECMEM > + help > + Select this for executable (trampoline) memory. Can be enabled when either > + module allocator or arch-specific allocator is available. > + > config ARCH_CORRECT_STACKTRACE_ON_KRETPROBE > bool > help > diff --git a/include/linux/execmem.h b/include/linux/execmem.h > new file mode 100644 > index 000000000000..ae2ff151523a > --- /dev/null > +++ b/include/linux/execmem.h > @@ -0,0 +1,13 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +#ifndef _LINUX_EXECMEM_H > +#define _LINUX_EXECMEM_H > + It should include moduleloader.h otherwise the user of alloc_execmem() must include both this header and moduleloader.h to use alloc_execmem() > +#ifdef CONFIG_HAVE_ALLOC_EXECMEM > +void *alloc_execmem(unsigned long size, gfp_t gfp); > +void free_execmem(void *region); > +#else > +#define alloc_execmem(size, gfp) module_alloc(size) Then gfp is silently ignored in the case. Is that expected ? > +#define free_execmem(region) module_memfree(region) > +#endif > + > +#endif /* _LINUX_EXECMEM_H */ > diff --git a/kernel/kprobes.c b/kernel/kprobes.c > index 9d9095e81792..13bef5de315c 100644 > --- a/kernel/kprobes.c > +++ b/kernel/kprobes.c > @@ -44,6 +44,7 @@ > #include <asm/cacheflush.h> > #include <asm/errno.h> > #include <linux/uaccess.h> > +#include <linux/execmem.h> > > #define KPROBE_HASH_BITS 6 > #define KPROBE_TABLE_SIZE (1 << KPROBE_HASH_BITS) > @@ -113,17 +114,17 @@ enum kprobe_slot_state { > void __weak *alloc_insn_page(void) > { > /* > - * Use module_alloc() so this page is within +/- 2GB of where the > + * Use alloc_execmem() so this page is within +/- 2GB of where the This 2G stuff is x86 specific, might be different value on other arches. Can we remove ? > * kernel image and loaded module images reside. This is required > * for most of the architectures. > * (e.g. x86-64 needs this to handle the %rip-relative fixups.) > */ > - return module_alloc(PAGE_SIZE); > + return alloc_execmem(PAGE_SIZE, GFP_KERNEL); > } > > static void free_insn_page(void *page) > { > - module_memfree(page); > + free_execmem(page); > } > > struct kprobe_insn_cache kprobe_insn_slots = { > @@ -1592,6 +1593,7 @@ static int check_kprobe_address_safe(struct kprobe *p, > goto out; > } > > +#ifdef CONFIG_MODULES > /* > * If the module freed '.init.text', we couldn't insert > * kprobes in there. > @@ -1602,7 +1604,9 @@ static int check_kprobe_address_safe(struct kprobe *p, > *probed_mod = NULL; > ret = -ENOENT; > } > +#endif /* CONFIG_MODULES */ > } > + > out: > preempt_enable(); > jump_label_unlock(); > @@ -2482,24 +2486,6 @@ int kprobe_add_area_blacklist(unsigned long start, unsigned long end) > return 0; > } > > -/* Remove all symbols in given area from kprobe blacklist */ > -static void kprobe_remove_area_blacklist(unsigned long start, unsigned long end) > -{ > - struct kprobe_blacklist_entry *ent, *n; > - > - list_for_each_entry_safe(ent, n, &kprobe_blacklist, list) { > - if (ent->start_addr < start || ent->start_addr >= end) > - continue; > - list_del(&ent->list); > - kfree(ent); > - } > -} > - > -static void kprobe_remove_ksym_blacklist(unsigned long entry) > -{ > - kprobe_remove_area_blacklist(entry, entry + 1); > -} > - > int __weak arch_kprobe_get_kallsym(unsigned int *symnum, unsigned long *value, > char *type, char *sym) > { > @@ -2564,6 +2550,25 @@ static int __init populate_kprobe_blacklist(unsigned long *start, > return ret ? : arch_populate_kprobe_blacklist(); > } > > +#ifdef CONFIG_MODULES > +/* Remove all symbols in given area from kprobe blacklist */ > +static void kprobe_remove_area_blacklist(unsigned long start, unsigned long end) > +{ > + struct kprobe_blacklist_entry *ent, *n; > + > + list_for_each_entry_safe(ent, n, &kprobe_blacklist, list) { > + if (ent->start_addr < start || ent->start_addr >= end) > + continue; > + list_del(&ent->list); > + kfree(ent); > + } > +} > + > +static void kprobe_remove_ksym_blacklist(unsigned long entry) > +{ > + kprobe_remove_area_blacklist(entry, entry + 1); > +} > + > static void add_module_kprobe_blacklist(struct module *mod) > { > unsigned long start, end; > @@ -2660,6 +2665,9 @@ static int kprobes_module_callback(struct notifier_block *nb, > mutex_unlock(&kprobe_mutex); > return NOTIFY_DONE; > } > +#else > +#define kprobes_module_callback (NULL) Can you use a static inline instead ? It allows typechecking at build. > +#endif /* CONFIG_MODULES */ > > static struct notifier_block kprobe_module_nb = { > .notifier_call = kprobes_module_callback, > @@ -2724,7 +2732,8 @@ static int __init init_kprobes(void) > err = arch_init_kprobes(); > if (!err) > err = register_die_notifier(&kprobe_exceptions_nb); > - if (!err) > + > + if (!err && IS_ENABLED(CONFIG_MODULES)) > err = register_module_notifier(&kprobe_module_nb); > > kprobes_initialized = (err == 0); > diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c > index c4c6e0e0068b..13ea0d0994d5 100644 > --- a/kernel/trace/trace_kprobe.c > +++ b/kernel/trace/trace_kprobe.c > @@ -111,6 +111,7 @@ static nokprobe_inline bool trace_kprobe_within_module(struct trace_kprobe *tk, > return strncmp(module_name(mod), name, len) == 0 && name[len] == ':'; > } > > +#ifdef CONFIG_MODULES > static nokprobe_inline bool trace_kprobe_module_exist(struct trace_kprobe *tk) > { > char *p; > @@ -129,6 +130,9 @@ static nokprobe_inline bool trace_kprobe_module_exist(struct trace_kprobe *tk) > > return ret; > } > +#else > +#define trace_kprobe_module_exist(tk) false /* aka a module never exists */ same, can it be a static inline instead ? > +#endif /* CONFIG_MODULES */ > > static bool trace_kprobe_is_busy(struct dyn_event *ev) > { > @@ -670,6 +674,7 @@ static int register_trace_kprobe(struct trace_kprobe *tk) > return ret; > } > > +#ifdef CONFIG_MODULES > /* Module notifier call back, checking event on the module */ > static int trace_kprobe_module_callback(struct notifier_block *nb, > unsigned long val, void *data) > @@ -699,6 +704,9 @@ static int trace_kprobe_module_callback(struct notifier_block *nb, > > return NOTIFY_DONE; > } > +#else > +#define trace_kprobe_module_callback (NULL) Same > +#endif /* CONFIG_MODULES */ > > static struct notifier_block trace_kprobe_module_nb = { > .notifier_call = trace_kprobe_module_callback, > @@ -1897,8 +1905,11 @@ static __init int init_kprobe_trace_early(void) > if (ret) > return ret; > > - if (register_module_notifier(&trace_kprobe_module_nb)) > - return -EINVAL; > + if (IS_ENABLED(CONFIG_MODULES)) { > + ret = register_module_notifier(&trace_kprobe_module_nb); > + if (ret) > + return -EINVAL; > + } No need for that, register_module_notifier() returns 0 when CONFIG_MODULES is not set. > > return 0; > }
diff --git a/arch/Kconfig b/arch/Kconfig index a5af0edd3eb8..5e9735f60f3c 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -52,8 +52,8 @@ config GENERIC_ENTRY config KPROBES bool "Kprobes" - depends on MODULES depends on HAVE_KPROBES + depends on ALLOC_EXECMEM select KALLSYMS select TASKS_RCU if PREEMPTION help @@ -215,6 +215,21 @@ config HAVE_OPTPROBES config HAVE_KPROBES_ON_FTRACE bool +config HAVE_ALLOC_EXECMEM + bool + help + Architectures that select this option are capable of allocating trampoline + executable memory for tracing subsystems, indepedently of the kernel module + subsystem. + +config ALLOC_EXECMEM + bool "Executable (trampoline) memory allocation" + default y + depends on MODULES || HAVE_ALLOC_EXECMEM + help + Select this for executable (trampoline) memory. Can be enabled when either + module allocator or arch-specific allocator is available. + config ARCH_CORRECT_STACKTRACE_ON_KRETPROBE bool help diff --git a/include/linux/execmem.h b/include/linux/execmem.h new file mode 100644 index 000000000000..ae2ff151523a --- /dev/null +++ b/include/linux/execmem.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_EXECMEM_H +#define _LINUX_EXECMEM_H + +#ifdef CONFIG_HAVE_ALLOC_EXECMEM +void *alloc_execmem(unsigned long size, gfp_t gfp); +void free_execmem(void *region); +#else +#define alloc_execmem(size, gfp) module_alloc(size) +#define free_execmem(region) module_memfree(region) +#endif + +#endif /* _LINUX_EXECMEM_H */ diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 9d9095e81792..13bef5de315c 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -44,6 +44,7 @@ #include <asm/cacheflush.h> #include <asm/errno.h> #include <linux/uaccess.h> +#include <linux/execmem.h> #define KPROBE_HASH_BITS 6 #define KPROBE_TABLE_SIZE (1 << KPROBE_HASH_BITS) @@ -113,17 +114,17 @@ enum kprobe_slot_state { void __weak *alloc_insn_page(void) { /* - * Use module_alloc() so this page is within +/- 2GB of where the + * Use alloc_execmem() so this page is within +/- 2GB of where the * kernel image and loaded module images reside. This is required * for most of the architectures. * (e.g. x86-64 needs this to handle the %rip-relative fixups.) */ - return module_alloc(PAGE_SIZE); + return alloc_execmem(PAGE_SIZE, GFP_KERNEL); } static void free_insn_page(void *page) { - module_memfree(page); + free_execmem(page); } struct kprobe_insn_cache kprobe_insn_slots = { @@ -1592,6 +1593,7 @@ static int check_kprobe_address_safe(struct kprobe *p, goto out; } +#ifdef CONFIG_MODULES /* * If the module freed '.init.text', we couldn't insert * kprobes in there. @@ -1602,7 +1604,9 @@ static int check_kprobe_address_safe(struct kprobe *p, *probed_mod = NULL; ret = -ENOENT; } +#endif /* CONFIG_MODULES */ } + out: preempt_enable(); jump_label_unlock(); @@ -2482,24 +2486,6 @@ int kprobe_add_area_blacklist(unsigned long start, unsigned long end) return 0; } -/* Remove all symbols in given area from kprobe blacklist */ -static void kprobe_remove_area_blacklist(unsigned long start, unsigned long end) -{ - struct kprobe_blacklist_entry *ent, *n; - - list_for_each_entry_safe(ent, n, &kprobe_blacklist, list) { - if (ent->start_addr < start || ent->start_addr >= end) - continue; - list_del(&ent->list); - kfree(ent); - } -} - -static void kprobe_remove_ksym_blacklist(unsigned long entry) -{ - kprobe_remove_area_blacklist(entry, entry + 1); -} - int __weak arch_kprobe_get_kallsym(unsigned int *symnum, unsigned long *value, char *type, char *sym) { @@ -2564,6 +2550,25 @@ static int __init populate_kprobe_blacklist(unsigned long *start, return ret ? : arch_populate_kprobe_blacklist(); } +#ifdef CONFIG_MODULES +/* Remove all symbols in given area from kprobe blacklist */ +static void kprobe_remove_area_blacklist(unsigned long start, unsigned long end) +{ + struct kprobe_blacklist_entry *ent, *n; + + list_for_each_entry_safe(ent, n, &kprobe_blacklist, list) { + if (ent->start_addr < start || ent->start_addr >= end) + continue; + list_del(&ent->list); + kfree(ent); + } +} + +static void kprobe_remove_ksym_blacklist(unsigned long entry) +{ + kprobe_remove_area_blacklist(entry, entry + 1); +} + static void add_module_kprobe_blacklist(struct module *mod) { unsigned long start, end; @@ -2660,6 +2665,9 @@ static int kprobes_module_callback(struct notifier_block *nb, mutex_unlock(&kprobe_mutex); return NOTIFY_DONE; } +#else +#define kprobes_module_callback (NULL) +#endif /* CONFIG_MODULES */ static struct notifier_block kprobe_module_nb = { .notifier_call = kprobes_module_callback, @@ -2724,7 +2732,8 @@ static int __init init_kprobes(void) err = arch_init_kprobes(); if (!err) err = register_die_notifier(&kprobe_exceptions_nb); - if (!err) + + if (!err && IS_ENABLED(CONFIG_MODULES)) err = register_module_notifier(&kprobe_module_nb); kprobes_initialized = (err == 0); diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index c4c6e0e0068b..13ea0d0994d5 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c @@ -111,6 +111,7 @@ static nokprobe_inline bool trace_kprobe_within_module(struct trace_kprobe *tk, return strncmp(module_name(mod), name, len) == 0 && name[len] == ':'; } +#ifdef CONFIG_MODULES static nokprobe_inline bool trace_kprobe_module_exist(struct trace_kprobe *tk) { char *p; @@ -129,6 +130,9 @@ static nokprobe_inline bool trace_kprobe_module_exist(struct trace_kprobe *tk) return ret; } +#else +#define trace_kprobe_module_exist(tk) false /* aka a module never exists */ +#endif /* CONFIG_MODULES */ static bool trace_kprobe_is_busy(struct dyn_event *ev) { @@ -670,6 +674,7 @@ static int register_trace_kprobe(struct trace_kprobe *tk) return ret; } +#ifdef CONFIG_MODULES /* Module notifier call back, checking event on the module */ static int trace_kprobe_module_callback(struct notifier_block *nb, unsigned long val, void *data) @@ -699,6 +704,9 @@ static int trace_kprobe_module_callback(struct notifier_block *nb, return NOTIFY_DONE; } +#else +#define trace_kprobe_module_callback (NULL) +#endif /* CONFIG_MODULES */ static struct notifier_block trace_kprobe_module_nb = { .notifier_call = trace_kprobe_module_callback, @@ -1897,8 +1905,11 @@ static __init int init_kprobe_trace_early(void) if (ret) return ret; - if (register_module_notifier(&trace_kprobe_module_nb)) - return -EINVAL; + if (IS_ENABLED(CONFIG_MODULES)) { + ret = register_module_notifier(&trace_kprobe_module_nb); + if (ret) + return -EINVAL; + } return 0; }
Tracing with kprobes while running a monolithic kernel is currently impossible because CONFIG_KPROBES depends on CONFIG_MODULES. Introduce alloc_execmem() and free_execmem() for allocating executable memory. If an arch implements these functions, it can mark this up with the HAVE_ALLOC_EXECMEM kconfig flag. The second new kconfig flag is ALLOC_EXECMEM, which can be selected if either MODULES is selected or HAVE_ALLOC_EXECMEM is support by the arch. If HAVE_ALLOC_EXECMEM is not supported by an arch, module_alloc() and module_memfree() are used as a fallback, thus retaining backwards compatibility to earlier kernel versions. This will allow architecture to enable kprobes traces without requiring to enable module. The support can be implemented with four easy steps: 1. Implement alloc_execmem(). 2. Implement free_execmem(). 3. Edit arch/<arch>/Makefile. 4. Set HAVE_ALLOC_EXECMEM in arch/<arch>/Kconfig. Link: https://lore.kernel.org/all/20240325115632.04e37297491cadfbbf382767@kernel.org/ Suggested-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org> --- v7: - Use "depends on" for ALLOC_EXECMEM instead of "select" - Reduced and narrowed CONFIG_MODULES checks further in kprobes.c. v6: - Use null pointer for notifiers and register the module notifier only if IS_ENABLED(CONFIG_MODULES) is set. - Fixed typo in the commit message and wrote more verbose description of the feature. v5: - alloc_execmem() was missing GFP_KERNEL parameter. The patch set did compile because 2/2 had the fixup (leaked there when rebasing the patch set). v4: - Squashed a couple of unrequired CONFIG_MODULES checks. - See https://lore.kernel.org/all/D034M18D63EC.2Y11D954YSZYK@kernel.org/ v3: - A new patch added. - For IS_DEFINED() I need advice as I could not really find that many locations where it would be applicable. --- arch/Kconfig | 17 +++++++++++- include/linux/execmem.h | 13 +++++++++ kernel/kprobes.c | 53 ++++++++++++++++++++++--------------- kernel/trace/trace_kprobe.c | 15 +++++++++-- 4 files changed, 73 insertions(+), 25 deletions(-) create mode 100644 include/linux/execmem.h