| Message ID | 20230705181256.3539027-9-vschneid@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | context_tracking,x86: Defer some IPIs until a user->kernel transition | expand |
On Wed, Jul 05, 2023 at 07:12:50PM +0100, Valentin Schneider wrote: > BROKEN: the struct static_key lives in a read-only mapping after > mark_rodata_ro(), which falls apart when the KVM module is loaded after > init and a write to the struct happens due to e.g. guest_state_exit_irqoff() > relying on the static key: Right.. so whoever added the whole ro_after_init jump_label support did a very poor job of it. That said; I think it is fixable. Since the key cannot be changed, we don't actually need to track the entries list and can thus avoid the key update. Something like the completely untested below... --- Subject: jump_label: Seal __ro_after_init keys When a static_key is marked ro_after_init, its state will never change (after init), therefore jump_label_update() will never need to iterate the entries, and thus module load won't actually need to track this -- avoiding the static_key::next write. Therefore, mark these keys such that jump_label_add_module() might recognise them and avoid the modification. Use the special state: 'static_key_linked(key) && !static_key_mod(key)' to denote such keys. *UNTESTED* NOT-Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> --- include/asm-generic/sections.h | 5 +++++ include/linux/jump_label.h | 1 + init/main.c | 1 + kernel/jump_label.c | 44 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 51 insertions(+) diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h index db13bb620f52..c768de6f19a9 100644 --- a/include/asm-generic/sections.h +++ b/include/asm-generic/sections.h @@ -180,6 +180,11 @@ static inline bool is_kernel_rodata(unsigned long addr) addr < (unsigned long)__end_rodata; } +static inline bool is_kernel_ro_after_init(unsigned long addr) +{ + return addr >= (unsigned long)__start_ro_after_init && + addr < (unsigned long)__end_ro_after_init; +} /** * is_kernel_inittext - checks if the pointer address is located in the * .init.text section diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index f0a949b7c973..88ef9e776af8 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -216,6 +216,7 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); +extern void jump_label_ro(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, diff --git a/init/main.c b/init/main.c index ad920fac325c..cb5304ca18f4 100644 --- a/init/main.c +++ b/init/main.c @@ -1403,6 +1403,7 @@ static void mark_readonly(void) * insecure pages which are W+X. */ rcu_barrier(); + jump_label_ro(); mark_rodata_ro(); rodata_test(); } else diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d9c822bbffb8..40fb72d79d7a 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -530,6 +530,46 @@ void __init jump_label_init(void) cpus_read_unlock(); } +static inline bool static_key_sealed(struct static_key *key) +{ + return (key->type & JUMP_TYPE_LINKED) && !(key->type & ~JUMP_TYPE_MASK); +} + +static inline void static_key_seal(struct static_key *key) +{ + unsigned long type = key->type & JUMP_TYPE_TRUE; + key->type = JUMP_TYPE_LINKED | type; +} + +void jump_label_ro(void) +{ + struct jump_entry *iter_start = __start___jump_table; + struct jump_entry *iter_stop = __stop___jump_table; + struct static_key *key = NULL; + struct jump_entry *iter; + + if (WARN_ON_ONCE(!static_key_initialized)) + return; + + cpus_read_lock(); + jump_label_lock(); + + for (iter = iter_start; iter < iter_stop; iter++) { + struct static_key *iterk = jump_entry_key(iter); + + if (!is_kernel_ro_after_init(iterk)) + continue; + + if (static_key_sealed(iterk)) + continue; + + static_key_seal(iterk); + } + + jump_label_unlock(); + cpus_read_unlock(); +} + #ifdef CONFIG_MODULES enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -650,6 +690,9 @@ static int jump_label_add_module(struct module *mod) static_key_set_entries(key, iter); continue; } + if (static_key_sealed(key)) + goto do_poke; + jlm = kzalloc(sizeof(struct static_key_mod), GFP_KERNEL); if (!jlm) return -ENOMEM; @@ -675,6 +718,7 @@ static int jump_label_add_module(struct module *mod) static_key_set_linked(key); /* Only update if we've changed from our initial state */ +do_poke: if (jump_label_type(iter) != jump_label_init_type(iter)) __jump_label_update(key, iter, iter_stop, true); }
diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a09f1c19336ae..4e6cb14272fcb 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -432,7 +432,7 @@ static __always_inline void ct_kernel_enter(bool user, int offset) { } #define CREATE_TRACE_POINTS #include <trace/events/context_tracking.h> -DEFINE_STATIC_KEY_FALSE(context_tracking_key); +DEFINE_STATIC_KEY_FALSE_RO(context_tracking_key); EXPORT_SYMBOL_GPL(context_tracking_key); static noinstr bool context_tracking_recursion_enter(void)
objtool now warns about it: vmlinux.o: warning: objtool: enter_from_user_mode+0x4e: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: enter_from_user_mode+0x50: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x60: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x62: Non __ro_after_init static key "context_tracking_key" in .noinstr section [...] The key can only be enabled (and not disabled) in the __init function ct_cpu_tracker_user(), so mark it as __ro_after_init. BROKEN: the struct static_key lives in a read-only mapping after mark_rodata_ro(), which falls apart when the KVM module is loaded after init and a write to the struct happens due to e.g. guest_state_exit_irqoff() relying on the static key: jump_label_add_module() `\ static_key_set_mod() static_key_set_linked() Signed-off-by: Valentin Schneider <vschneid@redhat.com> --- kernel/context_tracking.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)