From patchwork Thu Jul 20 16:30:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C860C00528 for ; Thu, 20 Jul 2023 16:33:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADB4E280138; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A647128004C; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B759280138; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 794D128004C for ; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 52E96160210 for ; Thu, 20 Jul 2023 16:33:26 +0000 (UTC) X-FDA: 81032535612.05.D5F53A2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 5CD5440017 for ; Thu, 20 Jul 2023 16:33:24 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkTLPdFg; spf=pass (imf17.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870804; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5HBHaVT0pfIJAbbI8C24iwVVdenu59ApFnKMQXGGMQ=; b=djd0FhlwuY2U0cJ6PSsRvVI46UiuVG/GfgTLsXGVEaBjgQ8uq7Nzi/1HYrHtzStfcVHaTR Pw435yc/6kFI6pSyWIqBKuxgyKVe5OmJfRk0uKDNFFQg/IX6K1RFTf/nBZAe3cMSMHFVI+ iRtVOg7j/Iv5a31owHfGcJ5Hu6T7wEQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkTLPdFg; spf=pass (imf17.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870804; a=rsa-sha256; cv=none; b=XB7ELUQKxGZWFyPCqVYs8gUB4Xyy8P2mPlTHtgYccisVXnnYw4LPLBxASR1V0xihlX8EuI 9QtekAIlFoE3aRlZULHA4xYtC3KjOcsdenNpcDxxbhiSVjfmy2eRaPBT0VMmt8HIA+qB7p IQotwz5ijy2G3L1lQqmGlW0WDzb88aE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G5HBHaVT0pfIJAbbI8C24iwVVdenu59ApFnKMQXGGMQ=; b=PkTLPdFgleBGcxnnJQ8Tb0+vei7LSM4eGj6jWRj6xyCBPTfmDFltcVMLYaBYxXYc41Gve5 09CPyfrMJap5knsTV5fiL2SYblhtRkydia5ATZicDi6d4zwVF38jZy/AC2yB/4a6rjGytN STzFqzVY5W0LUU51gy2xwUdQM8nFNeg= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-543-hhF8YKTPPJa4Xr8UF_kvXw-1; Thu, 20 Jul 2023 12:33:21 -0400 X-MC-Unique: hhF8YKTPPJa4Xr8UF_kvXw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 679323815EF2; Thu, 20 Jul 2023 16:33:18 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E3F8940C206F; Thu, 20 Jul 2023 16:33:10 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 10/20] jump_label,module: Don't alloc static_key_mod for __ro_after_init keys Date: Thu, 20 Jul 2023 17:30:46 +0100 Message-Id: <20230720163056.2564824-11-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 5CD5440017 X-Rspam-User: X-Stat-Signature: 9irw839t5bbuu1rgnqkw7zbi8nknwyap X-Rspamd-Server: rspam01 X-HE-Tag: 1689870804-625382 X-HE-Meta: U2FsdGVkX18JdbDx+RAjqmeXZOCgNjCDYGKX1i5pzIl/By0OBM/wGYh1eMCI3ycvmr4Wsn2EKN+cNk4WITJ6OvkCNljexIByEdZAEWNbgE8GMO2Opb9gHQATJt5tn21EWSSupa6A3fPH4LTdw+h9sdKaC2HW347TDd4btFFZ+I2UsqrPzXFynE1v+kKFI81wRUbyiu76MeC+iQYT1gYgFn9SlfgkDqrzvRBkwueLo5SCSoq0C+NKHD8hciccheeyl9VL/MQKE7+J1ZJDbAuzojbsb6uzPBKWYUs1XCzPTe8RjzJVXgt3gr6xspo7W1YIxA5GZsrE6JnQpI3lYUx+ZAb/IWlWiKHpzrtZMKhKR8TT1c+rGzZOwWIZuSML01siSiX4PD/hfhbVAJoPJyPzpg613JAjj5aVAeQB0nDMVP/HrQL/kjubdy3XSehK0ay9NnXMwGxOlVvELp4F5KpDvbIE11QOdHN41IK+gTEMcP5V2xrB+K3GsWhCtouTm8ZnTGjgReHTei9qhUJNlJjOACXje5BCGxlGtLVWnYNl3vWOIwDQ1CV925yUhs/P4GMRsvb0wmOYdKz5U0jrOGcaApxyLembez4T4yFUEtWVF9AVk0M+bM2OuOh1lVPY2pQQECglWM9Xxzwim93Vhk8LLRn/TcnDkuW1+MGa1gedrnNUSXbX0EjRwKHt66tRdIMdrR8Uk4xuz9uWS2BmqBh9icMp0Q6/l29NmvE99U9mDtxYMC3TEAi7EleB5QeLkOZ1lTn59YxPEsuz/2MnsPS+4zFHqv2nIrcxUoD/tKhwpK1QKs3RModmoxrMynQ4Zg6ZaVVfANrdSgjoEMf2R7coSBZEaKWYWO2NbpScBEOXNVyHfMSXCTdtPC3dSzToUBjPw8QNzLA49Tmowi6m9mTwUtDDEqOySENarVGWXKFHtg4RJEWqLUMeYEO5TY5Z8D7UmB8ZXKwJVCZWmjeu8kX EFvBcpQD diw0j3K8jYu6mU98pwFAAHHJnTjah4pQn4h1bqQdqKS2DTdxmVixmBE3gHe78tOj+ndpjNVhrgyQcWfyeYLQ6W06OfsnC7BiFSZdEjL6AhkH5grffT58fz1/3vxCFDNeAy50I77ChQmfNee6xVh6UnsR43xB7y6un50dByYO5Cv6Xh+3ACuUPm9xmc0Kv6nU5NYR8qBOMoJootSMOj96MPuVE3KKqOhcUtP5MryBbDCeBnWC6ypV9LNO8zjet2W6JS+6d2nT7Arx/amjR/k2JAnKrsAoNHB96mc3R6FOs4nsxaKMGQGwFM1YVX7psZtP4eWUmiV3QLISpcihxjmr1xHlkyI0fRMDlkzT8meTJ/VYPiIlZ1Rm+QT8n5X6vYtn96/iJjUKSawW73gnBFSVhhenOCO4FA3pSmYY9E6NckLTs8qF/VvvsXAyPfgIFdXMWhWy7PimpwfTAIsGT5zlp7zNjTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Zijlstra When a static_key is marked ro_after_init, its state will never change (after init), therefore jump_label_update() will never need to iterate the entries, and thus module load won't actually need to track this -- avoiding the static_key::next write. Therefore, mark these keys such that jump_label_add_module() might recognise them and avoid the modification. Use the special state: 'static_key_linked(key) && !static_key_mod(key)' to denote such keys. Link: http://lore.kernel.org/r/20230705204142.GB2813335@hirez.programming.kicks-ass.net NOT-Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Valentin Schneider --- @Peter: I've barely touched this patch, it's just been writing a comment and fixing benign compilation issues, so credit's all yours really! --- include/asm-generic/sections.h | 5 ++++ include/linux/jump_label.h | 1 + init/main.c | 1 + kernel/jump_label.c | 49 ++++++++++++++++++++++++++++++++++ 4 files changed, 56 insertions(+) diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h index db13bb620f527..c768de6f19a9a 100644 --- a/include/asm-generic/sections.h +++ b/include/asm-generic/sections.h @@ -180,6 +180,11 @@ static inline bool is_kernel_rodata(unsigned long addr) addr < (unsigned long)__end_rodata; } +static inline bool is_kernel_ro_after_init(unsigned long addr) +{ + return addr >= (unsigned long)__start_ro_after_init && + addr < (unsigned long)__end_ro_after_init; +} /** * is_kernel_inittext - checks if the pointer address is located in the * .init.text section diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index f0a949b7c9733..88ef9e776af8d 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -216,6 +216,7 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); +extern void jump_label_ro(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, diff --git a/init/main.c b/init/main.c index ad920fac325c3..cb5304ca18f4d 100644 --- a/init/main.c +++ b/init/main.c @@ -1403,6 +1403,7 @@ static void mark_readonly(void) * insecure pages which are W+X. */ rcu_barrier(); + jump_label_ro(); mark_rodata_ro(); rodata_test(); } else diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d9c822bbffb8d..661ef74dee9b7 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -530,6 +530,45 @@ void __init jump_label_init(void) cpus_read_unlock(); } +static inline bool static_key_sealed(struct static_key *key) +{ + return (key->type & JUMP_TYPE_LINKED) && !(key->type & ~JUMP_TYPE_MASK); +} + +static inline void static_key_seal(struct static_key *key) +{ + unsigned long type = key->type & JUMP_TYPE_TRUE; + key->type = JUMP_TYPE_LINKED | type; +} + +void jump_label_ro(void) +{ + struct jump_entry *iter_start = __start___jump_table; + struct jump_entry *iter_stop = __stop___jump_table; + struct jump_entry *iter; + + if (WARN_ON_ONCE(!static_key_initialized)) + return; + + cpus_read_lock(); + jump_label_lock(); + + for (iter = iter_start; iter < iter_stop; iter++) { + struct static_key *iterk = jump_entry_key(iter); + + if (!is_kernel_ro_after_init((unsigned long)iterk)) + continue; + + if (static_key_sealed(iterk)) + continue; + + static_key_seal(iterk); + } + + jump_label_unlock(); + cpus_read_unlock(); +} + #ifdef CONFIG_MODULES enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -650,6 +689,15 @@ static int jump_label_add_module(struct module *mod) static_key_set_entries(key, iter); continue; } + + /* + * If the key was sealed at init, then there's no need to keep a + * a reference to its module entries - just patch them now and + * be done with it. + */ + if (static_key_sealed(key)) + goto do_poke; + jlm = kzalloc(sizeof(struct static_key_mod), GFP_KERNEL); if (!jlm) return -ENOMEM; @@ -675,6 +723,7 @@ static int jump_label_add_module(struct module *mod) static_key_set_linked(key); /* Only update if we've changed from our initial state */ +do_poke: if (jump_label_type(iter) != jump_label_init_type(iter)) __jump_label_update(key, iter, iter_stop, true); }