From patchwork Mon Jul 2 18:11:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10502255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 20D5A6035E for ; Mon, 2 Jul 2018 18:14:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1221528816 for ; Mon, 2 Jul 2018 18:14:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 05B1428CA1; Mon, 2 Jul 2018 18:14:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 50BCF28816 for ; Mon, 2 Jul 2018 18:14:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HglSKDQPzzMZVskeOv5pj6EVP95ll1jSA+mOSEUPnGQ=; b=VWskRwysjNKXBzpGRL6xN7A7QC N/Lij28JNPhEPfS0VZxuHIu2Fz3uecufDNeRAPojTG9Jc199S1ppRRDCLcPJXonAhYyWzLnBeKeT3 ZDUmOgCYTve5i3vHSCR7vdup+5ZWM+4WamNKVYG8bYR3Zc+uAe5LxRjnrck6Brv2b3vvWXq7xcbYR YBtEIRVXfc5HCj5MEU82HtlrS5zd/tPOli3UWNGuzSI7BpBfIwX+xdamDTqZHIrBtV8LsEy5+XX9T 4T7TsJAIOXWk1+8D8Q7F4dabnoW78T+BqUNdGY/cYjMyxftuvY7CFt/FJpoQIN3BMLdjy4sJAXGfG 7R2MhPAg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fa3LL-0001Gz-OY; Mon, 02 Jul 2018 18:14:31 +0000 Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fa3JB-0007yM-Pa for linux-arm-kernel@lists.infradead.org; Mon, 02 Jul 2018 18:13:23 +0000 Received: by mail-wr0-x242.google.com with SMTP id q10-v6so1707711wrd.4 for ; Mon, 02 Jul 2018 11:12:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cadlqJXdwe8CrO/JHhgh75R+ewEWBI40jWFpP2FbnG8=; b=K+NImYqveGE4IO8GbXZTULp648/OGzvx2TZX0mMwc+QsX0v0pZ6Xl02YW9+hOXl54c 43oAykSex1GmOB4cPy9RJWZKPu+eNEsUEk3EpbRlZiwGF6/vKaBTwWryWz7NWjLRiEcL 0K2tAYhRTfnZwqqVuDeVZG5bFsMxetHEPuBX4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cadlqJXdwe8CrO/JHhgh75R+ewEWBI40jWFpP2FbnG8=; b=l55ywwX27SEHQGFXctCtUnUDPHQM6ldJm8DMlFwLvhvLF5jXXQCCd5mQ97mQXrpYLw V6TLo2yaZRAFU6Ypi/Qk6yz6RnumsDyAndvjUhAzsiVDXtggAiqVU1Bs3zHW3y2JBHKE CZkxQbvQ1qWD891ZXx9yIr+LM4fQF+bsGi4jfknlBuvS8Evlnovye5fIH2U+ZOcfUJdm IupGWirXAwt3RtL/4rHY8I++JoV2jfYoyh4X5KFfV7MnYIAWL+LcvqXgORGiOjVVkPnr IuunM6wPIbnnoZ64rhMxvRKISzqZwXDPu9vYoPBE1v8kt4jIFSLpSNWyjS1YFwAgpgP1 1SKg== X-Gm-Message-State: APt69E1xPFRJ+HLty+oz7gfrBEVGwPkpW1L0tHaMzVZu3xt2YFaLbiMg k9RwdYFkBGLs2H8eu1cu3d0g0/Phr2s= X-Google-Smtp-Source: AAOMgpf0pCd6+J1BeKm/pj/SofhXtAV+CSYkgvpLmgwPGA5GXCf3DY4B/zHU2A6JzwbeQcLC2FQaag== X-Received: by 2002:adf:c70e:: with SMTP id k14-v6mr19184697wrg.16.1530555128491; Mon, 02 Jul 2018 11:12:08 -0700 (PDT) Received: from localhost.localdomain (151.21.90.92.rev.sfr.net. [92.90.21.151]) by smtp.gmail.com with ESMTPSA id 189-v6sm10582822wmd.17.2018.07.02.11.12.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Jul 2018 11:12:07 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, linux-arch@vger.kernel.org Subject: [PATCH v2 7/8] jump_label: annotate entries that operate on __init code earlier Date: Mon, 2 Jul 2018 20:11:44 +0200 Message-Id: <20180702181145.4799-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180702181145.4799-1-ard.biesheuvel@linaro.org> References: <20180702181145.4799-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180702_111218_215042_E6FEF7F8 X-CRM114-Status: GOOD ( 23.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Heiko Carstens , Jessica Yu , Will Deacon , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Jump table entries are mostly read-only, with the exception of the init and module loader code that defuses entries that point into init code when the code being referred to is freed. For robustness, it would be better to move these entries into the ro_after_init section, but clearing the 'code' member of each jump table entry referring to init code at module load time races with the module_enable_ro() call that remaps the ro_after_init section read only, so we'd like to do it earlier. So given that whether such an entry refers to init code can be decided much earlier, we can pull this check forward. Since we may still need the code entry at this point, let's switch to setting a low bit in the 'key' member just like we do to annotate the default state of a jump table entry. Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook --- include/linux/jump_label.h | 11 ++--- init/main.c | 1 - kernel/jump_label.c | 48 ++++++-------------- 3 files changed, 18 insertions(+), 42 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 871826fd0c3b..feee8abc96be 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -141,7 +141,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - long offset = entry->key & ~1L; + long offset = entry->key & ~3L; return (struct static_key *)((unsigned long)&entry->key + offset); } @@ -160,7 +160,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + return (struct static_key *)((unsigned long)entry->key & ~3UL); } #endif @@ -172,12 +172,12 @@ static inline bool jump_entry_is_branch(const struct jump_entry *entry) static inline bool jump_entry_is_init(const struct jump_entry *entry) { - return entry->code == 0; + return (unsigned long)entry->key & 2UL; } static inline void jump_entry_set_init(struct jump_entry *entry) { - entry->code = 0; + entry->key |= 2; } #endif @@ -213,7 +213,6 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); -extern void jump_label_invalidate_initmem(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, @@ -261,8 +260,6 @@ static __always_inline void jump_label_init(void) static_key_initialized = true; } -static inline void jump_label_invalidate_initmem(void) {} - static __always_inline bool static_key_false(struct static_key *key) { if (unlikely(static_key_count(key) > 0)) diff --git a/init/main.c b/init/main.c index e59a01f163d6..d1a6b8a896e5 100644 --- a/init/main.c +++ b/init/main.c @@ -1062,7 +1062,6 @@ static int __ref kernel_init(void *unused) /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); ftrace_free_init_mem(); - jump_label_invalidate_initmem(); free_initmem(); mark_readonly(); system_state = SYSTEM_RUNNING; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d424e1d22d63..7cdd49aeaf6a 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -373,14 +373,15 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) static void __jump_label_update(struct static_key *key, struct jump_entry *entry, - struct jump_entry *stop) + struct jump_entry *stop, + bool init) { for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) { /* * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (!jump_entry_is_init(entry)) { + if (init || !jump_entry_is_init(entry)) { if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else @@ -420,6 +421,9 @@ void __init jump_label_init(void) if (jump_label_type(iter) == JUMP_LABEL_NOP) arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -432,19 +436,6 @@ void __init jump_label_init(void) cpus_read_unlock(); } -/* Disable any jump label entries in __init/__exit code */ -void __init jump_label_invalidate_initmem(void) -{ - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)jump_entry_code(iter), 1)) - jump_entry_set_init(iter); - } -} - #ifdef CONFIG_MODULES static enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -524,7 +515,8 @@ static void __jump_label_mod_update(struct static_key *key) stop = __stop___jump_table; else stop = m->jump_entries + m->num_jump_entries; - __jump_label_update(key, mod->entries, stop); + __jump_label_update(key, mod->entries, stop, + m->state == MODULE_STATE_COMING); } } @@ -570,6 +562,9 @@ static int jump_label_add_module(struct module *mod) for (iter = iter_start; iter < iter_stop; iter++) { struct static_key *iterk; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -605,7 +600,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ if (jump_label_type(iter) != jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop); + __jump_label_update(key, iter, iter_stop, true); } return 0; @@ -661,19 +656,6 @@ static void jump_label_del_module(struct module *mod) } } -/* Disable any jump label entries in module init code */ -static void jump_label_invalidate_module_init(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(jump_entry_code(iter), mod)) - jump_entry_set_init(iter); - } -} - static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data) @@ -695,9 +677,6 @@ jump_label_module_notify(struct notifier_block *self, unsigned long val, case MODULE_STATE_GOING: jump_label_del_module(mod); break; - case MODULE_STATE_LIVE: - jump_label_invalidate_module_init(mod); - break; } jump_label_unlock(); @@ -767,7 +746,8 @@ static void jump_label_update(struct static_key *key) entry = static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop); + __jump_label_update(key, entry, stop, + system_state < SYSTEM_RUNNING); } #ifdef CONFIG_STATIC_KEYS_SELFTEST