From patchwork Wed Sep 19 06:51:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605379 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 977DB112B for ; Wed, 19 Sep 2018 06:53:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68034283BF for ; Wed, 19 Sep 2018 06:53:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B53128747; Wed, 19 Sep 2018 06:53:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B2BBC283BF for ; Wed, 19 Sep 2018 06:53:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aHf61tMuKF6F19+1NgFFp1yS7l2QUNP4c22F3qKOX6s=; b=aBuA53KNJblEEiSp69GOxg3tPA tBl6kAX5Bk5uEUJVkKWW4PdXtY17WRSbq6y03+1gnIspA0ku+etUZAi2QGXJ/sRNzJ4VNcKjm3TOy cs7/RxZoKCR+N4Pv+wXv9UxkP4TfW4ekUQ+UPtomX3p3dJaZE8Q03DZPGn6CJgGwwMGZof4Lngke9 JulZXgl7eGACccMYkZNn9asr3RUS0j7YSNuRLkGt7Bm+BWzq5beqWoLi6yu86QINiZgzF1s/UEDTd h80cUw9wcA8mnanyzRBmxZxzs3ZJDgGakHsZMdI5hJCFjGYCVJRrsJuVkel9XX5Mcb1oY5471Qq6y fTErtAQA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WM8-0001sk-Lb; Wed, 19 Sep 2018 06:53:00 +0000 Received: from mail-io1-xd43.google.com ([2607:f8b0:4864:20::d43]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLX-0001Y4-4h for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:35 +0000 Received: by mail-io1-xd43.google.com with SMTP id 75-v6so3597861iou.11 for ; Tue, 18 Sep 2018 23:52:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FZJNEVNflrAxT+1Z0Hv5uIC0eQ5VzZc2FwvWytlu3MQ=; b=EhK3obIXXfTsziRS4vydKb2tw/b8QbSumbZwfBdocf8KWlnGbvytD6YkoFzpwgpz8o YW5z7ju0gy9iwK67mFRVf1NQlXfH+qCIOW+LgDoyAhmatpvTE7mGtXg7VqGeuSes5EKK N6syiiph9gKTA+xyTGgM1CO1Gvw/wJNtuCdgc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FZJNEVNflrAxT+1Z0Hv5uIC0eQ5VzZc2FwvWytlu3MQ=; b=SBWRt7xrdpDbnbN2Kaxq6uU78mvOJFlwpibSt4tc/N17zukM/FUeNyFPu2gTkdijvD jL9Pz8xMXU4IhaDI+m9GgAVQST75l43/b/i5uoR/VdrRjR1ektJpAI+nI14llPNXcpyO Owy99Byv6CnYTmS+nvMM8fDXXS3Xdy8cHtph44lWDQDShVPU6WRAp5j9XSFrH6x4mEEr VrQhFRY8hgfvTWy3zgHYYqxKe/wu96exvJBaVKWcAXr2RZ7rWk0tkfIrEUpoGAiCM6Eh YaqcO+pqjryssJPHDMQEyrm8Qg7jodHZVflhD6K0EEppA13lv3VGy0DYRmFKJr6fmkFg 32gA== X-Gm-Message-State: APzg51AJ6k1oaPUKfnu5DoBWhc8yYD2cZAYi6PCBIiHc3ky6uwnC7Y8R bfey9CVqJW/WevgQYVGu32nRQA== X-Google-Smtp-Source: ANB0VdaIiJrzgxXWBxuIAnUhl/0ErBhuYUX5OmCNWJXtIloBOoTXwE50R1pAazveXxSMGzcWHXk2rA== X-Received: by 2002:a6b:9d95:: with SMTP id g143-v6mr28303195ioe.46.1537339931935; Tue, 18 Sep 2018 23:52:11 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:11 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 1/9] kernel/jump_label: abstract jump_entry member accessors Date: Tue, 18 Sep 2018 23:51:36 -0700 Message-Id: <20180919065144.25010-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235223_236150_F9A5667C X-CRM114-Status: GOOD ( 20.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 34 +++++++++++++++++ kernel/jump_label.c | 40 ++++++++------------ 2 files changed, 49 insertions(+), 25 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 1a0b6f17a5d6..2eadff9b3b90 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -119,6 +119,40 @@ struct static_key { #ifdef HAVE_JUMP_LABEL #include + +#ifndef __ASSEMBLY__ + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#endif #endif #ifndef __ASSEMBLY__ diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 2e62503bea0d..834e43de0daf 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -38,10 +38,10 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if (jump_entry_key(jea) < jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if (jump_entry_key(jea) > jump_entry_key(jeb)) return 1; return 0; @@ -261,8 +261,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -321,16 +321,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -355,7 +345,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -370,12 +360,12 @@ static void __jump_label_update(struct static_key *key, * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (entry->code) { - if (kernel_text_address(entry->code)) + if (!jump_entry_is_init(entry)) { + if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else WARN_ONCE(1, "can't patch jump_label at %pS", - (void *)(unsigned long)entry->code); + (void *)jump_entry_code(entry)); } } } @@ -430,8 +420,8 @@ void __init jump_label_invalidate_initmem(void) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)(unsigned long)iter->code, 1)) - iter->code = 0; + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); } } @@ -441,7 +431,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -565,7 +555,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -615,7 +605,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -659,8 +649,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); } } From patchwork Wed Sep 19 06:51:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605383 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE63F5A4 for ; Wed, 19 Sep 2018 06:54:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAB702B41C for ; Wed, 19 Sep 2018 06:54:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCE3D2B45B; Wed, 19 Sep 2018 06:54:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5EF762B41C for ; Wed, 19 Sep 2018 06:54:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Xr8abvgyk3+pRz6KWX9r3uZPXN28BGmgkRVsD+XPMjE=; b=QBtQ+//pbLVLA/Mp7pW/UO90Sq +tNuJC3X5SztdRWKlC6Kf5NWKAZXCdm81g2zJxqehrpHNDrTGK4Xhl44608x9iKKrMkwm72H6WNlS szqjX5rRJp100943sF6wSpn4oGMkfsYhqZlY7fypdaqE93il1/c2RErdN3bJoYMzWAjA/Kq7a/Xy9 l9SUT742NW2UEnN2ELk9yErA+5YJb49dRnljwu+2b5hBc18BW+gsK4VDz20mXldWNIiRgJwCaV8je Oq6+Wavr9eEIVdx6Vct94dE79vTuLq+r/dUql1GOA0C9Eq3QXZ6lS7/7M9rR+cG6BX1uu+uIOjjJv iU3GmyQQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WNf-0002Vj-4t; Wed, 19 Sep 2018 06:54:35 +0000 Received: from mail-it0-x242.google.com ([2607:f8b0:4001:c0b::242]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLb-0001YX-3H for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:55 +0000 Received: by mail-it0-x242.google.com with SMTP id u13-v6so6398824iti.1 for ; Tue, 18 Sep 2018 23:52:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ve/tzdX0J+Uuck6FHDdYfUVN2t3T9LHXWJmCGGfmOhA=; b=XSBs8g1FXx8+MYOqu4x8BUS5rCmHRqZYI9aGmmBiXo0QwJ2kIR6WDnc9UUL/Bu+m+y zVeou4wFWxg8G6IoGz4AeO3MicrYY+zjoYV6Eiy/DHCjJnu8y++FAsajVh0ji7RNYmex cJ5/VtZNVe4avLVH3jZwR66wSdQsdSIkkVsjg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ve/tzdX0J+Uuck6FHDdYfUVN2t3T9LHXWJmCGGfmOhA=; b=Nkmcq9uNJdlqYw2lfTSgLRTHAA7bUvLijl1W40QOcbVbq+T5VbU/sRrSglsOiRiSkl h/kFLg/eTmeh4xsQO77SR18xIvlq//R0hoZbtknoWKDA+rgH690N4ZwRbA8I1IT61soU JXnSkyIPMPv+5gtMROwGaY3UOi9XLSTDMGaEiKBPP57feVC8rYxtRUEmvKOhMuZPh+Eb Z/KWuCxurGvKbVer2UnD3XOv+CAkhfF/uLj6TCtfuWiHU3ZUE4Nq5BGKaXwjvPX9Z8Iu +4jUBnsj9axgecO5zU3LTu8tsYdsodzwGgY6AUtqxReEjAtxaAFXUDnitg4T9Wf/SbtS QzIQ== X-Gm-Message-State: APzg51CskBQtbrYpiXC/Y/SvCXeiDce+DXq0zvIJvxP0N62ICHr+M5xi 3xtmdSpBcpcXPhI1QFeG8zNtos42hdwUpw== X-Google-Smtp-Source: ANB0VdaxaSE7+lLZp2sHqUZazu+Y8NwzM6CBX7SKJH/HJqGnlPtID1ZOU+qbIx9RvxvuX94U/rYaZA== X-Received: by 2002:a24:6e0f:: with SMTP id w15-v6mr20900860itc.77.1537339936033; Tue, 18 Sep 2018 23:52:16 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:15 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 2/9] kernel/jump_label: implement generic support for relative references Date: Tue, 18 Sep 2018 23:51:37 -0700 Message-Id: <20180919065144.25010-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235227_189632_88F642D7 X-CRM114-Status: GOOD ( 17.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP To reduce the size taken up by absolute references in jump label entries themselves and the associated relocation records in the .init segment, add support for emitting them as relative references instead. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 3 +++ include/linux/jump_label.h | 28 ++++++++++++++++++++ kernel/jump_label.c | 22 ++++++++++++++- 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9d329608913e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -359,6 +359,9 @@ config HAVE_PERF_USER_STACK_DUMP config HAVE_ARCH_JUMP_LABEL bool +config HAVE_ARCH_JUMP_LABEL_RELATIVE + bool + config HAVE_RCU_TABLE_FREE bool diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2eadff9b3b90..2768a925bafa 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -121,6 +121,32 @@ struct static_key { #include #ifndef __ASSEMBLY__ +#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE + +struct jump_entry { + s32 code; + s32 target; + long key; // key may be far away from the core kernel under KASLR +}; + +static inline unsigned long jump_entry_code(const struct jump_entry *entry) +{ + return (unsigned long)&entry->code + entry->code; +} + +static inline unsigned long jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + long offset = entry->key & ~1L; + + return (struct static_key *)((unsigned long)&entry->key + offset); +} + +#else static inline unsigned long jump_entry_code(const struct jump_entry *entry) { @@ -137,6 +163,8 @@ static inline struct static_key *jump_entry_key(const struct jump_entry *entry) return (struct static_key *)((unsigned long)entry->key & ~1UL); } +#endif + static inline bool jump_entry_is_branch(const struct jump_entry *entry) { return (unsigned long)entry->key & 1UL; diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 834e43de0daf..898a1d0c38dc 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -47,14 +47,34 @@ static int jump_label_cmp(const void *a, const void *b) return 0; } +static void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + static void jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) { unsigned long size; + void *swapfn = NULL; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE)) + swapfn = jump_label_swap; size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, swapfn); } static void jump_label_update(struct static_key *key); From patchwork Wed Sep 19 06:51:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B50CD14BD for ; Wed, 19 Sep 2018 06:54:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A131E2B41C for ; Wed, 19 Sep 2018 06:54:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 950092B45B; Wed, 19 Sep 2018 06:54:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F3EDD2B41C for ; Wed, 19 Sep 2018 06:54:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=W2mepAe+zrzFoJsgKcLACVFWIrhjxS4cleOJEUwOF5E=; b=MQGsSnDcy/m63J/UiboJPc4IP2 y8ztgYo3Vg0m5KF+UBLUbVfIHnUFnBPZVC26bvVcgBv/3McY9hG3HzhE4qSq0tCD/331Kic32L26b g9J/rlv8nJr1pFYZfJOFYwC81JxKPim+Go+qOLx4/wxTvMTc6Q5zjnFi/FJUzJcTMOS4rLEsl3ij2 hjne2/1aLrqq1sM+UsCNQyovCwVoJQipE6sZ+/4dznGC1pthAEJgNgGqJux9OT0PRw5md4EQ+EGOU zBQMiwK6BBOmWgkLFIphRPisxTWCG6U8bkPf3y6lZkyISVngcGw8TbZDU3McBBPKR+wYe0yLA0kx5 f1xxqwiA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WN4-0002F1-Vs; Wed, 19 Sep 2018 06:53:59 +0000 Received: from mail-it0-x242.google.com ([2607:f8b0:4001:c0b::242]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLb-0001Yo-3J for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:40 +0000 Received: by mail-it0-x242.google.com with SMTP id j81-v6so6863289ite.0 for ; Tue, 18 Sep 2018 23:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TGyPYi2VRIFEPcRhb8tIzZMhU5s7IYbBe3KULe7Oy1M=; b=RX689+zqYWhkAwVV303gAnGbrJ9CzrBF+KgU2p+y7qbKoXHWycb1HUUPkpNb3+j9xw nYaYaat5echYppzD0W+cO6BYfEwuoSP6FJyKDc6DONSA8nY79Jtkstw4r8B3d+GPZ+r6 W5Ip07ogGOZ8mv9fOKV/zmGUvyGDwwqxr+AXY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TGyPYi2VRIFEPcRhb8tIzZMhU5s7IYbBe3KULe7Oy1M=; b=jC1uOfokvSe3xiypFgiafBq7ibRwBJP4gWeOwaUyLmSJQjXStnbeHh5HZnrDhhEDHK vVD3oFd6XgLuchVwnmW5w9JCjl2tsTOmnwCsdn4xv/JfKxBUo1hsf0dIXYJQ8sCXPckC FopSSLVUSsStZaxOhMsz0qfSeWav5m68hU4E+72hKj4KINhwveIOghFLrYW+R/dlNBzT cZvLZhYR4sIxHHf59xKVppDj9gJy05FUjULmKzj76a9sNRyGA/grQmwSxR6ZGcjwOXtc DbyY4L6ppuMlwehoeDQ4uCZdLk7YcknWXeQc/tXKLJu7GbCn3SYte1oeOVZEBRPOB3iM C2rA== X-Gm-Message-State: APzg51DylTJ0dE5Tw3WxGYh/4QFJzUw+lNM7XwJFagPF1+JmXK4pjfET RRIEM9A9KVvSvohX0ze/vv7KhQ== X-Google-Smtp-Source: ANB0VdagX7kllJRGS9IYdSb38gxySP9dOzV8AfuT5GxWvk5nahqxpeZSqPl8VpCjukI55T4KAqfoFQ== X-Received: by 2002:a02:8647:: with SMTP id e65-v6mr30385365jai.62.1537339940402; Tue, 18 Sep 2018 23:52:20 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:19 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 3/9] arm64/kernel: jump_label: switch to relative references Date: Tue, 18 Sep 2018 23:51:38 -0700 Message-Id: <20180919065144.25010-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235227_167681_69B76FC3 X-CRM114-Status: GOOD ( 16.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references for the code and target field, and a 64-bit relative reference for the 'key' field (which may reside in another module or the core kernel, which may be more than 4 GB way on arm64 when running with KASLR enable): this reduces the size of the __jump_table by 33%, and gets rid of the RELA section entirely. Acked-by: Will Deacon Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/jump_label.h | 38 ++++++++++---------- arch/arm64/kernel/jump_label.c | 6 ++-- 3 files changed, 22 insertions(+), 23 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1b1a0e95c751..16e1e2c1e77b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -104,6 +104,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 7e2b3e360086..472023498d71 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -26,13 +26,16 @@ #define JUMP_LABEL_NOP_SIZE AARCH64_INSN_SIZE -static __always_inline bool arch_static_branch(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch(struct static_key *key, + bool branch) { - asm_volatile_goto("1: nop\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm_volatile_goto( + "1: nop \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -40,13 +43,16 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran return true; } -static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) +static __always_inline bool arch_static_branch_jump(struct static_key *key, + bool branch) { - asm_volatile_goto("1: b %l[l_yes]\n\t" - ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" - ".popsection\n\t" + asm_volatile_goto( + "1: b %l[l_yes] \n\t" + " .pushsection __jump_table, \"aw\" \n\t" + " .align 3 \n\t" + " .long 1b - ., %l[l_yes] - . \n\t" + " .quad %c0 - . \n\t" + " .popsection \n\t" : : "i"(&((char *)key)[branch]) : : l_yes); return false; @@ -54,13 +60,5 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -typedef u64 jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index e0756416e567..646b9562ee64 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); From patchwork Wed Sep 19 06:51:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1BCE15A4 for ; Wed, 19 Sep 2018 06:55:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0809B2B41C for ; Wed, 19 Sep 2018 06:55:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F046A2B45B; Wed, 19 Sep 2018 06:55:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 445822B41C for ; Wed, 19 Sep 2018 06:55:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=QJGgGipgeY2DrvCYDE95ngETyWYjdDXbCf5zlyOaPBw=; b=IAyKKLti/gnMJwmIWIYQpPNUb7 0JHsPdcvN8zmZTN7+Hmc8OPK/oh9xs6r07YishmrGxM732OM7RzU7MtDhpMOODDlNclAvsSM+1mZh 0h3Vuv9tCDUYuQyEF1AYa0AgFfQnW08ENUdiu1aaIhGQ5A/vXZ0ce80ypBlJjHp8i9LG2yF5vWLjO dxZY3bde54e7oefI8x7S7fy5tepN6GJjBNXR2yLVraG9Dw0idp1Fwq7KwgSVU1l4URjLt2C41RVga /h7f+81ERMKYvcgTueLW4TOgtq2pYvt5uY6vXbKhrgrg7O0/ee9OErBTEhk+tLxWoWrYDqTkMMgEb fEpObaaA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WON-0003HK-N9; Wed, 19 Sep 2018 06:55:19 +0000 Received: from mail-it0-x243.google.com ([2607:f8b0:4001:c0b::243]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLj-0001aj-A2 for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:58 +0000 Received: by mail-it0-x243.google.com with SMTP id e14-v6so6858434itf.1 for ; Tue, 18 Sep 2018 23:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vjlNGgKK+hdQ2oIOvWFpRpQO96t5oT9NlwIzOhMdM8o=; b=DhDtIlRo84pAR/lLfZZdTtSpY4IbWrE2M7Q3fbmY0W8iqGZiJJMONls0dIy7dmO7yE VpiQJ9v47+5DZd1eceatrZ7qN+PeX+XwJhjOIasK6oYUVfSk2VmSzZffnSJbDgZ302fQ 3dX12Q4QmLBeuTPo7bvGVMxg5fjcnMLlZmt1I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vjlNGgKK+hdQ2oIOvWFpRpQO96t5oT9NlwIzOhMdM8o=; b=RGE+tVuiipyulsVeZAM5j3SISoIeTtyvLdB46dFT/U47WUFycTFr6CVdhLCOy6zu1g 2Q9eRQh0CHCXTz76wrxfVS9JYeOKKLw3NZdnt9WPB1A2wxDnnbnt2e2F9k7BsRMLNRrs +0uxkRJwlwYqAy3SnLHcIEwVvlmGyTSYXuTM+PX5Laa/uoO6RLqS47ARGHPR0sQvT3jw uMw9AlnmiDdDgELvXl861hbML6fQbBjxZG44B9p3aIpf1pNHVSsU1+/DybQ3wGaNWhYA iH73RWJmoX3tSNa9Kj32xHaQLla3qtWg6rWtAiLA5dlDilAf2JWjt2WR1pqQXVCG7TgZ 0fAA== X-Gm-Message-State: APzg51DQLxEzda8uUAKpChHde6VniC9ZpfEEDsX2hoBLiIT4jub4zDkO m5X3Ut44u3Savi6OCX+IVyfOQw== X-Google-Smtp-Source: ANB0VdZWATphTVUC1+4j99sUxsajCdozbe7MZlE824KEJq2nWkZ2xrHiXZt4gPP7GQ6W6ipwFJV7ZA== X-Received: by 2002:a24:4a83:: with SMTP id k125-v6mr19923669itb.121.1537339944194; Tue, 18 Sep 2018 23:52:24 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:23 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 4/9] x86: add support for 64-bit place relative relocations Date: Tue, 18 Sep 2018 23:51:39 -0700 Message-Id: <20180919065144.25010-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235235_406914_10777AE8 X-CRM114-Status: GOOD ( 16.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Add support for R_X86_64_PC64 relocations, which operate on 64-bit quantities holding a relative symbol reference. Also remove the definition of R_X86_64_NUM: given that it is currently unused, it is unclear what the new value should be. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/elf.h | 3 +-- arch/x86/kernel/module.c | 6 ++++++ arch/x86/tools/relocs.c | 10 ++++++++++ arch/x86/um/asm/elf.h | 3 +-- 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..a357031d85b5 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -62,8 +62,7 @@ typedef struct user_fxsr_struct elf_fpxregset_t; #define R_X86_64_PC16 13 /* 16 bit sign extended pc relative */ #define R_X86_64_8 14 /* Direct 8 bit sign extended */ #define R_X86_64_PC8 15 /* 8 bit sign extended pc relative */ - -#define R_X86_64_NUM 16 +#define R_X86_64_PC64 24 /* Place relative 64-bit signed */ /* * These are used to set parameters in the core dumps. diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index f58336af095c..b052e883dd8c 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -201,6 +201,12 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, goto overflow; #endif break; + case R_X86_64_PC64: + if (*(u64 *)loc != 0) + goto invalid_relocation; + val -= (u64)loc; + *(u64 *)loc = val; + break; default: pr_err("%s: Unknown rela relocation: %llu\n", me->name, ELF64_R_TYPE(rel[i].r_info)); diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c index 3a6c8ebc8032..0b08067c45f3 100644 --- a/arch/x86/tools/relocs.c +++ b/arch/x86/tools/relocs.c @@ -196,6 +196,7 @@ static const char *rel_type(unsigned type) #if ELF_BITS == 64 REL_TYPE(R_X86_64_NONE), REL_TYPE(R_X86_64_64), + REL_TYPE(R_X86_64_PC64), REL_TYPE(R_X86_64_PC32), REL_TYPE(R_X86_64_GOT32), REL_TYPE(R_X86_64_PLT32), @@ -782,6 +783,15 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym, add_reloc(&relocs32neg, offset); break; + case R_X86_64_PC64: + /* + * Only used by jump labels + */ + if (is_percpu_sym(sym, symname)) + die("Invalid R_X86_64_PC64 relocation against per-CPU symbol %s\n", + symname); + break; + case R_X86_64_32: case R_X86_64_32S: case R_X86_64_64: diff --git a/arch/x86/um/asm/elf.h b/arch/x86/um/asm/elf.h index 548197212a45..413f3519d9a1 100644 --- a/arch/x86/um/asm/elf.h +++ b/arch/x86/um/asm/elf.h @@ -116,8 +116,7 @@ do { \ #define R_X86_64_PC16 13 /* 16 bit sign extended pc relative */ #define R_X86_64_8 14 /* Direct 8 bit sign extended */ #define R_X86_64_PC8 15 /* 8 bit sign extended pc relative */ - -#define R_X86_64_NUM 16 +#define R_X86_64_PC64 24 /* Place relative 64-bit signed */ /* * This is used to ensure we don't load something for the wrong architecture. From patchwork Wed Sep 19 06:51:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4240F14BD for ; Wed, 19 Sep 2018 06:56:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D7712B41C for ; Wed, 19 Sep 2018 06:56:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 211302B45B; Wed, 19 Sep 2018 06:56:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 937BD2B41C for ; Wed, 19 Sep 2018 06:56:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EdmQOwti+5/HTdlVsH3weMpYAgka2jS+ENYJWHBwQRo=; b=HLkbKkU2ZgCCKrmpMHVEKp2qXt aM0/O+cDA2u4+P8PMhP7NAxkbTRbeOa+G5o2/mYDGbSZT3/wUQ/LQsbRcogkN3ZUMSKnpnvgoyZjM OAwRE434m1YLp7kWUGKnlKYLTI44JBXcYOi+v83tN3Z9kHme2TGALIxIh6Jjsdig6OPPwMqSX5aNk CGJxMwQ1xKs7Q5rpiLqZpnbnm25JDRX9Yv3Ji+y5GiN0fhpYB6VJbMcHquvjVWKAYl9NpTtVr+/jy ngvMSiEQ8wPr5lRT+qf+D0X8zdPPh9bbFBrxZWj5Y/chL1EqxSxwviRPpDG2xL/1dk/DFiUEqZXsQ OFL3/bfQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WP2-0004M6-3l; Wed, 19 Sep 2018 06:56:00 +0000 Received: from mail-it0-x243.google.com ([2607:f8b0:4001:c0b::243]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLj-0001cv-9x for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:59 +0000 Received: by mail-it0-x243.google.com with SMTP id j198-v6so16355907ita.0 for ; Tue, 18 Sep 2018 23:52:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NQAhy2Ey7Mb6Ox0BCHyj7wjUntgBhLLd89LZtlZjBHU=; b=NsN3AGnv7VsZoqzNrVLUZ4ZX6yXb4cVP5GslK5jPZ0Y4tbBxC1B7e1XNykxHAR/3nT Xwjbvtf7nZ33/GqY6GYPE+PxBEbnXSd7i1+38SmgzfPHMcuuuVMJTKvhYZemWDB1m4QH Ja5JsrtYPbYr70dgGUwnDQM80n6TlQiQHp9AQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NQAhy2Ey7Mb6Ox0BCHyj7wjUntgBhLLd89LZtlZjBHU=; b=H54IUPVcZPd302KjzkiIgxKVut7iKm3RWowl1xHq5Dz3bTEX5wkVr1uJXr+/p2icFL 0UPdL3NGIbeBOpXtF5ZUQvmtf7hQUKNEhBKiSvZEW4mTTK9x3pRx0Fu8c/+mUJ46lttC HiXkvNLz9eSAORsBPmkOx3Epv6/DaFpd5kHEI+3y1JelyOfGf2fYo4aqfPNigpIjc5tD q35dorJaPwLj6Z4gyiUA6tjp1F6j8v2Dwb1lYiDonLcQjZfqRTeIvt8GqOLO/iWO9D+o X0NsphIhOdV3unvVA8kOSUgzD99gHTKlPK42uEu+iRlJuETjSMSQgJqKNGqxu+S05KSn VtiQ== X-Gm-Message-State: APzg51DOTYW7Wm3DCq5ONovs4j3vRdU6bHWVfPzBHovWoT9ur49Xz0kB PG8GLZPoeLuliOYiPm6HVt0XkA== X-Google-Smtp-Source: ANB0VdYEogMo8sPSravQfn7FYvX8lZmccL3rlR5nUpxZNJFm1pFyOodUDp8b37oLtQTDBbhwYH8leA== X-Received: by 2002:a24:1ad2:: with SMTP id 201-v6mr20493825iti.18.1537339948813; Tue, 18 Sep 2018 23:52:28 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:28 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 5/9] x86: jump_label: switch to jump_entry accessors Date: Tue, 18 Sep 2018 23:51:40 -0700 Message-Id: <20180919065144.25010-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235235_390256_63964919 X-CRM114-Status: GOOD ( 18.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct members with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. This incorporates a cleanup of __jump_label_transform() proposed by Peter. Cc: Peter Zijlstra Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 62 ++++++++------------ 1 file changed, 25 insertions(+), 37 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index eeea935e9bb5..aac0c1f7e354 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -42,55 +42,40 @@ static void __ref __jump_label_transform(struct jump_entry *entry, void *(*poker)(void *, const void *, size_t), int init) { - union jump_code_union code; + union jump_code_union jmp; const unsigned char default_nop[] = { STATIC_KEY_INIT_NOP }; const unsigned char *ideal_nop = ideal_nops[NOP_ATOMIC5]; + const void *expect, *code; + int line; + + jmp.jump = 0xe9; + jmp.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); if (early_boot_irqs_disabled) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { if (init) { - /* - * Jump label is enabled for the first time. - * So we expect a default_nop... - */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - /* - * ...otherwise expect an ideal_nop. Otherwise - * something went horribly wrong. - */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + expect = ideal_nop; line = __LINE__; } - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code = &jmp.code; } else { - /* - * We are disabling this jump label. If it is not what - * we think it is, then something must have gone wrong. - * If this is the first initialization call, then we - * are converting the default nop to the ideal nop. - */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = default_nop; line = __LINE__; } else { - code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + expect = &jmp.code; line = __LINE__; } - memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); + + code = ideal_nop; } + if (memcmp((void *)jump_entry_code(entry), expect, JUMP_LABEL_NOP_SIZE)) + bug_at((void *)jump_entry_code(entry), line); + /* * Make text_poke_bp() a default fallback poker. * @@ -99,11 +84,14 @@ static void __ref __jump_label_transform(struct jump_entry *entry, * always nop being the 'currently valid' instruction * */ - if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); - else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + if (poker) { + (*poker)((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); + return; + } + + text_poke_bp((void *)jump_entry_code(entry), code, JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Wed Sep 19 06:51:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BC5E5A4 for ; Wed, 19 Sep 2018 06:56:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 164392B43C for ; Wed, 19 Sep 2018 06:56:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 073362B5C1; Wed, 19 Sep 2018 06:56:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 74E722B43C for ; Wed, 19 Sep 2018 06:56:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nbGbNYjI/1iIoLPzy/uwua13MJHK00eVxYhBiTvjH/s=; b=DpUF5zufbK97BasCxdR82GxT2Z iFEhns7rFaDshcNZR0CE/DuxD+3mooQCMKUblZuFn+RxZLzgUa2m5ET3MpBYHThZ5W9qQAasH8tBf 8Yj4/jI9oheOFiXtFZrmuVncZzZs4hHgYh28q7QwmCJPKEl4rPldep12sf/B++3OK3B7JYxm2CQUb n1C4FPRfmR96VWCDsfiWRqJgzUlzJf7CPyR7OZjHOhiDu0W3Du9ly1OE42c+wuzjA2C+g7acGt+Up Z+3MguTej7SoxsZKOp1s0nNinStOIsCG+qzwwkQWnInu6FtUCUKVNpZA19am2Ni6e3kSllGa/d5cX tYUPBRVg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WPe-0004cG-RS; Wed, 19 Sep 2018 06:56:38 +0000 Received: from mail-io1-xd44.google.com ([2607:f8b0:4864:20::d44]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLs-0001fi-RK for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:52:59 +0000 Received: by mail-io1-xd44.google.com with SMTP id q4-v6so3614706iob.8 for ; Tue, 18 Sep 2018 23:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XGRY1T3h+gs2S6eNeT9ZaVSn/BnvjE5DrD6FeDRoeQ4=; b=NB7mskp/qv9FTKKiv/NIH3QElhQgT74KLCGGPtWIXEbKuGh+Mn7QgIY8P0rP2earA6 HGPls0WytLUlXWc6UX60x6rMJgjdziPfxccBg+oM7l/vTE9ydrroiYYiRC+mZAFx3Pl3 mkSPKyvp48KYUfPkN/+fCiXjZWlfxNVTGFUGg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XGRY1T3h+gs2S6eNeT9ZaVSn/BnvjE5DrD6FeDRoeQ4=; b=SnCRj+ycKmIag0nrrqmdNBetYp9a9HepoxFpfPntKotwJW3XArs/0SuEBg+JVNaKqG x/FhP16YbYEKz3vLlcFE0+aRT65oi8Sp+3u6bN1Tj5Me2hIkfCZsajCQtVyngurZVLr/ BvvdvHYVilJdDYi5cceIbWvwY6PAI++rHVRMr+zepgDCQ1tJoHVKLhYhrXACkRnmhPfb bEGdz/UpZFJryshsK9/KvLzKOHxZek7UBC5OkNlAgsNoA9aAo3SzatCRIvfIvbvso4gT 21Zk3sNQJnPvFPVnInWRjNzsoq8FQ4zsY30JepmIU+eyYJfg5Kd9Pi0hMDnp0dL5r6sr PB+A== X-Gm-Message-State: APzg51BpciqLQxwRrH7Wz33FNPGJ8oU8P9suwZ74DMvO/H/5Exndf/6O K8kTZP992CCxDyeak1e9A2tKHQ== X-Google-Smtp-Source: ANB0VdYZmcooAa65xYvgxeEeJ184S8IKmJEkvPLtm2qYaaQ5RGALeFjwN9NBwKVhneUZCfdB/VsYSQ== X-Received: by 2002:a5e:db07:: with SMTP id q7-v6mr28850533iop.81.1537339953658; Tue, 18 Sep 2018 23:52:33 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:33 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 6/9] x86/kernel: jump_table: use relative references Date: Tue, 18 Sep 2018 23:51:41 -0700 Message-Id: <20180919065144.25010-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235244_949265_3606C8FD X-CRM114-Status: GOOD ( 15.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Similar to the arm64 case, 64-bit x86 can benefit from using relative references rather than absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 33%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 24 +++++++------------- tools/objtool/special.c | 4 ++-- 3 files changed, 11 insertions(+), 18 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..603f374f871d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -119,6 +119,7 @@ config X86 select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..21efc9d07ed9 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -37,7 +37,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -53,7 +54,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".long 1b - ., %l[l_yes] - . \n\t" + _ASM_PTR "%c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -62,18 +64,6 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool return true; } -#ifdef CONFIG_X86_64 -typedef u64 jump_label_t; -#else -typedef u32 jump_label_t; -#endif - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def @@ -88,7 +78,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key - . .popsection .endm @@ -104,7 +95,8 @@ struct jump_entry { .endif .pushsection __jump_table, "aw" _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .long .Lstatic_jump_\@ - ., \target - . + _ASM_PTR \key + 1 - . .popsection .endm diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..50af4e1274b3 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 16 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0 From patchwork Wed Sep 19 06:51:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 97A915A4 for ; Wed, 19 Sep 2018 06:57:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 82EFE2B618 for ; Wed, 19 Sep 2018 06:57:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 772362B622; Wed, 19 Sep 2018 06:57:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BEB2D2B618 for ; Wed, 19 Sep 2018 06:57:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dhyyOIg7idNwJMHwdsCuiK8/8hxJdBTNZ05kOjQzRNo=; b=MmbPpa4WrCglaNLbNJbMSTmaEq I/gQ20rRQYHfkt6Imd4sFQ4r3Aiw0L1aKaDE6f0SyndoBA+4adjYhJU09/HRrs3tHKejeRown9Q9z O9LYyb9GkQaiNyy1SnUkoxC7+7yfw3yLsV8f1Xcs5DXVqK/RB9RZCbKPpX1XO4Sv1k1ebNyVl/S4t JTiH1aive9Qae4gRvxG87oB2RscoNXAgw0f4zsKhQD0fDFch9e0dJuyt+BdpJGVpP2LKc9HobA8IB UtLDHr5JPyypNYNmI7sIwVjvB/YUUizc4DenM/uBwixMXNV92m7z6hU2yMGcTNjjvjMFKP8S3soJX p4qexG+g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WQO-0004ws-R8; Wed, 19 Sep 2018 06:57:24 +0000 Received: from mail-it0-x241.google.com ([2607:f8b0:4001:c0b::241]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WLy-0001j2-26 for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:53:20 +0000 Received: by mail-it0-x241.google.com with SMTP id d10-v6so6815332itj.5 for ; Tue, 18 Sep 2018 23:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=ePc7YC7AmNfXSr4FpXbzsfuPGom2b9uu9QTkOtVTTpvbQnmlm+Ab5Fda4DSvDkiNSW 2n930+omZG2krqTyIvZHbqPMRfYLPEPCf7CqbnlVZak11lPxt0+JZGVX0RaAkWHUW1Qg 5Yzihq5i7+V3yXSxUwy87y990PW7TLphcsVXA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Je2400ni9A5s80feHs0HBjY1gZ6UX9x0bE0fIx90rAE=; b=qPu4uiq8C54PVkoEn1vMzdmm15yIa8pV7kdQYA2zVh7lXZrNDR6fcXimwOn0BmEgTF heDqYk55h9XN8fFf3Fzhmf02dASxZRbJc08yGjB3gtmHbSEGWo1gUkWB2/c7mYCM3+7Z yUpHJzH73U87fdZcc4Y3A5n8OuOaaIk7VP4XBEInAb/xmDXShmOZKFPFd9awxmHORskB PXcGDLVerwHCZal7q4Zz9AV81XurvuI86h3qAKH1vbj5EFaMdWIMe81KtuaMXkopB1vt hQeOxRBnAhlTsmu8Rhi6ZB3x8nH3w+rcqQ8ySdLGb7SMwPIOCVTUepL2SflnvF/JGXpE RQ6g== X-Gm-Message-State: APzg51CSOw1qy9o5NZaB9Xt38D6/4bTbVaMAt3cYsJnLeyYViH07dT1k RiuYUC0WugZVo3GPAqf7TcNrbQ== X-Google-Smtp-Source: ANB0VdbJIyPZkvihO7OB4sEFaf7dVCb85ROt+pj2C1IGG7N6NsvNoY+cNRUaQF9RxhbGxdC+soAo0Q== X-Received: by 2002:a24:6401:: with SMTP id t1-v6mr21062079itc.135.1537339958595; Tue, 18 Sep 2018 23:52:38 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:37 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 7/9] jump_label: annotate entries that operate on __init code earlier Date: Tue, 18 Sep 2018 23:51:42 -0700 Message-Id: <20180919065144.25010-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235250_280225_3A21D05C X-CRM114-Status: GOOD ( 25.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Jump table entries are mostly read-only, with the exception of the init and module loader code that defuses entries that point into init code when the code being referred to is freed. For robustness, it would be better to move these entries into the ro_after_init section, but clearing the 'code' member of each jump table entry referring to init code at module load time races with the module_enable_ro() call that remaps the ro_after_init section read only, so we'd like to do it earlier. So given that whether such an entry refers to init code can be decided much earlier, we can pull this check forward. Since we may still need the code entry at this point, let's switch to setting a low bit in the 'key' member just like we do to annotate the default state of a jump table entry. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/jump_label.h | 11 ++--- init/main.c | 1 - kernel/jump_label.c | 48 ++++++-------------- 3 files changed, 18 insertions(+), 42 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2768a925bafa..5df6a621e464 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -141,7 +141,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - long offset = entry->key & ~1L; + long offset = entry->key & ~3L; return (struct static_key *)((unsigned long)&entry->key + offset); } @@ -160,7 +160,7 @@ static inline unsigned long jump_entry_target(const struct jump_entry *entry) static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + return (struct static_key *)((unsigned long)entry->key & ~3UL); } #endif @@ -172,12 +172,12 @@ static inline bool jump_entry_is_branch(const struct jump_entry *entry) static inline bool jump_entry_is_init(const struct jump_entry *entry) { - return entry->code == 0; + return (unsigned long)entry->key & 2UL; } static inline void jump_entry_set_init(struct jump_entry *entry) { - entry->code = 0; + entry->key |= 2; } #endif @@ -213,7 +213,6 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); -extern void jump_label_invalidate_initmem(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, @@ -261,8 +260,6 @@ static __always_inline void jump_label_init(void) static_key_initialized = true; } -static inline void jump_label_invalidate_initmem(void) {} - static __always_inline bool static_key_false(struct static_key *key) { if (unlikely(static_key_count(key) > 0)) diff --git a/init/main.c b/init/main.c index 18f8f0140fa0..a664246450d1 100644 --- a/init/main.c +++ b/init/main.c @@ -1064,7 +1064,6 @@ static int __ref kernel_init(void *unused) /* need to finish all async __init code before freeing the memory */ async_synchronize_full(); ftrace_free_init_mem(); - jump_label_invalidate_initmem(); free_initmem(); mark_readonly(); diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 898a1d0c38dc..e8cf3ff3149c 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -373,14 +373,15 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) static void __jump_label_update(struct static_key *key, struct jump_entry *entry, - struct jump_entry *stop) + struct jump_entry *stop, + bool init) { for (; (entry < stop) && (jump_entry_key(entry) == key); entry++) { /* * An entry->code of 0 indicates an entry which has been * disabled because it was in an init text area. */ - if (!jump_entry_is_init(entry)) { + if (init || !jump_entry_is_init(entry)) { if (kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); else @@ -420,6 +421,9 @@ void __init jump_label_init(void) if (jump_label_type(iter) == JUMP_LABEL_NOP) arch_jump_label_transform_static(iter, JUMP_LABEL_NOP); + if (init_section_contains((void *)jump_entry_code(iter), 1)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -432,19 +436,6 @@ void __init jump_label_init(void) cpus_read_unlock(); } -/* Disable any jump label entries in __init/__exit code */ -void __init jump_label_invalidate_initmem(void) -{ - struct jump_entry *iter_start = __start___jump_table; - struct jump_entry *iter_stop = __stop___jump_table; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (init_section_contains((void *)jump_entry_code(iter), 1)) - jump_entry_set_init(iter); - } -} - #ifdef CONFIG_MODULES static enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -524,7 +515,8 @@ static void __jump_label_mod_update(struct static_key *key) stop = __stop___jump_table; else stop = m->jump_entries + m->num_jump_entries; - __jump_label_update(key, mod->entries, stop); + __jump_label_update(key, mod->entries, stop, + m->state == MODULE_STATE_COMING); } } @@ -570,6 +562,9 @@ static int jump_label_add_module(struct module *mod) for (iter = iter_start; iter < iter_stop; iter++) { struct static_key *iterk; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_init(iter); + iterk = jump_entry_key(iter); if (iterk == key) continue; @@ -605,7 +600,7 @@ static int jump_label_add_module(struct module *mod) /* Only update if we've changed from our initial state */ if (jump_label_type(iter) != jump_label_init_type(iter)) - __jump_label_update(key, iter, iter_stop); + __jump_label_update(key, iter, iter_stop, true); } return 0; @@ -661,19 +656,6 @@ static void jump_label_del_module(struct module *mod) } } -/* Disable any jump label entries in module init code */ -static void jump_label_invalidate_module_init(struct module *mod) -{ - struct jump_entry *iter_start = mod->jump_entries; - struct jump_entry *iter_stop = iter_start + mod->num_jump_entries; - struct jump_entry *iter; - - for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(jump_entry_code(iter), mod)) - jump_entry_set_init(iter); - } -} - static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data) @@ -695,9 +677,6 @@ jump_label_module_notify(struct notifier_block *self, unsigned long val, case MODULE_STATE_GOING: jump_label_del_module(mod); break; - case MODULE_STATE_LIVE: - jump_label_invalidate_module_init(mod); - break; } jump_label_unlock(); @@ -767,7 +746,8 @@ static void jump_label_update(struct static_key *key) entry = static_key_entries(key); /* if there are no users, entry can be NULL */ if (entry) - __jump_label_update(key, entry, stop); + __jump_label_update(key, entry, stop, + system_state < SYSTEM_RUNNING); } #ifdef CONFIG_STATIC_KEYS_SELFTEST From patchwork Wed Sep 19 06:51:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D728B5A4 for ; Wed, 19 Sep 2018 06:58:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C48A72B59A for ; Wed, 19 Sep 2018 06:58:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6AD62B4F0; Wed, 19 Sep 2018 06:58:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2211D2B4F0 for ; Wed, 19 Sep 2018 06:58:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Uf0exNihCkqVw/dg61YMyMJE+eJ5EECLBw7DhZawC1o=; b=Nkqmvk8gAt/tb6vJXFLCpzzDVF ZpIqltkkaF+LyVJOQA0Rpp8qJTPs3pLZmzV3oyP7Ur6Rgd+Gzdj+iSpDG+F4KPJMvek5yzSGqt75l 1aRNiMPpg2WcNhoyspHTX8J/zoHXqIvAgBlmNvnntO+m4TIkxdDGhfKhW2RtLQv8w9ynjpl+MV5xU KL9fKwTVphG5QGOcjMkH4w5qHyDjLrAyNQgg8k4fkOkPJJVhl6i3k/2ZLzRUX8NIk0OakuGBp0M3E QFBaR/KHg9cB6aSa3CLmFBpVQI2EfUSlQCGrYjwuMeYEfIal5Fbcxym9Rl5PpTdjWfkY5hXiZnD9u 9ZhBjxJw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WR2-0005Ex-FP; Wed, 19 Sep 2018 06:58:04 +0000 Received: from mail-it0-x244.google.com ([2607:f8b0:4001:c0b::244]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WM2-0001kz-7E for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:53:34 +0000 Received: by mail-it0-x244.google.com with SMTP id e14-v6so6859260itf.1 for ; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6WcczBvykg2Z58bAiHqwLTs7BS22B+N9kwtSqthViSE=; b=Focb2vMv0Nwcyy8JpgHJ+eawzudhCpL77NjYEXyCv5owfpRIMB9U+XM+EUUsK4ZAgM FzzcDvaN97C+D4oF8Abvfa1yIacasMqfez3ah7v1BBLBqkVVyQMiR4oCyqB/pi8iYLns qqs7M3GyoO3OUXvzfKgFJjlZ98/QYCN5z09n0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6WcczBvykg2Z58bAiHqwLTs7BS22B+N9kwtSqthViSE=; b=cpnhBg4bAb5OHccrWuMvUXVkam9mZYF33Dqe9zEigbXWeb4AVtFRNERT2xPAgVrSw6 pRJ1PBZ9W3bLmfjm+sIJmRFGh8dvRFHlc+k6AleT0F3Lgxdix32p3WO5AWy4EquQx1yb ozJneP0EkD/W3VLtMuCKBrdmQs6qc9oanyA4nYz0czTvhIAcEGQ6kE2stjgyX3CAiCWz QhwA12vf8heP6BEiF2C9Uqv1g1okk0DmE1Er+t7jW5/Bnilf/i9nh6Qf7h3+NtoY0MAZ YUhE7/mcsO+Y+KmkkMkuDSJpt9xpuNfbN6XfACBZeS9jLcihaGYJvRfH+/zBkBZ5NhlD iQ0g== X-Gm-Message-State: APzg51AJXL6VUovhcy2yp3b+kS55K+Drwg6BozR5S1gsb5TXSJqON1zB gF0PGmkMhvSgELSeFf6X10zv3A== X-Google-Smtp-Source: ANB0Vdav2eJIjyrERUEmiGtnItipG3yg96dIXIH+VceUJ53gJklJE8jmvQvZXExnKIj05TmFCizbcQ== X-Received: by 2002:a24:1355:: with SMTP id 82-v6mr20940938itz.74.1537339963109; Tue, 18 Sep 2018 23:52:43 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:42 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 8/9] jump_table: move entries into ro_after_init region Date: Tue, 18 Sep 2018 23:51:43 -0700 Message-Id: <20180919065144.25010-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235254_321633_F8FED6DD X-CRM114-Status: GOOD ( 14.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The __jump_table sections emitted into the core kernel and into each module consist of statically initialized references into other parts of the code, and with the exception of entries that point into init code, which are defused at post-init time, these data structures are never modified. So let's move them into the ro_after_init section, to prevent them from being corrupted inadvertently by buggy code, or deliberately by an attacker. Reviewed-by: Kees Cook Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/s390/kernel/vmlinux.lds.S | 1 + include/asm-generic/vmlinux.lds.h | 11 +++++++---- kernel/module.c | 9 +++++++++ 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S index b43f8d33a369..4042bbf3f9ad 100644 --- a/arch/s390/kernel/vmlinux.lds.S +++ b/arch/s390/kernel/vmlinux.lds.S @@ -66,6 +66,7 @@ SECTIONS *(.data..ro_after_init) } EXCEPTION_TABLE(16) + JUMP_TABLE_DATA . = ALIGN(PAGE_SIZE); __end_ro_after_init = .; diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 7b75ff6e2fce..f09ee3c544bc 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -253,10 +253,6 @@ STRUCT_ALIGN(); \ *(__tracepoints) \ /* implement dynamic printk debug */ \ - . = ALIGN(8); \ - __start___jump_table = .; \ - KEEP(*(__jump_table)) \ - __stop___jump_table = .; \ . = ALIGN(8); \ __start___verbose = .; \ KEEP(*(__verbose)) \ @@ -300,6 +296,12 @@ . = __start_init_task + THREAD_SIZE; \ __end_init_task = .; +#define JUMP_TABLE_DATA \ + . = ALIGN(8); \ + __start___jump_table = .; \ + KEEP(*(__jump_table)) \ + __stop___jump_table = .; + /* * Allow architectures to handle ro_after_init data on their * own by defining an empty RO_AFTER_INIT_DATA. @@ -308,6 +310,7 @@ #define RO_AFTER_INIT_DATA \ __start_ro_after_init = .; \ *(.data..ro_after_init) \ + JUMP_TABLE_DATA \ __end_ro_after_init = .; #endif diff --git a/kernel/module.c b/kernel/module.c index 6746c85511fe..49a405891587 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -3315,6 +3315,15 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) * Note: ro_after_init sections also have SHF_{WRITE,ALLOC} set. */ ndx = find_sec(info, ".data..ro_after_init"); + if (ndx) + info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT; + /* + * Mark the __jump_table section as ro_after_init as well: these data + * structures are never modified, with the exception of entries that + * refer to code in the __init section, which are annotated as such + * at module load time. + */ + ndx = find_sec(info, "__jump_table"); if (ndx) info->sechdrs[ndx].sh_flags |= SHF_RO_AFTER_INIT; From patchwork Wed Sep 19 06:51:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 10605399 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19DC15A4 for ; Wed, 19 Sep 2018 06:58:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 042812B5BA for ; Wed, 19 Sep 2018 06:58:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EBE802B5DC; Wed, 19 Sep 2018 06:58:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 809352B5BA for ; Wed, 19 Sep 2018 06:58:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=E9/J33jRxhDGrHpYNBHkJEBMzn9k5LLJXBQ1963FdZY=; b=HC5G8PF+4rcxwfzU3wmJ8T+89j ofCpuWWbGJ0f7ZIkiTIEyme6GMaTfYE2l7DoLDxbkhwGKD1+EvE4nI0rRd77eY9A0wYeHoR/OMWpu U1HecGuS5zHMWh+GDa49j31QDDX3oqZYB98F7UgRlk2go7zwMTCqu3gdzmkPhzLrGKItEiF3Fcrps /yDBaS2/M5uuYWKcOM6aNskw7YLptds6znrtvRJOkypw1vfXOqPUen3MVSE2WbuydujrQ9TmwG6rJ KGQvf1voqLaaXCUOjaUKF8dQ3KnDSfbkEB66RXgSUILcazc1Ax4osic8yX5QE4JRA7dqc2vtHjPhm /nBfGNeg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WRe-0005UK-JC; Wed, 19 Sep 2018 06:58:42 +0000 Received: from mail-it0-x243.google.com ([2607:f8b0:4001:c0b::243]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1g2WM6-0001mp-6x for linux-arm-kernel@lists.infradead.org; Wed, 19 Sep 2018 06:53:52 +0000 Received: by mail-it0-x243.google.com with SMTP id d10-v6so6815710itj.5 for ; Tue, 18 Sep 2018 23:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iA34PvoXnv/5jnydRzA1GyetmcM2pytY3HL8SIbi1DA=; b=hX3BTpM6OFVDYHhbaduRyiEvKVa1h5kwWbYpjAoXCspqeB/8w7+LnC7RvhgfoKauDh K/qeYeiNjLC3g0sGJS48hHaypqf2IsTsjQVJVmwnZUT1yHVeaa9UqpQsRT936/9JiXi2 dZmpGMvJAVhcNCShHhhW5p2Y0X6JZN8AwlEvA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iA34PvoXnv/5jnydRzA1GyetmcM2pytY3HL8SIbi1DA=; b=leC8CO85eQMr/aKr5RCFvRpSyWZbm5sikv7VViYS2S84DHV0RFbnqEjNnDqoTQn3SB GM/zMdIfk6RWModl300hMMbJ+GXgn9oxK2Fkfb3Mon53tHqB4M45REQaROmrlYNkzUDX KeRplNl+rbefseL1HjfzX8qFaNnpRalS490/quxj7KgPhneksqg40d7bKvnzmwwEz+T1 G22NekKFfM6OFl+QjBHOy2IAmqZTb+1T47R6Fjr50i05vnzDC4h8alQCHWtNlftctbLB xhcetuES8p07AKIUul9pHPtm16crOXmzsV8qyrqAvMgH0tBrfHe2JlbcLpJGFEsWA21Z VkZA== X-Gm-Message-State: APzg51BFZL0IIAcd0knkOI87bznbALvgncjTw/mMF3tXzqHVC+z+rahS MxvbWMtM1ygIyVUlKKpFUeJteQ== X-Google-Smtp-Source: ANB0VdbHgyQN/E94/b0fZazm9JVFBSMPzoF08/jTJct0BqxIbnvWsEkdPNxM/CD6Iz4TD2PbVPjMNg== X-Received: by 2002:a24:cf06:: with SMTP id y6-v6mr19881811itf.5.1537339966590; Tue, 18 Sep 2018 23:52:46 -0700 (PDT) Received: from localhost.localdomain ([209.82.80.116]) by smtp.gmail.com with ESMTPSA id x68-v6sm7939477ita.2.2018.09.18.23.52.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Sep 2018 23:52:45 -0700 (PDT) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v3 9/9] s390/jump_label: switch to relative references Date: Tue, 18 Sep 2018 23:51:44 -0700 Message-Id: <20180919065144.25010-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180919065144.25010-1-ard.biesheuvel@linaro.org> References: <20180919065144.25010-1-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180918_235258_303991_78D0122D X-CRM114-Status: GOOD ( 17.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-s390@vger.kernel.org, Kees Cook , Arnd Bergmann , Ard Biesheuvel , Peter Zijlstra , Will Deacon , Jessica Yu , Heiko Carstens , Steven Rostedt , Ingo Molnar , Catalin Marinas , Martin Schwidefsky , Thomas Gleixner , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Heiko Carstens Enable support for relative references in jump_label entries. Signed-off-by: Heiko Carstens Signed-off-by: Ard Biesheuvel --- arch/s390/Kconfig | 1 + arch/s390/include/asm/jump_label.h | 40 ++++++++------------ arch/s390/kernel/jump_label.c | 11 +++--- 3 files changed, 23 insertions(+), 29 deletions(-) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 9a9c7a6fe925..12dc4c1af96c 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -120,6 +120,7 @@ config S390 select HAVE_ALIGNED_STRUCT_PAGE if SLUB select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_JUMP_LABEL + select HAVE_ARCH_JUMP_LABEL_RELATIVE select CPU_NO_EFFICIENT_FFS if !HAVE_MARCH_Z9_109_FEATURES select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_SOFT_DIRTY diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 40f651292aa7..e2d3e6c43395 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -14,41 +14,33 @@ * We use a brcl 0,2 instruction for jump labels at compile time so it * can be easily distinguished from a hotpatch generated instruction. */ -static __always_inline bool arch_static_branch(struct static_key *key, bool branch) +static inline bool arch_static_branch(struct static_key *key, bool branch) { - asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" - ".pushsection __jump_table, \"aw\"\n" - ".balign 8\n" - ".quad 0b, %l[label], %0\n" - ".popsection\n" - : : "X" (&((char *)key)[branch]) : : label); - + asm_volatile_goto("0: brcl 0,"__stringify(JUMP_LABEL_NOP_OFFSET)"\n" + ".pushsection __jump_table,\"aw\"\n" + ".balign 8\n" + ".long 0b-.,%l[label]-.\n" + ".quad %0-.\n" + ".popsection\n" + : : "X" (&((char *)key)[branch]) : : label); return false; label: return true; } -static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch) +static inline bool arch_static_branch_jump(struct static_key *key, bool branch) { - asm_volatile_goto("0: brcl 15, %l[label]\n" - ".pushsection __jump_table, \"aw\"\n" - ".balign 8\n" - ".quad 0b, %l[label], %0\n" - ".popsection\n" - : : "X" (&((char *)key)[branch]) : : label); - + asm_volatile_goto("0: brcl 15,%l[label]\n" + ".pushsection __jump_table,\"aw\"\n" + ".balign 8\n" + ".long 0b-.,%l[label]-.\n" + ".quad %0-.\n" + ".popsection\n" + : : "X" (&((char *)key)[branch]) : : label); return false; label: return true; } -typedef unsigned long jump_label_t; - -struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; -}; - #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/s390/kernel/jump_label.c b/arch/s390/kernel/jump_label.c index 43f8430fb67d..50a1798604a8 100644 --- a/arch/s390/kernel/jump_label.c +++ b/arch/s390/kernel/jump_label.c @@ -33,13 +33,13 @@ static void jump_label_make_branch(struct jump_entry *entry, struct insn *insn) { /* brcl 15,offset */ insn->opcode = 0xc0f4; - insn->offset = (entry->target - entry->code) >> 1; + insn->offset = (jump_entry_target(entry) - jump_entry_code(entry)) >> 1; } static void jump_label_bug(struct jump_entry *entry, struct insn *expected, struct insn *new) { - unsigned char *ipc = (unsigned char *)entry->code; + unsigned char *ipc = (unsigned char *)jump_entry_code(entry); unsigned char *ipe = (unsigned char *)expected; unsigned char *ipn = (unsigned char *)new; @@ -59,6 +59,7 @@ static void __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, int init) { + void *code = (void *)jump_entry_code(entry); struct insn old, new; if (type == JUMP_LABEL_JMP) { @@ -69,13 +70,13 @@ static void __jump_label_transform(struct jump_entry *entry, jump_label_make_nop(entry, &new); } if (init) { - if (memcmp((void *)entry->code, &orignop, sizeof(orignop))) + if (memcmp(code, &orignop, sizeof(orignop))) jump_label_bug(entry, &orignop, &new); } else { - if (memcmp((void *)entry->code, &old, sizeof(old))) + if (memcmp(code, &old, sizeof(old))) jump_label_bug(entry, &old, &new); } - s390_kernel_write((void *)entry->code, &new, sizeof(new)); + s390_kernel_write(code, &new, sizeof(new)); } static int __sm_arch_jump_label_transform(void *data)