From patchwork Tue Mar 24 20:32:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11456463 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F381913 for ; Tue, 24 Mar 2020 20:33:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 29BF920735 for ; Tue, 24 Mar 2020 20:33:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="IolzPvUG"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BcF5fu0w" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 29BF920735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ubRhZQNTRrEujNc6p511/JgKEaXbNiDKZLHosrBSiU0=; b=IolzPvUGdtka8O 0R/gUIwiEO1p0TPHSjm2YOcKM5FW5lZLlDgLD77pzxd9Y+6s1umXmn04A/qun8tRgOhm8ge4wkoE5 XW8Bs/kdWIrIFaQNdu5WFuctr5UTvIBmYt3rFBLsls4Pjrfc690AT2/uDpO7y7lMIY4SH5EHTk+Tn eKmPRRFOnHmoJC15U2A1QE+gHE8Dnw0B5gl8e3FIfZYpGYFkkVOOr92EBv4OeLzeWFIs3vs6lXWFa k0yuhKHbFsEHn5tOKeuGW4kDjX6WegxO3zEpDQj+PIpWjjFcKESqVBRosPLYjjMHW60Eb4h1rsFSw BeFoOVastpvQylrRq5uw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqEb-0002QS-7u; Tue, 24 Mar 2020 20:33:13 +0000 Received: from mail-pj1-x1043.google.com ([2607:f8b0:4864:20::1043]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE5-0001wV-S1 for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2020 20:32:43 +0000 Received: by mail-pj1-x1043.google.com with SMTP id w9so47574pjh.1 for ; Tue, 24 Mar 2020 13:32:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1MVGnSojuGRbIf385C9jQfbyEyUVxBYpFXB8EsCcrYw=; b=BcF5fu0w+3RjSRBoJOMuJwRkQ7K/mRa1BZHJeNserbqzlapS73yhJkI9BAFD3QlZia /tK6Q6a0Xix0108WsK916BMXctBv6x/nqlbYs0Fqc2Fl/c07lkSHe4EI2UglGuLZrWzk OsxF7OZPK7kzoWTSxKgurb1BP56sAT/bnrrp0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1MVGnSojuGRbIf385C9jQfbyEyUVxBYpFXB8EsCcrYw=; b=bUumf39c4LU35JIeut47fiKIjTaHH5wk0jWd4B4+aVB47RJJAk1Jtui8vq+EqOGIUe AQAR6MSII6Zcte5/W5j3SVYvlWZZqn2OnDbkzFtmGdpTsH813WPQ+loABMDJ9BU20DUq xgJfymkxFZrUxuSZcMCeaU66kQe4lpbUd/sznKgvpwUVcpPzAu7q0aoZo4DfG6ut9Ne0 DIyoNpCXOy9rjmi1WNfoWbM7scs7T5rPZMfvzw8L+P2Ivw+u1Z6N2jiOV75H747eTWZ7 YEQG+2JoD0jtKwdq9UgN8HnYx7go7aGFc2ixfGYjE3IQQC1rdlYLie+S+YgKWalmS9yQ iePg== X-Gm-Message-State: ANhLgQ25j4oa8HT8a2VkurGRDapmeA/EPIjjQQ4obKmNVw9+HHqnGaKf LXXOt09YRj2UsCE6guLY6xbBug== X-Google-Smtp-Source: ADFU+vvnVRb00GhJN60Odz51h8y0qLnFMPgufkKkVflkAhmJwmU/2do8qdmBufOmVDll0iyTu12GSw== X-Received: by 2002:a17:902:61:: with SMTP id 88mr28126394pla.313.1585081961408; Tue, 24 Mar 2020 13:32:41 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id na18sm3276384pjb.31.2020.03.24.13.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2020 13:32:37 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v2 1/5] jump_label: Provide CONFIG-driven build state defaults Date: Tue, 24 Mar 2020 13:32:27 -0700 Message-Id: <20200324203231.64324-2-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324203231.64324-1-keescook@chromium.org> References: <20200324203231.64324-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200324_133241_904805_0445944D X-CRM114-Status: GOOD ( 13.91 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1043 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, "Perla, Enrico" , Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Choosing the initial state of static branches changes the assembly layout (if the condition is expected to be likely, inline, or unlikely, out of line via a jump). A few places in the kernel use (or could be using) a CONFIG to choose the default state, so provide the infrastructure to do this and convert the existing cases (init_on_alloc and init_on_free) to the new macros. Signed-off-by: Kees Cook Acked-by: Peter Zijlstra (Intel) --- include/linux/jump_label.h | 19 +++++++++++++++++++ include/linux/mm.h | 12 ++---------- mm/page_alloc.c | 12 ++---------- 3 files changed, 23 insertions(+), 20 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 3526c0aee954..615fdfb871a3 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -382,6 +382,21 @@ struct static_key_false { [0 ... (count) - 1] = STATIC_KEY_FALSE_INIT, \ } +#define _DEFINE_STATIC_KEY_1(name) DEFINE_STATIC_KEY_TRUE(name) +#define _DEFINE_STATIC_KEY_0(name) DEFINE_STATIC_KEY_FALSE(name) +#define DEFINE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_, IS_ENABLED(cfg))(name) + +#define _DEFINE_STATIC_KEY_RO_1(name) DEFINE_STATIC_KEY_TRUE_RO(name) +#define _DEFINE_STATIC_KEY_RO_0(name) DEFINE_STATIC_KEY_FALSE_RO(name) +#define DEFINE_STATIC_KEY_MAYBE_RO(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_RO_, IS_ENABLED(cfg))(name) + +#define _DECLARE_STATIC_KEY_1(name) DECLARE_STATIC_KEY_TRUE(name) +#define _DECLARE_STATIC_KEY_0(name) DECLARE_STATIC_KEY_FALSE(name) +#define DECLARE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DECLARE_STATIC_KEY_, IS_ENABLED(cfg))(name) + extern bool ____wrong_branch_error(void); #define static_key_enabled(x) \ @@ -482,6 +497,10 @@ extern bool ____wrong_branch_error(void); #endif /* CONFIG_JUMP_LABEL */ +#define static_branch_maybe(config, x) \ + (IS_ENABLED(config) ? static_branch_likely(x) \ + : static_branch_unlikely(x)) + /* * Advanced usage; refcount, branch is enabled when: count != 0 */ diff --git a/include/linux/mm.h b/include/linux/mm.h index c54fb96cb1e6..059658604dd6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2662,11 +2662,7 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_alloc); -#else -DECLARE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc) && @@ -2675,11 +2671,7 @@ static inline bool want_init_on_alloc(gfp_t flags) return flags & __GFP_ZERO; } -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_free); -#else -DECLARE_STATIC_KEY_FALSE(init_on_free); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { return static_branch_unlikely(&init_on_free) && diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3c4eb750a199..1f625e5a03c0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -135,18 +135,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_alloc); -#else -DEFINE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_free); -#else -DEFINE_STATIC_KEY_FALSE(init_on_free); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static int __init early_init_on_alloc(char *buf) From patchwork Tue Mar 24 20:32:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11456445 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DDAA174A for ; Tue, 24 Mar 2020 20:32:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F174E20735 for ; Tue, 24 Mar 2020 20:32:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Y/89HkBH"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Vpxmpa7y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F174E20735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=reWOJIvKxexJOEjV2NCXcwGR6DBw/qMcwrQC3DIrRZ8=; b=Y/89HkBHkVXJeK xeW4b+kGW8UTILH8DiKXnqTxD52nb0cY1itrYxSZ5uJJLUOVw51Nuyoi/dxD80CXXjXb5dgT1LRQD ME941yG5HFaGWW3IPPJ1/5Z0eFZEs1S8q5E0KcUOIpi5lp0Ck1VEr4jP+co4iYQUFjV6p9aqp5Ex8 2HF6vnxG1neoebWkzyt3lRG52rfnk3m4ruHcyroNSersuxrJWF+AFkVed4qD0HoWRnMDB6nHG5c5v f5+IfZICwKNNZL3qsG05Rfkqdu4aRuW/BqAmXi3tGRIEnNClIvURPyINg1/ddaJIQjW8Xj1f8Dhvg lBOO4O6WyhNpa/BcAWdw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE7-0001xX-TJ; Tue, 24 Mar 2020 20:32:43 +0000 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE4-0001vd-ER for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2020 20:32:42 +0000 Received: by mail-pl1-x644.google.com with SMTP id h11so7886724plr.11 for ; Tue, 24 Mar 2020 13:32:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QtcXhojXdvVAaDeWfB/KlcDfMVh9eKi/T1lC0sMYq8g=; b=Vpxmpa7ys0bod0SNGpU9HyY796q3xE/M3orNp8yw116/0ZyHjlZKj+WNR92bTLQ7N7 mX5GB7DrOSUiWFgL9zFZ9VSSzsyJoLgWbC4sD0apXV2JjP55GWJ/SQRkk+tLe9BdjdAA RdWbcxQUkap+TOsmpSxq5+j6/AOKEweKlqbt8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QtcXhojXdvVAaDeWfB/KlcDfMVh9eKi/T1lC0sMYq8g=; b=nf0EwuD1lKr9CS4FK4LXuEowuBI+NueXWElwWtI6UVZjmjRD6+kcdQ0BxrH4OZfVB+ rZEc4hQrAvlrrvCFTlvyFMhjUCZSDeXUeRVDjlmzasyjo75DVeDqJEpjzI+DT1WXD2e5 xqp3tBaTbYLU3iNCgW+kdhgoOPTPn/+BRMLkkTtpCgDQNuNKD6spDa/HIw/uPI9yFy22 FCwtKWNYq8qTUNrO1Itl/OiHA2kjk5ungFmdjJzTzNuALt2C2eGPGGpPYQuvtdWBDas1 A5Y6d2Vbb4q/yWr8v9bYgn+UgToGQpwtDzTymoVmF7sALiexWxjh3sBneZ9Y9MkzEl9a Lk7w== X-Gm-Message-State: ANhLgQ23J5N46IHkqHeuPJLr+G9/dpzyBEK6YrkP4erm+Nb11qDVtVTQ vTZiUyNvWXp4IBwAr9Lvl/HGoA== X-Google-Smtp-Source: ADFU+vtPSvkZ9yb3NUUQBsFHC9q7zD81H0YjIqBGW1aGNMgqBP877n5HImnbQzQDRnrTS9OMqHYFgg== X-Received: by 2002:a17:90b:124a:: with SMTP id gx10mr7434930pjb.117.1585081958466; Tue, 24 Mar 2020 13:32:38 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id y17sm10418685pfl.104.2020.03.24.13.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2020 13:32:37 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v2 2/5] init_on_alloc: Unpessimize default-on builds Date: Tue, 24 Mar 2020 13:32:28 -0700 Message-Id: <20200324203231.64324-3-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324203231.64324-1-keescook@chromium.org> References: <20200324203231.64324-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200324_133240_490720_D1AE2491 X-CRM114-Status: GOOD ( 12.90 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, "Perla, Enrico" , Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Right now, the state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branch tests. Use the new jump_label macro to check CONFIG settings to default to the "expected" state, unpessimizes the resulting assembly code. Signed-off-by: Kees Cook Reviewed-by: Alexander Potapenko --- include/linux/mm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 059658604dd6..64e911159ffa 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2665,7 +2665,8 @@ static inline void kernel_poison_pages(struct page *page, int numpages, DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc) && + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc) && !page_poisoning_enabled()) return true; return flags & __GFP_ZERO; @@ -2674,7 +2675,8 @@ static inline bool want_init_on_alloc(gfp_t flags) DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free) && + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free) && !page_poisoning_enabled(); } From patchwork Tue Mar 24 20:32:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11456467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E17DA913 for ; Tue, 24 Mar 2020 20:33:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AA00D2074D for ; Tue, 24 Mar 2020 20:33:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="MCC2DTnA"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="UJ8E9FEM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA00D2074D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oRe0yM131ZVUFBzfxKKsfM8jTiCL6rUbyqIYMpaBCWE=; b=MCC2DTnA5JHB0O j/BUtdSz1x+4M0VH35JLnko92KyEz4xfYrYzuQ0zxpX7G5ES7o7Dgi1wDHQNm/vPe79OBT5l7hHUr oFluZ2GZOhIcxZzSHzxICXCmK4i/kM7I3UWqZdOiuNvJ+WD3kxziqEkvV5vOmicBDAxPlc9IeEqEm RGqk/gTZu2yHr5gXVxMbgzRs3LCRbDRO6BPBUgkHS4jY4nOdYpt1UqzivWT+bAyrKf6lJFOUtZ0e3 DjyJN5WtovkuarOS/t2zV5y76s31ahiM21HD9W5Tlg+Nz10Pf0PTYeyrp4J403T2cEtNXhu+x79zA DnnpuUWaA5ZSU+qYPq9A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqEo-0002ni-9M; Tue, 24 Mar 2020 20:33:26 +0000 Received: from mail-pj1-x1043.google.com ([2607:f8b0:4864:20::1043]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE4-0001vk-IO for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2020 20:32:44 +0000 Received: by mail-pj1-x1043.google.com with SMTP id ng8so45116pjb.2 for ; Tue, 24 Mar 2020 13:32:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZpbSabNgeqQFGGWbdjXwu6Z/5tgvjlUVk+jmvcDMVbI=; b=UJ8E9FEM21rF2t4iJN9KMT1vZeOgYW14fs4KJlrdkYrKJPObtDcxrT3UbkXZ4CUJOk k5th1fbqGRozvVjSrW5ZXd88KFy9869kIJa1egEs+lGJlGxXxyC+gMfluZ8Ky7z1BhFG U4Ls0yMLPD3BnrUhkq067AdPvN+XlfPgSUjak= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZpbSabNgeqQFGGWbdjXwu6Z/5tgvjlUVk+jmvcDMVbI=; b=uhFze9mAKZwoP5uaMZLic9OyF4Tmymv4tt8RKdwilvEAL6W9ksMwp9P1eIEWaslxrQ M0ZHoE7zmJCSD4XuMGMOFwwpOLCsXLRdXoFOs+iXtkyoVh6zTj9xkulh4PMaRRl5jNCS XhDsyKnSq5lM5i6YQKt8FIplcNS5A68xaAZoVOIQsND57Xm89tcbGqdNLqwyWfP4MA/M dA+EK2AtvfZRGIOwhQ4jXYuki1I7MW1iPmiTJxWzKOcpFkygwwMj6V8t9AsUPb2gjv81 4BXCTAbcbLVGtnsNudQN7vlc50tHfCrxWc1bYmXr74HT+4UCVRFPmfkc2gs2ZNbmrWdN PMWA== X-Gm-Message-State: ANhLgQ13guavm7glfcBNrwPS6QEaPnALD5J8XYHcoeHtlGhnJdjjSPOZ xAjewnHhLicLA29uPoYN5cCO1A== X-Google-Smtp-Source: ADFU+vssG85d9uw7w/dZYKVY8iWcX36ihTmaX9l/jn8ofzXnh2OIhkrrNAHbLEj4qRZelyfdl7DkKw== X-Received: by 2002:a17:902:b703:: with SMTP id d3mr9747893pls.39.1585081959545; Tue, 24 Mar 2020 13:32:39 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id y131sm16913766pfg.25.2020.03.24.13.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2020 13:32:37 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v2 3/5] stack: Optionally randomize kernel stack offset each syscall Date: Tue, 24 Mar 2020 13:32:29 -0700 Message-Id: <20200324203231.64324-4-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324203231.64324-1-keescook@chromium.org> References: <20200324203231.64324-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200324_133240_606884_9A0141DC X-CRM114-Status: GOOD ( 27.92 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1043 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, "Perla, Enrico" , Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This provides the ability for architectures to enable kernel stack base address offset randomization. This feature is controlled by the boot param "randomize_kstack_offset=on/off", with its default value set by CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. This feature is based on the original idea from the last public release of PaX's RANDKSTACK feature: https://pax.grsecurity.net/docs/randkstack.txt All the credit for the original idea goes to the PaX team. Note that the design and implementation of this upstream randomize_kstack_offset feature differs greatly from the RANDKSTACK feature (see below). Reasoning for the feature: This feature aims to make harder the various stack-based attacks that rely on deterministic stack structure. We have had many such attacks in past (just to name few): https://jon.oberheide.org/files/infiltrate12-thestackisback.pdf https://jon.oberheide.org/files/stackjacking-infiltrate11.pdf https://googleprojectzero.blogspot.com/2016/06/exploiting-recursion-in-linux-kernel_20.html As Linux kernel stack protections have been constantly improving (vmap-based stack allocation with guard pages, removal of thread_info, STACKLEAK), attackers have to find new ways for their exploits to work. They have done so, continuing to rely on the kernel's stack determinism, in situations where VMAP_STACK and THREAD_INFO_IN_TASK_STRUCT were not relevant. For example, the following recent attacks would have been hampered if the stack offset was non-deterministic between syscalls: https://repositorio-aberto.up.pt/bitstream/10216/125357/2/374717.pdf https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html The main idea is that since the stack offset is randomized upon each system call, it is hard for an attack to reliably land in any particular place on the thread stack, even with address exposures, as the stack base will change on the next syscall. Also, since randomization is performed after placing pt_regs, the ptrace-based approach[1] to discover the randomized offset during a long-running syscall should not be possible. Design description: During most of the kernel's execution, it runs on the "thread stack", which is allocated at fork.c/dup_task_struct() and stored in a per-task variable (tsk->stack). Since stack is growing downward, the stack top can be always calculated using task_top_of_stack(tsk) function, which essentially returns an address of tsk->stack + stack size. When VMAP_STACK is enabled, the thread stack is allocated from vmalloc space. The thread stack is pretty deterministic in its structure -- fixed in size, and upon every entry from a userspace to kernel on a syscall the thread stack is started to be constructed from an address fetched from a per-cpu cpu_current_top_of_stack variable. The first element to be pushed to the thread stack is the pt_regs struct that stores all required CPU registers and syscall parameters. The goal of randomize_kstack_offset feature is to add a random offset after the pt_regs has been pushed to the stack and the rest of thread stack (used during the syscall processing) every time a process issues a syscall. The source of randomness is currently arch-defined (but x86 is using the low byte of rdtsc()). Future improvements for different entropy sources is possible, but out of scope for this patch. The offset is added using alloca() call since it helps avoiding changes in assembly syscall entry code and unwinder, and provides correct stack alignment as defined by the compiler. In order to make this available by default with zero performance impact for those that don't want it, now it is selectable with static branches. This way, if the overhead is not wanted, it can just be turned off. Using the per-cpu variable as the entropy source and __builtin_alloc() for stack adjustment and alignment, the generated assembly for x86_64 with GCC looks like this: ... ffffffff81003977: 65 8b 05 02 ea 00 7f mov %gs:0x7f00ea02(%rip),%eax # 12380 ffffffff8100397e: 25 ff 03 00 00 and $0x3ff,%eax ffffffff81003983: 48 83 c0 0f add $0xf,%rax ffffffff81003987: 25 f8 07 00 00 and $0x7f8,%eax ffffffff8100398c: 48 29 c4 sub %rax,%rsp ffffffff8100398f: 48 8d 44 24 0f lea 0xf(%rsp),%rax ffffffff81003994: 48 83 e0 f0 and $0xfffffffffffffff0,%rax ... As a result of the above stack alignment, this patch introduces about 5 bits of randomness after pt_regs is spilled to the thread stack on x86_64, and 6 bits on x86_32 (since its has 1 fewer bits required for stack alignment). The amount of entropy could be adjusted based on how much of the stack space we wish to trade for security. My measure of syscall performance overhead (on x86_64): lmbench: /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_syscall -N 10000 null randomize_kstack_offset=y Simple syscall: 0.7082 microseconds randomize_kstack_offset=n Simple syscall: 0.7016 microseconds So, roughly 0.9% overhead growth for a no-op syscall, which is very manageable. And for people that don't want this, it's off by default. Comparison to PaX RANDKSTACK feature: The RANDKSTACK feature randomizes the location of the stack start (cpu_current_top_of_stack), i.e. including the location of pt_regs structure itself on the stack. Initially this patch followed the same approach, but during the recent discussions[2], it has been determined to be of a little value since, if ptrace functionality is available for an attacker, they can use PTRACE_PEEKUSR/PTRACE_POKEUSR to read/write different offsets in the pt_regs struct, observe the cache behavior of the pt_regs accesses, and figure out the random stack offset. Another difference is that the random offset is stored in a per-cpu variable, rather than having it be per-thread. As a result, these implementations differ a fair bit in their implementation details and results, though obviously the intent is similar. [1] https://lore.kernel.org/kernel-hardening/2236FBA76BA1254E88B949DDB74E612BA4BC57C1@IRSMSX102.ger.corp.intel.com/ [2] https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ Co-developed-by: Elena Reshetova Signed-off-by: Elena Reshetova Link: https://lore.kernel.org/r/20190415060918.3766-1-elena.reshetova@intel.com Signed-off-by: Kees Cook --- v2: - move to per-cpu rdtsc() saved on syscall exit - add static branches for zero-cost dynamic enabling - Kconfig just selects the default state of static branch - __builtin_alloca() produces ugly asm without -fno-stack-clash-protection - made arch agnostic rfc: https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ --- Makefile | 4 ++++ arch/Kconfig | 19 +++++++++++++++ include/linux/randomize_kstack.h | 40 ++++++++++++++++++++++++++++++++ init/main.c | 23 ++++++++++++++++++ 4 files changed, 86 insertions(+) create mode 100644 include/linux/randomize_kstack.h diff --git a/Makefile b/Makefile index 171f2b004c8a..c99463406522 100644 --- a/Makefile +++ b/Makefile @@ -779,6 +779,10 @@ ifdef CONFIG_INIT_STACK_ALL KBUILD_CFLAGS += -ftrivial-auto-var-init=pattern endif +# While VLAs have been removed, GCC produces unreachable stack probes +# for the random_kstack_offset feature. Disable it for all compilers. +KBUILD_CFLAGS += $(call cc-option,-fno-stack-clash-protection,) + DEBUG_CFLAGS := $(call cc-option, -fno-var-tracking-assignments) ifdef CONFIG_DEBUG_INFO diff --git a/arch/Kconfig b/arch/Kconfig index 17fe351cdde0..619a56da4b76 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -854,6 +854,25 @@ config VMAP_STACK virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + def_bool n + help + An arch should select this symbol if it can support kernel stack + offset randomization with calls to add_random_kstack_offset() + during syscall entry and choose_random_kstack_offset() during + syscall exit. + +config RANDOMIZE_KSTACK_OFFSET_DEFAULT + bool "Randomize kernel stack offset on syscall entry" + depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + help + The kernel stack offset can be randomized (after pt_regs) by + roughly 5 bits of entropy, frustrating memory corruption + attacks that depend on stack address determinism or + cross-syscall address exposures. This feature is controlled + by kernel boot param "randomize_kstack_offset=on/off", and this + config chooses the default boot state. + config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kstack.h new file mode 100644 index 000000000000..651ba9504568 --- /dev/null +++ b/include/linux/randomize_kstack.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_RANDOMIZE_KSTACK_H +#define _LINUX_RANDOMIZE_KSTACK_H + +#include +#include +#include + +DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DECLARE_PER_CPU(u32, kstack_offset); + +/* + * Do not use this anywhere else in the kernel. This is used here because + * it provides an arch-agnostic way to grow the stack with correct + * alignment. Also, since this use is being explicitly masked to a max of + * 10 bits, stack-clash style attacks are unlikely. For more details see + * "VLAs" in Documentation/process/deprecated.rst + */ +void *__builtin_alloca(size_t size); + +#define add_random_kstack_offset() do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + char *ptr = __builtin_alloca(offset & 0x3FF); \ + asm volatile("" : "=m"(*ptr)); \ + } \ +} while (0) + +#define choose_random_kstack_offset(rand) do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + offset ^= (rand); \ + this_cpu_write(kstack_offset, offset); \ + } \ +} while (0) + +#endif diff --git a/init/main.c b/init/main.c index ee4947af823f..78fe3aea00b0 100644 --- a/init/main.c +++ b/init/main.c @@ -777,6 +777,29 @@ static void __init mm_init(void) pti_init(); } +#ifdef CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET +DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DEFINE_PER_CPU(u32, kstack_offset); + +static int __init early_randomize_kstack_offset(char *buf) +{ + int ret; + bool bool_result; + + ret = kstrtobool(buf, &bool_result); + if (ret) + return ret; + + if (bool_result) + static_branch_enable(&randomize_kstack_offset); + else + static_branch_disable(&randomize_kstack_offset); + return 0; +} +early_param("randomize_kstack_offset", early_randomize_kstack_offset); +#endif + void __init __weak arch_call_rest_init(void) { rest_init(); From patchwork Tue Mar 24 20:32:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11456455 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35EE11731 for ; Tue, 24 Mar 2020 20:32:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 155F820735 for ; Tue, 24 Mar 2020 20:32:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="HxTJ0Yr5"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="T/YXNFJY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 155F820735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CNj0RmqafA9jFkHNOg1T2nqT+tYhSq+J6YWagN86sDg=; b=HxTJ0Yr5bxaJfF UVHY4CuhKVYcn2B+Bxt389IRfVeMgNXQwVZDkJhGpJbK6ytk8kMcsGL8s6Nj30lR5FUn4Ngplc2iX LdzM4mBc0c3h+TlrXYTc4RoRRTP+zc/bkNEYC+h4+WaQYI9WFkvGLXnU0A+dRD7T/RDDshqKkEjzo KqzYJh6PnG5oVZhqj/3ybK6KxiFXH25Y+A9sGafmxdRC3aWKPC/OgbA7ZVRXNFBi5Nccr1f+hs3T6 m5ESJYmDwoLTDIkFmuYTJDcybR/yj+62UPSY7sO6O/RJ4cocBsKWbfdmESIhUJp6THSlS2wdd2XNW W566imnM7csmZa9VTObQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqEI-00027s-Kl; Tue, 24 Mar 2020 20:32:54 +0000 Received: from mail-pj1-x1042.google.com ([2607:f8b0:4864:20::1042]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE4-0001vg-El for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2020 20:32:42 +0000 Received: by mail-pj1-x1042.google.com with SMTP id ng8so45106pjb.2 for ; Tue, 24 Mar 2020 13:32:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tXxslwRt84lo46s4AfUIy0jwZL6qW/F8aLPlGo+Vuzs=; b=T/YXNFJYjvxaejU1cNn6KiZWcTChX5JBh8J6KYpqLsCb4Zi/vVIMOgsSl+o1HY1WJ2 vWHAi9J3xlxnvLtkfEmpWoe/DsoFGZbmYUNMbqksBmf+pmqHgLFCCUoxSz+q/p2uCZ5s HUkzy9HRyx5OmeeEAodSW6aIsMJPc3Pvav6+U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tXxslwRt84lo46s4AfUIy0jwZL6qW/F8aLPlGo+Vuzs=; b=InZswGQAfGptJ/97s82fppwYH1Hpj+cWhK4IlxeczpTuBxaQuqX9o7Uf18xZRRM3Wi 61ZjsIiDmOcaKBW/ugvru+aEcnajJmT6yQ8hzCTj9ZVQ4/4dLQZSON9sFVJBWEsBqwB0 6mb8LJKKU1uex4bO4FC7XFf9pHJVs03RAS71OTZqgKjYblldagbgIrosah1gGhpqtygt 3EiEr1gpqvizcl2H0DV4xEDVtD/Nbeiqoa5Vp7YhL4LAv3hti8MzniUy2llrhc9lFkMo gq58/QvhfcBqPrG2paND/7FQf7x9OjJN86pVOy6LzmdYqC5TvNRxCH0OUxKqUOehAUHs 5PPg== X-Gm-Message-State: ANhLgQ0uzzpJQdqrP4hNO0h8pivJqCricuCIcj1hItZqsQEG4WELEtsi z7LTrl9D2Zin9LRYTpVHkkyscw== X-Google-Smtp-Source: ADFU+vuPz1pOMJKdmnm1JY1Cd9NWe+ToQ10sdatj73bSqAyNgc+HS1jzGaTvvHVtjKzMIO9M2dV0uA== X-Received: by 2002:a17:902:8bc8:: with SMTP id r8mr27618425plo.48.1585081958998; Tue, 24 Mar 2020 13:32:38 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id 135sm17497623pfu.207.2020.03.24.13.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2020 13:32:37 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v2 4/5] x86/entry: Enable random_kstack_offset support Date: Tue, 24 Mar 2020 13:32:30 -0700 Message-Id: <20200324203231.64324-5-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324203231.64324-1-keescook@chromium.org> References: <20200324203231.64324-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200324_133240_492606_E6DAB84D X-CRM114-Status: GOOD ( 14.45 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1042 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, "Perla, Enrico" , Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. Signed-off-by: Kees Cook --- arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index beea77046f9b..b9d449581eb6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -150,6 +150,7 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 9747876980b5..086d7af570af 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -189,6 +190,13 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs) lockdep_assert_irqs_disabled(); lockdep_sys_exit(); + /* + * x86_64 stack alignment means 3 bits are ignored, so keep + * the top 5 bits. x86_32 needs only 2 bits of alignment, so + * the top 6 bits will be used. + */ + choose_random_kstack_offset(rdtsc() & 0xFF); + cached_flags = READ_ONCE(ti->flags); if (unlikely(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) @@ -283,6 +291,7 @@ __visible void do_syscall_64(unsigned long nr, struct pt_regs *regs) { struct thread_info *ti; + add_random_kstack_offset(); enter_from_user_mode(); local_irq_enable(); ti = current_thread_info(); @@ -355,6 +364,7 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) /* Handles int $0x80 */ __visible void do_int80_syscall_32(struct pt_regs *regs) { + add_random_kstack_offset(); enter_from_user_mode(); local_irq_enable(); do_syscall_32_irqs_on(regs); @@ -378,8 +388,8 @@ __visible long do_fast_syscall_32(struct pt_regs *regs) */ regs->ip = landing_pad; + add_random_kstack_offset(); enter_from_user_mode(); - local_irq_enable(); /* Fetch EBP from where the vDSO stashed it. */ From patchwork Tue Mar 24 20:32:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11456473 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D6E1913 for ; Tue, 24 Mar 2020 20:33:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E65C52080C for ; Tue, 24 Mar 2020 20:33:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UW5sT42F"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="S9Srk3vt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E65C52080C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lBwmX6m8RZxjD1zr1ZU86KFvj4Yx4LxjWtVsHo9i6h0=; b=UW5sT42FjisypD nTA9TwobqADBgiCRzisB/k4EnoElrAs1lnjijns0xSqIyzxx85PWpzPwm9tfikgzeOAdzhEtKIP45 OUPXoE5+6hBczZbypIQEOJpYwA/E6hCISC9wl/oIhEDlPC0U8lxpmY9X41jcRLxfUhQtNqfGHBU7Q klX72t9RVccL8cVEtyVP3msqSV6RbnaeI2NEuj9N/cNpu4solwxVTw6QS1vIKzrCGubVf+Fy8fbPp 9CjeGuD9Jjcek0dQVeDVk0JE8zYcebMzquRvlalQEqxJuSmROIs7hRIw1z7TjC7AgitZK8A/tXypu xt9WsHnUyZedUMhRzesg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqFH-0003Hw-1O; Tue, 24 Mar 2020 20:33:55 +0000 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jGqE8-0001xW-3i for linux-arm-kernel@lists.infradead.org; Tue, 24 Mar 2020 20:32:45 +0000 Received: by mail-pl1-x642.google.com with SMTP id w3so7900893plz.5 for ; Tue, 24 Mar 2020 13:32:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=c3b1sjWFJGWk+HhbHvBn4evu301M7fqA/kMQ4pyC4tI=; b=S9Srk3vtm/u5AWxb5Lq0VLBe/+/+CUvcc/nWuriifzbcDxo3ZwBkLe8Zpl+BamGg/O HO1v1zWZwiU7plLhdxR37s1Jaqc0uJSA8G7tsY4VxqWCO/G2fiHTxNvERagnEIgId99M jOOdb0wJUrZaXGZR5SVzv7rR8cyPHnbBh6gDc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=c3b1sjWFJGWk+HhbHvBn4evu301M7fqA/kMQ4pyC4tI=; b=J+TFYYZVpYtdpHKV+7bYUJLrNRgCAcJs3XJWmNprnnkmQrckHKd3CMpfQsGuAcXEJf 7f8VovNlzu9p5s+1L3kqhEfNv2IgOPKfX2OeKRBltwpI2OnAQCNrPcuDn0asur00CCW7 sWHuGkJwbMYmjZ/K2wyvLpF3844cnhmmimEpjLiAwO79zjXJ21GQTg+oPth1waI30omt Y6Ovr88Y4ExaOpUKQV0yh5LIpYVsBQQG3NuOcn1EdYYer+NtZUgf/WMhdTd9RJ4T56/u e165RD8RFrfHw/Bg9kNg+0fP/a6sMgIvlUzlKUmBnRZaXlsWRj9a5dtnHeXzBuwRwdpt ujIQ== X-Gm-Message-State: ANhLgQ2meKgQS2P7GEbYsZa6rpz5o9DHPsDwyExeKJddf6BJm6fsm+Pn Uhhnr+89CkrXdZLOQXXEjblJZw== X-Google-Smtp-Source: ADFU+vtXBHEgdg26RkMBmCkSMYW4AGLyko9MzjPWEETXma0en8jpa6fm5D16S9175yT7TTEUnSU+xQ== X-Received: by 2002:a17:90a:1911:: with SMTP id 17mr205762pjg.65.1585081963230; Tue, 24 Mar 2020 13:32:43 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id v26sm3813241pfn.51.2020.03.24.13.32.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Mar 2020 13:32:41 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v2 5/5] arm64: entry: Enable random_kstack_offset support Date: Tue, 24 Mar 2020 13:32:31 -0700 Message-Id: <20200324203231.64324-6-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200324203231.64324-1-keescook@chromium.org> References: <20200324203231.64324-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200324_133244_192085_1766C480 X-CRM114-Status: GOOD ( 14.08 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:642 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, "Perla, Enrico" , Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. Signed-off-by: Kees Cook Acked-by: Mark Rutland Acked-by: Will Deacon --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/syscall.c | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 0b30e884e088..4d5aa4959f72 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -127,6 +127,7 @@ config ARM64 select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT select HAVE_ARCH_PREL32_RELOCATIONS + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_STACKLEAK select HAVE_ARCH_THREAD_STRUCT_WHITELIST diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index a12c0c88d345..238dbd753b44 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,8 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, { long ret; + add_random_kstack_offset(); + if (scno < sc_nr) { syscall_fn_t syscall_fn; syscall_fn = syscall_table[array_index_nospec(scno, sc_nr)]; @@ -51,6 +54,13 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, } regs->regs[0] = ret; + + /* + * Since the compiler chooses a 4 bit alignment for the stack, + * let's save one additional bit (9 total), which gets us up + * near 5 bits of entropy. + */ + choose_random_kstack_offset(get_random_int() & 0x1FF); } static inline bool has_syscall_work(unsigned long flags)